Dexterous hand motion capture system and method and readable storage medium

Abstract

The invention discloses a dexterous hand motion capture system and method and a readable storage medium, and belongs to the technical field of motion capture. The motion capture system comprises a dexterous hand motion capture sensor group, a dexterous hand reference template creation unit and a dexterous hand calculation unit, wherein the dexterous hand motion capture sensor group comprises a near-infrared or infrared band image sensor and a visible light band image sensor; the dexterous hand reference template creating unit is used for outputting a dexterous hand reference template based on the dexterous hand image in the near-infrared or infrared band in the reference state; and the dexterous hand calculation unit is used for processing output data of the dexterous hand motion capture sensor group based on the dexterous hand reference template and determining motion data of the dexterous hand. According to the invention, the motion data of the dexterous hand is obtained based on two different modes of dexterous hand images, namely, the near-infrared or infrared band image and the visible light band image, so that the precision and robustness of capturing the motion of the dexterous hand by a dexterous hand motion capturing system are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of motion capture, and particularly relates to a dexterous hand motion capture system, method and readable storage medium. BACKGROUND

[0002] In traditional optical motion capture, optical devices such as near-infrared motion capture cameras are mainly relied on to monitor and track a dexterous hand. In order to facilitate identification and processing, a specially designed light-emitting or light-reflecting marker point is usually attached to the surface of the target to be measured (such as a dexterous hand). However, due to the complex motion trajectory and deformation of the dexterous hand, when the optical motion capture technology is applied, the target tracking is prone to failure, the marker point cannot be re-identified or is misidentified, thereby causing inaccurate and non-robust results such as motion data abnormalities or calculation failures. Therefore, how to capture the motion of the dexterous hand to meet the requirements of high precision and high robustness has become a major challenge in the field of motion capture. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the related art. To this end, the present application provides a dexterous hand motion capture system, method and readable storage medium, which obtains motion data of a dexterous hand based on dexterous hand images of two different modalities of near-infrared or infrared band images and visible light band images, thereby improving the precision and robustness of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0004] In a first aspect, the present application provides a motion capture system, which comprises:

[0005] a dexterous hand motion capture sensor group, a dexterous hand reference template creation unit and a dexterous hand calculation unit,

[0006] The dexterous hand motion capture sensor group comprises a plurality of near-infrared or infrared band image sensors and a plurality of visible light band image sensors. The near-infrared or infrared band image sensor is used to perceive a dexterous hand in a near-infrared or infrared band, and outputs a near-infrared or infrared band dexterous hand image. The visible light band image sensor is used to perceive the dexterous hand in a visible light band, and outputs a visible light band dexterous hand image.

[0007] The dexterous hand reference template creation unit is used to take the palm and finger straight state of the dexterous hand as a reference state, and output a dexterous hand reference template based on the near-infrared or infrared band dexterous hand image of the reference state collected by the near-infrared or infrared band image sensor.

[0008] The dexterous hand computing unit is connected with the dexterous hand motion capture sensor group and the dexterous hand reference template creating unit, and is configured to process output data of at least one of the dexterous hand motion capture sensor groups based on the dexterous hand reference template to determine motion data of the dexterous hand.

[0009] The dexterous hand is provided with a plurality of marker points.

[0010] In the technical solution, the dexterous hand motion capture system includes a dexterous hand motion capture sensor group, a dexterous hand reference template creating unit and a dexterous hand computing unit. The dexterous hand motion capture sensor group includes a plurality of near-infrared or infrared waveband image sensors and a plurality of visible light waveband image sensors. The dexterous hand reference template creating unit is configured to output a dexterous hand reference template based on a near-infrared or infrared waveband dexterous hand image in a reference state acquired by the near-infrared or infrared waveband image sensor. The dexterous hand computing unit is configured to process output data of at least one of the dexterous hand motion capture sensor groups based on the dexterous hand reference template to determine motion data of the dexterous hand. The dexterous hand motion capture system can obtain the motion data of the dexterous hand based on the near-infrared or infrared waveband image and the visible light waveband image of the dexterous hand in two different modalities, thereby improving the accuracy and robustness of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0011] According to an embodiment of the present application, the marker points include palm marker points and finger joint marker points.

[0012] The palm of the dexterous hand is taken as a root joint, and at least six palm marker points are arranged at the wrist of the dexterous hand and the connection between each finger and the palm. According to the finger joint topology of the dexterous hand, at least two finger joint marker points are arranged on each finger of the dexterous hand, and each joint on a finger shares one finger joint marker point with its parent joint.

[0013] In the technical solution, the palm of the dexterous hand is taken as a root joint, and at least six palm marker points are arranged at the wrist of the dexterous hand and the connection between each finger and the palm. According to the finger joint topology of the dexterous hand, at least two finger joint marker points are arranged on each finger of the dexterous hand, and each joint on a finger shares one finger joint marker point with its parent joint. Subsequently, the spatial coordinates and IDs of the marker points can be obtained through epipolar geometry constraint, three-dimensional reconstruction and tracking of the marker points, and then the motion of the dexterous hand can be calculated to determine the pose data of the dexterous hand. The palm marker points and the finger joint marker points provide a basis for determining the pose data of the dexterous hand based on the near-infrared or infrared waveband dexterous hand image.

[0014] According to an embodiment of the present application, the dexterous hand reference template includes initial pose data of the dexterous hand, initial spatial coordinates and initial identification information ID of each marker point.

[0015] outputting a dexterous hand reference template based on the near-infrared or infrared band image of the dexterous hand in the reference state collected by the near-infrared or infrared band image sensor, comprising:

[0016] performing epipolar geometry constraint and three-dimensional reconstruction on each of the marker points in the near-infrared or infrared band image of the dexterous hand in the reference state to determine initial spatial coordinates and initial ID of each of the marker points;

[0017] performing motion solving on each joint of the dexterous hand based on each of the marker points in the near-infrared or infrared band image of the dexterous hand in the reference state to obtain initial pose data of the dexterous hand.

[0018] In the above technical solution, each marker point in the near-infrared or infrared band image of the dexterous hand in the reference state is subjected to epipolar geometry constraint and three-dimensional reconstruction to determine initial spatial coordinates and initial ID of each marker point, and motion solving is performed on the dexterous hand to obtain initial pose data of the dexterous hand. The initial pose data of the dexterous hand, the initial spatial coordinates of each marker point, and the initial identification information ID of each marker point form a dexterous hand reference template. The dexterous hand reference template provides a basis and reference for subsequent determination of motion data of the dexterous hand, can reduce the deviation of the dexterous hand motion capture system in capturing the motion of the dexterous hand, and improves the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0019] According to one embodiment of the present application, the dexterous hand motion capture sensor set further comprises at least one first processing unit connected with the near-infrared or infrared band image sensor, for performing epipolar geometry constraint, three-dimensional reconstruction, and tracking on each of the marker points in the near-infrared or infrared band image of the dexterous hand, determining spatial coordinates and ID of each of the marker points, comprising:

[0020] performing epipolar geometry constraint on each of the marker points in the near-infrared or infrared band image of the dexterous hand to determine a two-dimensional homonymous point coordinate point set in different near-infrared or infrared band images of the dexterous hand;

[0021] performing three-dimensional reconstruction and tracking on each homonymous point based on the two-dimensional homonymous point coordinate point set in different near-infrared or infrared band images of the dexterous hand to determine spatial coordinates and ID of each of the marker points.

[0022] In the technical solution, the first processing unit is configured to screen the same-named points in the near-infrared or infrared waveband nimble hand image through polar geometric constraint, determine a two-dimensional same-named point coordinate set, improve the accuracy of the corresponding relationship of the mark points in the two-dimensional image, and provide a reliable basis for subsequent three-dimensional reconstruction, perform three-dimensional reconstruction and tracking, determine the spatial coordinates and ID of the mark points, and realize the recovery of the three-dimensional spatial coordinates of the mark points from the two-dimensional image. The tracking further enables the dynamic changes of the mark points in the continuous image frames or time sequence to be accurately captured. The determined spatial coordinates and ID of the mark points provide a data basis for subsequent motion calculation of the nimble hand.

[0023] According to an embodiment of the present application, the first processing unit is further connected with a nimble hand reference template creating unit, and the first processing unit is further configured to:

[0024] According to the spatial coordinates and ID of the palm mark points, perform motion calculation on the palm of the nimble hand based on the nimble hand reference template, and obtain palm pose data of the nimble hand;

[0025] According to the palm pose data and the parent-child link relationship between each finger joint and the palm root joint of the nimble hand, perform motion calculation on the finger joints of the nimble hand based on hand kinematic characteristics, and obtain first pose data of the nimble hand. The hand kinematic characteristics include at least one of the following: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and parent-child joint relative motion restriction characteristics.

[0026] In the technical solution, based on the spatial coordinates and ID of the palm mark points, the motion calculation is performed in combination with the nimble hand reference template to obtain the palm pose data. Then, in combination with the palm pose data, the parent-child link relationship between the finger and the palm root joint, and the hand kinematic characteristics, the motion calculation is performed on the nimble hand fingers to obtain the first pose data of the nimble hand, thereby realizing the determination of the pose data of the nimble hand based on the near-infrared or infrared waveband nimble hand image.

[0027] According to an embodiment of the present application, the nimble hand motion capture sensor group further includes at least one second processing unit connected with the visible light waveband image sensor, configured to perform key point detection and motion pose estimation on the visible light waveband nimble hand image based on a deep learning method, and send the key point detection result and second pose data of the nimble hand to the nimble hand computing unit.

[0028] In the technical solution, the second processing unit is connected with the visible light band image sensor, performs key point detection and motion data inference prediction on the visible light band dexterous hand image based on a deep learning method, and sends the key point ID detection result and the second pose data to the dexterous hand computing unit. The second processing unit can be an embedded processing unit or in the form of a host computer in the dexterous hand motion capture system, thereby improving the flexibility of the dexterous hand motion capture system.

[0029] According to an embodiment of the present application, the dexterous hand computing unit is configured to:

[0030] Based on the spatial coordinates of each marker point, each marker point is re-projected onto the visible light band dexterous hand image, and the ID tracking result of each marker point is verified or the ID of a lost tracking marker point is identified by comparing the re-projected marker points with the key point detection result based on projection distance features, time domain motion features of each finger joint, and motion axis angle and motion direction constraint features of each finger joint.

[0031] For image frames with marker point ID tracking errors or lost marker point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the features between the marker points and the key points of each joint, the spatial perspective features, and the motion axis angle and motion direction constraint features of each finger joint, to obtain the motion data of the dexterous hand.

[0032] In the technical solution, the dexterous hand computing unit re-projects each marker point onto the visible light band dexterous hand image according to the spatial coordinates of each marker point, compares the re-projected marker points with the key point detection result based on projection distance features, time domain motion features of each finger joint, and motion axis angle and motion direction constraint features of each finger joint, verifies the correctness of marker point ID tracking, and identifies the ID of a lost tracking marker point. For image frames with ID tracking errors or lost marker point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the features between the marker points and the key points of each joint, the spatial perspective features, and the motion axis angle and motion direction constraint features of each finger joint, to obtain the motion data of the dexterous hand. The motion data of the dexterous hand is obtained based on the perception data of the dual modal, and in the case of lost marker point information or tracking errors, the first motion data obtained by motion calculation based on the visible light band dexterous hand image can also be fused to obtain the motion data of the dexterous hand, thereby improving the accuracy and robustness of the dexterous hand motion capture system in capturing dexterous hand motion.

[0033] According to an embodiment of the present application, the dexterous hand motion capture sensor group further comprises at least one light supplementing light source for illuminating the dexterous hand so that each marker point forms a contrast with the background in the imaging of the near-infrared or infrared band image sensor under the illumination of the light supplementing light source.

[0034] In the technical solution, the dexterous hand motion capture system further comprises at least one light supplementing light source for irradiating the dexterous hand, so that the marker points form a contrast with the background in the imaging of the near-infrared or infrared band image sensor under the irradiation of the light supplementing light source, the marker points are more prominent, the accuracy of identifying and locating the marker points in the near-infrared or infrared band dexterous hand image is improved, and the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand is further improved.

[0035] In a second aspect, the application provides a dexterous hand motion capture method based on the dexterous hand motion capture system as described in the first aspect, comprising:

[0036] perceiving the dexterous hand in the near-infrared or infrared band and outputting a near-infrared or infrared band dexterous hand image;

[0037] performing epipolar geometry constraint, three-dimensional reconstruction and tracking on each of the marker points in the near-infrared or infrared band dexterous hand image to determine the spatial coordinates and ID of each of the marker points;

[0038] based on the spatial coordinates and ID of each of the marker points, performing motion solving on each joint of the dexterous hand based on a dexterous hand reference template to obtain first pose data of the dexterous hand;

[0039] perceiving the dexterous hand in the visible light band and outputting a visible light band dexterous hand image;

[0040] performing key point detection and motion pose estimation on the visible light band dexterous hand image based on a deep learning method to obtain key point detection results and second pose data of the dexterous hand;

[0041] based on the spatial coordinates, ID of each of the marker points, the first pose data of the dexterous hand, and the key point detection results and the second pose data of the dexterous hand, determining motion data of the dexterous hand.

[0042] According to an embodiment of the application, the performing epipolar geometry constraint, three-dimensional reconstruction and tracking on each of the marker points in the near-infrared or infrared band dexterous hand image to determine the spatial coordinates and ID of each of the marker points comprises:

[0043] performing epipolar geometry constraint on each of the marker points in the near-infrared or infrared band dexterous hand image to determine a two-dimensional homonymous point coordinate point set in different near-infrared or infrared band dexterous hand images;

[0044] based on the two-dimensional homonymous point coordinate point set in different near-infrared or infrared band dexterous hand images, performing three-dimensional reconstruction and tracking on each homonymous point to determine the spatial coordinates and ID of each of the marker points.

[0045] According to one embodiment of the present application, the motion of each joint of the dexterous hand is solved based on the spatial coordinates and ID of each marker point and the dexterous hand reference template to obtain the first pose data of the dexterous hand, including:

[0046] The motion of the palm of the dexterous hand is solved based on the spatial coordinates and ID of the palm marker point and the dexterous hand reference template to obtain the palm pose data of the dexterous hand.

[0047] The motion of the finger joints of the dexterous hand is solved based on the hand kinematics characteristics according to the palm pose data and the parent-child link relationship of each finger and palm root joint of the dexterous hand to obtain the first pose data of the dexterous hand, and the hand kinematics characteristics include at least one of the following: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and relative motion restriction characteristics of parent-child joints.

[0048] According to one embodiment of the present application, the motion data of the dexterous hand is determined based on the spatial coordinates and ID of each marker point, the first pose data of the dexterous hand, and the key point detection result and the second pose data of the dexterous hand, including:

[0049] Each marker point is re-projected onto the visible light waveband dexterous hand image based on the spatial coordinates of each marker point, and the ID tracking result of each marker point is verified for correctness or the lost marker point is identified based on the projection distance characteristics, the time domain motion characteristics of each finger joint, and the motion axis angle and motion direction constraint characteristics of each finger joint, and the key point detection result is compared.

[0050] For image frames with marker ID tracking errors or lost marker point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the characteristics and spatial perspective characteristics between the marker points and the joint key points, and the motion axis angle and motion direction constraint characteristics of each finger joint to obtain the motion data of the dexterous hand.

[0051] According to one embodiment of the present application, before the dexterous hand is perceived in the near-infrared or infrared waveband and the near-infrared or infrared waveband dexterous hand image is output, the method further includes:

[0052] The palm and finger straight state of the dexterous hand is taken as a reference state, and the dexterous hand reference template is output based on the near-infrared or infrared waveband dexterous hand image in the reference state collected by the near-infrared or infrared waveband image sensor.

[0053] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dexterous hand motion capture method according to the second aspect.

[0054] In a fourth aspect, the present application provides a non-transitory computer readable storage medium, having stored thereon a computer program, wherein the computer program is executable by a processor to implement the dexterous hand motion capture method according to the second aspect.

[0055] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or an instruction to implement the dexterous hand motion capture method according to the second aspect.

[0056] In a sixth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executable by a processor to implement the dexterous hand motion capture method according to the second aspect.

[0057] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings.

[0059] Figure 1 is one of structural schematic diagrams of a dexterous hand motion capture system provided by some embodiments of the present application;

[0060] Figure 2 is another of structural schematic diagrams of a dexterous hand motion capture system provided by some embodiments of the present application;

[0061] Figure 3 is a third of structural schematic diagrams of a dexterous hand motion capture system provided by some embodiments of the present application;

[0062] Figure 4 is a fourth of structural schematic diagrams of a dexterous hand motion capture system provided by some embodiments of the present application;

[0063] Figure 5 is a flowchart of a dexterous hand motion capture method provided by some embodiments of the present application;

[0064] Figure 6 is a structural schematic diagram of an electronic device provided by some embodiments of the present application.

[0065] REFERENCE SIGNS

[0066] 100: dexterous hand motion capture system; 101: dexterous hand motion capture sensor group;

[0067] 102: dexterous hand reference template creation unit; 103: dexterous hand calculation unit;

[0068] 104: near-infrared or infrared waveband image sensor; 105: visible light waveband image sensor;

[0069] 106: first processing unit; 107: second processing unit; 108: light supplement light source;

[0070] 600: electronic device; 601: processor; 602: memory. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0072] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0073] The dexterous hand motion capture system, method and readable storage medium provided by the embodiments of the present application will be described in detail below in conjunction with the drawings and specific embodiments and application scenarios.

[0074] Figure 1 is one of the structural schematic diagrams of the dexterous hand motion capture system provided by some embodiments of the present application. As shown in Figure 1 , the dexterous hand motion capture system 100 includes:

[0075] a dexterous hand motion capture sensor group 101, a dexterous hand reference template creation unit 102, and a dexterous hand calculation unit 103,

[0076] The dexterous hand motion capture sensor group 101 comprises a plurality of near-infrared or infrared waveband image sensors 104 and a plurality of visible light waveband image sensors 105, the near-infrared or infrared waveband image sensors 104 are used to perceive the dexterous hand in the near-infrared or infrared waveband and output near-infrared or infrared waveband dexterous hand images, and the visible light waveband image sensors 105 are used to perceive the dexterous hand in the visible light waveband and output visible light waveband dexterous hand images.

[0077] The dexterous hand reference template creation unit 102 is used to take the palm and finger straight state of the dexterous hand as a reference state, output a dexterous hand reference template based on the near-infrared or infrared waveband dexterous hand images collected by the near-infrared or infrared waveband image sensors 104 in the reference state.

[0078] The dexterous hand computing unit 103 is connected with the dexterous hand motion capture sensor group 101 and the dexterous hand reference template creation unit 102, and is used to process the output data of at least one dexterous hand motion capture sensor group 101 based on the dexterous hand reference template, and determine the motion data of the dexterous hand.

[0079] The dexterous hand is provided with a plurality of marker points.

[0080] It can be understood that the dexterous hand finger has a highly flexible and precise control ability, and is widely used in the field of robot technology, and can be used to simulate the complex motion of human hands to achieve fine operation and control.

[0081] Optionally, the dexterous hand can also refer to the whole composed of a glove and a human hand worn on the human hand, wherein the glove is provided with a plurality of marker points.

[0082] The near-infrared or infrared waveband image sensor 104 is used to perceive the dexterous hand in the near-infrared or infrared waveband and output near-infrared or infrared waveband dexterous hand images, and since the light in the near-infrared and infrared waveband is invisible to the human eye, the near-infrared or infrared waveband image sensor 104 is not limited by the visible light illumination condition and is more suitable for use in an environment where the visible light illumination condition is not ideal. The dexterous hand motion capture sensor group comprises a plurality of near-infrared or infrared waveband image sensors 104, thereby improving the applicability of the dexterous hand motion capture system under different illumination conditions. The visible light waveband image sensor 105 is used to perceive the dexterous hand in the visible light waveband and output visible light waveband dexterous hand images, and in the case of good illumination condition, the visible light waveband image sensor 105 can output high-resolution color images. Optionally, the near-infrared or infrared waveband image sensor 104 and the visible light waveband image sensor 105 are calibrated by using Zhang Zhengyou calibration method.

[0083] The dexterous hand motion capture sensor group 101 includes a plurality of near-infrared or infrared waveband image sensors 104 and a plurality of visible light waveband image sensors 105, so that the dexterous hand can be perceived from two modalities of optics and vision through the dexterous hand motion capture sensor group 101, improving the comprehensiveness and accuracy of perceiving the dexterous hand.

[0084] The dexterous hand reference template creation unit 102 is configured to take the palm and finger straight state of the dexterous hand as the reference state, and output a dexterous hand reference template based on the near-infrared or infrared waveband dexterous hand image in the reference state collected by the near-infrared or infrared waveband image sensor 104. The reference state is also called a calibration pose. In addition to taking the palm and finger straight state of the dexterous hand as the reference state, other states of the dexterous hand can also be taken as the reference state, such as the state in which the thumb and index finger of the dexterous hand are in a pinch state or the state in which the dexterous hand is in a half-fist state, etc. The embodiments of the present application do not limit this, and the specific determination can be made according to the actual application requirements.

[0085] The dexterous hand reference template determined by the dexterous hand reference template creation unit 102 can be understood as the initial motion data of the dexterous hand when starting to capture the motion of the dexterous hand, such as the initial pose data of the dexterous hand, the initial coordinates and identification information of the mark points arranged on the dexterous hand, etc. The dexterous hand reference template provides a basis and reference for subsequent implementation of motion capture of the dexterous hand, can reduce the deviation of the dexterous hand motion capture system in capturing the motion of the dexterous hand, and improve the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0086] The dexterous hand computing unit 103 is configured to process the output data of at least one dexterous hand motion capture sensor group 101 based on the dexterous hand reference template to obtain motion data of the dexterous hand, which describes the motion state of the dexterous hand in three-dimensional space, such as gesture pose data of the dexterous hand, etc., thereby realizing the motion capture of the dexterous hand. It can be understood that the motion data is obtained based on the dexterous hand images in two different modalities of near-infrared or infrared waveband images and visible light waveband images, realizing the fusion of perception data in different modalities, improving the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand. A single modality sensor may be affected by occlusion or other factors, resulting in inaccurate output perception data. Fusion of perception data from different modalities can improve the robustness of the dexterous hand motion capture system. In some embodiments, in the case that the motion data of the dexterous hand determined based on the perception data of a certain modality is abnormal (such as not conforming to the hand structure feature), the motion data of the dexterous hand can be constrained or corrected based on the perception data of another modality, so that the motion data of the dexterous hand is more reasonable.

[0087] The dexterous hand is provided with a plurality of marker points, which are special markers or light-emitting points, also known as markers, and have good recognizability in the near-infrared or infrared wave band. The shape, size and material of the marker points are determined according to specific requirements.

[0088] The dexterous hand motion capture system provided by the embodiment of the application comprises a dexterous hand motion capture sensor group, a dexterous hand reference template creation unit and a dexterous hand calculation unit. The dexterous hand motion capture sensor group comprises a plurality of near-infrared or infrared wave band image sensors and a plurality of visible light wave band image sensors. The dexterous hand reference template creation unit is used to output a dexterous hand reference template based on a near-infrared or infrared wave band dexterous hand image in a reference state acquired by the near-infrared or infrared wave band image sensor. The dexterous hand calculation unit is used to process output data of at least one dexterous hand motion capture sensor group based on the dexterous hand reference template, so as to realize the motion data of the dexterous hand based on the dexterous hand images in two different modalities of the near-infrared or infrared wave band image and the visible light wave band image, and improve the precision and robustness of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0089] In some embodiments of the application, the marker points comprise palm marker points and finger joint marker points.

[0090] The palm of the dexterous hand is taken as a root joint, and at least 6 palm marker points are arranged at the wrist of the dexterous hand and the connection between each finger and the palm. According to the finger joint topology of the dexterous hand, at least 2 finger joint marker points are arranged on each finger of the dexterous hand, wherein each joint on the finger shares one finger joint marker point with its parent joint.

[0091] At least 6 palm marker points are arranged at the wrist of the dexterous hand and the connection between each finger and the palm. Specifically, there are 5 connections between the fingers and the palm of the dexterous hand, and there are 6 connections in total including the wrist of the dexterous hand, and at least one palm marker point is arranged at each of the 6 positions.

[0092] Each finger of the dexterous hand comprises a plurality of joints, such as proximal interphalangeal joints and distal interphalangeal joints. The finger joint marker points are arranged on these joints, and each joint on the finger shares one finger joint marker point with its parent joint. At least 2 finger joint marker points are arranged on each finger. When the fingers of the dexterous hand perform various actions such as bending, stretching or grasping, the angles and positions of the joints will change.

[0093] In the embodiments of the present application, the palm of the dexterous hand is taken as the root joint, at least six palm markers are arranged at the wrist of the dexterous hand and the connection between each finger and the palm, and at least two finger joint markers are arranged on each finger of the dexterous hand according to the finger joint topology of the dexterous hand, wherein the joint on each finger shares a finger joint marker with its parent joint. Subsequently, through epipolar geometry constraint, three-dimensional reconstruction and tracking on these markers, the spatial coordinates and IDs of these markers can be obtained, and then the motion of the dexterous hand can be solved to determine the pose data of the dexterous hand. The palm markers and the finger joint markers provide a basis for determining the pose data of the dexterous hand based on the near-infrared or infrared band dexterous hand image.

[0094] In some embodiments of the present application, the dexterous hand reference template includes initial pose data of the dexterous hand, initial spatial coordinates and initial identification information ID of each marker;

[0095] The dexterous hand reference template is output based on the near-infrared or infrared band dexterous hand image in the reference state collected by the near-infrared or infrared band image sensor 104, and includes:

[0096] The initial spatial coordinates and initial IDs of each marker are determined by performing epipolar geometry constraint and three-dimensional reconstruction on each marker in the near-infrared or infrared band dexterous hand image in the reference state;

[0097] Based on each marker in the near-infrared or infrared band dexterous hand image in the reference state, the motion of each joint of the dexterous hand is solved to obtain the initial pose data of the dexterous hand.

[0098] It can be understood that the initial pose data of the dexterous hand is the position data and attitude data of the dexterous hand in the reference state (initial state), which can be considered as the initial reference for motion capture of the dexterous hand. The initial spatial coordinates of each marker are the coordinate positions of each marker in the three-dimensional space when the dexterous hand is in the reference state, which are the starting points for subsequent calculation of the motion trajectory and displacement of each marker. The identification information ID of each marker can be used to distinguish each marker to avoid confusion between markers.

[0099] In some embodiments, the initial spatial coordinates and initial IDs of each marker are determined by performing epipolar geometry constraint and three-dimensional reconstruction on each marker in the near-infrared or infrared band dexterous hand image in the reference state, including:

[0100] The two-dimensional homonymous point coordinate point sets in the near-infrared or infrared band dexterous hand images in different reference states are determined by performing epipolar geometry constraint on each marker in the near-infrared or infrared band dexterous hand image in the reference state;

[0101] Based on the two-dimensional homonymic point coordinate point sets in the near-infrared or infrared waveband dexterous hand images in different reference states, three-dimensional reconstruction is performed on each homonymic point to determine the initial spatial coordinates and the ID of each marker point.

[0102] Epipolar geometry constraint is a geometric model for determining the relationship between homonymic points (physically identical points, which can be understood as the projection points of the same marker point in different images) in two views. For example, if the projection points of the marker point in the near-infrared or infrared waveband dexterous hand images in different reference states collected at different angles satisfy the epipolar geometry constraint, it represents that there is a specific geometric relationship between the two projection points. Through the application of the epipolar geometry constraint, the matching homonymic point set in the two near-infrared or infrared waveband dexterous hand images can be screened out. Similarly, the homonymic point set of the near-infrared or infrared waveband dexterous hand images in different reference states collected at different angles can be further determined through the epipolar geometry constraint. The application of the epipolar geometry constraint effectively improves the accuracy and reliability of the marker point matching in different near-infrared or infrared waveband dexterous hand images.

[0103] Three-dimensional reconstruction is a process of restoring three-dimensional structure from two-dimensional images collected at different angles. In the embodiments of the present application, the initial spatial coordinates of each marker point in the three-dimensional space can be calculated by using methods such as triangulation, based on the multiple near-infrared or infrared waveband dexterous hand images in the reference state and the homonymic point set determined through the epipolar geometry constraint. In the process of three-dimensional reconstruction, the marker points are identified and distinguished to determine the identification information ID of each marker point. For example, different marker points can be identified and distinguished through the geometric layout and relative position relationship of the marker points, so as to determine the ID of each marker point.

[0104] After obtaining the initial spatial coordinates and the ID of each marker point through three-dimensional reconstruction, the motion of each joint of the dexterous hand can be calculated based on the initial spatial coordinates and the ID of each marker point on the dexterous hand, and the position data and the attitude data of the dexterous hand in the reference state can be calculated. For example, the rotation angle and the displacement of the dexterous hand can be determined by calculating the relative position relationship between the marker points.

[0105] In the embodiments of the present application, the epipolar geometry constraint and the three-dimensional reconstruction are performed on each marker point in the near-infrared or infrared waveband dexterous hand images in the reference state to determine the initial spatial coordinates and the initial ID of each marker point. Based on the initial spatial coordinates and the initial ID of each marker point, the motion of the dexterous hand in the reference state is calculated to obtain the initial pose data of the dexterous hand. The initial pose data of the dexterous hand, the initial spatial coordinates and the identification information ID of each marker point form a dexterous hand reference template. The dexterous hand reference template provides a basis and a reference for subsequent determination of the motion data of the dexterous hand, which can reduce the deviation of the dexterous hand motion capture system in capturing the motion of the dexterous hand and improve the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0106] Figure 2 is a structural schematic diagram of a dexterous hand motion capture system provided by some embodiments of the present application. As shown in the figure, the dexterous hand motion capture sensor group 101 further comprises at least one first processing unit 106 connected with the near-infrared or infrared waveband image sensor 104, for performing epipolar geometry constraint, three-dimensional reconstruction and tracking on each of the marker points in the near-infrared or infrared waveband dexterous hand image, determining the spatial coordinates and ID of each of the marker points, comprising: Figure 2

[0107] performing epipolar geometry constraint on each of the marker points in the near-infrared or infrared waveband dexterous hand image, to determine a two-dimensional homonym coordinate point set of the same-named points in different near-infrared or infrared waveband dexterous hand images;

[0108] based on the two-dimensional homonym coordinate point set of the same-named points in different near-infrared or infrared waveband dexterous hand images, performing three-dimensional reconstruction and tracking on each of the same-named points, to determine the spatial coordinates and ID of each of the marker points.

[0109] For example, if the projection points of a marker point in two near-infrared or infrared waveband dexterous hand images collected at different angles satisfy the epipolar geometry constraint, it means that there is a specific geometric relationship between the two projection points. By applying epipolar geometry constraint, a matching same-named point (physically the same point) set in the two near-infrared or infrared waveband dexterous hand images can be screened out. Similarly, the same-named point set of different near-infrared or infrared waveband dexterous hand images can be further generated by epipolar geometry constraint.

[0110] Three-dimensional reconstruction is a process of recovering three-dimensional structure from two-dimensional images collected at different angles. In the embodiments of the present application, by using multiple near-infrared or infrared waveband dexterous hand images, combined with the same-named point set determined by epipolar geometry constraint, each same-named point can be reconstructed in three dimensions by using methods such as triangulation. In the process of three-dimensional reconstruction, the marker points are identified and distinguished to determine the identification information ID of each marker point. For example, different marker points can be identified and distinguished by their geometric layout and relative position relationship, so as to determine the ID of each marker point.

[0111] ​It should be noted that the near-infrared or infrared waveband nimble hand images in the embodiments of the present application include nimble hand images in multiple different states (not limited to the reference state) and the spatial coordinates of the attention mark points need to be tracked in the nimble hand movement process. Therefore, in the embodiments of the present application, the three-dimensional points obtained by the three-dimensional reconstruction need to be tracked after the three-dimensional reconstruction to determine the change of the spatial coordinates in the continuous image frames or time sequence. Optionally, Kalman filtering or other methods are used to track the mark points to obtain mark point ID tracking results, which include the spatial coordinates, ID and corresponding image frame sequence or time sequence of each mark point.

[0112] The first processing unit in the nimble hand motion capture system provided by the embodiments of the present application is used to screen the matching homonymic points in the near-infrared or infrared waveband nimble hand images through epipolar geometry constraints, determine the two-dimensional homonymic point coordinate set, improve the accuracy of the corresponding relationship of the mark points in the two-dimensional images, provide a reliable basis for subsequent three-dimensional reconstruction, perform three-dimensional reconstruction and tracking, determine the spatial coordinates and ID of the mark points, and realize the recovery of the three-dimensional spatial coordinates of the mark points from the two-dimensional images. The tracking further enables the dynamic change of the mark points in the continuous image frames or time sequence to be accurately captured, and the determined spatial coordinates and ID of each mark point provide a data basis for subsequent motion calculation of the nimble hand.

[0113] According to some embodiments of the present application, the first processing unit 106 is also connected with the nimble hand reference template creation unit 102, and the first processing unit 106 is further used to:

[0114] According to the spatial coordinates and ID of the palm mark points, the motion of the palm of the nimble hand is calculated based on the nimble hand reference template to obtain palm pose data of the nimble hand;

[0115] According to the palm pose data and the parent-child link relationship of each finger joint and the palm root joint of the nimble hand, the motion of the finger joints of the nimble hand is calculated based on the hand kinematics characteristics to obtain first pose data of the nimble hand, and the hand kinematics characteristics include at least one of the following: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and parent-child joint relative motion restriction characteristics.

[0116] The mark points include palm mark points and finger joint mark points, the motion of the nimble hand is calculated based on the spatial coordinates and ID of the palm mark points in the mark points to calculate the position data and pose data (palm pose data of the nimble hand) of the palm of the nimble hand. In the process of motion calculation, the nimble hand reference template provides a basis and a reference.

[0117] Optionally, the palm landmark points are compared and matched with the dexterous hand reference template, through the ID of the palm landmark points, the corresponding landmark points in the dexterous hand reference template can be accurately determined, and the position data and attitude data of the landmark points in the reference state are determined, by comparing the current spatial coordinates of the landmark points with the spatial coordinates in the reference state, the spatial position change of each palm landmark point can be calculated.

[0118] In the process of motion solving of the dexterous hand, according to the palm pose data and the parent-child linkage relationship of each finger joint and the palm root joint of the dexterous hand, the finger joints are motion solved based on the hand kinematics characteristics, and the first pose data of the dexterous hand is obtained. The hand kinematics characteristics involved in this process include but are not limited to: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and relative motion restriction characteristics of parent-child joints.

[0119] The finger distance characteristics are that the dexterous hand includes a thumb, an index finger, a middle finger, a ring finger and a little finger, each finger includes three joints except the thumb, the lengths of the joints are different, and the distance change ranges of the joints are also inconsistent in the process of finger movement, in the process of creating the dexterous hand reference template, the reference lengths of the joints can be measured, and the upper and lower limits of the joint movement can be set by statistical method;

[0120] The time domain motion characteristics of each finger joint are that the motion of the finger joint relative to the parent joint can be approximately regarded as uniform motion with a certain acceleration displacement in the time domain, by analyzing the matching results of each finger joint at the previous time and the dexterous hand reference template, the spatial coordinates of each finger joint at the current time can be predicted, which helps the matching of each finger joint at the current time with the dexterous hand reference template;

[0121] The motion axis angle and motion direction constraint characteristics of each finger joint are that the other four fingers of the dexterous hand except the thumb basically move in the plane perpendicular to the palm plane (allowing a certain range of angle tolerance), and have a certain spatial range restriction, the motion direction of each joint also has a restriction, and abnormal gestures contrary to hand physiology will not appear, in the time domain, the motion of the finger presents the characteristics of continuous change, and the difference between adjacent frames is relatively small;

[0122] The relative motion restriction characteristics of parent-child joints are that when the finger joint is a child joint, the motion range of the child joint relative to the parent joint is limited.

[0123] In the process of matching each landmark point of the dexterous hand with the dexterous hand reference template, if the palm of the dexterous hand has been successfully matched, the first pose data of the entire dexterous hand can be determined by starting from the first joint connected between the palm of the dexterous hand and each finger based on the above characteristics.

[0124] In the embodiment of the present application, motion calculation is performed based on the spatial coordinates and ID of the palm landmark points in combination with the dexterous hand reference template to obtain palm posture data. Then, combined with the palm posture data, the parent-child link relationship between the fingers and the palm root joints, and the hand kinematic characteristics, the motion calculation of the dexterous hand fingers is performed to obtain the first-order posture data of the dexterous hand, thereby realizing the determination of the dexterous hand posture data based on the dexterous hand image in the near-infrared or infrared band.

[0125] Optionally, the first processing unit 106 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the dexterous hand motion capture sensor group 101, connected to the near-infrared or infrared band image sensor 104, and can also exist in the dexterous hand motion capture system 100 in the form of a host computer.

[0126] Optionally, the first processing unit 106 may be one, connected to all the near-infrared or infrared band image sensors 104 , and processes the near-infrared or infrared band dexterous hand image output by each near-infrared or infrared band image sensor 104 .

[0127] Optionally, there may be multiple first processing units 106 , corresponding one-to-one or one-to-many to the near-infrared or infrared band image sensors 104 , to process the near-infrared or infrared band dexterous hand images output by the connected near-infrared or infrared band image sensors 104 .

[0128] Optionally, the first processing unit 106 is connected to the dexterous hand computing unit 103, and is used to send the spatial coordinates, ID of each of the marker points and the first position data of the dexterous hand to the dexterous hand computing unit.

[0129] Figure 3 This is the third structural diagram of the dexterous hand motion capture system provided by some embodiments of the present application. Figure 3 As shown, in some embodiments, the dexterous hand motion capture sensor group 101 also includes at least one second processing unit 107, which is connected to the visible light band image sensor 105, and is used to perform key point detection and motion posture estimation on the visible light band dexterous hand image based on a deep learning method, and send the key point detection results and the second posture data of the dexterous hand to the dexterous hand computing unit 103.

[0130] Optionally, the second processing unit 107 is an embedded processing unit, such as a field programmable gate array (FPGA), embedded in the dexterous hand motion capture sensor group 101, or may exist in the dexterous hand motion capture system 100 in the form of a host computer.

[0131] Optionally, the second processing unit 107 uses a deep learning algorithm, such as YOLOv8, to perform key point detection on the dexterous hand image in the visible light band to obtain a key point ID detection result. The key point ID detection result includes the coordinates and ID of the key point, where the ID is a unique identifier of the key point.

[0132] Optionally, the second processing unit 107 uses algorithms such as MediaPipe to estimate the motion posture of the dexterous hand in the dexterous hand image in the visible light band to obtain second posture data. The posture data of the dexterous hand includes position data and posture data of the dexterous hand in three-dimensional space.

[0133] Optionally, the second processing unit 107 may be one, connected to all visible light band image sensors 105 , and processes the near-infrared or infrared band dexterous hand image output by each visible light band image sensor 105 .

[0134] Optionally, there may be multiple second processing units 107 , corresponding one-to-one or one-to-many to the visible light band image sensor 105 , to process the near-infrared or infrared band dexterous hand images output by the connected visible light band image sensor 105 .

[0135] In the embodiment of the present application, the second processing unit is connected to the visible light band image sensor, performs key point detection and motion data inference and prediction on the visible light band dexterous hand image based on the deep learning method, and sends the key point ID detection results and the second posture data to the dexterous hand computing unit. The second processing unit can be an embedded processing unit or can exist in the dexterous hand motion capture system in the form of a host computer, thereby improving the flexibility of the dexterous hand motion capture system.

[0136] like Figure 3 As shown, when the dexterous hand motion capture sensor group includes the first processing unit 106 and the second processing unit 107, the dexterous hand calculation unit 103 is used to:

[0137] Based on the spatial coordinates of each marker point, each marker point is reprojected onto the visible light band dexterous hand image, and compared with the key point detection results based on the projection distance characteristics, the time domain motion characteristics of each finger joint, and the motion axis angle and motion direction constraint characteristics of each finger joint to verify the correctness of the ID tracking results of each marker point or perform ID identification on the marker point that has lost tracking;

[0138] For the image frame with a landmark point ID tracking error or lost landmark point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the features and spatial perspective features between the landmark points and the joint key points, and the motion axis angle and motion direction constraint features of the joint of each finger, to obtain the motion data of the dexterous hand.

[0139] In some embodiments, the dexterous hand motion capture sensor group 101 includes a plurality of near-infrared or infrared waveband image sensors 104 and a plurality of visible light waveband image sensors 105, which can perceive the dexterous hand from multiple perspectives to obtain near-infrared or infrared waveband dexterous hand images and visible light waveband dexterous hand images. The first processing unit 106 performs epipolar geometry constraint, three-dimensional reconstruction and tracking on the near-infrared or infrared waveband dexterous hand images to obtain the spatial coordinates and IDs of the landmark points, and performs motion solving on the dexterous hand to obtain the first pose data of the dexterous hand. The second processing unit 107 performs key point detection and motion pose estimation on the visible light waveband dexterous hand images based on a deep learning method, and sends the key point ID detection result and the second pose data of the dexterous hand to the dexterous hand computing unit 103.

[0140] The dexterous hand computing unit 103 projects the landmark points onto the visible light waveband dexterous hand images according to the spatial coordinates of the landmark points (to obtain landmark point spatial coordinate projection points), realizes the conversion of the three-dimensional spatial coordinates of the landmark points into two-dimensional re-projection coordinates, and compares (matches) the projections of the landmark points in the visible light waveband dexterous hand images with the detected key points in the visible light waveband dexterous hand images. In this process, the projection distance feature, the time domain motion feature of each finger joint, and the motion axis angle and motion direction constraint feature of each finger joint can be used for comparison to verify the correctness of the ID tracking result of each landmark point or to identify the ID of the lost tracking landmark point. In this process, the projection distance feature, the time domain motion feature of each finger joint, and the motion axis angle and motion direction constraint feature of each finger joint can be used for comparison to verify the correctness of the ID tracking result of each landmark point or to identify the ID of the lost tracking landmark point.

[0141] Verifying the correctness of the ID tracking result refers to verifying whether there is ambiguity or mismatch in the matching of the landmark point ID;

[0142] The projection distance feature is that the spatial coordinates of the landmark points that have been determined to match are projected into the visible light image, and the distance between the projection points and the detected key points in the visible light waveband dexterous hand images is matched. If the distance between all the projection points of the spatial coordinates of the landmark points and the corresponding key points is the closest, it is considered that the matching is successful. If there are multiple key points with similar distances to the projection points of the spatial coordinates of the landmark points, or it is difficult to distinguish the closest points, it is considered that there is ambiguity or mismatch in the matching;

[0143] The motion axis angle and motion direction constraint features of each finger joint are that, in the second pose data of the dexterous hand obtained by performing key point detection and motion pose estimation on the visible light band dexterous hand image based on a deep learning method, no abnormal finger bending phenomenon appears, and if the difference between the first pose data obtained by matching and solving the spatial coordinates of the marker points and the second pose data is too large, it is considered that there is a mismatch.

[0144] Optionally, the ID tracking result of the marker point can be verified or the lost marker point can be identified by ID by comparing the coordinate positions and IDs of the marker point spatial coordinate projection points and the key points, and combining the projection distance features, the time domain motion features of each finger joint, and the motion axis angle and motion direction constraint features of each finger joint.

[0145] For image frames in which the marker point ID tracking result is incorrect (the marker point ID matching is ambiguous or mismatched) or the marker point information is lost, based on the features between the marker points and the key points of each joint and the spatial perspective features, and the motion axis angle and motion direction constraint features of each finger joint, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused to obtain the motion data of the dexterous hand.

[0146] In some embodiments, the features between the marker points and the key points of each joint are that, the distance between the spatial coordinate projection points of the marker points and the key points is used as the basis for the fusion weight proportion, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused, and the closer the distance between the spatial coordinate projection points of the marker points and the key points, the more reliable the second pose data of the dexterous hand determined based on the visible light band dexterous hand image.

[0147] In some embodiments, the spatial perspective features are that, in the case that the imaging of the dexterous hand in the visible light band dexterous hand image is not orthographic (for example, the fingers of the dexterous hand are blocked in the image), the reliability of the visible light band dexterous hand image will be reduced.

[0148] In the fusion process, various methods can be used, including weighted fusion and optimal selection fusion. In some embodiments, in the fusion process, if a certain modality loses the current frame image, the lost current frame image can be predicted based on the previous frame image of the modality and the current frame image of another modality.

[0149] Optionally, the specific technical implementation of the fusion process can be direct fusion or filtering, such as Kalman filtering.

[0150] In the embodiments of the present application, the dexterous hand computing unit projects each marker point to the visible light band dexterous hand image according to the spatial coordinates of each marker point, combines the projection distance feature, the time domain motion feature of the finger joint, and the motion axis angle and direction constraint feature with the key point detection result for comparison, verifies the correctness of the marker point ID tracking, and identifies the lost marker point ID. For the image frame with ID tracking error or lost marker point information, the first pose data and the second pose data of the dexterous hand are fused based on the feature between the marker point and each joint key point, the spatial perspective feature, and the motion axis angle and direction constraint feature of the finger joint, so as to obtain the motion data of the dexterous hand, realize the motion data of the dexterous hand based on the perception data of the dual modal, and obtain the motion data of the dexterous hand based on the first motion data obtained by the motion calculation of the dexterous hand on the visible light band dexterous hand image in the case of marker point information loss or tracking error, thereby improving the accuracy and robustness of the dexterous hand motion capture system in capturing the motion of the dexterous hand.

[0151] In some embodiments of the present application, the dexterous hand computing unit 103 is configured to perform at least one of the following:

[0152] Performing epipolar geometry constraint on each marker point in the near-infrared or infrared band dexterous hand image to determine a two-dimensional homonymous point coordinate point set in different near-infrared or infrared band dexterous hand images;

[0153] Performing three-dimensional reconstruction and tracking on each homonymous point based on the two-dimensional homonymous point coordinate point set in different near-infrared or infrared band dexterous hand images to determine the spatial coordinates and ID of each marker point;

[0154] Performing motion calculation on the palm of the dexterous hand based on the spatial coordinates and ID of the palm marker point and the dexterous hand reference template to obtain palm pose data of the dexterous hand;

[0155] Performing motion calculation on the finger joint of the dexterous hand based on the palm pose data and the parent-child link relationship between each finger and palm root joint of the dexterous hand to obtain first pose data of the dexterous hand, wherein the hand kinematics feature includes at least one of the following: finger distance feature, time domain motion feature of each finger joint, motion axis angle and motion direction constraint feature of each finger joint, and parent-child joint relative motion restriction feature;

[0156] Performing key point detection and motion pose estimation on the visible light band dexterous hand image based on a deep learning method, and sending the key point detection result and second pose data of the dexterous hand to the dexterous hand computing unit;

[0157] Based on the spatial coordinates of each of the marker points, each of the marker points is re-projected onto the visible light band dexterous hand image, and based on a projection distance feature, a time domain motion feature of each of the finger joints, and a motion axis angle and motion direction constraint feature of each of the finger joints, the ID tracking result of each of the marker points is verified or a lost tracking marker point is identified.

[0158] For an image frame in which a marker point ID tracking error or marker point information is lost, based on a feature between a marker point and each of the joint key points and a spatial perspective feature, and a motion axis angle and motion direction constraint feature of each of the finger joints, first pose data of the dexterous hand and second pose data of the dexterous hand are fused to obtain motion data of the dexterous hand.

[0159] Figure 4 Fig. 4 is a structural schematic diagram of a dexterous hand motion capture system according to some embodiments of the present application. Figure 4 As shown in Fig. 4, in some embodiments, the dexterous hand motion capture sensor group 101 further comprises at least one light supplementing light source 108 for irradiating the dexterous hand so that the marker points form a contrast with the background in imaging of the near-infrared or infrared band image sensor 104 under irradiation of the light supplementing light source 108.

[0160] It can be understood that the light supplementing light source 108 is used to provide near-infrared or infrared band light to irradiate the dexterous hand, so that the near-infrared or infrared band image sensor 104 can more clearly perceive the dexterous hand, and the quality of the near-infrared or infrared band dexterous hand image output by the near-infrared or infrared band image sensor 104 is improved, and the near-infrared or infrared band image sensor 104 can still perceive the dexterous hand in the case that the near-infrared or infrared band light irradiation condition is not ideal.

[0161] The dexterous hand is arranged with a plurality of marker points, and under irradiation of the light supplementing light source 108, the marker points form a contrast with the background in imaging of the near-infrared or infrared band image sensor, so that the marker points are more prominent in the near-infrared or infrared band dexterous hand image output by the near-infrared or infrared band image sensor 104, and form a contrast with the surrounding background, so that the marker points are more easily identified and tracked in subsequent image processing and analysis, and the accuracy of identifying and locating the marker points is improved.

[0162] The dexterous hand motion capture system in the embodiments of the present application further comprises at least one light supplementing light source for irradiating the dexterous hand, so that the marker points form a contrast with the background in imaging of the near-infrared or infrared band image sensor under irradiation of the light supplementing light source, so that the marker points are more prominent, the accuracy of identifying and locating the marker points in the near-infrared or infrared band dexterous hand image is improved, and the accuracy of the dexterous hand motion capture system in capturing the motion of the dexterous hand is further improved.

[0163] Figure 5 is a flowchart of a dexterous hand motion capture method provided by some embodiments of the present application. As shown in the figure, the dexterous hand motion capture method includes steps 510, 520, 530, 540, 550, and 560. Figure 5

[0164] Step 510: perceiving the dexterous hand in the near-infrared or infrared waveband and outputting a near-infrared or infrared waveband dexterous hand image.

[0165] In some embodiments, the landmark points include palm landmark points and finger joint landmark points.

[0166] The palm of the dexterous hand is taken as a root joint, and at least 6 palm landmark points are arranged at the wrist of the dexterous hand and the connection between each finger and the palm. According to the finger joint topology of the dexterous hand, at least 2 finger joint landmark points are arranged on each finger of the dexterous hand, wherein each joint on the finger shares one finger joint landmark point with its parent joint.

[0167] Step 520: performing epipolar geometry constraint, three-dimensional reconstruction, and tracking on each landmark point in the near-infrared or infrared waveband dexterous hand image to determine the spatial coordinates and ID of each landmark point.

[0168] Step 530: performing motion solving on each joint of the dexterous hand based on a dexterous hand reference template according to the spatial coordinates and ID of each landmark point to obtain first pose data of the dexterous hand.

[0169] In some embodiments, the dexterous hand reference template includes initial pose data of the dexterous hand, initial spatial coordinates, and initial identification information ID of each landmark point, and the method further includes:

[0170] performing epipolar geometry constraint and three-dimensional reconstruction on each landmark point in the near-infrared or infrared waveband dexterous hand image in the reference state to determine the initial spatial coordinates and initial ID of each landmark point;

[0171] performing motion solving on each joint of the dexterous hand based on each landmark point in the near-infrared or infrared waveband dexterous hand image in the reference state to obtain initial pose data of the dexterous hand.

[0172] In some embodiments, the performing epipolar geometry constraint, three-dimensional reconstruction, and tracking on each landmark point in the near-infrared or infrared waveband dexterous hand image to determine the spatial coordinates and ID of each landmark point includes:

[0173] ​The epipolar geometry constraint is performed on each of the mark points in the near-infrared or infrared waveband dexterous hand image, so as to determine a two-dimensional homonymy point coordinate point set in different near-infrared or infrared waveband dexterous hand images.

[0174] Based on the two-dimensional homonymy point coordinate point set in different near-infrared or infrared waveband dexterous hand images, three-dimensional reconstruction and tracking are performed on each homonymy point, so as to determine the spatial coordinates and ID of each mark point.

[0175] In some embodiments, based on the spatial coordinates and ID of each mark point, the motion of each joint of the dexterous hand is solved based on a dexterous hand reference template, so as to obtain first pose data of the dexterous hand, including:

[0176] Based on the spatial coordinates and ID of the palm mark point, the motion of the palm of the dexterous hand is solved based on the dexterous hand reference template, so as to obtain palm pose data of the dexterous hand.

[0177] Based on the palm pose data and the parent-child linkage relationship of each finger joint and palm root joint of the dexterous hand, the motion of the finger joint of the dexterous hand is solved based on hand kinematic characteristics, so as to obtain first pose data of the dexterous hand, the hand kinematic characteristics including at least one of the following: finger distance characteristics, time-domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and parent-child joint relative motion restriction characteristics.

[0178] Step 540, perceiving the dexterous hand in the visible light waveband and outputting a visible light waveband dexterous hand image.

[0179] Step 550, performing key point detection and motion pose estimation on the visible light waveband dexterous hand image based on a deep learning method, so as to obtain key point detection results and second pose data of the dexterous hand.

[0180] Step 560, based on the spatial coordinates, ID of each mark point, and first pose data of the dexterous hand, and the key point detection results and second pose data of the dexterous hand, determining motion data of the dexterous hand.

[0181] In some embodiments, based on the spatial coordinates, ID of each mark point, and first pose data of the dexterous hand, and the key point detection results and second pose data of the dexterous hand, determining motion data of the dexterous hand, including:

[0182] Based on the spatial coordinates of each of the landmark points, each of the landmark points is re-projected onto the visible light band dexterous hand image, and based on a projection distance feature, a time domain motion feature of each of the finger joints, and a motion axis angle and motion direction constraint feature of each of the finger joints, the ID tracking result of each of the landmark points is verified or an ID of a lost tracking landmark point is identified by comparing with the key point detection result.

[0183] For an image frame in which a landmark point ID tracking is incorrect or a landmark point information is lost, first pose data of the dexterous hand and second pose data of the dexterous hand are fused based on a feature between the landmark points and the joint key points and a spatial perspective feature, and a motion axis angle and motion direction constraint feature of each of the finger joints, to obtain motion data of the dexterous hand.

[0184] In some embodiments, before the dexterous hand in the near-infrared or infrared band is perceived and the near-infrared or infrared band dexterous hand image is output, the method further includes:

[0185] Taking a palm and finger straightened state of the dexterous hand as a reference state, a near-infrared or infrared band dexterous hand image in the reference state is output based on the near-infrared or infrared band image sensor, to output a dexterous hand reference template.

[0186] The dexterous hand motion capture method provided by the embodiments of the present application can be understood with reference to the description of the foregoing embodiments, and the same technical effects can be achieved. To avoid repetition, no further description is given here.

[0187] Figure 6 is a structural schematic diagram of an electronic device provided by some embodiments of the present application. In some embodiments, as shown in Figure 6 the embodiments of the present application also provide an electronic device 600, which includes a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned dexterous hand motion capture method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, no further description is given here.

[0188] The embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, each process of the above-mentioned dexterous hand motion capture method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, no further description is given here.

[0189] The processor is the processor of the electronic device in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0190] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the dexterous hand motion capture method.

[0191] The processor is the processor in the electronic device in the above embodiment. The readable storage medium comprises a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0192] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, wherein the communication interface is coupled with the processor, and the processor is configured to run a program or an instruction, so as to implement each process of the dexterous hand motion capture method embodiment and achieve the same technical effects. To avoid repetition, details are not described herein.

[0193] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip, etc.

[0194] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0195] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk, etc.), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server or network device, etc.) execute the method described in each embodiment of the present application.

[0196] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection of the present application.

[0197] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0198] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A dexterous hand motion capture system, characterized in that: include: Dexterous hand motion capture sensor group, dexterous hand reference template creation unit and dexterous hand calculation unit, The dexterous hand motion capture sensor group includes a plurality of near-infrared or infrared band image sensors and a plurality of visible light band image sensors. The near-infrared or infrared band image sensors are used to sense the dexterous hand in the near-infrared or infrared band and output the dexterous hand image in the near-infrared or infrared band. The visible light band image sensors are used to sense the dexterous hand in the visible light band and output the dexterous hand image in the visible light band. The dexterous hand reference template creation unit is configured to output a dexterous hand reference template based on a near-infrared or infrared band dexterous hand image in the reference state acquired by the near-infrared or infrared band image sensor, with the palm and fingers of the dexterous hand stretched as a reference state; The dexterous hand calculation unit is connected to the dexterous hand motion capture sensor group and the dexterous hand reference template creation unit, and is used to process output data of at least one of the dexterous hand motion capture sensor groups based on the dexterous hand reference template to determine motion data of the dexterous hand; A plurality of landmarks are arranged on the dexterous hand.

2. The dexterous hand motion capture system according to claim 1, characterized in that: The landmarks include palm landmarks and finger joint landmarks; Taking the palm of the dexterous hand as the root joint, at least 6 palm landmark points are arranged at the wrist of the dexterous hand and at the connection between each finger and the palm. According to the finger joint topology of the dexterous hand, at least 2 finger joint landmark points are also arranged on each finger of the dexterous hand, wherein each joint on the finger shares one finger joint landmark point with its parent joint.

3. The dexterous hand motion capture system according to claim 2, characterized in that: The dexterous hand reference template includes the initial posture data of the dexterous hand, the initial spatial coordinates of each of the landmarks and the initial identification information ID; The outputting of the dexterous hand reference template based on the near-infrared or infrared band dexterous hand image in the reference state acquired by the near-infrared or infrared band image sensor comprises: Performing epipolar geometry constraints and three-dimensional reconstruction on each of the marker points in the near-infrared or infrared band dexterous hand image in the reference state to determine the initial spatial coordinates and initial ID of each of the marker points; Based on the landmark points in the near-infrared or infrared band image of the dexterous hand in the reference state, motion calculation is performed on the joints of the dexterous hand to obtain the initial posture data of the dexterous hand.

4. The dexterous hand motion capture system according to claim 3, characterized in that: The dexterous hand motion capture sensor group further includes at least one first processing unit connected to the near-infrared or infrared band image sensor, and configured to perform epipolar geometry constraint, three-dimensional reconstruction, and tracking on each of the marker points in the near-infrared or infrared band dexterous hand image, and determine the spatial coordinates and ID of each of the marker points, including: Performing epipolar geometry constraints on each of the landmark points in the near-infrared or infrared band dexterous hand image to determine a set of two-dimensional coordinate points of the same name in different near-infrared or infrared band dexterous hand images; Based on the two-dimensional coordinate point sets of the same-name points in the different near-infrared or infrared band dexterous hand images, each of the same-name points is three-dimensionally reconstructed and tracked to determine the spatial coordinates and ID of each of the marker points.

5. The dexterous hand motion capture system according to claim 4, characterized in that: The first processing unit is further connected to the dexterous hand reference template creation unit, and the first processing unit is further configured to: According to the spatial coordinates and IDs of the palm landmarks, based on the dexterous hand reference template, the palm of the dexterous hand is subjected to motion calculation to obtain the palm posture data of the dexterous hand; According to the palm posture data and the parent-child link relationship between the fingers and palm root joints of the dexterous hand, the motion of the finger joints of the dexterous hand is solved based on the hand kinematic characteristics to obtain the first position data of the dexterous hand. The hand kinematic characteristics include at least one of the following: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and relative motion restriction characteristics of parent-child joints.

6. The dexterous hand motion capture system according to claim 5, characterized in that: The dexterous hand motion capture sensor group also includes at least one second processing unit, which is connected to the visible light band image sensor and is used to perform key point detection and motion posture estimation on the visible light band dexterous hand image based on a deep learning method, and send the key point detection results and the second posture data of the dexterous hand to the dexterous hand computing unit.

7. The dexterous hand motion capture system according to claim 6, characterized in that: The dexterous hand computing unit is used for: Based on the spatial coordinates of each marker point, each marker point is reprojected onto the visible light band dexterous hand image, and compared with the key point detection results based on the projection distance characteristics, the time domain motion characteristics of each finger joint, and the motion axis angle and motion direction constraint characteristics of each finger joint to verify the correctness of the ID tracking results of each marker point or perform ID identification on the marker point that has lost tracking; For image frames with incorrect marker point ID tracking or missing marker point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the features between the marker points and the key points of each joint and the spatial perspective features, as well as the motion axis angle and motion direction constraint features of each finger joint to obtain the motion data of the dexterous hand.

8. The dexterous hand motion capture system according to claim 1, characterized in that: The dexterous hand motion capture sensor group also includes at least one fill light source for illuminating the dexterous hand so that each of the marker points forms a contrast with the background in the imaging of the near-infrared or infrared band image sensor under the illumination of the fill light source.

9. A dexterous hand motion capture method based on the dexterous hand motion capture system according to any one of claims 1 to 8, characterized in that: include: Sense the dexterous hand in the near-infrared or infrared band, and output the dexterous hand image in the near-infrared or infrared band; Performing epipolar geometry constraints, three-dimensional reconstruction, and tracking on each marker point in the near-infrared or infrared band dexterous hand image to determine the spatial coordinates and ID of each marker point; According to the spatial coordinates and IDs of each of the landmark points, based on the dexterous hand benchmark template, the motion of each joint of the dexterous hand is solved to obtain the first position pose data of the dexterous hand; sensing the dexterous hand in the visible light band and outputting an image of the dexterous hand in the visible light band; Perform key point detection and motion posture estimation on the dexterous hand image in the visible light band based on a deep learning method to obtain key point detection results and second posture data of the dexterous hand; The motion data of the dexterous hand is determined based on the spatial coordinates, ID and first posture data of each of the marker points, as well as the key point detection results and second posture data of the dexterous hand.

10. The method for capturing dexterous hand motion according to claim 9, wherein: The step of performing epipolar geometry constraint, three-dimensional reconstruction, and tracking on each marker point in the near-infrared or infrared band dexterous hand image to determine the spatial coordinates and ID of each marker point includes: Performing epipolar geometry constraints on each of the landmark points in the near-infrared or infrared band dexterous hand image to determine a set of two-dimensional coordinate points of the same name in different near-infrared or infrared band dexterous hand images; Based on the two-dimensional coordinate point sets of the same-name points in the different near-infrared or infrared band dexterous hand images, each of the same-name points is three-dimensionally reconstructed and tracked to determine the spatial coordinates and ID of each of the marker points.

11. The method for capturing dexterous hand motion according to claim 9 or 10, characterized in that: The method of performing motion calculation on each joint of the dexterous hand according to the spatial coordinates and ID of each landmark point and based on the dexterous hand reference template to obtain the first position pose data of the dexterous hand includes: According to the spatial coordinates and IDs of the palm landmarks, based on the dexterous hand reference template, the palm of the dexterous hand is subjected to motion calculation to obtain the palm posture data of the dexterous hand; According to the palm posture data and the parent-child link relationship between the fingers and palm root joints of the dexterous hand, the motion of the finger joints of the dexterous hand is solved based on the hand kinematic characteristics to obtain the first position data of the dexterous hand. The hand kinematic characteristics include at least one of the following: finger distance characteristics, time domain motion characteristics of each finger joint, motion axis angle and motion direction constraint characteristics of each finger joint, and relative motion restriction characteristics of parent-child joints.

12. The method for capturing dexterous hand motion according to claim 9, wherein: The determining of the motion data of the dexterous hand based on the spatial coordinates and ID of each of the marker points and the first pose data of the dexterous hand, as well as the key point detection result and the second pose data of the dexterous hand, comprises: Based on the spatial coordinates of each marker point, each marker point is reprojected onto the visible light band dexterous hand image, and compared with the key point detection results based on the projection distance characteristics, the time domain motion characteristics of each finger joint, and the motion axis angle and motion direction constraint characteristics of each finger joint to verify the correctness of the ID tracking results of each marker point or perform ID identification on the marker point that has lost tracking; For image frames with incorrect marker point ID tracking or missing marker point information, the first pose data of the dexterous hand and the second pose data of the dexterous hand are fused based on the features between the marker points and the key points of each joint and the spatial perspective features, as well as the motion axis angle and motion direction constraint features of each finger joint to obtain the motion data of the dexterous hand.

13. The method for capturing dexterous hand motion according to claim 9, wherein: Before sensing the dexterous hand in the near-infrared or infrared band and outputting the dexterous hand image in the near-infrared or infrared band, the method further includes: Taking the palm and fingers of the dexterous hand as the reference state, a dexterous hand reference template is output based on the near-infrared or infrared band dexterous hand image in the reference state acquired by the near-infrared or infrared band image sensor.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dexterous hand motion capture method according to any one of claims 9 to 13 is implemented.

Images