Parameter adaptive three-dimensional hand reconstruction system and method based on inverse kinematics

Abstract

The invention relates to a parameter adaptive three-dimensional hand reconstruction system and method based on inverse kinematics, and the method is characterized in that an inverse kinematics calculation module calculates the pose of a palm root node by using a singular value decomposition algorithm for an inverse kinematics chain of a hand, and calculates the pose of a finger key point by using a Rodrigues rotation formula; constructing a three-dimensional hand model parameter control module, adaptively adjusting weights omega1 and omega2 of a pose parameter and a position parameter according to a use scene, when a pose parameter value is unreliable, increasing the proportion of the position parameter, and otherwise, reducing the proportion of the position parameter; the parameter dynamic updating module uses a three-dimensional point estimation algorithm to estimate positions of three-dimensional key points of the hand at multiple angles, continuously compares the positions with key points of a reconstructed three-dimensional hand, and adjusts position parameters, omega1 and omega2 in real time if there is a large difference, so that the accuracy of three-position reconstruction of the hand is improved. According to the method, the use flexibility of the three-dimensional hand model can be improved, and the accuracy of a three-dimensional hand reconstruction task is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional hand reconstruction, and relates to a parameter-adaptive three-dimensional hand reconstruction system and method based on inverse kinematics. Background Art

[0002] With the rapid development of human-computer interaction, 3D hand reconstruction has been widely used in virtual reality, augmented reality, large-screen interaction, etc. The flexibility of hand joint movement makes 3D hand reconstruction a challenging task. For this reason, researchers have constructed a 3D hand model (MANO), which can realize the 3D hand reconstruction task by inputting the parameters of the 3D hand model.

[0003] However, the pose parameters (relative rotation angles between the three-dimensional key points of the hand) of the MANO model in the prior art are relatively abstract and difficult to estimate directly from the image. Moreover, the pose parameters, as the main input parameters of the MANO model, limit the flexibility of the MANO model. Therefore, designing a flexible method for using the three-dimensional hand model has great practical significance and application value. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the problem that the control parameters of the MANO model in the prior art are single and the posture parameters are difficult to estimate, and to propose a parameter adaptive three-dimensional hand reconstruction system and method based on inverse kinematics, which improves the flexibility of use of the MANO model and improves the reliability of three-dimensional hand reconstruction.

[0005] The technical solution of the present invention is: a parameter adaptive three-dimensional hand reconstruction system based on inverse kinematics, including an inverse kinematics calculation module, a three-dimensional hand model parameter control module, a three-dimensional hand model module and a parameter dynamic update module;

[0006] The inverse kinematics calculation module uses the preset key point position coordinates of the hand as input, and based on the inverse kinematics law of the hand joints, uses the singular value decomposition algorithm and the Rodriguez rotation formula to solve the palm posture and finger joint posture respectively;

[0007] The input of the 3D hand model parameter control module includes two parts: the palm posture and finger joint posture solved by the inverse kinematics calculation module, and the palm posture and finger joint posture acquired by the sensor in real time. The weights of the two parts of the input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of the input is used as the input control information of the 3D hand model module;

[0008] The 3D hand model module reconstructs the posture of the 3D hand model according to the received input control information, and outputs the position coordinates of the key points of the 3D hand model after reconstruction to the parameter dynamic update module;

[0009] The parameter dynamic update module has two input parts: the key point position coordinates after the 3D hand model is reconstructed, and the key point position coordinates estimated by the 3D key point estimation algorithm on the multi-view image of the hand. When the difference between the two key point position coordinates exceeds the error threshold, the 3D key point position obtained by the 3D key point estimation algorithm is used to solve the palm posture and finger joint posture, or the weights in the 3D hand model parameter control module are adjusted, and the key point position coordinates after the 3D hand model is reconstructed are updated and compared again until the difference between the two key point position coordinates is within the error threshold.

[0010] Furthermore, the method for solving the palm posture by the inverse kinematics calculation module is:

[0011] Three key points on the palm are selected to form a rigid body. The problem of solving the palm posture is converted into the solution of the rigid body transformation between the template hand and the reconstructed three-dimensional hand, that is, solving the least squares distance between the two point sets. The singular value decomposition algorithm is used to calculate the least squares distance. Since the three-dimensional hand reconstruction only considers the rotation transformation of the point set and does not consider the translation transformation, the obtained rotation transformation matrix of the rigid body is the palm posture.

[0012] Furthermore, the method for solving the finger joint posture by the inverse kinematics calculation module is: based on the inverse kinematics law of the finger joint, the relative rotation angle of each finger joint is iteratively solved in sequence through the Rodriguez rotation formula to obtain the finger joint posture.

[0013] Furthermore, the 3D key point positions obtained by the 3D key point estimation algorithm are used to solve the palm posture and finger joint posture, or to adjust the weights in the 3D hand model parameter control module, as follows:

[0014] Extract the coordinates of N 3D key points from the reconstructed 3D hand model;

[0015] Generate several groups of N key point coordinates using multi-view images and 3D key point estimation algorithm;

[0016] When the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm exceeds a second threshold, the 3D key point positions obtained by the 3D key point estimation algorithm are used to solve the palm posture and finger joint posture; when the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm is between the first threshold and the second threshold, the weights in the 3D hand model parameter control module are adjusted to generate new 3D hand model module input control information; wherein the first threshold < the second threshold.

[0017] Furthermore, the difference between the position coordinates of the two key points is the average distance error of each three-dimensional key point after the general alignment.

[0018] A parameter adaptive 3D hand reconstruction method based on inverse kinematics comprises the following steps:

[0019] Taking the preset key point position coordinates of the hand as input, based on the inverse kinematics law of the hand joints, the singular value decomposition algorithm and Rodriguez rotation formula are used to solve the palm posture and finger joint posture respectively;

[0020] Taking the palm posture and finger joint posture solved by calculation and the palm posture and finger joint posture obtained by the sensor in real time as input, the weights of the two parts of input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of input is used as the posture control information for reconstructing the three-dimensional hand model;

[0021] Reconstructing the posture of the three-dimensional hand model according to the posture control information, and outputting the position coordinates of key points of the reconstructed three-dimensional hand model;

[0022] The key point position coordinates after reconstructing the three-dimensional hand model and the key point position coordinates estimated by using the three-dimensional key point estimation algorithm on the multi-view image of the hand are used as input. When the difference between the two key point position coordinates exceeds the error threshold, the three-dimensional key point positions obtained by the three-dimensional key point estimation algorithm are used to recalculate the palm posture and finger joint posture, or the weights are adjusted to update the key point position coordinates after reconstructing the three-dimensional hand model and compare them again until the difference between the two key point position coordinates is within the error threshold.

[0023] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a parameter-adaptive three-dimensional hand reconstruction method based on inverse kinematics are implemented.

[0024] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a parameter adaptive three-dimensional hand reconstruction method based on inverse kinematics are implemented.

[0025] A computer program product comprises a computer program, which, when executed by a processor, implements the steps of a parameter adaptive three-dimensional hand reconstruction method based on inverse kinematics.

[0026] The beneficial effects of the present invention compared with the prior art are:

[0027] (1) The inverse kinematics calculation module of the present invention is based on the kinematics of the hand. It realizes the calculation of the three-dimensional posture from the three-dimensional key points through the singular value decomposition algorithm and the Rodriguez rotation formula, and then reconstructs the three-dimensional hand. Compared with the method of using the three-dimensional posture to control the MANO model to reconstruct the three-dimensional hand in the prior art, the MANO model can be controlled using the three-dimensional key point position while meeting the accuracy of the reconstructed three-dimensional hand.

[0028] (2) The three-dimensional hand model parameter control module of the present invention can freely select and adjust the control parameters of the MANO model according to the usage scenario by adjusting the proportion of posture parameters and position parameters, thereby improving the usage flexibility of the MANO model.

[0029] (3) The 3D hand model parameter dynamic update module of the present invention monitors the status of 3D hand reconstruction in real time through a high-accuracy, multi-view 3D key point estimation algorithm, thereby ensuring the reliability of 3D hand reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a parameter adaptive 3D hand reconstruction system based on inverse kinematics;

[0031] Figure 2 Schematic diagram of the inverse kinematics calculation module. DETAILED DESCRIPTION

[0032] like Figure 1 As shown, a parameter adaptive 3D hand reconstruction system based on inverse kinematics proposed by the present invention includes an inverse kinematics calculation module, a 3D hand model parameter control module, a 3D hand model module and a parameter dynamic update module;

[0033] The inverse kinematics calculation module uses the preset key point position coordinates of the hand as input, and based on the inverse kinematics law of the hand joints, uses the singular value decomposition algorithm and the Rodriguez rotation formula to solve the palm posture and finger joint posture respectively;

[0034] The input of the 3D hand model parameter control module includes two parts: the palm posture and finger joint posture solved by the inverse kinematics calculation module, and the palm posture and finger joint posture acquired by the sensor in real time. The weights of the two parts of the input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of the input is used as the input control information of the 3D hand model module;

[0035] The 3D hand model module reconstructs the posture of the 3D hand model according to the received input control information, and outputs the position coordinates of the key points of the 3D hand model after reconstruction to the parameter dynamic update module;

[0036] The parameter dynamic update module has two input parts: the key point position coordinates after the 3D hand model is reconstructed, and the key point position coordinates estimated by the 3D key point estimation algorithm on the multi-view image of the hand. When the difference between the two key point position coordinates (the average distance error of each 3D key point after universal alignment) exceeds the error threshold, the 3D key point position obtained by the 3D key point estimation algorithm is used to solve the palm posture and finger joint posture, or the weights in the 3D hand model parameter control module are adjusted, and the key point position coordinates after the 3D hand model is reconstructed are updated and compared again until the difference between the two key point position coordinates is within the error threshold.

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0038] Example 1

[0039] A parameter adaptive three-dimensional hand reconstruction system based on inverse kinematics includes an inverse kinematics calculation module, a three-dimensional hand model parameter control module, a three-dimensional hand model module and a parameter dynamic update module.

[0040] (1) The inverse kinematics calculation module takes the preset key point position coordinates of the hand as input, and based on the inverse kinematics law of the hand joints, uses the singular value decomposition algorithm and the Rodriguez rotation formula to solve the palm posture and finger joint posture respectively.

[0041] The inverse kinematics calculation module uses inverse kinematics technology, such as Figure 2 As shown, the inverse kinematics process refers to the process of solving the three-dimensional key point posture from the three-dimensional key point position, which is divided into the calculation of the palm root node posture and the calculation of the finger joint posture. In the process of calculating the palm root node posture, the three key points on the palm are used for solving through the singular value decomposition algorithm. The three key points can form a rigid body structure. The posture of the palm root node can be obtained by calculating the rigid body transformation between the three key points between the template hand and the reconstructed hand. The details are as follows:

[0042] The method for solving the palm posture includes: selecting three key points on the palm to form a rigid body. The palm posture solution problem can be converted into the solution of the rigid body transformation between the template hand (human hand) and the reconstructed three-dimensional hand, that is, solving the least squares distance between the two point sets, and using the singular value decomposition algorithm to calculate the least squares distance. Since the three-dimensional hand reconstruction only considers the rotation transformation of the point set and does not consider the translation transformation, the obtained rigid body rotation transformation matrix is ​​the palm posture (relative rotation angle).

[0043] The method of solving the finger joint posture comprises: based on the inverse kinematics law of the finger joint, iteratively solving the relative rotation angle of each finger joint in turn through the Rodriguez rotation formula to obtain the finger joint posture.

[0044] (2) The input of the three-dimensional hand model parameter control module includes two parts: the palm posture and finger joint posture solved by the inverse kinematics calculation module, and the palm posture and finger joint posture acquired by the sensor in real time. The weights of the two parts of the input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of the input is used as the input control information of the three-dimensional hand model module.

[0045] The 3D hand model parameter control module can adaptively adjust the weights of the two parts of the input according to the usage scenario. It supports three methods of reconstructing the 3D hand model: calculating the posture control information only using the palm posture and finger joint posture solved by the inverse kinematics calculation module, calculating the posture control information only using the palm posture and finger joint posture obtained in real time by the sensor, and calculating the posture control information using the two parts of the input together. The proportion of posture parameters and position parameters can be freely adjusted according to the accuracy of the sensor, thereby improving the flexibility of using the 3D hand model.

[0046] (3) A three-dimensional hand model module reconstructs the posture of the three-dimensional hand model according to the received input control information, and outputs the position coordinates of the key points of the reconstructed three-dimensional hand model to the parameter dynamic update module.

[0047] (4) Parameter dynamic update module, the input includes two parts: the key point position coordinates after the 3D hand model is reconstructed, and the key point position coordinates estimated by using the 3D key point estimation algorithm on the multi-view image of the hand. When the difference between the two key point position coordinates exceeds the error threshold, the 3D key point position obtained by the 3D key point estimation algorithm is used to solve the palm posture and finger joint posture, or the weight in the 3D hand model parameter control module is adjusted to update the key point position coordinates after the 3D hand model is reconstructed and compared again until the difference between the two key point position coordinates is within the error threshold.

[0048] In this embodiment, the parameter dynamic update module uses the 3D key point estimation algorithm and multi-view images to perform real-time supervision on the reconstructed 3D hand model and dynamically adjust the position parameters to improve the accuracy of the reconstructed 3D hand. The process is as follows:

[0049] The first step is to extract the coordinates of 21 3D key points from the reconstructed 3D hand model.

[0050] The second step is to generate several groups of 21 key point coordinates using multi-view images and 3D key point estimation algorithm;

[0051] Step 3: When the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm exceeds a second threshold, the 3D key point positions obtained by the 3D key point estimation algorithm are used to solve the palm posture and finger joint posture; when the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm is between the first threshold and the second threshold, the weights in the 3D hand model parameter control module are adjusted to generate new 3D hand model module input control information;

[0052] Step 4: Update the coordinates of the key points after the 3D hand model reconstruction, calculate the difference between the key points generated by the 3D key point estimation algorithm, and repeat steps 3 to 4 until the difference is less than the first threshold. Where: the first threshold < the second threshold.

[0053] According to the system, a parameter adaptive 3D hand reconstruction method based on inverse kinematics is proposed, comprising the following steps:

[0054] S1. Taking the preset key point position coordinates of the hand as input, based on the inverse kinematics law of the hand joints, the singular value decomposition algorithm and the Rodriguez rotation formula are used to solve the palm posture and finger joint posture respectively;

[0055] S2, taking the palm posture and finger joint posture solved by calculation and the palm posture and finger joint posture acquired by the sensor in real time as input, adaptively setting the weights of the two parts of input according to the usage scenario, and performing weighted sum operation on the two parts of input to obtain the result as the posture control information for reconstructing the three-dimensional hand model;

[0056] S3, reconstructing the posture of the three-dimensional hand model according to the posture control information, and outputting the position coordinates of the key points of the reconstructed three-dimensional hand model;

[0057] S4. Taking the key point position coordinates after reconstructing the three-dimensional hand model and the key point position coordinates estimated by the three-dimensional key point estimation algorithm on the multi-view image of the hand as input, when the difference between the two key point position coordinates exceeds the error threshold, use the three-dimensional key point position obtained by the three-dimensional key point estimation algorithm to recalculate the palm posture and finger joint posture, or adjust the weights, update the key point position coordinates after reconstructing the three-dimensional hand model, and compare them again until the difference between the two key point position coordinates is within the error threshold.

[0058] The present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.

[0059] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0060] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0064] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A parameter-adaptive 3D hand reconstruction system based on inverse kinematics, characterized in that: It includes an inverse kinematics calculation module, a three-dimensional hand model parameter control module, a three-dimensional hand model module and a parameter dynamic update module; The inverse kinematics calculation module uses the preset key point position coordinates of the hand as input, and based on the inverse kinematics law of the hand joints, uses the singular value decomposition algorithm and the Rodriguez rotation formula to solve the palm posture and finger joint posture respectively; The input of the 3D hand model parameter control module includes two parts: the palm posture and finger joint posture solved by the inverse kinematics calculation module, and the palm posture and finger joint posture acquired by the sensor in real time. The weights of the two parts of the input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of the input is used as the input control information of the 3D hand model module; The 3D hand model module reconstructs the posture of the 3D hand model according to the received input control information, and outputs the position coordinates of the key points of the 3D hand model after reconstruction to the parameter dynamic update module; The parameter dynamic update module has two input parts: the key point position coordinates after the 3D hand model is reconstructed, and the key point position coordinates estimated by the 3D key point estimation algorithm on the multi-view image of the hand. When the difference between the two key point position coordinates exceeds the error threshold, the 3D key point position obtained by the 3D key point estimation algorithm is used to solve the palm posture and finger joint posture, or the weights in the 3D hand model parameter control module are adjusted, and the key point position coordinates after the 3D hand model is reconstructed are updated and compared again until the difference between the two key point position coordinates is within the error threshold.

2. The parameter-adaptive 3D hand reconstruction system based on inverse kinematics according to claim 1, characterized in that: The method for solving the palm posture by the inverse kinematics calculation module is: Three key points on the palm are selected to form a rigid body. The problem of solving the palm posture is converted into the solution of the rigid body transformation between the template hand and the reconstructed three-dimensional hand, that is, solving the least squares distance between the two point sets. The singular value decomposition algorithm is used to calculate the least squares distance. Since the three-dimensional hand reconstruction only considers the rotation transformation of the point set and does not consider the translation transformation, the obtained rotation transformation matrix of the rigid body is the palm posture.

3. The parameter-adaptive 3D hand reconstruction system based on inverse kinematics according to claim 1, characterized in that: The method for solving the finger joint posture by the inverse kinematics calculation module is: based on the inverse kinematics law of the finger joint, the relative rotation angle of each finger joint is iteratively solved in turn by the Rodriguez rotation formula to obtain the finger joint posture.

4. The parameter-adaptive 3D hand reconstruction system based on inverse kinematics according to claim 1, characterized in that: Use the 3D key point positions obtained by the 3D key point estimation algorithm to solve the palm pose and finger joint pose, or adjust the weights in the 3D hand model parameter control module, as follows: Extract the coordinates of N 3D key points from the reconstructed 3D hand model; Generate several groups of N key point coordinates using multi-view images and 3D key point estimation algorithm; When the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm exceeds a second threshold, the 3D key point positions obtained by the 3D key point estimation algorithm are used to solve the palm posture and finger joint posture; when the difference between the key point coordinates of the reconstructed 3D hand model and the key point coordinates generated by the 3D key point estimation algorithm is between the first threshold and the second threshold, the weights in the 3D hand model parameter control module are adjusted to generate new 3D hand model module input control information; wherein the first threshold < the second threshold.

5. The parameter-adaptive 3D hand reconstruction system based on inverse kinematics according to claim 1, characterized in that: The difference between the position coordinates of the two key points is the average distance error of each three-dimensional key point after the general alignment.

6. A parameter-adaptive three-dimensional hand reconstruction method based on inverse kinematics based on the system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Taking the preset key point position coordinates of the hand as input, based on the inverse kinematics law of the hand joints, the singular value decomposition algorithm and Rodriguez rotation formula are used to solve the palm posture and finger joint posture respectively; Taking the palm posture and finger joint posture solved by calculation and the palm posture and finger joint posture obtained by the sensor in real time as input, the weights of the two parts of input are adaptively set according to the usage scenario, and the result of the weighted sum operation of the two parts of input is used as the posture control information for reconstructing the three-dimensional hand model; Reconstructing the posture of the three-dimensional hand model according to the posture control information, and outputting the position coordinates of key points of the reconstructed three-dimensional hand model; The key point position coordinates after reconstructing the three-dimensional hand model and the key point position coordinates estimated by using the three-dimensional key point estimation algorithm on the multi-view image of the hand are used as input. When the difference between the two key point position coordinates exceeds the error threshold, the three-dimensional key point positions obtained by the three-dimensional key point estimation algorithm are used to recalculate the palm posture and finger joint posture, or the weights are adjusted to update the key point position coordinates after reconstructing the three-dimensional hand model and compare them again until the difference between the two key point position coordinates is within the error threshold.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method as claimed in claim 6 are implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method as claimed in claim 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 6 are implemented.

Images