Hand three-dimensional key point detection method and device, electronic equipment and medium

By obtaining the first plane vector of the hand and the world coordinate system of the wrist, combined with the deep learning model, the problem of predicting deviations in high-speed motion or occlusion scenes of three-dimensional key point detection of the hand is solved, and the detection accuracy and anti-interference ability are improved .

CN118918641APending Publication Date: 2024-11-08MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411151299.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing three-dimensional key point detection scheme for the hand is prone to prediction deviations in high-speed hand movement or occlusion scenarios, resulting in low prediction accuracy.

Method used

By obtaining the three-dimensional key points of the human body corresponding to the image frame, obtaining the first plane vector of the hand and the world coordinate system of the wrist, as input to the deep learning model, outputting the wrist local rotation matrix and the finger local rotation matrix, combining the plane vector deviation, determine The confidence of three-dimensional key points of the hand filters unnatural hand postures and improves detection accuracy.

Benefits of technology

It improves the accuracy of hand 3D key point detection, is suitable for high-speed hand movement or occlusion application scenarios, reduces the rate of misjudgment, and makes the output hand 3D key point more in line with artificial engineering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918641A_ABST
    Figure CN118918641A_ABST
Patent Text Reader

Abstract

The invention discloses a hand three-dimensional key point detection method and device, electronic equipment and a medium. The method comprises the following steps: acquiring a first plane vector of a hand and a world coordinate system of a wrist according to a human body three-dimensional key point corresponding to an image frame; wherein the first plane vector is used for representing the preliminarily estimated palm plane attitude; and determining a hand three-dimensional key point corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist. According to the scheme, hand three-dimensional key point detection under guidance of priori knowledge of limbs can be achieved, the output hand three-dimensional key points better conform to artificial engineering, unnatural conditions such as wrist rotation distortion are reduced, and the detection accuracy of the hand three-dimensional key points is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic equipment and medium for detecting three-dimensional key points of a hand. Background Art

[0002] At present, the main solution for detecting three-dimensional (3D) key points of hands in videos is to use a deep learning model (MANO) for modeling and analyzing human hand postures to derive parameters for each frame of image, obtain the hand model parameters corresponding to each frame of image, input the parameters into the hand model, and output the corresponding 3D key points of the hand.

[0003] Since MANO predicts the rotation information of the hand based on the world coordinate system alone, this prediction method can easily cause prediction deviations of the three-dimensional key points of the hand when the hand is in high-speed motion or in scenes with occlusion. Summary of the invention

[0004] Embodiments of the present invention provide a method, device, electronic device and medium for detecting three-dimensional key points of a hand, so as to solve the problem that existing three-dimensional key point detection schemes for a hand are prone to prediction bias, resulting in low prediction accuracy.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting three-dimensional key points of a hand, comprising:

[0006] According to the three-dimensional key points of the human body corresponding to the image frame, a first plane vector of the hand and a world coordinate system of the wrist are obtained; wherein the first plane vector is used to represent a preliminary estimated plane posture of the palm;

[0007] The three-dimensional key points of the hand corresponding to the image frame are determined according to the first plane vector and the world coordinate system of the wrist.

[0008] Optionally, acquiring a first plane vector of the hand according to the three-dimensional key point of the human body corresponding to the image frame includes:

[0009] Obtaining the coordinates of the first wrist key point and the first palm key point in the three-dimensional key points of the human body;

[0010] A direction vector pointing from the first wrist key point coordinates to the first palm key point coordinates is determined as the first plane vector.

[0011] Optionally, determining the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist includes:

[0012] Taking the first plane vector and the world coordinate system of the wrist as inputs of a deep learning model, and outputting a wrist local rotation matrix and a finger local rotation matrix;

[0013] The wrist local rotation matrix and the finger local rotation matrix are used as inputs of a parameterized hand model, and three-dimensional key points of the hand corresponding to the image frame are output.

[0014] Optionally, the output of the deep learning model also includes a plane vector deviation;

[0015] The above-mentioned hand three-dimensional key point detection method also includes:

[0016] Determine a second plane vector according to the three-dimensional key point of the hand, wherein the second plane vector is used to characterize the predicted plane posture of the palm;

[0017] Determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector;

[0018] The confidence of the three-dimensional key point of the hand is determined according to the second plane vector and the third plane vector.

[0019] Optionally, determining the second plane vector according to the three-dimensional key point of the hand includes:

[0020] Acquire the coordinates of the second wrist key point and the second palm key point in the three-dimensional key points of the hand;

[0021] The direction vector pointing from the second wrist key point coordinates to the second palm key point coordinates is determined as the second plane vector.

[0022] Optionally, determining the confidence of the three-dimensional key point of the hand according to the second plane vector and the third plane vector includes:

[0023] determining an angle between the second plane vector and the third plane vector;

[0024] The confidence level of the three-dimensional key point of the hand is determined according to the angle.

[0025] Optionally, determining the confidence of the three-dimensional key point of the hand according to the angle includes:

[0026] Determine the quotient of the angle and a preset maximum tolerance angle to obtain a first value;

[0027] The difference between the preset value and the first value is determined as the confidence level.

[0028] In a second aspect, an embodiment of the present invention further provides a hand three-dimensional key point detection device, comprising:

[0029] A first acquisition module, used to acquire a first plane vector of the hand and a world coordinate system of the wrist according to the three-dimensional key points of the human body corresponding to the image frame; wherein the first plane vector is used to represent a preliminarily estimated plane posture of the palm;

[0030] The first processing module is used to determine the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist.

[0031] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the three-dimensional key point detection method of the hand as described above.

[0032] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method for detecting three-dimensional key points of a hand are implemented.

[0033] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the above-mentioned method for detecting three-dimensional key points of a hand.

[0034] In the above scheme, the first plane vector of the hand and the world coordinate system of the wrist are obtained according to the three-dimensional key points of the human body corresponding to the image frame; wherein, the first plane vector is used to characterize the preliminary estimated plane posture of the palm; the first plane vector of the hand and the world coordinate system of the wrist are used as preliminary reference values ​​to determine the three-dimensional key points of the hand corresponding to the image frame. In this way, the three-dimensional key points of the hand can be guided by the three-dimensional key points of the limbs, so that the output three-dimensional key points of the hand are more in line with human engineering. At the same time, by introducing the first plane vector of the hand, the effect of filtering unnatural hand postures can be achieved, and the occurrence of unnatural situations such as wrist rotation and distortion can be reduced, thereby improving the detection accuracy of the three-dimensional key points of the hand, which is suitable for application scenarios where the hand moves at high speed or is occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0036] Figure 1 This is one of the flow charts of the hand three-dimensional key point detection method provided by an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of key points of a hand provided by an embodiment of the present invention;

[0038] Figure 3 This is the second flowchart of the hand three-dimensional key point detection method provided by the embodiment of the present invention;

[0039] Figure 4 is a structural diagram of a hand three-dimensional key point detection device provided by an embodiment of the present invention;

[0040] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 , Figure 1 is a flow chart of a method for detecting three-dimensional key points of a hand provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0043] Step 101, obtaining a first plane vector of the hand and a world coordinate system of the wrist according to the three-dimensional key points of the human body corresponding to the image frame; wherein the first plane vector is used to represent a preliminarily estimated plane posture of the palm.

[0044] For example, the first plane vector can be determined based on the wrist key points and the palm key points, and the first plane vector is used to describe the rotation information or relative position of the palm relative to the wrist to represent the initially estimated palm plane posture.

[0045] For example, the first plane vector can be obtained by calculating the normal vector based on the key points at the bases of multiple pinky fingers, the bases of the index fingers and the wrists, and the orientation of the palm can be described by the first plane vector to represent the preliminarily estimated palm plane posture.

[0046] In specific implementation, the image frame is input into the human body model, and the three-dimensional key points of the human body corresponding to the image frame are output, wherein the human body model refers to a computer model for modeling and describing the structure and movement of the human body; the three-dimensional key points of the human body are analyzed using the Inverse Kinematics (IK) algorithm to determine the world coordinate system of the wrist; and the rough position coordinates of the hand key points in the body are extracted from the three-dimensional key points of the human body, and the first plane vector is obtained based on the rough position coordinates of the hand key points. Among them, the world coordinate system of the wrist refers to the reference system of the position and direction of the wrist in the entire three-dimensional space, and the origin of the coordinates is the key point corresponding to the wrist.

[0047] Step 102: Determine the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist.

[0048] In specific implementation, the first plane vector and the world coordinate system of the wrist can be used as prior knowledge information of the deep learning model to guide the deep learning model to output the three-dimensional key points of the hand.

[0049] In the above embodiment, the first plane vector of the hand and the world coordinate system of the wrist are first obtained according to the three-dimensional key points of the human body corresponding to the image frame; further, the first plane vector of the hand and the world coordinate system of the wrist are used as preliminary reference values ​​to determine the three-dimensional key points of the hand corresponding to the image frame. In this way, the three-dimensional key point detection of the hand can be achieved under the guidance of prior knowledge of the limbs, so that the output three-dimensional key points of the hand are more in line with human engineering. At the same time, by introducing the first plane vector of the hand, the effect of filtering unnatural hand postures can be achieved, and the occurrence of unnatural situations such as wrist rotation and distortion can be reduced, thereby improving the detection accuracy of the three-dimensional key points of the hand, which can be applied to application scenarios where the hand moves at high speed or is blocked.

[0050] In some optional embodiments, in the above step 101, obtaining the first plane vector of the hand according to the three-dimensional key points of the human body corresponding to the image frame includes:

[0051] Obtaining the coordinates of the first wrist key point and the first palm key point in the three-dimensional key points of the human body;

[0052] A direction vector pointing from the first wrist key point coordinates to the first palm key point coordinates is determined as the first plane vector.

[0053] Among them, the wrist key point is also called the root node of the hand, such as Figure 2 The point corresponding to "0" in the figure; the key point of the palm is Figure 2 The point corresponding to "16" in .

[0054] In a specific implementation, the two-dimensional coordinate a=(x1, y1) corresponding to the first wrist key point and the two-dimensional coordinate b=(x2, y2) corresponding to the first palm key point are subtracted to obtain the first plane vector ab=(x1-x2, y1-y2).

[0055] In the above embodiment, the wrist key point coordinates and the palm key point coordinates can be obtained from the three-dimensional key point coordinates of the human body, and the direction vector from the first wrist key point to the first palm key point is used to represent the preliminary estimated palm plane posture. The algorithm is simple and has high accuracy.

[0056] In some optional embodiments, in the above step 102, determining the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist includes:

[0057] Taking the first plane vector and the world coordinate system of the wrist as inputs of a deep learning model, and outputting a wrist local rotation matrix and a finger local rotation matrix;

[0058] The wrist local rotation matrix and the finger local rotation matrix are used as inputs of a parameterized hand model, and three-dimensional key points of the hand corresponding to the image frame are output.

[0059] It should be pointed out that the deep learning model is obtained after training based on a large amount of training sample data. During the training process, the model will learn how to predict the wrist local rotation matrix and the finger local rotation matrix from the input first plane vector and the world coordinate system of the wrist.

[0060] Among them, the parametric hand model is specifically a computer model for modeling and analyzing human hand postures (Model for Articulated Hands, MANO), which can generate realistic three-dimensional hand postures and capture the complex joint structure and movement of the hand.

[0061] The wrist local rotation matrix is ​​a 3×3 hand root node rotation matrix, and the finger local rotation matrix includes a 3×3 selection matrix corresponding to 15 finger keys. Figure 2 As shown, it shows a schematic diagram of finger joints, each finger includes 3 joint points, and 5 fingers have a total of 15 joint points, such as Figure 2 The points corresponding to the numbers "1 to 15" in the figure.

[0062] In specific implementation, the hand rotation information includes: a rotation matrix corresponding to 1 wrist key point and a rotation matrix corresponding to 15 finger joints. After the hand rotation information is converted into rotation Euler angles, it is input into the MANO model, and the MANO model is used to predict the three-dimensional key points of the hand.

[0063] In the above embodiment, the first plane vector of the hand obtained based on the three-dimensional key points of the human body and the world coordinate system of the wrist are used as the input of the deep learning model, which can be used as prior knowledge to guide the deep learning model to obtain the local rotation matrix of the wrist and the local rotation matrix of the fingers; further, the local rotation matrix of the wrist and the local rotation matrix of the fingers are used as the input of the parameterized hand model, which provides a prediction benchmark and a general prediction direction for the hand parameterized model, so that the output result of the hand parameterized model can retain the joint connection relationship of the three-dimensional key points of the hand, constrain the output range of the model, and realize the combination of hand movements and limbs, which can achieve the technical effect of reducing misjudgment, improving anti-interference ability and detection accuracy.

[0064] In some optional embodiments, the output of the deep learning model also includes a plane vector deviation;

[0065] The above-mentioned hand three-dimensional key point detection method also includes:

[0066] Determine a second plane vector according to the three-dimensional key point of the hand, wherein the second plane vector is used to characterize the predicted plane posture of the palm;

[0067] Determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector;

[0068] The confidence of the three-dimensional key point of the hand is determined according to the second plane vector and the third plane vector.

[0069] In the above embodiment, a plane vector deviation is output through a deep learning model; the sum of the first plane vector and the plane vector deviation is calculated to obtain a third plane vector, which can improve the estimation accuracy of the palm plane posture, and further determine the confidence of the three-dimensional key points of the hand based on the second plane vector and the third plane vector, so that the three-dimensional key points of the hand output by the parameterized hand model can be supervised to avoid using the three-dimensional key points of the hand with a confidence lower than a preset threshold for hand driving, thereby reducing the misjudgment rate of hand driving and improving the accuracy of hand driving.

[0070] In some optional embodiments, determining the second plane vector according to the three-dimensional key point of the hand includes:

[0071] Acquire the coordinates of the second wrist key point and the second palm key point in the three-dimensional key points of the hand;

[0072] The direction vector pointing from the second wrist key point coordinates to the second palm key point coordinates is determined as the second plane vector.

[0073] In specific implementation, the two-dimensional coordinate c=(x11, y11) corresponding to the second wrist key point and the two-dimensional coordinate d=(x22, y22) corresponding to the second palm key point are subtracted to obtain the second plane vector cd=(x11-x22, y11-y22).

[0074] In the above embodiment, the coordinates of the second wrist key point and the second palm key point can be obtained from the three-dimensional hand key points output by the hand parameterized model; the direction vector from the second wrist key point to the second palm key point is used to represent the predicted palm plane posture.

[0075] In some other optional embodiments, the output of the deep learning model further includes: a second plane vector, the third plane vector being the sum of the first plane vector and the plane vector deviation;

[0076] The method further comprises:

[0077] The confidence of the three-dimensional key point of the hand is determined according to the second plane vector and the third plane vector.

[0078] In the above embodiment, the first plane vector is corrected through a deep learning model to obtain a third plane vector, which can improve the estimation accuracy of the palm plane posture; further, based on the second plane vector and the third plane vector, the confidence of the three-dimensional key points of the hand is determined, and the three-dimensional key points of the hand output by the parameterized hand model can be supervised to avoid using the three-dimensional key points of the hand with a confidence level lower than a preset threshold for hand driving, thereby reducing the misjudgment rate of hand driving and making hand driving more accurate.

[0079] In some optional embodiments, determining the confidence of the three-dimensional key point of the hand according to the second plane vector and the third plane vector includes:

[0080] determining an angle between the second plane vector and the third plane vector;

[0081] The confidence level of the three-dimensional key point of the hand is determined according to the angle.

[0082] It should be noted that the angle between the second plane vector and the third plane vector can reflect the difference between the initial estimated palm plane posture and the palm plane posture predicted by the model. The confidence calculated based on the angle can characterize the credibility of the three-dimensional key points of the hand. If the confidence is lower than the preset threshold, it means that the predicted three-dimensional key points of the hand are unavailable. In this way, the misjudgment of the hand can be reduced, making the hand drive smoother and more natural.

[0083] In some optional embodiments, determining the confidence of the three-dimensional key point of the hand according to the angle includes:

[0084] Determine the quotient of the angle and a preset maximum tolerance angle to obtain a first value;

[0085] The difference between the preset value and the first value is determined as the confidence level.

[0086] For example, if the angle between the second plane vector and the third plane vector is 10°, and the preset maximum tolerance angle is 100°, the confidence calculation formula can be: It can be understood that if the included angle exceeds the preset maximum tolerance angle, the confidence level will be a negative value.

[0087] It should be pointed out that the preset maximum tolerance angle and the preset value are calibration quantities, which can be adjusted according to actual needs and are not limited thereto.

[0088] like Figure 3 , which shows a flow chart of a hand key point detection method, which may include the following steps:

[0089] Step 21: Input the image frame into the human body model and output the three-dimensional key points of the human body.

[0090] Step 22: Determine the first plane vector of the hand according to the wrist key point and the palm key point in the three-dimensional key points of the human body.

[0091] Step 23: Analyze the three-dimensional key points of the human body through the IK algorithm to obtain the world coordinate system of the wrist.

[0092] Step 24: Input the first plane vector and the world coordinate system of the wrist into the deep learning model, and output the wrist local rotation matrix, the finger local rotation matrix and the plane vector deviation.

[0093] Step 25: Input the wrist local rotation matrix and the finger local rotation matrix into the MANO model and output the coordinates of the hand key points.

[0094] Step 26: Determine a second plane vector based on the three-dimensional key points of the hand; determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector; determine the confidence of the three-dimensional key points of the hand based on the second plane vector and the third plane vector.

[0095] See also Figure 4 The embodiment of the present invention also provides a three-dimensional key point detection device for a hand. Figure 4 As shown, the hand three-dimensional key point detection device 400 includes:

[0096] A first acquisition module 401 is used to acquire a first plane vector of the hand and a world coordinate system of the wrist according to the three-dimensional key points of the human body corresponding to the image frame; wherein the first plane vector is used to represent a preliminarily estimated plane posture of the palm;

[0097] The first processing module 402 is used to determine the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist.

[0098] Optionally, the first acquisition module 401 is specifically configured to:

[0099] Obtaining the coordinates of the first wrist key point and the first palm key point in the three-dimensional key points of the human body;

[0100] A direction vector pointing from the first wrist key point coordinates to the first palm key point coordinates is determined as the first plane vector.

[0101] Optionally, the first processing module 402 is specifically configured to:

[0102] Taking the first plane vector and the world coordinate system of the wrist as inputs of a deep learning model, and outputting a wrist local rotation matrix and a finger local rotation matrix;

[0103] The wrist local rotation matrix and the finger local rotation matrix are used as inputs of a parameterized hand model, and three-dimensional key points of the hand corresponding to the image frame are output.

[0104] Optionally, the output of the deep learning model also includes a plane vector deviation;

[0105] The apparatus 400 further comprises:

[0106] A first determination module, used for determining a second plane vector according to the three-dimensional key point of the hand, wherein the second plane vector is used for representing the predicted plane posture of the palm;

[0107] A second determination module is used to determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector;

[0108] The third determination module is used to determine the confidence of the three-dimensional key point of the hand according to the second plane vector and the third plane vector.

[0109] Optionally, the first determining module is specifically configured to:

[0110] Acquire the coordinates of the second wrist key point and the second palm key point in the three-dimensional key points of the hand;

[0111] The direction vector pointing from the second wrist key point coordinates to the second palm key point coordinates is determined as the second plane vector.

[0112] Optionally, the third determining module includes:

[0113] A first determining unit, configured to determine an angle between the second plane vector and the third plane vector;

[0114] The second determining unit is used to determine the confidence of the three-dimensional key point of the hand according to the angle.

[0115] Optionally, the second determining unit is specifically configured to:

[0116] Determine the quotient of the angle and a preset maximum tolerance angle to obtain a first value;

[0117] The difference between the preset value and the first value is determined as the confidence level.

[0118] Since the principle of solving the problem by the three-dimensional key point detection device for the hand is similar to the three-dimensional key point detection method for the hand in the embodiment of the present invention, the implementation of the three-dimensional key point detection device for the hand can refer to the implementation of the method, and the repeated parts will not be repeated.

[0119] like Figure 5 As shown, in order to better achieve the above purpose, Figure 5 As shown, an embodiment of the present invention further provides an electronic device, including:

[0120] A processor 500; and a memory 520 connected to the processor 500 via a bus interface, the memory 520 being used to store programs and data used by the processor 500 when performing operations, and the processor 500 calling and executing the programs and data stored in the memory 520.

[0121] The transceiver 510 is connected to the bus interface and is used to receive and send data under the control of the processor 500; the processor 500 is used to read the program in the memory 520 to implement the following steps:

[0122] According to the three-dimensional key points of the human body corresponding to the image frame, a first plane vector of the hand and a world coordinate system of the wrist are obtained; wherein the first plane vector is used to represent a preliminary estimated plane posture of the palm;

[0123] The three-dimensional key points of the hand corresponding to the image frame are determined according to the first plane vector and the world coordinate system of the wrist.

[0124] Among them, Figure 5In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 500 and various circuits of memory represented by memory 520 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 510 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. For different terminals, the user interface 530 may also be an interface capable of externally connecting to required internal devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 when performing operations.

[0125] Optionally, the processor 500 is further configured to read a program in the memory 520 to implement the following steps:

[0126] Obtaining the coordinates of the first wrist key point and the first palm key point in the three-dimensional key points of the human body;

[0127] A direction vector pointing from the first wrist key point coordinates to the first palm key point coordinates is determined as the first plane vector.

[0128] Optionally, the processor 500 is further configured to read a program in the memory 520 to implement the following steps:

[0129] Taking the first plane vector and the world coordinate system of the wrist as inputs of a deep learning model, and outputting a wrist local rotation matrix and a finger local rotation matrix;

[0130] The wrist local rotation matrix and the finger local rotation matrix are used as inputs of a parameterized hand model, and three-dimensional key points of the hand corresponding to the image frame are output.

[0131] Optionally, the output of the deep learning model also includes a plane vector deviation; the processor 500 is also used to read the program in the memory 520 to implement the following steps:

[0132] Determine a second plane vector according to the three-dimensional key point of the hand, wherein the second plane vector is used to characterize the predicted plane posture of the palm;

[0133] Determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector;

[0134] The confidence of the three-dimensional key point of the hand is determined according to the second plane vector and the third plane vector.

[0135] Optionally, the processor 500 is further configured to read a program in the memory 520 to implement the following steps:

[0136] Acquire the coordinates of the second wrist key point and the second palm key point in the three-dimensional key points of the hand;

[0137] The direction vector pointing from the second wrist key point coordinates to the second palm key point coordinates is determined as the second plane vector.

[0138] Optionally, the processor 500 is further configured to read a program in the memory 520 to implement the following steps:

[0139] determining an angle between the second plane vector and the third plane vector;

[0140] The confidence level of the three-dimensional key point of the hand is determined according to the angle.

[0141] Optionally, the processor 500 is further configured to read a program in the memory 520 to implement the following steps:

[0142] Determine the quotient of the angle and a preset maximum tolerance angle to obtain a first value;

[0143] The difference between the preset value and the first value is determined as the confidence level.

[0144] The electronic device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0145] In addition, the computer-readable storage medium of the embodiment of the present invention is used to store a computer program, and the computer program can be executed by a processor to implement the steps of the above three-dimensional key point detection method of the hand.

[0146] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0149] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0150] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting three-dimensional key points of hands, It is characterized in that include: According to the three-dimensional key points of the human body corresponding to the image frame, a first plane vector of the hand and a world coordinate system of the wrist are obtained; wherein the first plane vector is used to represent a preliminary estimated plane posture of the palm; The three-dimensional key points of the hand corresponding to the image frame are determined according to the first plane vector and the world coordinate system of the wrist.

2. The method according to claim 1, It is characterized in that The step of obtaining a first plane vector of the hand according to the three-dimensional key points of the human body corresponding to the image frame includes: Obtaining the coordinates of the first wrist key point and the first palm key point in the three-dimensional key points of the human body; A direction vector pointing from the first wrist key point coordinates to the first palm key point coordinates is determined as the first plane vector.

3. The method according to claim 1, It is characterized in that Determining the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist includes: Taking the first plane vector and the world coordinate system of the wrist as inputs of a deep learning model, and outputting a wrist local rotation matrix and a finger local rotation matrix; The wrist local rotation matrix and the finger local rotation matrix are used as inputs of a parameterized hand model, and three-dimensional key points of the hand corresponding to the image frame are output.

4. The method according to claim 3, It is characterized in that The output of the deep learning model also includes a plane vector deviation; The method further comprises: Determine a second plane vector according to the three-dimensional key point of the hand, wherein the second plane vector is used to characterize the predicted plane posture of the palm; Determine the sum of the first plane vector and the plane vector deviation to obtain a third plane vector; The confidence of the three-dimensional key point of the hand is determined according to the second plane vector and the third plane vector.

5. The method according to claim 4, It is characterized in that The determining of the second plane vector according to the three-dimensional key point of the hand includes: Acquire the coordinates of the second wrist key point and the second palm key point in the three-dimensional key points of the hand; The direction vector pointing from the second wrist key point coordinates to the second palm key point coordinates is determined as the second plane vector.

6. The method according to claim 4, It is characterized in that The step of determining the confidence of the three-dimensional key point of the hand according to the second plane vector and the third plane vector comprises: determining an angle between the second plane vector and the third plane vector; The confidence level of the three-dimensional key point of the hand is determined according to the angle.

7. The method according to claim 6, It is characterized in that Determining the confidence of the three-dimensional key point of the hand according to the angle includes: Determine the quotient of the angle and a preset maximum tolerance angle to obtain a first value; The difference between a preset value and the first value is determined as the confidence level.

8. A three-dimensional key point detection device for a hand, It is characterized in that include: A first acquisition module, used to acquire a first plane vector of the hand and a world coordinate system of the wrist according to the three-dimensional key points of the human body corresponding to the image frame; wherein the first plane vector is used to represent a preliminarily estimated plane posture of the palm; The first processing module is used to determine the three-dimensional key points of the hand corresponding to the image frame according to the first plane vector and the world coordinate system of the wrist.

9. An electronic device, include: A transceiver, a memory, a processor and a computer program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the method for detecting three-dimensional key points of a hand as described in any one of claims 1 to 7.

10. A computer program product, It is characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps of the hand three-dimensional key point detection method as described in any one of claims 1 to 7.