Method and device for determining three-dimensional key points of hand

Abstract

The present invention relates to a method for determining three-dimensional key points of a hand, an apparatus for determining three-dimensional key points of a hand, a computer storage medium and a computer device. According to one aspect of the present invention, a method for determining three-dimensional key points of a hand is proposed, which comprises the following steps: acquiring an RGB image of a hand and a depth image of a hand aligned with the RGB image of the hand; determining data of two-dimensional key points of the hand according to the acquired RGB image of the hand; constructing a 2.5D model of the hand based at least on the data of the determined two-dimensional key points of the hand; and determining three-dimensional information of the three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically to a method for determining three-dimensional key points of a hand, an apparatus for determining three-dimensional key points of a hand, a computer storage medium and a computer device. Background Art

[0002] In recent years, due to the rapid development of 3D application scenarios such as augmented reality (AR), virtual reality (VR), games and animations, the application of 3D vision has gradually become popular. The determination of 3D key points of the hand is very important in the 3D reconstruction process in AR, VR, games and animations.

[0003] In the current method of determining the three-dimensional key points of the hand, the correspondence between the depth image and the RGB image is usually used to correspond the two-dimensional key points in the RGB image to the depth image, so as to use the information in the depth image to calculate the three-dimensional key points. However, since the depth image can only represent the depth of the object surface, the occluded hand key points in the depth image may not be labeled or may be labeled incorrectly. In addition, three-dimensional positioning can be performed by binding multi-degree-of-freedom sensors on the hand. However, this method is complicated to operate and requires a lot of hardware and human resources. In addition, since different operators operate the fixed position of the sensor multiple times, it is difficult to keep it consistent, resulting in inaccurate three-dimensional key points. Summary of the invention

[0004] In order to solve or at least alleviate one or more of the above problems, the following technical solutions are provided.

[0005] According to a first aspect of the present invention, a method for determining three-dimensional key points of a hand is provided, the method comprising the following steps: acquiring an RGB image of the hand and a depth image of the hand aligned with the RGB image of the hand; determining data of two-dimensional key points of the hand based on the acquired RGB image of the hand; constructing a 2.5D model of the hand based at least on the determined data of the two-dimensional key points of the hand; and determining three-dimensional information of the three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand.

[0006] According to a method for determining three-dimensional key points of a hand according to an embodiment of the present invention, obtaining an RGB image of the hand and a depth image of the hand aligned with the RGB image of the hand comprises: obtaining the RGB image of the hand using an RGB camera; obtaining the depth image of the hand using a depth camera; and aligning the RGB image of the hand with the depth image of the hand based on intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera.

[0007] According to the method for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, determining the data of the two-dimensional key points of the hand based on the acquired RGB image of the hand includes: training a first convolutional neural network model based on the data of the pre-labeled two-dimensional key points of the hand; using the trained first convolutional neural network model to process the acquired RGB image of the hand; and determining the data of the two-dimensional key points of the hand based on the processing results of the trained first convolutional neural network model, wherein the data of the two-dimensional key points of the hand includes one or more of the two-dimensional coordinate information of the two-dimensional key points of the hand, the occlusion information of the two-dimensional key points of the hand, and the heat map information of the two-dimensional key points of the hand.

[0008] According to the method for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, constructing a 2.5D hand model based at least on the data of the determined two-dimensional key points of the hand includes: determining the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand; and constructing the 2.5D hand model using the shape parameters and gesture parameters of the hand.

[0009] According to the method for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, determining the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand includes: inputting the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand into a multi-layer perceptual neural network model; and determining the shape parameters and gesture parameters of the hand based on the output result of the multi-layer perceptual neural network model.

[0010] According to the method for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, constructing a 2.5D hand model using the shape parameters and gesture parameters of the hand includes: training a second convolutional neural network model based on pre-labeled 2.5D hand key point data; using the trained second convolutional neural network model to process the shape parameters and gesture parameters of the hand; and constructing a 2.5D hand model corresponding to the two-dimensional key points of the hand based on the processing results of the trained second convolutional neural network model.

[0011] According to the method for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any one of the above embodiments, determining the three-dimensional information of the three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand includes: determining the visible key points among the two-dimensional key points of the hand based on the data of the two-dimensional key points of the hand; determining the depth information of the corresponding key points of the visible key points among the two-dimensional key points of the hand in the depth image of the hand; and updating the depth information of the two-dimensional key points of the hand in the constructed 2.5D model of the hand based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

[0012] According to the method for determining three-dimensional key points of the hand according to one embodiment of the present invention or any one of the above embodiments, the method further includes: discarding outliers in the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; determining the mean of the depth information of the remaining corresponding key points; and updating the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model to the sum of the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model and the mean of the depth information of the remaining corresponding key points.

[0013] According to the method for determining three-dimensional key points of a hand according to one embodiment of the present invention or any one of the above embodiments, outliers among the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand are determined in the following manner: determining the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; determining the deviation value of each corresponding key point from the mean of the corresponding key points; and determining that the corresponding key point is an outlier in response to determining that the deviation value between the corresponding key point and the mean of the corresponding key points is greater than a preset value.

[0014] According to a second aspect of the present invention, there is provided an apparatus for determining three-dimensional key points of a hand, the apparatus comprising: an acquisition unit configured to acquire an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand; a determination unit configured to determine data of two-dimensional key points of the hand based on the acquired RGB image of the hand; a construction unit configured to construct a 2.5D model of the hand based at least on the data of the determined two-dimensional key points of the hand; and a processing unit configured to determine three-dimensional information of the three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand.

[0015] According to the device for determining the three-dimensional key points of the hand described in one embodiment of the present invention, the acquisition unit includes an RGB camera and a depth camera, and the acquisition unit is further configured to: acquire an RGB image of the hand using the RGB camera; acquire a depth image of the hand using the depth camera; and align the RGB image of the hand with the depth image of the hand based on the intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera.

[0016] According to the device for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, the determination unit is further configured to: train a first convolutional neural network model based on the data of pre-labeled two-dimensional key points of the hand; use the trained first convolutional neural network model to process the acquired RGB image of the hand; and determine the data of the two-dimensional key points of the hand based on the processing results of the trained first convolutional neural network model, wherein the data of the two-dimensional key points of the hand includes one or more of the two-dimensional coordinate information of the two-dimensional key points of the hand, the occlusion information of the two-dimensional key points of the hand, and the heat map information of the two-dimensional key points of the hand.

[0017] According to the device for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, the construction unit is further configured to: determine the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand; and construct a 2.5D hand model using the shape parameters and gesture parameters of the hand.

[0018] According to the device for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, the construction unit is further configured to: input the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand into a multi-layer perceptual neural network model; and determine the shape parameters and gesture parameters of the hand based on the output results of the multi-layer perceptual neural network model.

[0019] According to the device for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any of the above embodiments, the construction unit is further configured to: train a second convolutional neural network model based on pre-labeled 2.5D hand key point data; use the trained second convolutional neural network model to process the shape parameters and gesture parameters of the hand; and construct a 2.5D hand model corresponding to the two-dimensional key points of the hand based on the processing results of the trained second convolutional neural network model.

[0020] According to the device for determining the three-dimensional key points of the hand according to one embodiment of the present invention or any one of the above embodiments, the processing unit is further configured to: determine the visible key points among the two-dimensional key points of the hand based on the data of the two-dimensional key points of the hand; determine the depth information of the corresponding key points of the visible key points among the two-dimensional key points of the hand in the depth image of the hand; and update the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

[0021] According to the device for determining three-dimensional key points of the hand according to one embodiment of the present invention or any one of the above embodiments, the processing unit is further configured to: discard abnormal points in the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; determine the mean of the depth information of the remaining corresponding key points; and update the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model to the sum of the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model and the mean of the depth information of the remaining corresponding key points.

[0022] According to the device for determining three-dimensional key points of a hand according to one embodiment of the present invention or any one of the above embodiments, the processing unit is configured to determine the outliers among the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand in the following manner: determining the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; determining the deviation value of each corresponding key point from the mean of the corresponding key points; and determining that the corresponding key point is an outlier in response to determining that the deviation value between the corresponding key point and the mean of the corresponding key points is greater than a preset value.

[0023] According to a third aspect of the present invention, there is provided a computer storage medium, the computer storage medium comprising instructions, which when run, execute the steps of the method for determining three-dimensional key points of a hand according to the first aspect of the present invention.

[0024] According to a fourth aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of the method for determining three-dimensional key points of a hand according to the first aspect of the present invention are implemented.

[0025] According to one or more embodiments of the present invention, the scheme for determining the three-dimensional key points of the hand uses an RGB camera and a depth camera to obtain the RGB image and depth image of the hand respectively, and processes the RGB image of the hand with the help of a trained convolutional neural network model and a multi-layer perceptual neural network model to construct a 2.5D model of the hand, so that the three-dimensional key points of the hand can be quickly and accurately determined based on the depth information in the depth image of the hand. In this way, the accuracy of the three-dimensional information of the three-dimensional key points of the hand can be improved, while the calculation speed is improved and the hardware cost and labor cost are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or other aspects and advantages of the present invention will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar units are represented by the same reference numerals. In the accompanying drawings:

[0027] Figure 1 The flowchart is a method for determining three-dimensional key points of a hand according to one or more embodiments of the present invention.

[0028] Figure 2 The block diagram is a device for determining three-dimensional key points of a hand according to one or more embodiments of the present invention.

[0029] Figure 3A and Figure 3B A schematic diagram of hand key points according to one or more embodiments of the present invention is shown.

[0030] Figure 4 A block diagram of a computer device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0031] The description of the following specific embodiments is merely exemplary in nature and is not intended to limit the disclosed technology or the application and use of the disclosed technology. In addition, it is not intended to be bound by any express or implied theory presented in the aforementioned technical field, background technology or the following specific embodiments.

[0032] In the following detailed description of the embodiments, many specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it is apparent to one of ordinary skill in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0033] Terms such as "comprising" and "including" indicate that in addition to the units and steps directly and clearly stated in the specification, the technical solution of the present invention does not exclude the situation of having other units and steps that are not directly or clearly stated. Terms such as "first" and "second" do not indicate the order of units in terms of time, space, size, etc., but are only used to distinguish between units.

[0034] It should be noted that in the context of the present invention, the term "RGB image" refers to an image composed of a three-dimensional array in the format of M×N×3, where "3" can be understood as three M×N two-dimensional images (grayscale images) that capture the same content. These three images represent the R, G, and B components, respectively, and the pixel value range of each component is [0,255], where R represents red, G represents green, and B represents blue. The term "depth image" refers to an image that uses the distance (depth) from the depth camera to each point on the object as a pixel value, which directly reflects the geometry of the visible surface of the object. The term "hand 2.5D model" refers to a model that can accurately represent the position information x and y of the key points of the hand and partially represent the depth information z of the key points of the hand. In this model, the depth information z of the key points of the hand is the difference between the key points of the hand and a fixed position of the wrist.

[0035] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[0036] Figure 1 The flowchart is a method for determining three-dimensional key points of a hand according to one or more embodiments of the present invention.

[0037] like Figure 1 As shown in , in step 101, an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand are obtained.

[0038] Optionally, in step 101, an RGB image of the hand may be acquired using an RGB camera and a depth image of the hand may be acquired using a depth camera, and the RGB image of the hand may be aligned with the depth image of the hand according to the intrinsic and extrinsic parameters of the RGB camera and the depth camera. Exemplarily, the intrinsic and extrinsic parameters of the RGB camera and the depth camera may be calibrated in the same world coordinate system. By acquiring the RGB image of the hand using an RGB camera and acquiring the depth image of the hand using a depth camera, respectively, strong image data support is provided for determining high-precision three-dimensional information of three-dimensional key points of the hand in step 107.

[0039] The intrinsic parameters of the camera indicate the projection relationship from the three-dimensional space to the two-dimensional image, and the extrinsic parameters of the camera indicate the relative position relationship between the camera coordinate system and the world coordinate system, such as the parameters indicating the conversion from the world coordinate system to the camera coordinate system, including the rotation matrix R and the translation vector T. It should be noted that the world coordinate system is a three-dimensional rectangular coordinate system, which can be used as a reference to describe the three-dimensional position of the camera and the object to be measured. It is an absolute coordinate system of the objective three-dimensional world, and its coordinate values ​​are usually represented by Pw (Xw, Yw, Zw). The camera coordinate system is a three-dimensional rectangular coordinate system, with the optical center of the camera as the coordinate origin, the Z axis as the optical axis of the camera, and the X axis and Y axis parallel to the X axis and Y axis in the image coordinate system, respectively. Its coordinate values ​​are usually represented by Pc (Xc, Yc, Zc). Exemplarily, taking pinhole imaging as an example, the camera extrinsic parameters, world coordinates and camera coordinates satisfy the following relationship Pc = RPw + T. In one or more embodiments of the present invention, the three-dimensional information of the three-dimensional key points of the hand can be represented in the camera coordinate system or the world coordinate system.

[0040] In step 103, data of two-dimensional key points of the hand are determined based on the acquired RGB image of the hand.

[0041] In one or more embodiments, the key points of the hand may include the bone nodes of the fingers, the fingertips, the connection points between the fingers and the palm, the wrist joints where the hand and the arm are connected, etc. According to different application scenarios, different positions and numbers of the key points of the hand can be set. For example, the number of key points of the hand can be set to 21 key points from the palm to the fingertips. For example, Figure 3A As shown in FIG. 1 , it shows a schematic diagram of 21 hand key points predefined for the palm area. Figure 3B As shown, it shows a schematic diagram of 21 hand key points pre-defined for the back of the hand area.

[0042] In one or more embodiments, the data of the two-dimensional key points of the hand may include two-dimensional coordinate information of the two-dimensional key points of the hand, occlusion information of the two-dimensional key points of the hand, and heat map information of the two-dimensional key points of the hand. Among them, the occlusion information is used to indicate whether the corresponding two-dimensional key points of the hand are occluded, and the unoccluded two-dimensional key points of the hand are visible two-dimensional key points of the hand. The heat map information is used to indicate the confidence of the corresponding two-dimensional key points of the hand.

[0043] Optionally, in step 103, the first convolutional neural network model can be trained based on the data of the pre-labeled two-dimensional key points of the hand, the trained first convolutional neural network model is used to process the acquired RGB image of the hand, and the data of the two-dimensional key points of the hand are determined based on the processing results of the trained first convolutional neural network model. Exemplarily, the first convolutional neural network model can be trained using the gradient return loss method of the neural network to improve the accuracy of the processing results of the first convolutional neural network model. Alternatively, in step 103, the data of the two-dimensional key points of the hand in the RGB image can also be determined by various other machine learning methods.

[0044] In step 105, a 2.5D hand model is constructed based on at least the data of the determined two-dimensional key points of the hand.

[0045] Optionally, in step 105, the shape parameters and gesture parameters of the hand can be determined based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand, and the shape parameters and gesture parameters of the hand are used to construct a 2.5D model of the hand. Exemplarily, the shape parameters of the hand can indicate the shape attributes of the hand, such as the thickness and length of the fingers and the thickness of the palm, etc., and the gesture parameters of the hand can indicate the posture attributes of the hand, such as the deformation of the hand through the joints.

[0046] In one embodiment, the heat map information of the two-dimensional key points of the hand and the RGB image of the hand in the data of the two-dimensional key points of the hand can be input into the multi-layer perceptual neural network model, and the shape parameters and gesture parameters of the hand are determined based on the output results of the multi-layer perceptual neural network model. As an example, the shape parameters and gesture parameters of the hand may include 10 shape coefficients and 16*3 gesture coefficients, wherein the shape coefficients may be represented by 10 numerical values, and the gesture coefficients may be represented by 16 groups of three-dimensional vectors.

[0047] In one embodiment, a second convolutional neural network model can be trained based on pre-labeled 2.5D key point data of the hand, the shape parameters and gesture parameters of the hand can be processed using the trained second convolutional neural network model, and a 2.5D hand model corresponding to the data of the two-dimensional key points of the hand can be constructed based on the processing results of the trained second convolutional neural network model. Exemplarily, the second convolutional neural network model can be trained using a gradient return loss method of a neural network to improve the accuracy of the processing results of the second convolutional neural network model.

[0048] In one embodiment, the hand 2.5D model can be implemented as a hinged non-rigid deformable hand model (Hand Model with Articulated and Non-rigiddeformations, MANO model), which is a priori parameterized hand model that can decompose the hand mesh into two parts: shape and posture. The shape part mainly models the shape attributes of the hand, such as the thickness, length and thickness of the fingers, and the posture part mainly models the posture attributes of the hand.

[0049] A 2.5D hand model corresponding to the two-dimensional key points of the hand is constructed by obtaining the shape parameters and gesture parameters of the hand from the heat map information of the two-dimensional key points of the hand and the RGB image of the hand, so that the 2.5D hand model matches the hand in the RGB image, ensuring the accuracy of the constructed 2.5D hand model, thereby improving the accuracy of the three-dimensional information of the three-dimensional key points of the hand determined subsequently.

[0050] In step 107, three-dimensional information of three-dimensional key points of the hand is determined based at least on the depth image of the hand and the constructed 2.5D hand model.

[0051] Optionally, in step 107, the visible key points in the two-dimensional key points of the hand can be determined based on the data of the two-dimensional key points of the hand, the depth information of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand can be determined, and the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model can be updated based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

[0052] In one embodiment, the visible key points in the two-dimensional key points of the hand determined in step 103 can be traversed to determine the key points corresponding to the visible key points in the two-dimensional key points of the hand in the constructed 2.5D model of the hand and the depth image of the hand, respectively, and the abnormal points in the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand are discarded.

[0053] In one embodiment, the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand can be determined, the deviation value of each corresponding key point from the mean of the corresponding key points can be determined, and in response to determining that the deviation value of the corresponding key point from the mean of the corresponding key points is greater than a preset value, the corresponding key point is determined to be an abnormal point. Exemplarily, if the deviation value of a certain corresponding key point from the mean of the corresponding key points is greater than a preset value (for example, 30 cm), the corresponding key point is discarded and the mean of the remaining corresponding key points is recalculated, the deviation value of each remaining corresponding key point from the recalculated mean of the remaining corresponding key points is determined, and the corresponding key points whose deviation value from the recalculated mean of the remaining corresponding key points is greater than the preset value (for example, 30 cm) are discarded, and so on, until there are no abnormal points.

[0054] Optionally, in step 107, after discarding the abnormal points in the corresponding key points of the visible key points in the hand two-dimensional key points in the hand depth image, the mean of the depth information of the remaining corresponding key points can be determined, and the depth information of the hand two-dimensional key points in the constructed hand 2.5D model can be updated to the sum of the depth information of the hand two-dimensional key points in the constructed hand 2.5D model and the mean of the depth information of the remaining corresponding key points. Exemplarily, the depth information of the data of the hand two-dimensional key points in the constructed hand 2.5D model can be updated by the formula Znew=Zold+Zwris, where Znew represents the depth information of the data of the hand two-dimensional key points in the updated hand 2.5D model, Zold represents the depth information of the data of the hand two-dimensional key points in the original hand 2.5D model, and Zwris represents the mean of the depth information of the non-abnormal corresponding key points of the visible key points in the hand two-dimensional key points in the hand depth image.

[0055] It should be noted that, as mentioned above, in the constructed 2.5D hand model, the position information x and y of the hand key points are accurate, and the depth information z of the hand key points is the difference between the hand key points and a certain fixed position of the wrist. Therefore, by updating the depth information of the two-dimensional hand key points in the constructed 2.5D hand model through the formula Znew=Zold+Zwris, the three-dimensional information of the three-dimensional hand key points can be determined.

[0056] According to one aspect of the present invention, the method for determining the three-dimensional key points of the hand can obtain the RGB image and the depth image of the hand by using the RGB camera and the depth camera respectively, and process the RGB image of the hand by means of the trained convolutional neural network model and the multi-layer perceptron neural network model to construct a 2.5D model of the hand, so that the three-dimensional key points of the hand can be determined quickly and accurately based on the depth information in the depth image of the hand. In this way, the accuracy of the three-dimensional information of the three-dimensional key points of the hand can be improved, while the calculation speed can be improved and the hardware cost and labor cost can be reduced.

[0057] Figure 2 The figure is a block diagram of a device for determining three-dimensional key points of a hand according to an embodiment of the present invention.

[0058] like Figure 2 As shown in , the device 200 for determining three-dimensional key points of a hand includes an acquisition unit 201, a determination unit 203, a construction unit 205 and a processing unit 207.

[0059] The acquisition unit 201 is configured to acquire an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand.

[0060] Optionally, the acquisition unit 201 may include an RGB camera and a depth camera, which may be configured to acquire an RGB image of the hand using the RGB camera and acquire a depth image of the hand using the depth camera, and align the RGB image of the hand with the depth image of the hand according to the intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera. Exemplarily, the intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera may be calibrated in the same world coordinate system. By acquiring the RGB image of the hand using the RGB camera and acquiring the depth image of the hand using the depth camera, respectively, strong image data support is provided for the processing unit 207 to determine the three-dimensional information of the three-dimensional key points of the hand with high precision.

[0061] The determination unit 203 is configured to determine the data of the two-dimensional key points of the hand according to the acquired RGB image of the hand.

[0062] Optionally, the determination unit 203 can be configured to train a first convolutional neural network model based on the data of the pre-labeled two-dimensional key points of the hand, use the trained first convolutional neural network model to process the acquired RGB image of the hand, and determine the data of the two-dimensional key points of the hand based on the processing results of the trained first convolutional neural network model. Exemplarily, the first convolutional neural network model can be trained using the gradient return loss method of the neural network to improve the accuracy of the processing results of the first convolutional neural network model.

[0063] The construction unit 205 is configured to construct a 2.5D hand model based at least on the data of the determined two-dimensional key points of the hand.

[0064] Optionally, the construction unit 205 can be configured to determine the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand, and to construct a 2.5D model of the hand using the shape parameters and gesture parameters of the hand. Exemplarily, the shape parameters of the hand can indicate the shape attributes of the hand, such as the thickness, length and thickness of the palm of the hand, etc., and the gesture parameters of the hand can indicate the posture attributes of the hand, such as the deformation of the hand through the joints.

[0065] In one embodiment, the construction unit 205 may be configured to input the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand into the multi-layer perceptual neural network model, and determine the shape parameters and gesture parameters of the hand based on the output results of the multi-layer perceptual neural network model. As an example, the shape parameters and gesture parameters of the hand may include 10 shape coefficients and 16*3 gesture coefficients, wherein the shape coefficients may be represented by 10 numerical values, and the gesture coefficients may be represented by 16 groups of three-dimensional vectors.

[0066] A 2.5D hand model corresponding to the two-dimensional key points of the hand is constructed by obtaining the shape parameters and gesture parameters of the hand from the heat map information of the two-dimensional key points of the hand and the RGB image of the hand, so that the 2.5D hand model matches the hand in the RGB image, ensuring the accuracy of the constructed 2.5D hand model, thereby improving the accuracy of the three-dimensional information of the three-dimensional key points of the hand determined subsequently.

[0067] The processing unit 207 is configured to determine three-dimensional information of three-dimensional key points of the hand based on at least the depth image of the hand and the constructed 2.5D model of the hand.

[0068] Optionally, the processing unit 207 can be configured to determine the visible key points among the two-dimensional key points of the hand based on the data of the two-dimensional key points of the hand, determine the depth information of the corresponding key points of the visible key points among the two-dimensional key points of the hand in the depth image of the hand, and update the depth information of the two-dimensional key points of the hand in a constructed 2.5D model of the hand based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

[0069] In one embodiment, the processing unit 207 can be configured to traverse the visible key points in the two-dimensional key points of the hand determined by the determination unit 203 to determine the key points corresponding to the visible key points in the two-dimensional key points of the hand in the constructed 2.5D model of the hand and in the depth image of the hand, respectively, and discard abnormal points in the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand.

[0070] In one embodiment, the processing unit 207 may be configured to determine the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand, determine the deviation value of each corresponding key point from the mean of the corresponding key points, and determine the corresponding key point as an abnormal point in response to determining that the deviation value of the corresponding key point from the mean of the corresponding key points is greater than a preset value. Exemplarily, if the deviation value of a corresponding key point from the mean of the corresponding key points is greater than a preset value (e.g., 30 cm), the corresponding key point is discarded and the mean of the remaining corresponding key points is recalculated, the deviation value of each remaining corresponding key point from the recalculated mean of the remaining corresponding key points is determined, and the corresponding key points whose deviation value from the recalculated mean of the remaining corresponding key points is greater than the preset value (e.g., 30 cm) are discarded, and so on, until there are no abnormal points.

[0071] Optionally, the processing unit 207 can be configured to determine the mean of the depth information of the remaining corresponding key points after discarding the abnormal points in the corresponding key points of the visible key points in the hand two-dimensional key points in the hand depth image, and update the depth information of the hand two-dimensional key points in the constructed hand 2.5D model to the sum of the depth information of the hand two-dimensional key points in the constructed hand 2.5D model and the mean of the depth information of the remaining corresponding key points. Exemplarily, the depth information of the data of the hand two-dimensional key points in the constructed hand 2.5D model can be updated by the formula Znew=Zold+Zwris, where Znew represents the depth information of the data of the hand two-dimensional key points in the updated hand 2.5D model, Zold represents the depth information of the data of the hand two-dimensional key points in the original hand 2.5D model, and Zwris represents the mean of the depth information of the non-abnormal corresponding key points of the visible key points in the hand two-dimensional key points in the hand depth image.

[0072] According to one aspect of the present invention, the device for determining the three-dimensional key points of the hand can obtain the RGB image and the depth image of the hand by using the RGB camera and the depth camera respectively, and process the RGB image of the hand by means of the trained convolutional neural network model and the multi-layer perceptron neural network model to construct a 2.5D model of the hand, so that the three-dimensional key points of the hand can be determined quickly and accurately based on the depth information in the depth image of the hand. In this way, the accuracy of the three-dimensional information of the three-dimensional key points of the hand can be improved, while the calculation speed can be improved and the hardware cost and labor cost can be reduced.

[0073] Figure 4 FIG. 1 is a block diagram of a computer device according to an embodiment of the present invention. Figure 4As shown in FIG. 4 , the computer device 400 includes a memory 410, a processor 420, and a computer program 430 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 430, the steps of the method for determining three-dimensional key points of a hand according to an embodiment of the present invention are implemented.

[0074] In addition, as described above, the present invention may also be implemented as a computer storage medium in which a program for causing a computer to execute the method for determining three-dimensional key points of a hand according to one aspect of the present invention is stored.

[0075] Here, as computer storage media, various types of computer storage media can be used, such as disks (for example, magnetic disks, optical disks, etc.), cards (for example, memory cards, optical cards, etc.), semiconductor memories (for example, ROMs, non-volatile memories, etc.), and tapes (for example, magnetic tapes, cassette tapes, etc.).

[0076] Where applicable, hardware, software or a combination of hardware and software may be used to implement the various embodiments provided by the present invention. Moreover, where applicable, without departing from the scope of the present invention, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware and / or both. Where applicable, without departing from the scope of the present invention, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware or both. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0077] Software according to the present invention (such as program code and / or data) may be stored on one or more computer storage media. It is also contemplated that the software identified herein may be implemented using one or more general or special purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein may be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0078] The embodiments and examples set forth herein are provided to best illustrate embodiments according to the present invention and its specific applications, and thereby enable those skilled in the art to make and use the present invention. However, those skilled in the art will appreciate that the above description and examples are provided for ease of illustration and example only. The description set forth is not intended to cover all aspects of the present invention or to limit the present invention to the precise form disclosed.

Claims

1. A method for determining three-dimensional key points of a hand, characterized in that: The method comprises the following steps: Acquire an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand; Determine data of two-dimensional key points of the hand according to the acquired RGB image of the hand; Constructing a 2.5D hand model based on at least the data of the determined two-dimensional key points of the hand; and Determine three-dimensional information of three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand, The 2.5D model indicates the (x, y, z) coordinates of the two-dimensional key points of the hand, and z is the difference between the fixed position of the two-dimensional key points of the hand relative to the wrist.

2. The method according to claim 1, wherein acquiring an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand comprises: Use an RGB camera to obtain an RGB image of the hand; Using a depth camera to obtain a depth image of the hand; as well as The RGB image of the hand is aligned with the depth image of the hand according to the intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera.

3. The method according to claim 1, wherein determining the data of the two-dimensional key points of the hand according to the acquired RGB image of the hand comprises: The first convolutional neural network model is trained based on the pre-labeled 2D key points of the hand; Processing the acquired RGB image of the hand using the trained first convolutional neural network model; and Based on the processing results of the trained first convolutional neural network model, the data of the two-dimensional key points of the hand are determined, wherein the data of the two-dimensional key points of the hand include one or more of the two-dimensional coordinate information of the two-dimensional key points of the hand, the occlusion information of the two-dimensional key points of the hand, and the heat map information of the two-dimensional key points of the hand.

4. The method according to claim 1, wherein constructing a 2.5D hand model based at least on the determined data of the two-dimensional key points of the hand comprises: Determine the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand; as well as A 2.5D hand model is constructed using the shape parameters and gesture parameters of the hand.

5. The method according to claim 4, wherein determining the shape parameters and gesture parameters of the hand based on the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand comprises: Inputting the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand into a multi-layer perception neural network model; as well as The shape parameters and gesture parameters of the hand are determined based on the output results of the multi-layer perceptron neural network model.

6. The method according to claim 4, wherein constructing a 2.5D hand model using shape parameters and gesture parameters of the hand comprises: Train the second convolutional neural network model based on the pre-labeled 2.5D key point data of the hand; Processing the shape parameters and gesture parameters of the hand using the trained second convolutional neural network model; and A 2.5D hand model corresponding to the two-dimensional key points of the hand is constructed based on the processing results of the trained second convolutional neural network model.

7. The method according to claim 1, wherein determining the three-dimensional information of the three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D model of the hand comprises: Determine the visible key points in the two-dimensional key points of the hand based on the data of the two-dimensional key points of the hand; Determine depth information of corresponding key points of visible key points among the two-dimensional key points of the hand in the depth image of the hand; as well as The depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model is updated based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

8. The method according to claim 7, wherein the method further comprises: Discarding outliers among the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; Determine the mean of the depth information of the remaining corresponding key points; as well as The depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model is updated to the sum of the mean values ​​of the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model and the depth information of the remaining corresponding key points.

9. The method according to claim 8, wherein the outliers of the visible key points in the two-dimensional key points of the hand in the corresponding key points in the depth image of the hand are determined by: Determine the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; Determine the deviation value of each corresponding key point from the mean value of the corresponding key point; as well as In response to determining that a deviation value between a corresponding key point and a mean value of the corresponding key point is greater than a preset value, the corresponding key point is determined to be an abnormal point.

10. A device for determining three-dimensional key points of a hand, characterized in that: The device comprises: an acquisition unit configured to acquire an RGB image of a hand and a depth image of the hand aligned with the RGB image of the hand; A determination unit configured to determine data of two-dimensional key points of the hand according to the acquired RGB image of the hand; A construction unit configured to construct a 2.5D hand model based at least on the determined data of the two-dimensional key points of the hand; and a processing unit configured to determine three-dimensional information of three-dimensional key points of the hand based at least on the depth image of the hand and the constructed 2.5D hand model, The 2.5D model indicates the (x, y, z) coordinates of the two-dimensional key points of the hand, and z is the difference between the fixed position of the two-dimensional key points of the hand relative to the wrist.

11. The apparatus according to claim 10, wherein the acquisition unit comprises an RGB camera and a depth camera, and the acquisition unit is further configured to: Use an RGB camera to obtain an RGB image of the hand; Acquire a depth image of the hand using a depth camera; and The RGB image of the hand is aligned with the depth image of the hand according to the intrinsic parameters and extrinsic parameters of the RGB camera and the depth camera.

12. The apparatus according to claim 10, wherein the determining unit is further configured to: The first convolutional neural network model is trained based on the pre-labeled 2D key points of the hand; Processing the acquired RGB image of the hand using the trained first convolutional neural network model; and Based on the processing results of the trained first convolutional neural network model, the data of the two-dimensional key points of the hand are determined, wherein the data of the two-dimensional key points of the hand include one or more of the two-dimensional coordinate information of the two-dimensional key points of the hand, the occlusion information of the two-dimensional key points of the hand, and the heat map information of the two-dimensional key points of the hand.

13. The device according to claim 10, wherein the building unit is further configured to: Determining shape parameters and gesture parameters of the hand based on heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand; and A 2.5D hand model is constructed using the shape parameters and gesture parameters of the hand.

14. The device according to claim 13, wherein the building unit is further configured to: Inputting the heat map information of the two-dimensional key points of the hand in the data of the two-dimensional key points of the hand and the RGB image of the hand into a multi-layer perception neural network model; and The shape parameters and gesture parameters of the hand are determined based on the output results of the multi-layer perceptron neural network model.

15. The device according to claim 13, wherein the building unit is further configured to: Train the second convolutional neural network model based on the pre-labeled 2.5D key point data of the hand; Processing the shape parameters and gesture parameters of the hand using the trained second convolutional neural network model; and A 2.5D hand model corresponding to the two-dimensional key points of the hand is constructed based on the processing results of the trained second convolutional neural network model.

16. The apparatus according to claim 10, wherein the processing unit is further configured to: Determine the visible key points in the two-dimensional key points of the hand based on the data of the two-dimensional key points of the hand; Determine depth information of corresponding key points of visible key points among the two-dimensional key points of the hand in the depth image of the hand; as well as The depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model is updated based on the depth information of the corresponding key points to determine the three-dimensional information of the three-dimensional key points of the hand.

17. The apparatus according to claim 16, wherein the processing unit is further configured to: Discarding outliers among the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; Determine the mean of the depth information of the remaining corresponding key points; and The depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model is updated to the sum of the mean values ​​of the depth information of the two-dimensional key points of the hand in the constructed 2.5D hand model and the depth information of the remaining corresponding key points.

18. The apparatus according to claim 17, wherein the processing unit is configured to determine outliers of visible key points in the two-dimensional key points of the hand in corresponding key points in the depth image of the hand by: Determine the mean of the corresponding key points of the visible key points in the two-dimensional key points of the hand in the depth image of the hand; Determine the deviation value of each corresponding key point from the mean value of the corresponding key point; as well as In response to determining that a deviation value between a corresponding key point and a mean value of the corresponding key point is greater than a preset value, the corresponding key point is determined to be an abnormal point.

19. A computer storage medium, characterized in that: The computer storage medium comprises instructions which, when executed, perform the method according to any one of claims 1 to 9.

20. A computer device, characterized in that: The computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the program.

Images