Personalized human hand geometric reconstruction method based on single image
Through neural network prediction of camera parameters and three-dimensional manual model, combined with MANO model and spiral convolution operator, the structure of the manual model is optimized, and the problem of poor shape details and artifacts in manual reconstruction of single images is solved, achieving high reduction and Reconstruction of authenticity of human-hand model.
Patent Information
- Application Number
- CN202411017979.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
AI Technical Summary
The existing human hand reconstruction method based on single images is difficult to accurately present the human hand shape details, resulting in low model restoration and grid fitting without model methods is prone to artifacts.
A neural network is used to predict camera parameters and a three-dimensional manual model, combined with the MANO model and spiral convolution operator, the model structure is optimized and the contour point matching is used to optimize the model structure and realize the precise reconstruction of the manual model.
The generated manual model is more realistic and reasonable, with the topological structure optimized, eliminating the contour deviation between the three-dimensional model and the two-dimensional image, and achieving the precise fit between the contour edge of the manual model and the contour of the input image.
Smart Images

Figure CN118918261A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a personalized human hand geometry reconstruction method based on a single image. Background Art
[0002] Most current research focuses on the dynamic behavior of virtual hands, such as gesture prediction, but less on the detailed modeling of hand shape. Three-dimensional geometric reconstruction is an important stage in virtual hand modeling. The reconstruction of a realistic hand model is based on a highly restored hand model outline. However, in practice, there are the following problems:
[0003] (1) Vision-based 3D hand reconstruction methods are mainly divided into two categories: one is model-based methods, and the other is model-free methods. Among them, the 3D hand reconstruction method based on parameterized models can ensure that the generated hand model is reasonable, but due to the high nonlinearity of the mapping from the image to the model parameter space, it is very challenging to directly and accurately estimate the model parameters. The model-free 3D hand reconstruction method can perform direct and accurate mesh fitting. However, the reconstructed mesh is prone to artifacts, resulting in unreasonable hand shapes;
[0004] (2) Due to the complexity of the human hand structure, the hand reconstruction method based on a single image often cannot accurately present the shape details of the target hand when constructing the geometric model of the hand, resulting in a low degree of model restoration. Summary of the invention
[0005] The present invention proposes a personalized human hand geometry reconstruction method based on a single image, the purpose of which is to solve the problems (1) and (2) described in the background technology.
[0006] To achieve the above object, the technical solution of the present invention is:
[0007] A personalized human hand geometry reconstruction method based on a single image, comprising the following steps:
[0008] (1) Using neural networks to predict camera parameters and the 3D model of the target hand;
[0009] (2) Obtain binary masks of the model and hand images;
[0010] (3) Detect and match contour points of the model and image;
[0011] (4) Use the nearest point iteration algorithm to optimize the model structure.
[0012] Preferably, the step (1) comprises the following specific steps:
[0013] (11) Use a convolutional neural network to extract the features of a single hand image and embed it into a latent space, and then use a spiral convolution operator to decode it; use the MANO model to supervise the prediction results so that the predicted three-dimensional vertex order follows the MANO vertex order;
[0014] (12) A camera parameter prediction module is introduced during the network training process to couple the camera parameter prediction and 3D vertex prediction to achieve joint training. Finally, the network outputs the predicted 3D vertex coordinates and camera parameters of the hand model.
[0015] Preferably, the step (11) comprises the following specific steps:
[0016] Step a: Input a 224×224 resolution image of the back of the target hand, perform random center cropping and apply random rotation; add noise to the image to simulate real-world interference;
[0017] Step b: Input the processed data into the image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image;
[0018] Step c: Use the spiral operator decoder D spiral Recover the hand mesh coordinates from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model;
[0019] Step d: Input the decoder result into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam;
[0020] Step e: Input the true value of the human hand model based on MANO of the corresponding human hand image, which includes the two-dimensional and three-dimensional coordinates of several hand joints and the three-dimensional hand posture annotation based on MANO; calculate the three-dimensional vertex loss by comparing the vertex coordinates in the predicted human hand mesh with their corresponding true values; extract the joint points in the predicted vertices according to the order of the MANO model, and calculate the three-dimensional joint loss by comparing the joint points with the corresponding true values; project the predicted three-dimensional human hand joint coordinates onto a two-dimensional plane to obtain the coordinates of the corresponding two-dimensional joints, and then calculate the two-dimensional joint loss by comparing them with the true values of the two-dimensional joint coordinates; calculate the model mesh edge loss by comparing the edges of each triangular facet of the predicted three-dimensional human hand model mesh with the corresponding edges in the model true value; calculate the model mesh normal loss by comparing the normal vectors of the triangular facets in the predicted model mesh with the corresponding facet normal vectors in the model true value:
[0021] Step f: Perform weighted summation of the above losses to obtain the total loss function, and perform back propagation;
[0022] The step (12) comprises the following specific steps:
[0023] Step a: Input an image of the back of the target hand with the same length and width, perform cropping processing centered on the hand, and scale the entire image so that the size of the network input image is unified to 224×224;
[0024] Step b: Input the processed data into the trained image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image;
[0025] Step c: Use the pre-trained spiral operator decoder D spiral Recover the hand mesh coordinates from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model;
[0026] Step d: The decoder result is input into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam.
[0027] Preferably, the step (2) comprises the following specific steps:
[0028] (21) Use the SAM (Segment Anything Model) algorithm to perform image segmentation and obtain the binary mask of the target hand and the hand foreground;
[0029] (22) The perspective projection transformation is used to obtain the coordinates of the two-dimensional projection points of the model. Then, the projection image of the three-dimensional human hand mesh model is drawn according to the coordinates of the two-dimensional projection points. After depth value filling and binarization, the binary mask of the projection image is obtained.
[0030] Preferably, the step (21) comprises the following specific steps:
[0031] In the step (21), the SAM (Segment Anything Model) model is used as an image segmentation model. When a binary mask of the input human hand image is obtained, all pixels in the binary mask image and the original human hand image are traversed at the same time, and the pixel points judged as white in the binary mask are filled with the pixel values at the corresponding positions in the original human hand image to obtain the human hand foreground of the input image;
[0032] The step (22) comprises the following specific steps:
[0033] Step a: Determine the coordinates of each vertex on the hand mesh model F in the world coordinate system (x w ,y w ,z w );
[0034] Step b: Calculate the vertices (x w ,y w ,z w ) is the two-dimensional coordinate in the image after perspective projection transformation;
[0035] Step c: Draw a projection image of the 3D hand mesh model according to each 2D point after projection transformation and fill the pixels with depth values;
[0036] Step d: Since the background pixel value of the projection image is zero and the pixel value of the human hand is non-zero, the mask image is binarized by setting a threshold.
[0037] Preferably, the step (3) comprises the following specific steps:
[0038] (31) Using the binary mask of the projection image of the three-dimensional human hand mesh model and the binary mask of the target human hand image, the edge points of the model contour and the contour points of the image are obtained according to the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour points;
[0039] (32) Based on the nearest point iteration algorithm, vector constraints are introduced to achieve the pairing of the edge points of the model contour and the image contour points.
[0040] Preferably, the step (31) comprises the following specific steps:
[0041] Step a: Set a 3×3 sliding window to traverse the points with pixel value 1 in the binary mask of the hand image;
[0042] Step b: Using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour points and obtain the contour point set of the hand image;
[0043] The step b comprises the following specific steps:
[0044] Step a1: obtaining the projection image of the target three-dimensional human hand model predicted in step (1) and the two-dimensional pixel coordinates of the model vertices, and binarizing the projection image to obtain a binary mask of the projection image;
[0045] Step b1: On the binary mask of the model projection image, set a 3×3 sliding window to traverse the two-dimensional projection points of the model vertices;
[0046] Step c1: using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour point;
[0047] Step d1: Synchronize the detection results of the model projection points to their corresponding 3D model vertices to obtain the edge points of the contour of the human hand model;
[0048] The step (32) comprises the following specific steps:
[0049] Step a2: Convolve the Sobel operator template with each pixel in the binary mask of the hand image: for each pixel, multiply the 8 pixels around it with the corresponding position of the template, and finally add the products to get the result;
[0050] Step b2: For the Sobel operator in the horizontal direction, compare the pixel with the template G x The multiplication operation is performed and then added, and the result represents the gradient dx of the image in the horizontal direction; for the Sobel operator in the vertical direction, the pixel is added to the template G y After multiplication, the result is added, and the result represents the gradient dy of the image in the vertical direction;
[0051] Step c2: After the gradient calculations in the horizontal and vertical directions are completed, the normalized gradient direction of each pixel in the image is calculated:
[0052] Step d2: Traverse each triangle in the 3D model of the target hand predicted in step (1) (the file format is OBJ format, i.e. OBJ model), assuming that the vertices of the triangle are v0(x0, y0, z0), v1(x1, y1, z1), v2(x2, y2, z2), and the normal vector is Obtain the face normal vector from the vertex coordinates of the triangle surface according to the cross multiplication principle;
[0053] Step e2: traverse the vertex index of each facet in the three-dimensional model of the target hand predicted in step (1) to save the triangular facets constituted by each vertex;
[0054] Step f2: using the edge points of the acquired hand model contour to collect the triangular facets formed by each point, summing and normalizing the normal vectors of these facets to obtain the normal vectors of the corresponding vertices;
[0055] Step g2: The edge point set V of the hand model contour and its corresponding model vertex normal vector Transform the perspective projection into the two-dimensional image to obtain the edge projection point set of the human hand model contour and its corresponding 2D normal vector
[0056] Step h2: Points in Traverse each point p in the hand image contour point set P, and use the product of the two normal vectors as the criterion for further distance calculation; when When Is the distance to p within the threshold range? If point p j and The distance is within the threshold range and relative to the distance of other points within the threshold range Recently, click p j yes otherwise The pairing points of are empty:
[0057] Step i2: Get the one-dimensional matrix Ω 3D→2D , each element of the matrix is used to store a key-value pair, the key is the index of the vertex in V, and the value is the index of the image edge point that is successfully paired with the model vertex of the corresponding index.
[0058] Preferably, the step (4) comprises the following specific steps:
[0059] (41) constructing a least squares problem based on the point pair set to calculate the transformation matrix of the three-dimensional hand model at each iteration;
[0060] (42) The loss function and the number of iterations are used as metrics at the same time, and the three-dimensional model is iteratively updated until the convergence condition is reached.
[0061] Preferably, the step (41) comprises the following specific steps:
[0062] Step a3: Project the edge points of the 3D hand model contour Pair a point p with its image contour j Calculate the edge projection point loss of the hand model contour to constrain the optimization of the edge vertex coordinates of the 3D hand model contour:
[0063] Project the edge points of the 3D hand model contour The target contour point p that is successfully matched with the image j The difference vector and the normal vector corresponding to the paired point of the image contour Perform dot multiplication to calculate the projection point optimization direction loss, so that the edge points of the hand model contour are close to their corresponding image contour points in a reasonable optimization direction:
[0064] The edge of each triangular facet of the optimized three-dimensional hand model mesh is compared with the corresponding edge in the model before optimization to calculate the model mesh edge loss;
[0065] Step b3: Perform weighted summation of the above losses to obtain the total loss function and perform back propagation.
[0066] The beneficial effects of the personalized human hand geometry reconstruction method based on a single image of the present invention are:
[0067] The present invention realizes the joint optimization of the hand model and camera parameters by adding a camera parameter prediction module on the basis of the model-free three-dimensional hand reconstruction method. The existing parameterized model is used as supervision to ensure the regression accuracy of the model-free method while generating a more realistic and reasonable hand model. Then, on the basis of deep learning to reconstruct the hand model, the contour optimization operation is further performed on the hand model to optimize the topological structure of the entire model, eliminate the contour deviation between the three-dimensional model and the two-dimensional image, and more accurately achieve the fitting of the contour edge of the hand model with the contour of the input image.
[0068] Instruction Manual
[0069] Figure 1 A flowchart of a method for personalized hand geometry reconstruction based on a single image provided by the present invention.
[0070] Figure 2 The present invention provides a flowchart of a method for personalized human hand geometry reconstruction based on a single image.
[0071] Figure 3 An overview of the single-image-based three-dimensional human hand reconstruction method provided by the present invention.
[0072] Figure 4 The decoder network structure provided by the present invention.
[0073] Figure 5 This is a perspective projection model in the camera pose estimation module provided by the present invention.
[0074] Figure 6 The qualitative evaluation results of the existing mainstream human hand geometry reconstruction method and the method in this paper on the Freihand dataset are provided in the present invention.
[0075] Figure 7 This is a comparison chart of the two-dimensional joint prediction results of the existing mainstream human hand geometry reconstruction method provided by the present invention and the method in this paper on a self-collected data set.
[0076] Figure 8 This is a comparison chart of the model projection results of the existing mainstream human hand geometry reconstruction method provided by the present invention and the method in this paper on the self-collected data set.
[0077] Fig. 9 This is a comparison chart of the model postures of the existing mainstream human hand geometry reconstruction method provided by the present invention and the method in this paper on the self-collected data set.
[0078] Fig.10 This is a comparison chart before and after model optimization of the human hand model contour optimization method provided by the present invention. DETAILED DESCRIPTION
[0079] The following description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
[0080] The following embodiments may be understood as individually expressing a part of a local structure or method of the present invention, or may be understood as a combination of the embodiments to explain the connotation of a larger structure or method of the present invention.
[0081] Example 1
[0082] The present invention provides a personalized human hand geometry reconstruction method based on a single image, such as Figure 1 As shown, the following steps are included:
[0083] (1) Using neural networks to predict camera parameters and the 3D model of the target hand;
[0084] (2) Obtain binary masks of the model and hand images;
[0085] (3) Detect and match contour points of the model and image;
[0086] (4) Use the nearest point iteration algorithm to optimize the model structure.
[0087] Example 2
[0088] The step (1) comprises the following specific steps:
[0089] (11) Use a convolutional neural network to extract the features of a single hand image and embed it into a latent space, and then use a spiral convolution operator to decode it; use the MANO model to supervise the prediction results so that the predicted three-dimensional vertex order follows the MANO vertex order;
[0090] (12) A camera parameter prediction module is introduced during the network training process to couple the camera parameter prediction and 3D vertex prediction to achieve joint training. Finally, the network outputs the predicted 3D vertex coordinates and camera parameters of the hand model.
[0091] Example 3
[0092] like Figure 2 As shown, the step (11) includes the following specific steps:
[0093] Step a: Input a 224×224 resolution image of the back of the target hand, perform random center cropping and apply random rotation. The cropping size based on the center point of the image and the rotation angle of the cropped area are randomly generated; then add noise to the image to simulate real-world interference, so that the model can better adapt to real scenes, thereby improving the robustness and generalization of the model;
[0094] Step b: Input the processed data into the image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image;
[0095] Step c: Use the spiral operator decoder D spiral The hand mesh coordinates are recovered from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model; the MANO model is a relatively mature parameterized model, which contains 1538 faces and 778 vertices, including 21 joint points with fixed numbers.
[0096] The overall structure of the decoder is as follows Figure 4 As shown in the figure, the input of the decoder is the image features extracted by the encoder, which are first aggregated through a fully connected layer to change the dimension of the expression features; then they are input into five spiral convolution blocks connected in series, each of which consists of an upsampling layer and a spiral convolution layer; the upsampling layer updates the number of vertices of the model by expanding the vertices of the previous layer. The spiral convolution layer performs a spiral convolution of a specified length L on each vertex of the current layer input, and reduces the channel dimension while aggregating features in a clockwise spiral order; the entire spiral convolution process is achieved by specifying the update target in each spiral convolution block; finally, the convolution result is input into the ReLU activation function for nonlinear activation;
[0097] Step d: The decoder result is input into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam.
[0098] The intrinsic parameter matrix of the perspective camera is shown in formula (1):
[0099]
[0100] Among them, f x , f y is the focal length of the camera on the x-axis and y-axis, [o x ,o y ] is the center position of the image, sk is a deviation coefficient; usually, the focal length of the camera is approximately f x =f y =f,[o x ,o y ]=[W / 2,H / 2], W and H are the width and height of the image respectively, s k =0; then the camera's intrinsic parameter matrix is expressed as:
[0101]
[0102] The camera pose estimation module in the reconstruction network defines that there is only a translation transformation between the world coordinate system and the camera coordinate system, that is, the x, y, z The directions of the axes are the same, and the translation vector is expressed as Among them, t x and t y Directly obtained from network prediction, t z Calculated indirectly by network prediction;
[0103] Assume that the origin of the world coordinate system is located at [-1,1] 3 The center of the cube and the three-dimensional joint points are all located in the cube; in order to facilitate the calculation of t z , let P c The coordinates are (0,1,t z ), its coordinates in the world coordinate system are P w (-t x ,1-t y ,0), such as Figure 5 As shown:
[0104] According to the triangle similarity principle, t is calculated. z The formula is as follows:
[0105]
[0106] Among them, s is obtained from the camera parameter cam regressed by the reconstruction network, which represents the proportion of the human hand in the image;
[0107] Since the camera parameters cam are estimated relative to the center of the cropped image, they cannot accurately represent the camera translation parameters of the real camera center. Therefore, the focal length is approximated using known conditions to more accurately estimate the camera parameters corresponding to the cropped image. In general, the focal length can be calculated from the camera's field of view (FOV), and the calculation formula is as follows:
[0108]
[0109] Among them, α is the diagonal field of view of the camera; the field of view of a general mobile phone camera is about 60 degrees, and the simple approximate focal length is:
[0110]
[0111] In summary, the projection process is expressed as the following two steps:
[0112] 1) For point P in the world coordinate system w (x w ,y w ,z w ) According to T = [tx ,t y ,t z ] is translated to obtain its representation P in the camera coordinate system c (x c ,y c ,z c ):
[0113]
[0114] 2) Construct the camera intrinsic parameter matrix and transform the joint points to obtain pixel coordinates:
[0115]
[0116] Where W is the size of the input image after cropping, set H = W = 224;
[0117] Step e: Input the true value of the human hand model based on MANO of the corresponding human hand image, which includes the 2D / 3D coordinates of 21 hand joints and 3D hand posture annotation based on MANO;
[0118] Calculate the 3D vertex loss by comparing the predicted vertex coordinates in the hand mesh with their corresponding true values;
[0119]
[0120] Where L is the number of vertices of the model, is the i-th vertex in the reconstructed mesh, v i is the i-th vertex in the true value of the mesh model;
[0121] Extract the joint points from the predicted vertices according to the order of the MANO model, and calculate the three-dimensional joint point loss by comparing the joint points with the corresponding true values;
[0122]
[0123] Where l is the number of joints of the human hand, is the i-th joint point in the reconstructed mesh, is the i-th joint point of the real annotation;
[0124] The predicted 3D hand joint point coordinates are projected onto the 2D plane according to formula (7) to obtain the corresponding 2D joint point coordinates, and then the 2D joint point loss is calculated by comparing the 2D joint point coordinates with the true values of the 2D joint point coordinates.
[0125]
[0126] where Π K is the projection matrix, i.e., formula (7), which realizes the projection transformation of the three-dimensional joint point from the three-dimensional space to the two-dimensional plane. is the two-dimensional coordinate of the i-th joint point that is actually annotated;
[0127] Calculate the model mesh edge loss by comparing the edge of each triangular facet of the predicted 3D hand model mesh with the corresponding edge in the model truth value;
[0128]
[0129] Where F is all the triangles in the model mesh, f is the index of the triangle in the model mesh, (i, j) is the vertex index of an edge in f, is the length of the edge of each triangle in the prediction model, ||v i -v j || is the edge length of each triangle in the true value of the model;
[0130] The normal vector of the triangular patch in the predicted model mesh is compared with the normal vector of the corresponding patch in the model truth to calculate the model mesh normal loss:
[0131]
[0132] where n f is the true value of the normal vector of the triangle patch f;
[0133] Step f: Perform weighted summation of the above losses to obtain the total loss function and perform back propagation.
[0134] L=λ Vert L Vert +λ 3DJ L 3DJ +λ 2DJ L 2DJ +λ Edge L Edge +λ Normal L Normal . (13)
[0135] where λ Vert , 3DJ , 2DJ , Edge and λ Normal It is the weight corresponding to each loss term, which is used to adjust the proportion of each error in the objective function;
[0136] The step (12) comprises the following specific steps:
[0137] Step a: Input an image of the back of the target hand with the same length and width, perform cropping processing centered on the hand, and scale the entire image so that the size of the network input image is unified to 224×224;
[0138] Step b: Input the processed data into the trained image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image;
[0139] Step c: Use the pre-trained spiral operator decoder D spiral The hand mesh coordinates are recovered from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model;
[0140] Step d: The decoder result is input into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam.
[0141] Example 4
[0142] The step (2) comprises the following specific steps:
[0143] (21) Use the SAM (Segment Anything Model) algorithm to perform image segmentation and obtain the binary mask of the target hand and the hand foreground;
[0144] (22) The perspective projection transformation is used to obtain the coordinates of the two-dimensional projection points of the model. Then, the projection image of the three-dimensional human hand mesh model is drawn according to the coordinates of the two-dimensional projection points. After depth value filling and binarization, the binary mask of the projection image is obtained.
[0145] Example 5
[0146] The step (21) comprises the following specific steps:
[0147] Step a: Use the SAM (Segment Anything Model) model as the image segmentation model to obtain the binary mask of the input hand image;
[0148] Step b: traverse all the pixels in the binary mask image and the original hand image at the same time, fill the pixel points judged as white in the binary mask with the pixel values at the corresponding positions in the original hand image, and obtain the hand foreground part of the input image;
[0149] The step (22) comprises the following specific steps:
[0150] Step a: Determine the coordinates of each vertex on the hand mesh model F in the world coordinate system (x w ,y w ,z w );
[0151] Step b: Use formula (7) to calculate the vertices (x w ,y w ,zw ) is the two-dimensional coordinate (u, v) in the image after perspective projection transformation, where:
[0152]
[0153] Step c: Draw a projection image of the 3D hand mesh model according to each 2D point after projection transformation and fill the pixels with depth values;
[0154] Step d: Since the background pixel value of the projection image is zero and the pixel value of the hand part is non-zero, the mask image can be binarized by setting a threshold. The most common binarization algorithm is as follows:
[0155]
[0156] Wherein, Pix represents pixel value, gray represents grayscale value, and T represents threshold value; when the grayscale value is lower than the set threshold value T, its pixel value is set to 0 and displayed as black; when the grayscale value is not lower than the threshold value T, its pixel value is set to 255 and displayed as white; setting T = 0.01 realizes the conversion of the projected image to the binary mask.
[0157] Example 6
[0158] The step (3) comprises the following specific steps:
[0159] (31) Using the binary mask of the projection image of the three-dimensional human hand mesh model and the binary mask of the target human hand image, the edge points of the model contour and the contour points of the image are obtained according to the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour points;
[0160] (32) Based on the nearest point iteration algorithm, vector constraints are introduced to achieve the pairing of the edge points of the model contour and the image contour points.
[0161] Example 7
[0162] The step (31) comprises the following specific steps:
[0163] Step a: Set a 3×3 sliding window to traverse the points with pixel value 1 in the binary mask of the hand image;
[0164] Step b: Using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour points and obtain the contour point set of the hand image.
[0165] The step b comprises the following specific steps:
[0166] Step a1: Use perspective projection formula (7) to obtain the projection image of the three-dimensional human hand model predicted in step (1) and the two-dimensional pixel coordinates of the model vertices, and use formula (16) to binarize the projection image to obtain a binary mask of the projection image;
[0167]
[0168] Step b1: On the binary mask of the model projection image, set a 3×3 sliding window to traverse the two-dimensional projection points of the model vertices;
[0169] Step c1: using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour point;
[0170] Step d1: Synchronize the detection results of the model projection points to their corresponding three-dimensional model vertices to obtain the edge points of the human hand model contour.
[0171] (32) Calculate the normal vector of the image contour point:
[0172] Step a2: Convolve the Sobel operator template with each pixel in the binary mask of the hand image: for each pixel, multiply the 8 pixels around it with the corresponding position of the template, and finally add the products to get the result;
[0173] Step b2: For the Sobel operator in the horizontal direction, compare the pixel with the template G x The multiplication operation is performed and then added, and the result represents the gradient dx of the image in the horizontal direction; for the Sobel operator in the vertical direction, the pixel is added to the template G y After multiplication, the result is added, and the result represents the gradient dy of the image in the vertical direction;
[0174] The Sobel operator consists of two 3 × 3 matrices, as shown in formula (17), which are used to calculate the gradient of the image in the horizontal and vertical directions respectively;
[0175]
[0176] Step c2: After the gradient calculations in the horizontal and vertical directions are completed, the normalized gradient direction of each pixel in the image is calculated using the following formula:
[0177]
[0178] in It is the gradient amplitude. A lower gradient amplitude is used as the judgment condition to filter out noise points. After experimental analysis, setting the threshold to 0.1 can eliminate most interference points.
[0179] Step d2: Traverse each triangular face in the 3D hand model predicted in step (1). Assume that the vertices of the triangular face are v0(x0, y0, z0), v1(x1, y1, z1), v2(x2, y2, z2), and the normal vector is According to the cross multiplication principle, the face normal vector is obtained from the vertex coordinates of the triangle face. The formula is:
[0180]
[0181] Step e2: traverse the vertex index of each face in the OBJ model to save the triangular face that each vertex participates in forming;
[0182] Step f2: using the edge points of the acquired hand model contour to collect the triangular facets formed by each point, summing and normalizing the normal vectors of these facets to obtain the normal vectors of the corresponding vertices;
[0183] Step g2: The edge point set V of the hand model contour and its corresponding model vertex normal vector Using formula (7), the perspective projection is transformed into a two-dimensional image to obtain the edge projection point set of the human hand model contour: and its corresponding 2D normal vector in,
[0184] Step h2: Points in Traverse each point p in the hand image contour point set P, and use the product of the two normal vectors as the criterion for further distance calculation; when When Is the distance to p within the threshold range? If point p j and The distance is within the threshold range and relative to the distance of other points within the threshold range Recently, click p j yes otherwise The pairing points of are empty:
[0185]
[0186] in, The angle difference between the normal vectors of the two points is about 30 degrees, and dist is the distance threshold;
[0187] Step i2: Get the one-dimensional matrix Ω 3D→2D , each element of the matrix is used to store a key-value pair, the key is the index of the vertex in V, and the value is the index of the image edge point that is successfully paired with the model vertex of the corresponding index.
[0188] Example 8
[0189] The step (4) comprises the following specific steps:
[0190] (41) constructing a least squares problem based on the point pair set to calculate the transformation matrix of the three-dimensional hand model at each iteration;
[0191] (42) The loss function and the number of iterations are used as metrics at the same time, and the three-dimensional model is iteratively updated until the convergence condition is reached.
[0192] Example 9
[0193] The step (41) comprises the following specific steps:
[0194] Step a3: Construct a least squares problem based on the point pair set to calculate the transformation matrix of the 3D hand model at each iteration. The least squares problem consists of three parts: project the edge points of the model contour Pair a point p with its image contour j Calculate the edge projection point loss of the hand model contour to constrain the edge vertex coordinates of the optimized model contour:
[0195]
[0196] Project the edge points of the model contour The target contour point p that is successfully matched with the image j The difference vector and the normal vector corresponding to the paired point of the image contour Perform dot multiplication to calculate the projection point optimization direction loss, so that the edge points of the hand model contour are close to their corresponding image contour points in a reasonable optimization direction:
[0197]
[0198] The edge of each triangular facet of the optimized three-dimensional hand model mesh is compared with the corresponding edge in the model before optimization to calculate the model mesh edge loss;
[0199]
[0200] Among them, N l is the lth vertex of the model The set of adjacent points, Vert is the set of vertices in the model, is the vector of each edge of the optimized model, is the vector of each edge of the model before optimization;
[0201] Step b3: Perform weighted summation of the above losses to obtain the total loss function and perform back propagation:
[0202] L=λprojvert L projvert +λ movedir L movedir +λ edge L edge . (twenty four)
[0203] Among them, λ projvert , movedir and λ edge It is the weight corresponding to each loss term, which is used to adjust the proportion of each error in the objective function;
[0204] Finally, step (42) is implemented, the loss function and the number of iterations are used as metrics at the same time, and the three-dimensional model is iteratively updated until the convergence condition is reached.
[0205] To highlight the advantages of the present invention, the following is described in conjunction with the effect diagram:
[0206] Table 1 lists the hand reconstruction evaluation indicators of the existing mainstream hand geometry reconstruction algorithms and the algorithm of the present invention on the Freihand dataset.
[0207] Table 1 Quantitative evaluation of hand geometry reconstruction on the Freihand dataset
[0208]
[0209] As can be seen from the table, the method of the present invention achieves the best results in all four evaluation indicators. It is worth mentioning that the CMR-SG and CMR-PG methods reconstruct the hand from a single image through root relative recovery and root recovery tasks, but both use traditional methods to predict camera poses and do not introduce them into the training process for joint training of hand reconstruction and pose prediction. In contrast, the method of the present invention introduces camera poses for joint training, which is 0.79% higher in F-5 and 0.23% higher in F-15 than the second-ranked CMR-PG method, and has better reconstruction effect.
[0210] The SMHR and S2HAND methods are hand reconstruction methods based on the MANO parameterized model, and the HiFiHR method is based on the NIMBLE parameterized model. They directly decode the parameters of the parameterized model from the latent space to restore the geometric shape of the hand, and both can restore the illumination and texture color. The SMHR method focuses on two-dimensional information supervision, HiFiHR additionally introduces texture consistency supervision, and S2HAND additionally introduces camera extrinsic supervision. The method of the present invention is a model-free method that uses a spiral convolution operator to decode the three-dimensional vertices and camera parameters of the model and perform joint optimization. Compared with SMHR, the AUC of the method of the present invention is j Improved by about 8%, AUC vThe AUC of the proposed method is improved by about 8%, F-5 by about 21%, and F-15 by about 4%. j Improved by about 10%, AUC v The AUC of the proposed method is improved by about 10%, F-5 by about 23%, and F-15 by about 6%. j Improved by about 17%, AUC v The F-5 increased by about 34%, and the F-15 increased by about 13%. The overall indicators have been significantly improved, which also strongly proves the effectiveness of the method of the present invention.
[0211] Table 2 shows the D before and after optimization using the hand model contour optimization method of the present invention. L2 Indicators (defined as the edge projection points of the model contour Pair a point p with its image contour j Quantitative evaluation results on the average L2 distance between , in pixels).
[0212] Table 2 Quantitative evaluation of hand model contour optimization
[0213]
[0214] It can be seen from Table 2 that D before optimization L2 The average value of the indicator on 6 samples is 10.053 pixels. L2 The average value of the index on the six samples is 0.087 pixels, which is 9.966 pixels lower than before optimization. L2 The indicators are relatively close, indicating that the method of the present invention can stably optimize input samples of different difficulty. The contour optimization method of the present invention can significantly reduce the distance between the edge points of the model contour and the matching image contour points on the two-dimensional plane, and can obtain a more accurate model contour.
[0215] Figure 6 The qualitative evaluation results of the existing mainstream human hand geometry reconstruction methods and the proposed method on the Freihand dataset are shown. As can be seen from the figure, the proposed method has achieved relatively excellent results in finger shape and overall hand posture reconstruction, and can accurately and realistically reconstruct different hand postures.
[0216] Figure 7 The comparison chart of the two-dimensional joint prediction results of the existing mainstream human hand geometry reconstruction method and the method in this paper on the self-collected data set is shown. It can be seen from the figure that compared with other methods, the two-dimensional joint points predicted by the method of the present invention are more accurate.
[0217] Figure 8The comparison chart of the model projection results of the existing mainstream human hand geometry reconstruction method and the method of the present invention on the self-collected data set is shown. It can be seen from the figure that compared with other methods, the model projection results of the method of the present invention are more consistent with the true value and have higher accuracy.
[0218] Fig. 9 The figure shows the comparison of the model postures of the existing mainstream human hand geometry reconstruction methods and the method of the present invention on the self-collected data set. It can be seen from the figure that compared with other methods, the model posture of the method of the present invention is more accurate and realistic.
[0219] Fig.10 The comparison diagram of the hand model contour optimization method of the present invention before and after model optimization is shown. It can be seen from the figure that the overall topological structure of the model before and after optimization by the method of this invention remains unchanged, ensuring the rationality of the hand model structure. At the same time, the contour of the hand model processed by the hand contour optimization algorithm of the present invention is significantly improved, ensuring the consistency of the hand model contour with the input image contour.
Claims
1. A personalized human hand geometry reconstruction method based on a single image, characterized by: The following steps are involved: (1) Using neural networks to predict camera parameters and the 3D model of the target hand; (2) Obtain binary masks of the model and hand images; (3) Detect and match contour points of the model and image; (4) Use the nearest point iteration algorithm to optimize the model structure.
2. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 1, characterized in that: The step (1) comprises the following specific steps: (11) Use a convolutional neural network to extract the features of a single hand image and embed it into a latent space, and then use a spiral convolution operator to decode it; use the MANO model to supervise the prediction results so that the predicted three-dimensional vertex order follows the MANO vertex order; (12) A camera parameter prediction module is introduced during the network training process to couple the camera parameter prediction and 3D vertex prediction to achieve joint training. Finally, the network outputs the predicted 3D vertex coordinates and camera parameters of the hand model.
3. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 2, characterized in that: The step (11) comprises the following specific steps: Step a: Input a 224×224 resolution image of the back of the target hand, perform random center cropping and apply random rotation; add noise to the image to simulate real-world interference; Step b: Input the processed data into the image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image; Step c: Use the spiral operator decoder D spiral Recover the hand mesh coordinates from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model; Step d: Input the decoder result into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam; Step e: Input the true value of the human hand model based on MANO of the corresponding human hand image, which includes the two-dimensional and three-dimensional coordinates of several hand joints and the three-dimensional hand posture annotation based on MANO; calculate the three-dimensional vertex loss by comparing the vertex coordinates in the predicted human hand mesh with their corresponding true values; extract the joint points in the predicted vertices according to the order of the MANO model, and calculate the three-dimensional joint loss by comparing the joint points with the corresponding true values; project the predicted three-dimensional human hand joint coordinates onto a two-dimensional plane to obtain the coordinates of the corresponding two-dimensional joints, and then calculate the two-dimensional joint loss by comparing them with the true values of the two-dimensional joint coordinates; calculate the model mesh edge loss by comparing the edges of each triangular facet of the predicted three-dimensional human hand model mesh with the corresponding edges in the model true value; calculate the model mesh normal loss by comparing the normal vectors of the triangular facets in the predicted model mesh with the corresponding facet normal vectors in the model true value: Step f: Perform weighted summation of the above losses to obtain the total loss function, and perform back propagation; The step (12) comprises the following specific steps: Step a: Input an image of the back of the target hand with the same length and width, perform cropping processing centered on the hand, and scale the entire image so that the size of the network input image is unified to 224×224; Step b: Input the processed data into the trained image encoder E image In the example, the encoder structure is a ResNet50 residual network, which regresses to obtain the feature vector Z of the hand image; Step c: Use the pre-trained spiral operator decoder D spiral Recover the hand mesh coordinates from the encoded features, and the predicted hand mesh conforms to the vertex order and topological structure of the MANO model; Step d: The decoder result is input into the camera prediction module of the fully connected layer to further regress the perspective camera parameters cam.
4. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 3, characterized in that: The step (2) comprises the following specific steps: (21) Use the SAM (Segment Anything Model) algorithm to perform image segmentation and obtain the binary mask of the target hand and the hand foreground; (22) The perspective projection transformation is used to obtain the coordinates of the two-dimensional projection points of the model. Then, the projection image of the three-dimensional human hand mesh model is drawn according to the coordinates of the two-dimensional projection points. After depth value filling and binarization, the binary mask of the projection image is obtained.
5. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 4, characterized in that: The step (21) comprises the following specific steps: In the step (21), when the SAM model is used as the image segmentation model and the binary mask of the input human hand image is obtained, all the pixels in the binary mask image and the original human hand image are traversed at the same time, and the pixel points judged as white in the binary mask are filled with the pixel values at the corresponding positions in the original human hand image to obtain the human hand foreground of the input image; The step (22) comprises the following specific steps: Step a: Determine the coordinates of each vertex on the hand mesh model F in the world coordinate system (x w ,y w ,z w ); Step b: Calculate the vertices (x w ,y w ,z w ) is the two-dimensional coordinate in the image after perspective projection transformation; Step c: Draw a projection image of the 3D hand mesh model according to each 2D point after projection transformation and fill the pixels with depth values; Step d: Since the background pixel value of the projection image is zero and the pixel value of the human hand is non-zero, the mask image is binarized by setting a threshold.
6. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 5, characterized in that: The step (3) comprises the following specific steps: (31) Using the binary mask of the projection image of the three-dimensional human hand mesh model and the binary mask of the target human hand image, the edge points of the model contour and the contour points of the image are obtained according to the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour points; (32) Based on the nearest point iteration algorithm, vector constraints are introduced to achieve the pairing of the edge points of the model contour and the image contour points.
7. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 6, characterized in that: The step (31) comprises the following specific steps: Step a: Set a 3×3 sliding window to traverse the points with pixel value 1 in the binary mask of the hand image; Step b: Using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour points and obtain the contour point set of the hand image; The step b comprises the following specific steps: Step a1: obtaining the projection image of the target three-dimensional human hand model predicted in step (1) and the two-dimensional pixel coordinates of the model vertices, and binarizing the projection image to obtain a binary mask of the projection image; Step b1: On the binary mask of the model projection image, set a 3×3 sliding window to traverse the two-dimensional projection points of the model vertices; Step c1: using the characteristic that there must be zero-value pixels in the first-order neighborhood of the contour point in the binary mask, it is determined whether all the pixel values in the 3×3 sliding window are 1 to detect the contour point; Step d1: Synchronize the detection results of the model projection points to their corresponding 3D model vertices to obtain the edge points of the contour of the human hand model; The step (32) comprises the following specific steps: Step a2: Convolve the Sobel operator template with each pixel in the binary mask of the hand image: for each pixel, multiply the 8 pixels around it with the corresponding position of the template, and finally add the products to get the result; Step b2: For the Sobel operator in the horizontal direction, compare the pixel with the template G x The multiplication operation is performed and then added, and the result represents the gradient dx of the image in the horizontal direction; for the Sobel operator in the vertical direction, the pixel is added to the template G y After multiplication, the result is added, and the result represents the gradient dy of the image in the vertical direction; Step c2: After the gradient calculations in the horizontal and vertical directions are completed, the normalized gradient direction of each pixel in the image is calculated: Step d2: Traverse each triangular face in the 3D model of the target hand predicted in step (1), assuming that the triangular face The vertices are v0(x0,y0,z0), v1(x1,y1,z1), v2(x2,y2,z2), and the normal vector is Obtain the face normal vector from the vertex coordinates of the triangle surface according to the cross multiplication principle; Step e2: traverse the vertex index of each facet in the three-dimensional model of the target hand predicted in step (1) to save the triangular facets constituted by each vertex; Step f2: using the edge points of the acquired hand model contour to collect the triangular facets formed by each point, summing and normalizing the normal vectors of these facets to obtain the normal vectors of the corresponding vertices; Step g2: The edge point set V of the hand model contour and its corresponding model vertex normal vector Transform the perspective projection into the two-dimensional image to obtain the edge projection point set of the human hand model contour and its corresponding 2D normal vector Step h2: Points in Traverse each point p in the hand image contour point set P, and use the product of the two normal vectors as the criterion for further distance calculation; when When Is the distance to p within the threshold range? If point p j and The distance is within the threshold range and relative to the distance of other points within the threshold range Recently, click p j yes otherwise The pairing points of are empty: Step i2: Get the one-dimensional matrix Ω 3D→2D , each element of the matrix is used to store a key-value pair, the key is the index of the vertex in V, and the value is the index of the image edge point that is successfully paired with the model vertex of the corresponding index.
8. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 7, characterized in that: The step (4) comprises the following specific steps: (41) constructing a least squares problem based on the point pair set to calculate the transformation matrix of the three-dimensional hand model at each iteration; (42) The loss function and the number of iterations are used as metrics at the same time, and the three-dimensional model is iteratively updated until the convergence condition is reached.
9. The method for personalized human hand geometry reconstruction based on a single image as claimed in claim 8, characterized in that: The step (41) comprises the following specific steps: Step a3: Project the edge points of the 3D hand model contour Pair a point p with its image contour j Calculate the edge projection point loss of the hand model contour to constrain the optimization of the edge vertex coordinates of the 3D hand model contour: Project the edge points of the 3D hand model contour The target contour point p that is successfully matched with the image j The difference vector and the normal vector n corresponding to the paired point of the image contour pj Perform dot multiplication to calculate the projection point optimization direction loss, so that the edge points of the hand model contour are close to their corresponding image contour points in a reasonable optimization direction: The edge of each triangular facet of the optimized three-dimensional hand model mesh is compared with the corresponding edge in the model before optimization to calculate the model mesh edge loss; Step b3: Perform weighted summation of the above losses to obtain the total loss function and perform back propagation.
