A three-dimensional reconstruction method under two-handed interaction
Features are extracted through a single color picture using 2.5D heat map and ResNet50 network, predict the relative position of both hands and parameterized model parameters, and iteratively optimize the cascade module, solving the accuracy problem of the three-dimensional model reconstruction of the two-hands, realizing the single color picture The three-dimensional model reconstruction of the two-hands is suitable for the close interaction and separation of hands, providing a more convenient and wide application scenario.
Patent Information
- Application Number
- CN202111144111.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-09-28
AI Technical Summary
It is difficult to accurately reconstruct a two-hand three-dimensional model, especially in the case of close interaction between the two-hands, and most methods rely on deep pictures, lacking methods to realize three-dimensional reconstruction of two-hands through a single color picture.
A single color picture was used to detect the key points of both hands through a 2.5D heat map extraction network, combined with ResNet50 shallow and deep residual modules to extract features, predict the relative position relationship of both hands and parameterized model parameters, and iterative cascade module optimizes the hand parameterized model , generate an accurate two-hand three-dimensional mesh model.
It realizes the accurate reconstruction of the three-dimensional model of the two-hands under a single color picture, which can handle the close interaction and separation of the hands. It is more convenient than the depth camera, has a wider application scenario, and ensures the accuracy and rationality of the reconstruction results through the supervision network.
Smart Images

Figure CN113888697B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of computer vision, human-computer interaction and computer graphics, and in particular, relates to a two-handed three-dimensional reconstruction technology in a two-handed interaction state. Background Art
[0002] 3D reconstruction of hands is a hot topic in the field of computer vision and human-computer interaction, and has wide applications in virtual reality, intelligent control, and terminal devices. The task of 3D reconstruction of hands is mainly to estimate the 3D shape of the hand based on the input image, and the shape is usually represented by a mesh model.
[0003] So far, many research results have been achieved in 3D reconstruction of hands. Existing technologies can be divided into three categories according to data acquisition equipment: 1) using multiple color cameras to achieve 3D reconstruction of hands; 2) using a single color camera to achieve 3D reconstruction of hands; 3) using a depth camera to capture a depth map for 3D reconstruction of hands.
[0004] Reconstruction methods can be divided into two main categories. 1) Directly estimate the coordinates of each vertex of the mesh model. This type of method uses a deep neural network to estimate the coordinates of each vertex of the mesh model through a graph convolution algorithm or a mesh convolution algorithm. The algorithm usually extracts fewer vertex features first, and designs an upsampling algorithm to gradually increase the number of vertices, and finally obtains a high-precision mesh model. The disadvantage of this method is that the mesh model of the hand usually contains hundreds or thousands of vertices, so directly predicting the coordinates of each vertex requires solving in a very large solution space. At the same time, directly predicting the position of each vertex does not utilize the prior of the hand shape. 2) Generate a mesh model using a parametric model. In order to solve the above problems, a large number of algorithms that use parametric models to estimate the three-dimensional mesh model of the hand have been proposed. MANO is the most commonly used parametric model of the hand, which contains two parameters: shape and posture. The 10-dimensional shape parameter describes the length, fatness and thinness of the hand, and the 48-dimensional posture parameter consists of the rotation vectors of the 16 joints of the hand. After the shape parameters and posture parameters of MANO are given, the hand mesh model can be generated and the key point positions of the hand can be obtained. It should be noted that the process of generating mesh models and key point locations from shape parameters and pose parameters is differentiable, which allows us to achieve the goal of training the network by supervising the network and skeleton output by the MANO model.
[0005] Currently, there are many works that reconstruct the 3D model of one hand through a single color image, but most of the 3D reconstruction of both hands relies on depth images. Existing works that reconstruct the 3D model of both hands through a single color image often cannot accurately reconstruct the 3D model of both hands in a close interaction situation. Summary of the invention
[0006] In view of the technical problems existing in the prior art, the purpose of the present invention is to provide a three-dimensional reconstruction method in a two-handed interactive state. The present invention reconstructs a two-handed three-dimensional model in a two-handed interactive state through a single color picture, and can use an ordinary color camera to reconstruct the two-handed three-dimensional model. It should be pointed out that although the present invention proposes an optimization algorithm for the situation where the two hands are closely interacting, the present invention can also realize the three-dimensional reconstruction of the two hands when the two hands are separated from each other.
[0007] In order to solve the above technical problems, the present invention provides a method for reconstructing a three-dimensional hand model. Figure 2 As shown, it includes the following steps:
[0008] Step 1, input a single color image;
[0009] Step 2, using a 2.5D heat map extraction network (such as InterNet) to detect the 2.5D heat map of the key points of both hands in the image; the 2.5D heat map reflects the position of the two-dimensional key points on the image and the depth of the two-dimensional key points in space;
[0010] Step 3: Use the ResNet50 shallow residual module to extract the features of the image in step 1;
[0011] Step 4: Input the image features extracted by the shallow residual module of ResNet50 and the 2.5D heat map into the subsequent deep residual module of ResNet50 to extract the feature map;
[0012] Step 5: Perform maximum pooling operation along the channel on the 2.5D heat maps of the left hand and the right hand detected in step 2 to obtain the attention maps of the left hand and the right hand respectively;
[0013] Step 6, using the feature map extracted in step 4 as input to predict the relative position relationship between the two hands using a fully connected network;
[0014] Step 7, multiply the feature map in step 4 by the left-hand attention map and the right-hand attention map obtained in step 5 point by point, and then obtain the left-hand feature map and the right-hand feature map by average pooling;
[0015] Step 8, using the feature maps of the left hand and the right hand, respectively predict the parameters of the parameterized model (such as MANO, etc.) of the left hand and the right hand;
[0016] Step 9, using the model parameters of the left hand and the right hand generated in step 8, regress the positions of the 3D key points of the left hand and the right hand, calculate the camera parameters of the weak perspective projection model using the positional relationship between the 2.5D heat map predicted in step 2 and the regressed 3D key points, and generate the attention maps of the left hand and the right hand respectively according to the projection positions of the 3D key points;
[0017] Step 10, multiply the attention map generated in step 9 by the low-level features extracted by the deep residual module in step 4 pixel by pixel to extract the features of the left hand and the right hand;
[0018] Step 11, input the left and right hand features in step 10, the model parameters of the left and right hands predicted by the network before, and the relative position relationship of the two hands predicted in step 6 into the cascade module, and the network predicts new parameterized model parameters of the left and right hands;
[0019] Step 12, iteratively repeat steps 9-11, use the latest predicted left and right hand model parameters to generate left and right hand attention maps, and multiply the newly generated left and right hand attention maps pixel by pixel with the lower-level features (features close to the input end) extracted by the deep residual module in step 4 to generate the input features of the next cascade module, which is used for the next cascade module to obtain a more accurate human hand parameterization model.
[0020] The present invention comprises M cascade modules. In addition to the result output by the Mth cascade module, each hand parameterized model parameter predicted by the network will be used to generate an attention map. The attention map generated by the cascade module close to the output end is multiplied pixel by pixel with the feature map closer to the input end in step 4 (similar to hourglass) to generate the input of the next cascade module.
[0021] Furthermore, the method also includes that the hand three-dimensional model used is a hand parameterized model including posture and shape parameters, and the corresponding mesh model can be generated by given parameters, and the mesh model is as follows: Figure 1 shown.
[0022] Furthermore, the method also includes that the three-dimensional hand model is represented as a mesh, which provides more information than skeleton nodes. Shape information has important applications in the field of human-computer interaction and is the basis of collision detection, bringing users a more realistic interaction experience.
[0023] Furthermore, the method also includes training the network using real data sets, including InterHand2.6M and CMUhaggling.
[0024] Furthermore, the method also includes step 9, generating an attention map using two-dimensional key points and three-dimensional key points. After specifying the parameters of the hand parameterized model, the three-dimensional key points of the hand are obtained, and the coordinates of the two-dimensional key points are obtained according to the position with the highest pixel value of the 2.5D heat map of each key point in step 2. Define the sum of the squares of the distances between the projection position of the three-dimensional key point and the two-dimensional key point under the weak perspective projection model as the loss, optimize the weak perspective projection parameters to minimize the loss, and obtain the aligned weak perspective projection parameters. Use the aligned weak perspective projection parameters to project the three-dimensional key points onto the two-dimensional image, generate a heat map with each projection position as the center, and then fuse the heat map of each key point through differentiable calculation to obtain the attention map.
[0025] Furthermore, the method further includes predicting a more accurate mesh model of both hands by combining the model parameters and relative position relationship of both hands with the cascade module used in step 11 and step 12. The neural network uses the semantic information between the two hands to achieve accurate three-dimensional model estimation.
[0026] Compared with the prior art, the present invention has the following advantages:
[0027] The present invention realizes 3D reconstruction of both hands only through a single color picture. Compared with depth cameras, color cameras are more convenient and have wider application scenarios and can be used outdoors.
[0028] In addition, compared with the traditional top-down method, the area where each hand is located is first detected and then the three-dimensional model of the hand is reconstructed in each area. The present invention directly reconstructs the three-dimensional model of both hands and estimates the relative position relationship between the two, so the two hands are not required to be separated from each other in the picture. Even when the two hands are closely interacting, the present invention can achieve the reconstruction of the three-dimensional model of both hands.
[0029] In addition, when supervising the network, the output 3D key point positions, the relative positions between the two hands, shape parameters, and attention maps are supervised, and shape consistency constraints and regularity constraints are added to ensure the accuracy and rationality of the 3D model reconstruction results.
[0030] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 Schematic diagram of the hand mesh model used in the present invention.
[0032] Figure 2 It is the overall architecture diagram of the present invention.
[0033] Figure 3 It is a schematic diagram of using attention map to extract features in the present invention. DETAILED DESCRIPTION
[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.
[0035] Step 1: Input a single color image.
[0036] The present invention requires the input of a single color image, and the hand should occupy a certain proportion in the image. The two hands in the image can be in a state of close interaction or in a state of separation. Before the image is input into the neural network, the size should be adjusted to 256×256. In order not to change the shape of the hand in the image, the length and width need to be scaled proportionally, the longer side is adjusted to a length of 256 pixels, and 128 pixels are used for padding to make the length and width of the image equal. Before the image is input into the network, the pixel value is divided by 256 and then subtracted by 0.5, and the range of pixel values becomes -0.5 to 0.5.
[0037] Step 2: Detect 2.5D heatmaps of key points of both hands using a neural network.
[0038] The present invention uses the existing InterNet network to predict the 2.5D heat map and estimates the heat maps of 42 key points of both hands. The output heat map resolution is 64×64×64. In order to reduce the space consumption of network testing and training, each heat map is downsampled to 16×32×32. The neural network used in this step to predict the 2.5D heat map can be trained separately, and the parameters of the InterNet network are fixed when training the two-hand three-dimensional model reconstruction network. All modules such as ResNet50 shallow residual module, ResNet50 deep residual module, and various cascade modules together constitute the three-dimensional model reconstruction network.
[0039] Step 3: Use the ResNet50 shallow residual module to extract the features of the image;
[0040] Step 4: Compare the image features extracted by the neural network with the 2.5D thermal Figure 1 The same is input into the subsequent ResNet50 deep residual module to extract features.
[0041] First, the image is convolved with a 7×7 convolution kernel with a stride of 2, and then the feature map dimension is 64×64×64 after maximum pooling. After two sets of residual modules, a 512×32×32 feature map is obtained. The downsampled heat map (42×16×32×32) obtained in step 2 is converted to 672×32×32 by merging the two dimensions of key points and depth. The heat map and the feature map are concatenated to form a 1184×32×32 feature map, which is sent to the subsequent residual module to output a 2048×8×8 feature map.
[0042] Step 5: Generate attention maps for the left and right hands using the 2.5D heat map.
[0043] The 2.5D heat maps of the left and right hands generated in step 2 are concatenated along the depth direction to obtain a tensor of dimension (21×64)×64×64, and the maximum value pooling is performed along the channel to obtain an attention map of dimension 64×64. Then, an 8×8 attention map is obtained by downsampling.
[0044] Step 6: predict the relative position relationship between the two hands.
[0045] The present invention will first predict the normalized three-dimensional model of each hand in its own coordinate system, and specify the metacarpophalangeal joint of the middle finger as the coordinate origin. And the distance from the joint to the wrist joint is normalized to 1. The advantage of doing this is that a model that is independent of absolute scale and absolute position can be obtained, which reduces the ambiguity of estimating a three-dimensional model from a two-dimensional color image. However, it is necessary to predict the relative position relationship of the two hands. The relative position relationship includes the relative position of the origin and the relative size of the two hands. The more important significance of introducing the origin is to eliminate the ambiguity of the hand position in the camera coordinate system, and pay more attention to the relative position of each point on the hand shape with respect to the origin.
[0046] The 2048×8×8 feature map obtained in step 4 is averaged and pooled to obtain a 2048-dimensional vector. This vector is passed through a two-layer fully connected network to obtain four parameters used to describe the relative position relationship between the two hands. The first parameter is the natural logarithm α of the scale ratio between the two hands, e α It is the ratio of the scale of the right hand to the left hand, and the last three parameters are the position offset T between the two hands (T is a vector in three-dimensional space containing three components x, y, and z). The predicted left-hand 3D model is added to the position offset T, and then multiplied by the scale ratio to transfer the left-hand 3D model to the coordinate system of the right-hand 3D model, thus unifying the 3D models of both hands into the same coordinate system.
[0047] The 7×7 convolution kernel used in step 2 and the residual modules in steps 2 to 4 above together constitute the Resnet50 network. The difference from the original Resnet50 is that the fully connected layer used for classification is removed, and the input dimension of the third group of residual modules of Resnet50 is modified to be equal to the dimension of the tensor obtained by concatenating the 2.5D heat map and the feature map.
[0048] Step 7: Extract the left hand feature map and the right hand feature.
[0049] Multiply the left and right hand attention maps obtained in step 5 by the 2048×8×8 feature map obtained in step 4 pixel by pixel in each channel. The left hand feature map and the right hand feature map are obtained respectively. The attention map is passed through the convolution network and average pooling to obtain the vector representing the left hand feature and the vector representing the right hand feature. The process of extracting features is as follows: Figure 3As shown in the figure. The way the attention map is generated determines that it has a higher response in the area where the hand is located. Therefore, the pixel values of the left hand feature map in the left hand area are retained, and the pixel values outside the area are suppressed. The result can better reflect the characteristics of the left hand. Similarly, the right hand feature map can retain more right hand features and remove other features.
[0050] Step 8: Predict the model parameters for the left and right hands.
[0051] Taking the left hand as an example, the feature map of the left hand is averaged and pooled to obtain a 2048-dimensional vector. The MANO model parameters of the left hand are predicted through a fully connected network. The network uses the MANO model, but the present invention can be extended to use other parameterized models. The MANO model parameters of the right hand are predicted in the same way. In order to supervise the prediction results of the MANO parameters, the loss function includes the following loss terms: joint offset loss of both hands, shape consistency loss of both hands, key point position loss, shape loss, skeleton length loss, shape parameter regularization term, and posture parameter regularization term.
[0052] J left,i and They represent the predicted value and true value of the position of the i-th 3D key point on the left hand. right,i and They represent the predicted value and true value of the position of the i-th 3D key point of the right hand respectively. left,b and They represent the predicted value and true value of the length of the bth bone in the left hand. right,b and They represent the predicted value and true value of the length of the bth bone of the right hand respectively.
[0053] β left and β right denote the shape parameters of the left-handed and right-handed MANO models respectively. left and θ right denote the shape parameters of the left-handed and right-handed MANO models, respectively.
[0054] The joint offset loss of both hands is
[0055] The shape consistency loss of both hands is
[0056] The key point location loss is
[0057] The shape loss is
[0058] The skeleton length loss is in It is the distance from the metacarpophalangeal joint of the middle finger to the wrist joint.
[0059] The shape parameter regularization term and the attitude parameter regularization term are the L2 regularization terms of the shape parameter and attitude parameter of MANO respectively.
[0060] Specifically, the hand key point offset loss is used to supervise the relative position of the key points with the same number on the left and right hands. The hand shape consistency loss is used to penalize the deviation of the shape parameters of the left and right hands, ensuring that the left and right hands of the same person have similar shape parameters. The key point position loss is used to supervise the key point position of the reconstructed 3D model. The shape loss is used to supervise the shape parameters of MANO. The skeleton length loss is used to supervise the skeleton length of the reconstructed 3D model. The shape parameter regularization term and the posture parameter regularization term ensure that the network outputs a reasonable hand shape.
[0061] Step 9: Generate an attention map using model parameters.
[0062] First, the three-dimensional key point coordinates of the hand are calculated by the parameters of the predicted hand parametric model. The three-dimensional key points of the hand are projected onto a two-dimensional plane using a weak perspective projection camera model. The two-dimensional coordinates of the key points on the picture can be obtained based on the position of the maximum pixel value of the 2.5D heat map in the picture (i.e., the single color image input in step 1, the 2.5D heat map can be transformed to the original picture through scale transformation). The sum of the squares of the distances between the two-dimensional coordinates and the projection position of the three-dimensional key point on the two-dimensional plane is regarded as the error. Minimizing the error is to solve the least squares problem with the weak perspective projection parameters as the independent variable, and the weak perspective projection parameters are solved by the least squares method so that the error between the predicted two-dimensional key point and the three-dimensional key point projection is minimized. Among them, the predicted value of the two-dimensional key point coordinates is obtained by the 2.5D heat map predicted in step 2. After the projected three-dimensional key point is projected onto the plane, the pixel value generated according to the preset variance is centered on the projection position and follows the Gaussian distribution. Two-dimensional heat map of x is the pixel coordinate, c is the center of the Gaussian distribution, i.e., the projection position of the key point, and σ is the variance of the Gaussian distribution. 2 =3. After obtaining the heat map of each key point, the heat map of each key point of each hand is fused by differentiable calculation. The present invention uses the method of cumulative multiplication to perform fusion Get the attention map.
[0063] Step 10, extract the left hand feature map and the right hand feature.
[0064] Taking the left hand as an example, take a 2048×8×8 dimension from the residual network and multiply it pixel by pixel with each channel of the left hand attention map obtained in step 9. The result is passed through a residual module to output a 2048×8×8 feature, which is averaged and pooled to obtain a 2048-dimensional left hand feature. The above method is used to obtain the features of the left and right hands respectively.
[0065] Step 11, predict new left-hand and right-hand model parameters.
[0066] Taking the left hand as an example, the left hand features predicted in step 10, the hand posture parameters and hand shape parameters predicted in step 8, and the relative position relationship of the hands predicted in step 6 are spliced together and sent to the left hand parameter prediction branch. This branch contains two fully connected layers. The first layer uses relu as the activation function, and the second layer outputs the MANO parameters of the left hand. The MANO parameters of the right hand are predicted in the same way. The same loss function as in step 8 is used to supervise the network.
[0067] The left-hand and right-hand branches that emerge in this step form the first-level cascade module.
[0068] Step 12: Improve the reconstruction accuracy of the two-hand model through cascading.
[0069] The network structure of each cascade module is the same. The attention map is generated using the MANO parameters predicted by the previous level. The attention map is multiplied pixel by pixel with the residual network feature map of the lower level (the second level uses 1024×16×16 dimensional features, and the third level uses 512×32×32 dimensional features). The result of the multiplication is passed through the residual module to obtain a 2048×8×8 feature, and the average value pooling is used to obtain a 2048-dimensional vector. The MANO parameters are predicted using the method of step 11.
[0070] It should be noted that the predicted 2D keypoint coordinates need to be multiplied by the corresponding coefficients according to the selected feature map size when aligning. For example, the original 2D keypoint prediction value corresponds to a 64×64 image. When the feature map is 16×16, the 2D coordinates need to be multiplied by
[0071] The present invention is explained above using a human hand as an example. It is easy to understand that the present invention is also applicable to objects such as the entire human body or a part thereof (such as a human hand or a human face), multiple people, multiple animals, multiple robots, etc. The present invention is also applicable to a variety of sensors, including but not limited to color cameras, depth cameras, lidars, etc.
[0072] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the principle and scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A three-dimensional reconstruction method in a two-handed interaction state, the steps of which are include: 1) Input a single color image, in which both hands are in an interactive state or a separated state; 2) Detecting a 2.5D heat map of the key points of both hands in the color image; 3) Using the ResNet50 shallow residual module to extract the features of the color image; 4) Inputting the 2.5D heat map and the features extracted in step 3) into the ResNet50 deep residual module to extract the feature map; 5) Perform maximum pooling operations on the 2.5D heat map of the left hand and the 2.5D heat map of the right hand along the channel to obtain the left hand attention map and the right hand attention map; 6) using the feature map extracted in step 4) as input to predict the relative position relationship between the two hands using a fully connected network; 7) Multiply the feature map extracted in step 4) by the left-hand attention map point by point, and then obtain the left-hand feature map by average pooling; multiply the feature map extracted in step 4) by the right-hand attention map point by point, and then obtain the right-hand feature map by average pooling; 8) Predicting the parameters of the left-hand parameterized model using the left-hand feature map; Predicting the parameters of the right-hand parameterized model using the right-hand feature map; 9) regressing the three-dimensional key point positions of both hands according to the parameters of the left hand parameterized model and the parameters of the right hand parameterized model, and then calculating the camera parameters of the weak perspective projection model using the positional relationship between the 2.5D heat map and the regressed three-dimensional key points, and generating the left hand attention map and the right hand attention map according to the projection position of the three-dimensional key points; The method for generating a left-hand attention map or a right-hand attention map is as follows: first, the coordinates of the three-dimensional key points of the hand are calculated by the parameters of the hand parameterization model; then, the three-dimensional key points of the hand are projected onto a two-dimensional plane using a weak perspective projection camera model; then, the two-dimensional coordinates of the key points on the color image are obtained according to the position of the pixel maximum value of the 2.5D heat map in the color image, and the sum of the squares of the distances between the two-dimensional coordinates and the projection position of the three-dimensional key points on the two-dimensional plane is regarded as an error; minimizing the error is to solve the least squares problem with weak perspective projection parameters as independent variables, and the weak perspective projection parameters are solved by the least squares method so that the error between the predicted two-dimensional key points and the projection of the three-dimensional key points is minimized, and after the three-dimensional key points are projected onto the two-dimensional plane, a two-dimensional heat map of each key point whose pixel value obeys a Gaussian function is generated according to a preset variance with the projection position as the center; then, the two-dimensional heat maps of each key point corresponding to each hand are fused to obtain the attention map of the corresponding hand; wherein the hand parameterization model is a left-hand parameterization model or a right-hand parameterization model; 10) Multiply the left-hand attention map and the right-hand attention map generated in step 9) by the feature map extracted in step 4) pixel by pixel to extract the left-hand feature and the right-hand feature; 11) inputting the parameters of the left-hand parameterized model and the right-hand parameterized model obtained in step 8), the left-hand features and the right-hand features obtained in step 10), and the relative position relationship predicted in step 6) into the cascade module, and the network predicts and updates the parameters of the left-hand parameterized model and the right-hand parameterized model; 12) Iteratively repeat steps 9) to 11), use the latest predicted left-hand parameterized model parameters and right-hand parameterized model parameters to generate the corresponding left-hand attention map and right-hand attention map, and then multiply the newly generated left-hand attention map and right-hand attention map with the feature map extracted in step 4) pixel by pixel and input them into the next cascade module to generate a human hand parameterized model.
2. The method according to claim 1, It is characterized in that Each of the cascade modules includes two branches, namely a left-hand parameter prediction branch and a right-hand parameter prediction branch; the left-hand features, the parameters of the left-hand parameterized model, the parameters of the right-hand parameterized model, and the relative position relationship are spliced together and sent to the left-hand parameter prediction branch; the left-hand parameter prediction branch includes two fully connected layers connected in sequence, the first fully connected layer uses relu as an activation function to process the input data and then inputs it into the second fully connected layer, and the second fully connected layer outputs the left-hand parameterized model parameters; the right-hand features, the parameters of the left-hand parameterized model, the parameters of the right-hand parameterized model, and the relative position relationship are spliced together and sent to the right-hand parameter prediction branch; the right-hand parameter prediction branch includes two fully connected layers connected in sequence, the first fully connected layer uses relu as an activation function to process the input data and then inputs it into the second fully connected layer, and the second fully connected layer outputs the right-hand parameterized model parameters.
3. The method according to claim 1, It is characterized in that Using the formula The two-dimensional heat maps of each key point corresponding to each hand are fused, N is the total number of key points, is the two-dimensional heat map of the i-th key point.
4. The method according to claim 1 or 2, It is characterized in that The left-hand parameterized model and the right-hand parameterized model are both MANO models.
5. The method according to claim 1, It is characterized in that The method for predicting the relative position relationship between the two hands is as follows: the feature map extracted in step 4) is average-pooled and the resulting vector is input into a two-layer fully connected network to obtain parameters for describing the relative position relationship between the two hands.
6. The method according to claim 1, It is characterized in that 2.5D heatmap extraction network detects 2.5D heatmaps of key points of both hands in the image.
7. A server, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
