Hand reconstruction model training method, hand reconstruction method and device

Abstract

The invention discloses a hand reconstruction model training method, a hand reconstruction method and a hand reconstruction device, which are applied to the field of three-dimensional reconstruction, and comprise the following steps: carrying out feature extraction on hand pictures in a training data set to obtain global features; extracting the global features to respectively obtain left and right hand positioning features, left and right hand joint region segmentation features and left and right hand interaction features; performing feature fusion on the left and right hand positioning features, the left and right hand joint region segmentation features and the left and right hand interaction features to obtain fusion features; based on the fusion features, performing regression on the left-right hand MANO coefficient and the camera parameters, and performing hand reconstruction by using the regressed left-right hand MANO coefficient and the regressed camera parameters to obtain a reconstructed hand; and training the hand reconstruction model according to the reconstructed hand and the real hand to obtain a trained hand reconstruction model. According to the method, hand reconstruction of any scene can be carried out based on a single picture, and the reconstructed hand posture and shape and the relative depth position between the two hands are accurate.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a hand reconstruction model training method, a hand reconstruction method and a hand reconstruction device. Background Art

[0002] 3D hand reconstruction based on a single RGB camera plays a vital role in various emerging applications such as virtual humans, AR (Augmented Reality) and VR (Virtual Reality), human-computer interaction, and 3D character animation for movies and games. However, 3D hand reconstruction is difficult due to limited labeled data, object occlusion, self-occlusion between hands, motion blur, and other issues.

[0003] Therefore, how to provide a hand reconstruction model that has good reconstruction effect, is universal, and can cope with any scenario is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a hand reconstruction model training method, a hand reconstruction method and a device, which solve the problems in the prior art that the hand reconstruction model is not universal enough, cannot cope with any hand scene, and has poor reconstruction accuracy.

[0005] In order to solve the above technical problems, the present invention provides a hand reconstruction model training method, comprising:

[0006] Use the feature extraction module to extract features from the hand images in the training data set to obtain global features;

[0007] The global features are extracted using a left and right hand positioning module, a left and right hand joint region segmentation module, and a left and right hand interaction feature extraction module to obtain left and right hand positioning features, left and right hand joint region segmentation features, and left and right hand interaction features;

[0008] Using a feature fusion module to fuse the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features to obtain a fused feature;

[0009] Based on the fusion features, the left-right-hand MANO coefficients and camera parameters are regressed using the left-right-hand MANO coefficient regression module, and the hand is reconstructed using the regressed left-right-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand;

[0010] The hand reconstruction model is trained according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module.

[0011] Optionally, a feature fusion module is used to fuse the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features to obtain a fused feature, including:

[0012] Fusing the left and right hand positioning features with the left and right hand joint area segmentation features to obtain left and right hand features;

[0013] Determining the number and positions of hands according to the left and right hand positioning features, and determining a fusion coefficient according to the number and positions of hands;

[0014] According to the fusion coefficient, the left and right hand features and the left and right hand interaction features are fused to obtain the fused features.

[0015] Optionally, the hand images in the training data set are images of the upper body region of the human body containing hands.

[0016] Optionally, the feature extraction module is a neural network with a Transformer structure.

[0017] Optionally, the process of acquiring the training data set includes:

[0018] The training dataset is constructed according to the acquired public hand reconstruction dataset, the actually collected hand motion dataset and the hand synthesis dataset.

[0019] Optionally, the process of acquiring the synthetic hand data set includes:

[0020] Use the actual collected limb and hand motion data, as well as the limb and hand motion data extracted from the public human motion data set to redirect and drive the 3D virtual digital human;

[0021] Based on the three-dimensional virtual digital human, randomly set camera parameters, light source parameters and background, and render to generate a two-dimensional digital human video;

[0022] Obtaining the three-dimensional key point coordinates of the hands of the three-dimensional virtual digital human, and calculating the MANO coefficients of the left and right hands, wherein the MANO coefficients of the left and right hands include the hand posture coefficient and the hand shape coefficient;

[0023] The hand synthesis data set is obtained according to the two-dimensional digital human video, the hand posture coefficient and the hand shape coefficient.

[0024] Optionally, training a hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model includes:

[0025] Regressing the center points of the left and right hands based on the left and right hand positioning features to obtain a regression center point, and calculating a hand center point loss value based on the regression center point and the center point in the hand image;

[0026] Calculating a hand joint loss value according to the left and right hand joint region segmentation features and the left and right hand joint region segmentation features in the hand image;

[0027] A hand posture coefficient loss value and a hand shape coefficient loss value are calculated based on the regressed left and right hand MANO coefficients and the left and right hand MANO coefficients in the hand picture;

[0028] Extracting the three-dimensional key point coordinates and the two-dimensional key point coordinates of the left and right hands from the reconstructed hand;

[0029] Calculating a 3D key point loss value according to the 3D key point coordinates and the 3D key point coordinates of the hand in the hand image;

[0030] Calculating a two-dimensional key point loss value according to the two-dimensional key point coordinates and the two-dimensional key point coordinates of the hand in the hand image;

[0031] Calculating a reconstructed bone length according to the reconstructed hand, and calculating the reconstructed bone length with the hand bone length in the hand image to obtain a bone length loss value;

[0032] A total loss value is calculated according to the hand center point loss value, the hand joint loss value, the hand posture coefficient loss value, the hand shape coefficient loss value, the three-dimensional key point loss value, the two-dimensional key point loss value and the bone length loss value;

[0033] The hand reconstruction model is subjected to negative feedback training according to the total loss value, and when a training end condition is reached, the training is ended to obtain the trained hand reconstruction model.

[0034] The present invention also provides a hand reconstruction method, comprising:

[0035] Acquire a picture to be reconstructed, where the picture to be reconstructed contains a hand;

[0036] The image to be reconstructed is input into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method described above.

[0037] The present invention also provides a hand reconstruction model training device, comprising:

[0038] A global feature extraction module is used to extract features from hand images in a training data set using the feature extraction module to obtain global features;

[0039] A fine-grained feature extraction module is used to extract the global features using the left and right hand positioning modules, the left and right hand joint area segmentation modules, and the left and right hand interaction feature extraction modules to obtain left and right hand positioning features, left and right hand joint area segmentation features, and left and right hand interaction features;

[0040] A fusion module, used for fusing the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features using a feature fusion module to obtain a fusion feature;

[0041] A training reconstruction module is used to regress the left-hand MANO coefficients and camera parameters using the left-hand MANO coefficient regression module based on the fusion features, and reconstruct the hand using the regressed left-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand;

[0042] A model training module is used to train a hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module.

[0043] The present invention also provides a hand reconstruction device, comprising:

[0044] A module for acquiring a picture to be reconstructed, used for acquiring a picture to be reconstructed, wherein the picture to be reconstructed includes a hand;

[0045] The reconstruction module is used to input the image to be reconstructed into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the above-mentioned hand reconstruction model training method.

[0046] The present invention also provides an electronic device, comprising:

[0047] Memory for storing computer programs;

[0048] The processor is used to execute the computer program to implement the above method.

[0049] The present invention also provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0050] It can be seen that the present invention uses a feature extraction module to extract features from hand images in a training data set to obtain global features; respectively uses a left and right hand positioning module, a left and right hand joint area segmentation module, and a left and right hand interaction feature extraction module to extract the global features to obtain left and right hand positioning features, left and right hand joint area segmentation features, and left and right hand interaction features; uses a feature fusion module to fuse the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features to obtain fused features; based on the fused features, uses a left and right hand MANO coefficient regression module to regress the left and right hand MANO coefficients and camera parameters, and uses the regressed left and right hand MANO coefficients and the regressed camera parameters to reconstruct the hand to obtain a reconstructed hand; trains a hand reconstruction model according to the reconstructed hand and the hand in the hand image to obtain the trained hand reconstruction model; the hand reconstruction model consists of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module, and the left and right hand MANO coefficient regression module. The hand reconstruction model provided by the method can be used to reconstruct hands in any scene based on a single image. Since the left and right hand positioning module, the left and right hand joint area segmentation module, and the left and right hand interaction feature extraction module in the hand reconstruction model of the method can extract the features of a single hand, and interacting hands, the hand reconstruction can be accurately performed for a variety of scenes such as a single hand, interacting hands, object occlusion, self-occlusion between hands, motion blur, etc. The reconstructed hand posture and shape are accurate, and the relative depth position between the hands is accurately predicted. The hand reconstruction in truncated and occluded scenes is greatly improved compared with the current methods.

[0051] In addition, the present invention also provides a hand reconstruction method, a hand reconstruction model training device, a hand reconstruction device, an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A flowchart of a hand reconstruction model training method provided by an embodiment of the present invention;

[0054] Figure 2 A flowchart of a data synthesis method provided by an embodiment of the present invention;

[0055] Figure 3 An example diagram of a hand reconstruction model provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a hand key point provided by an embodiment of the present invention;

[0057] Figure 5 A flowchart of a hand reconstruction method provided by an embodiment of the present invention;

[0058] Figure 6 A schematic diagram of a result of interactive two-handed and one-handed 3D hand reconstruction provided by an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of the result of 3D hand reconstruction in a truncated and occluded scene provided by an embodiment of the present invention;

[0060] Figure 8 A schematic diagram of the result of 3D hand reconstruction in a motion blurred scene provided by an embodiment of the present invention;

[0061] Fig. 9 A schematic diagram of the structure of a hand reconstruction model training device provided by an embodiment of the present invention;

[0062] Fig.10 A schematic diagram of the structure of a hand reconstruction device provided by an embodiment of the present invention;

[0063] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] 3D hand reconstruction has been widely used in various fields such as digital humans, augmented reality (AR) and virtual reality (VR), human-computer interaction, movies and games. However, the current mainstream method is to use hand detection algorithms, such as single-hand reconstruction and interactive double-hand reconstruction. (1) Single-hand reconstruction: First, use hand detection to detect the left and right hand regions, and then reconstruct the left and right hands separately. This type of method reduces the difficulty of the problem, only considers the scene where the two hands do not interact, and focuses on single-hand reconstruction. Although the single-hand reconstruction method can be extended to double-hand reconstruction, due to the confusion caused by the mutual occlusion between the two hands in the interactive state and the similarity between the skin colors, this type of method has very poor results for hand reconstruction in the interactive state. (2) Interactive double-hand reconstruction: First, use hand detection to detect the left and right hand regions, and treat the left and right hands in the interactive state as a whole, and reconstruct both hands at the same time. This type of method assumes that the input image must contain two hands, which inevitably introduces constraints on the input image. When the input image almost contains one hand or no hand, both hands will still be reconstructed, resulting in serious errors. In addition, when the hands are severely occluded or truncated, the reconstruction effect is very poor.

[0066] When the above method is finally applied, the accuracy of hand reconstruction is heavily dependent on the accuracy of the hand detection network. In addition, when reconstructing the left and right hands separately in a non-interactive scene, the relative position relationship between the two hands cannot be modeled, resulting in serious errors in the relative position between the two hands, especially in the relative position in the depth dimension.

[0067] In order to solve the above problems, the method of the present invention can use the same model to reconstruct hands in various hand scenes such as no hands, only one hand, non-interactive hands, and interactive hands in the image. It can not only reconstruct accurate hand posture and shape, but also accurately model the relative position relationship between the two hands. For details, please refer to Figure 1 , Figure 1 A flowchart of a hand reconstruction model training method provided by an embodiment of the present invention. The method may include:

[0068] S101: Use the feature extraction module to extract features from the hand images in the training data set to obtain global features.

[0069] This embodiment first extracts features from the hand images in the training data set to obtain global features. This embodiment does not limit the feature extraction model, as long as it can complete the hand feature extraction. This embodiment also does not limit the hand images in the training data set, as long as they contain hands.

[0070] Furthermore, in order to reduce the restrictions on the images and improve the applicability of the model, the hand images in the training data set in this embodiment can be images of the upper body region of the human body containing the hands. In this way, the model can be applied not only to hand images with only the hands, but also to hand images with backgrounds, thereby improving the wide application of the model.

[0071] Furthermore, in order to improve the accuracy of feature extraction, the feature extraction module can be a neural network with a Transformer (a deep learning model architecture) structure. Compared with conventional convolutional neural networks, the Transformer structure neural network has a powerful attention mechanism, which can focus the network's attention area on the hand area of ​​the image during training, thereby improving the accuracy of hand feature extraction.

[0072] Furthermore, in order to improve the accuracy of feature extraction and thus improve the accuracy of model reconstruction, the process of acquiring the above training data set may include the following steps:

[0073] The training dataset is constructed based on the acquired public hand reconstruction dataset, the actually collected hand motion dataset and the hand synthesis dataset.

[0074] The training data set used in this embodiment is richer in data, which can improve the training effect of the model. Specifically, it can include a public data set for hand reconstruction, a data set of hand motion actually collected, and a data set of hand synthesis. Among them, the hand motion data set actually collected is the hand motion data recorded by a motion capture device. Considering that the neural network with a Transformer structure may often require more data than a convolutional neural network to extract effective features, and accurate hand reconstruction data relies on high-cost motion capture equipment, which is very expensive and difficult to collect a large amount of data, this embodiment also provides a data synthesis method. Specifically, the acquisition process of the above-mentioned hand synthesis data set may include the following steps:

[0075] Step 11: Redirect and drive the 3D virtual digital human using the actual collected limb and hand motion data and the limb and hand motion data extracted from the public human motion dataset;

[0076] Step 12: Based on the 3D virtual digital human, randomly set the camera parameters, light source parameters and background, and render to generate a 2D digital human video;

[0077] Step 13: Obtain the three-dimensional key point coordinates of the hand of the three-dimensional virtual digital human, and calculate the MANO coefficients of the left and right hands, which include the hand posture coefficient and the hand shape coefficient;

[0078] Step 14: Obtain a hand synthesis dataset based on the two-dimensional digital human video, hand posture coefficients and hand shape coefficients.

[0079] Specifically, please refer to Figure 2 , Figure 2 A flowchart of a data synthesis method provided by an embodiment of the present invention. In the data synthesis process of the hand synthesis data set, the motion capture device is first used to perform motion capture, and the limb and hand motion data are also extracted from the public human motion data set. The large amount of motion data obtained by these two methods is redirected to drive the 3D virtual digital human. In the process of driving, digital humans of different genders, ages, skin colors, and heights can be randomly selected, and different hairstyles, accessories, clothes, etc. can be given to the selected digital humans to generate various digital humans. Then, the redirected driven 3D virtual digital human is rendered. During the rendering process, random camera parameters, light source parameters, background images, etc. can be added to increase the randomness of the viewing angle, brightness, and scene. After rendering, a two-dimensional digital human video containing hand motions is obtained. In the driving process, the 3D key point coordinates of the digital human's hand are saved, and the MANO (HandModel with Articulated and Non-rigid Deformations, a hand parameterized model with articulated and non-rigid deformations) coefficients (including hand posture coefficients and hand shape coefficients) are calculated using a least squares-based optimization method to obtain the label of the hand synthesis image.

[0080] S102: extracting global features using left and right hand positioning modules, left and right hand joint region segmentation modules, and left and right hand interaction feature extraction modules respectively, to obtain left and right hand positioning features, left and right hand joint region segmentation features, and left and right hand interaction features.

[0081] In order to achieve the purpose of hand reconstruction in any scene, it is necessary to locate the hand to determine whether there are hands and how many hands there are in the picture; it is also necessary to segment the hand to extract the separate features of the left and right hands for the regression of the MANO coefficients and camera parameters of the left and right hands; it is also necessary to extract the hand interaction features, that is, to extract the overall features of the left and right hands when they interact. Because when the two hands are not in contact and not interacting, the hand posture only depends on its own local features, and when the two hands interact, the hand postures of the left and right hands are interdependent, such as the New Year's greeting gesture of putting your hands together and the prayer gesture of crossing your fingers. Therefore, three branches are designed in the feature decoding part, which are respectively used for hand positioning, hand segmentation, and hand interaction feature extraction. That is, the left and right hand positioning module, the left and right hand joint area segmentation module, and the left and right hand interaction feature extraction module. Such fine-grained feature extraction can eliminate redundant features and can also facilitate accurate hand reconstruction based on various features in the later stage. The following is a detailed description of each module:

[0082] Left and right hand positioning module: In the hand positioning branch, the global features extracted by the feature extraction module are input into the left and right hand positioning modules. The left and right hand positioning modules are two branches with exactly the same structure. The branch structure is a 2x upsampling module and a 2D (two-dimensional) convolution layer to generate feature maps and perform feature recovery. In this module, the global features are separated into left-hand features and right-hand features for the regression of the left and right hand center key points. This branch can predict the 2D coordinates of the hand center point and the predicted confidence. In actual application, the number and position of the hands in the image can be determined based on the coordinates and confidence predicted by this branch. At the same time, this branch can use the left and right hand features to accurately locate the hands, which means that the extracted left and right hand features are accurate, which helps to separate the left and right hand features.

[0083] Left and right hand joint region segmentation module: In the hand segmentation branch, the global features extracted by the feature extraction module are input into the left and right hand joint region segmentation module. In this module, the global features also need to be separated into left hand features and right hand features for the segmentation of the left and right hand joint regions. This branch uses a segmentation decoding network with a UNet (a deep learning network architecture specifically for image segmentation). In the decoding network, multiple upsampling layers and convolution layers are superimposed, and upsampling is performed by 2 times each time to finally generate The purpose of this branch is to further separate the left and right hand features. When this branch can accurately segment the left and right hand regions, it means that the network can accurately separate the left and right hand features, which helps to extract the features of the left and right hands. In addition, this branch will also segment each joint area of ​​each hand, and the extracted features are more refined, which is helpful for hand posture recognition.

[0084] Left-right hand interaction feature extraction module: In the hand interaction feature extraction branch, the global features of the feature extraction module are input into the left-right hand interaction feature extraction module to further extract the global features of the interacting hands, namely the left-right hand interaction features, and improve the network's 3D reconstruction capabilities for the interacting hands. The first layer of this branch is a convolutional layer, which is used to convert global features into interaction features, followed by multiple superimposed upsampling and convolutional layers, each upsampling is 2 times, and the interaction features are increased to 128×128 size.

[0085] S103: Using a feature fusion module, the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features are fused to obtain fused features.

[0086] This embodiment performs feature fusion on various hand features extracted by the above-mentioned fine-grained modules. The specific fusion process can refer to the following steps:

[0087] Step 21: Fuse the left and right hand positioning features and the left and right hand joint area segmentation features to obtain the left and right hand features.

[0088] Specifically, the left and right hand features separated in the left and right hand positioning module are upsampled 2 times to 128×128, and the left and right hand features separated in the left and right hand joint area segmentation module are downsampled 2 times to 128×128, and then the features are fused to generate the left and right hand features.

[0089] Step 22: Determine the number and position of hands according to the left and right hand positioning features, and determine the fusion coefficient according to the number and position of hands; based on the fusion coefficient, perform feature fusion on the left and right hand features and the left and right hand interaction features to obtain fusion features.

[0090] Specifically, the left and right hand features will be fused with the left and right hand interaction features again. Here, the strength coefficient (i.e., fusion coefficient) λ of the feature fusion is converted according to the distance between the two hands. Specifically, the hand position can be determined according to the left and right hand positioning features, and the number of hands and the distance between the two hands can be determined according to the hand position, thereby generating a fusion coefficient. When there is only one hand in the image, or the distance between the two hands is too far, exceeding a certain threshold, there is no interaction between the two hands, and the strength coefficient λ is 0, that is, the two-hand interaction feature is not used; when the distance between the two hands is closer, there is interaction between the two hands, and the deeper the interaction between the two hands, the larger the fusion strength coefficient λ, and the more interactive features the features after fusion depend on.

[0091] S104: Based on the fusion features, the left and right hand MANO coefficients and camera parameters are regressed using the left and right hand MANO coefficient regression module, and the hand is reconstructed using the regressed left and right hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand.

[0092] Specifically, the fusion features obtained by the feature fusion module are input into the left and right hand MANO coefficient regression module to regress the left and right hand MANO coefficients (hand posture coefficients and hand shape coefficients) and camera parameters (rotation parameter r and translation parameter t). The regressed left and right hand MANO coefficients and camera parameters are used for hand reconstruction. The left and right hand MANO coefficient regression module is used to describe the posture and shape of the hand, where the hand posture coefficient is used to control the angle of the hand joint, and the hand shape coefficient is used to adjust the overall shape of the hand, such as the length of the fingers and the width of the palm.

[0093] Specifically, the hand reconstruction process based on MANO can refer to the following formula:

[0094] .

[0095] in is the average hand of MANO, is a hybrid model of the hand shape, is a hybrid model of hand posture, is the shape factor, is the attitude coefficient, where All are known quantities, and only the parameters need to be obtained The hand reconstruction can be performed. Subsequently, the reconstructed hand is rotated and translated according to the rotation parameter r and the translation parameter t to make it fit the hand position in the image, thus completing the entire hand reconstruction process.

[0096] S105: Training a hand reconstruction model according to the reconstructed hand and the hand in the hand image to obtain a trained hand reconstruction model.

[0097] It should be noted that the hand reconstruction model in this embodiment is composed of a feature extraction module, a left and right hand positioning module, a left and right hand joint region segmentation module, a left and right hand interaction feature extraction module, a feature fusion module and a left and right hand MANO coefficient regression module. Figure 3 .

[0098] Further, in order to improve the accuracy of model training, the above-mentioned training of the hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model may include the following steps:

[0099] Step 31: Perform left and right hand center point regression based on the left and right hand positioning features to obtain the regression center point, and calculate the hand center point loss value based on the regression center point and the center point in the hand image.

[0100] The hand center point regression is based on the left and right hand positioning features and the Heatmap (generating a Gaussian heat map to represent the position information of each key point). The loss function of the hand center point loss value is calculated as follows:

[0101] .

[0102] in, Indicates the loss of the hand center point; represents the weight of the loss, and They represent the true Heatmap label corresponding to the center point in the hand image and the predicted label generated based on the center point of the hand; examples of Heatmap labels overlaid on images are as follows Figure 3 Image schematic of left and right hand positioning.

[0103] Step 32: Calculate the hand joint loss value based on the left and right hand joint area segmentation features and the left and right hand joint area segmentation features in the hand image.

[0104] .

[0105] in, Indicates loss of hand joints; represents the weight of the loss, and They respectively represent the hand segmentation mask labels corresponding to the left and right hand joint area segmentation features in the hand image and the predicted labels based on the left and right hand joint area segmentation features.

[0106] Step 33: Calculate the hand posture coefficient loss value and the hand shape coefficient loss value based on the regressed left and right hand MANO coefficients and the left and right hand MANO coefficients in the hand image.

[0107] The hand posture coefficient loss value and the hand shape coefficient loss value both belong to the MANO coefficient loss. The MANO coefficients of the regressed left and right hands (hand posture coefficients Hand shape factor ) and the L2 distance (Euclidean distance) between the MANO coefficients of the left and right hands in the corresponding labels of the hand images. You can refer to the following formula:

[0108] ;

[0109] .

[0110] in, represents the hand posture coefficient loss; represents the weight of the loss; and They represent the hand posture coefficient in the label corresponding to the hand image and the regressed hand posture coefficient respectively. represents the hand shape factor loss; indicates the weight of the loss; and They represent the hand shape coefficient in the label corresponding to the hand image and the regressed hand shape coefficient respectively.

[0111] Step 34: Extract the 3D key point coordinates and 2D key point coordinates of the left and right hands from the reconstructed hand.

[0112] The number of key points is not limited in this embodiment. For example, the key points can refer to Figure 4 .

[0113] Step 35: Calculate the 3D key point loss value based on the 3D key point coordinates and the 3D key point coordinates of the hand in the hand image.

[0114] In this embodiment, the 3D key point loss is calculated based on 21 key points, such as extracting the coordinates of the 21 3D key points of the left and right hands from the reconstructed 3D hand. , corresponding to the 3D key point coordinates in the label of the hand image Calculate 3D keypoint loss , the specific formula is as follows:

[0115] ;

[0116] in, Represents the weight of the loss function.

[0117] Step 36: Calculate the two-dimensional key point loss value based on the two-dimensional key point coordinates and the two-dimensional key point coordinates of the hand in the hand image.

[0118] Specifically, regarding the 2D key point loss value, combined with the regressed camera parameters, the 3D key point coordinates are projected into the 2D image space to obtain the 2D key point coordinates of the hand , corresponding to the 2D key point coordinates in the label of the hand image Compute 2D keypoint loss , the specific formula is as follows:

[0119] .

[0120] in, Represents the weight of the loss function.

[0121] It needs to be explained that and It can not only supervise the regressed left-hand and right-hand MANO coefficients, but also assist the network in regressing the correct camera parameters.

[0122] Step 37: Calculate the reconstructed bone length based on the reconstructed hand, and calculate the reconstructed bone length with the hand bone length in the hand image to obtain the bone length loss value.

[0123] Specifically, regarding the hand bone length loss, the hand model bone length reconstructed by the network is calculated Bone length labels corresponding to the hand images The error between them gives the bone length loss , the specific formula is as follows:

[0124] .

[0125] in, Represents the weight of the loss function.

[0126] Step 38: The total loss value is calculated based on the hand center point loss value, hand joint loss value, hand posture coefficient loss value, hand shape coefficient loss value, three-dimensional key point loss value, two-dimensional key point loss value and bone length loss value.

[0127] The final total loss value is the sum of all the above loss values, that is:

[0128] .

[0129] Step 39: Perform negative feedback training on the hand reconstruction model according to the total loss value. When the training end condition is reached, the training is terminated to obtain a trained hand reconstruction model.

[0130] This embodiment does not limit the training end condition. For example, the training may be ended when the training times are reached; or the training may be ended when the total loss value meets a preset loss value.

[0131] The hand reconstruction model training method provided by the embodiment of the present invention is applied. The feature extraction module is used to extract features of the hand pictures in the training data set to obtain global features; the left and right hand positioning modules, the left and right hand joint area segmentation modules, and the left and right hand interaction feature extraction modules are used to extract the global features to obtain left and right hand positioning features, left and right hand joint area segmentation features, and left and right hand interaction features; the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features are fused by the feature fusion module to obtain fused features; based on the fused features, the left and right hand MANO coefficient regression modules are used to regress the left and right hand MANO coefficients and camera parameters, and the regressed left and right hand MANO coefficients and the regressed camera parameters are used to reconstruct the hand to obtain a reconstructed hand; the hand reconstruction model is trained according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning modules, the left and right hand joint area segmentation modules, the left and right hand interaction feature extraction module, the feature fusion module, and the left and right hand MANO coefficient regression module. The hand reconstruction model provided by the method can be used to reconstruct hands in any scene based on a single image. Since the left and right hand positioning module, the left and right hand joint area segmentation module, and the left and right hand interaction feature extraction module in the hand reconstruction model of the method can extract the features of a single hand, and interacting hands, the hand reconstruction can be accurately performed for a variety of scenes such as a single hand, interacting hands, object occlusion, self-occlusion between hands, motion blur, etc. The reconstructed hand posture and shape are accurate, and the relative depth position between the hands is accurately predicted. The hand reconstruction in truncated and occluded scenes is greatly improved compared with the current methods.

[0132] Please refer to Figure 5 , Figure 5 A flowchart of a hand reconstruction method provided by an embodiment of the present invention. The method may include:

[0133] S201: Obtain a picture to be reconstructed, where the picture to be reconstructed includes a hand;

[0134] S202: Inputting the image to be reconstructed into the trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the above-mentioned hand reconstruction model training method.

[0135] Specifically, the hand reconstruction model obtained by the above training method can be used to reconstruct 3D hands in any scene from any single RGB image. It can accurately reconstruct 3D hands for a variety of scenes such as single hand and double hand. The reconstructed hand posture and shape are accurate, and the relative depth position between the two hands is accurately predicted. The hand reconstruction for truncated and occluded scenes is greatly improved compared with the current methods. The 3D hand reconstruction results can be seen in Figure 6 , Figure 7 and Figure 8 , Figure 6 A schematic diagram of a result of interactive two-handed and one-handed 3D hand reconstruction provided by an embodiment of the present invention, Figure 7 A schematic diagram of the result of 3D hand reconstruction in a truncated and occluded scene provided by an embodiment of the present invention, Figure 8 A schematic diagram of the result of 3D hand reconstruction in a motion blurred scene provided by an embodiment of the present invention.

[0136] The hand reconstruction method provided by the embodiment of the present invention is applied, by obtaining a picture to be reconstructed, wherein the picture to be reconstructed includes a hand; the picture to be reconstructed is input into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method described above. The hand reconstruction model provided by this method can be used to reconstruct hands in any scene based on a single picture. Since the left and right hand positioning modules, the left and right hand joint area segmentation modules, and the left and right hand interaction feature extraction modules in the hand reconstruction model in this method can extract features of single and double hands, as well as interacting double hands, the hand reconstruction can be performed accurately for a variety of scenes such as single hand, interacting double hands, object occlusion, self-occlusion between hands, motion blur, etc. The reconstructed hand posture and shape are accurate, and the relative depth position prediction between the two hands is accurate. The reconstruction of hands in truncated and occluded scenes is greatly improved compared to the current methods.

[0137] The hand reconstruction model training device provided by an embodiment of the present invention is introduced below. The hand reconstruction model training device described below and the hand reconstruction model training method described above can be referred to each other.

[0138] Please refer to Fig. 9 , Fig. 9 A schematic diagram of the structure of a hand reconstruction model training device provided by an embodiment of the present invention may include:

[0139] A global feature extraction module 100 is used to extract features from hand images in a training data set using a feature extraction module to obtain global features;

[0140] A fine-grained feature extraction module 200 is used to extract the global features using the left and right hand positioning modules, the left and right hand joint region segmentation modules, and the left and right hand interaction feature extraction modules to obtain left and right hand positioning features, left and right hand joint region segmentation features, and left and right hand interaction features;

[0141] A fusion module 300 is used to fuse the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features using a feature fusion module to obtain a fusion feature;

[0142] A training reconstruction module 400 is used to regress the left-hand MANO coefficients and camera parameters using the left-hand MANO coefficient regression module based on the fusion features, and reconstruct the hand using the regressed left-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand;

[0143] The model training module 500 is used to train the hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module.

[0144] Based on the above embodiment, the fusion module 300 may include:

[0145] A first feature fusion unit is used to fuse the left and right hand positioning features and the left and right hand joint area segmentation features to obtain left and right hand features;

[0146] A fusion coefficient determination unit, used to determine the number and position of hands according to the left and right hand positioning features, and determine a fusion coefficient according to the number and position of hands;

[0147] The second feature fusion unit is used to perform feature fusion on the left and right hand features and the left and right hand interaction features according to the fusion coefficient to obtain the fused features.

[0148] Based on the above embodiment, the hand images in the training data set are images of the upper body area of ​​the human body containing hands.

[0149] Based on the above embodiment, the feature extraction module is a neural network with a Transformer structure.

[0150] Based on the above embodiment, the global feature extraction module 100 may include:

[0151] A three-dimensional virtual digital driving unit, which is used for redirecting and driving a three-dimensional virtual digital human using the limb and hand motion data actually collected, as well as the limb and hand motion data extracted from a public human motion data set;

[0152] A two-dimensional digital human video generation unit, used to randomly set camera parameters, light source parameters and background based on the three-dimensional virtual digital human, and render and generate a two-dimensional digital human video;

[0153] A coefficient calculation unit, used to obtain the three-dimensional key point coordinates of the hand of the three-dimensional virtual digital human, and calculate the MANO coefficients of the left and right hands, wherein the MANO coefficients of the left and right hands include the hand posture coefficient and the hand shape coefficient;

[0154] The hand synthesis data set acquisition unit is used to obtain the hand synthesis data set according to the two-dimensional digital human video, the hand posture coefficient and the hand shape coefficient.

[0155] Based on the above embodiment, the model training module 500 may include:

[0156] A first loss calculation unit, configured to perform left and right hand center point regression based on the left and right hand positioning features to obtain a regression center point, and calculate a hand center point loss value based on the regression center point and a center point in the hand image;

[0157] A second loss calculation unit, used for calculating a hand joint loss value according to the left and right hand joint region segmentation features and the left and right hand joint region segmentation features in the hand image;

[0158] A third loss calculation unit, configured to calculate a hand posture coefficient loss value and a hand shape coefficient loss value according to the regressed left and right hand MANO coefficients and the left and right hand MANO coefficients in the hand picture;

[0159] A coordinate extraction unit, used to extract the three-dimensional key point coordinates and the two-dimensional key point coordinates of the left and right hands from the reconstructed hand;

[0160] a fourth loss calculation unit, configured to calculate a three-dimensional key point loss value according to the three-dimensional key point coordinates and the three-dimensional key point coordinates of the hand in the hand image;

[0161] a fifth loss calculation unit, configured to calculate a two-dimensional key point loss value according to the two-dimensional key point coordinates and the two-dimensional key point coordinates of the hand in the hand image;

[0162] a sixth loss calculation unit, configured to calculate a reconstructed bone length according to the reconstructed hand, and calculate the reconstructed bone length with the hand bone length in the hand image to obtain a bone length loss value;

[0163] A total loss calculation unit, used for calculating a total loss value according to the hand center point loss value, the hand joint loss value, the hand posture coefficient loss value, the hand shape coefficient loss value, the three-dimensional key point loss value, the two-dimensional key point loss value and the bone length loss value;

[0164] A training unit is used to perform negative feedback training on the hand reconstruction model according to the total loss value, and when the training end condition is reached, the training is ended to obtain the trained hand reconstruction model.

[0165] It should be noted that the order of the modules and units in the above-mentioned hand reconstruction model training device can be changed without affecting the logic.

[0166] The hand reconstruction model training device provided by the embodiment of the present invention is applied, through the global feature extraction module 100, to extract features from the hand images in the training data set using the feature extraction module to obtain global features; The fine-grained feature extraction module 200 is used to extract the global features using the left and right hand positioning modules, the left and right hand joint area segmentation modules, and the left and right hand interaction feature extraction modules respectively to obtain the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features; the fusion module 300 is used to fuse the left and right hand positioning features, the left and right hand joint area segmentation features, and the left and right hand interaction features using the feature fusion module to obtain the fusion features; the training and reconstruction module 400 is used to regress the left and right hand MANO coefficients and camera parameters based on the fusion features using the left and right hand MANO coefficient regression modules, and reconstruct the hand using the regressed left and right hand MANO coefficients and the regressed camera parameters to obtain the reconstructed hand; the model training module 500 is used to train the hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning modules, the left and right hand joint area segmentation modules, the left and right hand interaction feature extraction module, the feature fusion module, and the left and right hand MANO coefficient regression module. The hand reconstruction model trained by this device can be used to reconstruct hands in any scene based on a single image. Since the left and right hand positioning module, the left and right hand joint area segmentation module, and the left and right hand interaction feature extraction module in the hand reconstruction model can extract the features of a single hand, as well as interacting hands, hands can be accurately reconstructed for a variety of scenes such as a single hand, interacting hands, object occlusion, self-occlusion between hands, motion blur, etc. The reconstructed hand posture and shape are accurate, and the relative depth position between the hands is accurately predicted. The hand reconstruction in truncated and occluded scenes is greatly improved compared with the current methods.

[0167] The hand reconstruction device provided by an embodiment of the present invention is introduced below. The hand reconstruction device described below and the hand reconstruction method described above can be referred to each other.

[0168] Please refer to Fig.10 , Fig.10 A schematic diagram of the structure of a hand reconstruction device provided by an embodiment of the present invention may include:

[0169] The to-be-reconstructed picture acquisition module 600 is used to acquire the to-be-reconstructed picture, where the to-be-reconstructed picture includes a hand;

[0170] The reconstruction module 700 is used to input the image to be reconstructed into the trained hand reconstruction model to obtain the hand reconstruction result; the trained hand reconstruction model is a model trained using the above-mentioned hand reconstruction model training method.

[0171] It should be noted that the order of the modules and units in the above hand reconstruction device can be changed without affecting the logic.

[0172] The hand reconstruction device provided by the embodiment of the present invention is used to obtain the image to be reconstructed through the image acquisition module 600, which includes the hand; the reconstruction module 700 is used to input the image to be reconstructed into the trained hand reconstruction model to obtain the hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method mentioned above. The hand reconstruction model trained by this device can reconstruct the hand of any scene based on a single picture. Since the left and right hand positioning module, the left and right hand joint area segmentation module, and the left and right hand interaction feature extraction module in the hand reconstruction model can extract the features of single and double hands, as well as interactive double hands, the hand reconstruction can be performed accurately for a variety of scenes such as single hand, interactive double hands, object occlusion, self-occlusion between hands, motion blur, etc. The reconstructed hand posture and shape are accurate, and the relative depth position prediction between the two hands is accurate. The reconstruction of hands in truncated and occluded scenes is greatly improved compared with the current methods.

[0173] The electronic device provided by the embodiment of the present invention is introduced below. The electronic device described below and the method described above can refer to each other.

[0174] Please refer to Fig.11 , Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention may include:

[0175] A memory 10, used for storing computer programs;

[0176] The processor 20 is used to execute the computer program to implement the above method.

[0177] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0178] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0179] Use the feature extraction module to extract features from the hand images in the training data set to obtain global features;

[0180] The global features are extracted using a left and right hand positioning module, a left and right hand joint region segmentation module, and a left and right hand interaction feature extraction module to obtain left and right hand positioning features, left and right hand joint region segmentation features, and left and right hand interaction features;

[0181] Using a feature fusion module to fuse the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features to obtain a fused feature;

[0182] Based on the fusion features, the left-right-hand MANO coefficients and camera parameters are regressed using the left-right-hand MANO coefficient regression module, and the hand is reconstructed using the regressed left-right-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand;

[0183] The hand reconstruction model is trained according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module;

[0184] and / or,

[0185] Acquire a picture to be reconstructed, where the picture to be reconstructed contains a hand;

[0186] The image to be reconstructed is input into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method described above.

[0187] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0188] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0189] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0190] The communication interface 31 may be an interface of a communication module, and is used to connect to other devices or systems.

[0191] Of course, it should be noted that Fig.11 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Fig.11 More or fewer components than shown, or combinations of certain components.

[0192] The computer-readable storage medium provided in the embodiments of the present invention is introduced below. The computer-readable storage medium described below and the method described above can be referenced to each other.

[0193] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0194] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0195] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0196] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0197] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0198] The above is a detailed introduction to a hand reconstruction model training and hand reconstruction method, a hand reconstruction model training and hand reconstruction device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A hand reconstruction model training method, characterized in that: include: Use the feature extraction module to extract features from the hand images in the training data set to obtain global features; The global features are extracted using a left and right hand positioning module, a left and right hand joint region segmentation module, and a left and right hand interaction feature extraction module to obtain left and right hand positioning features, left and right hand joint region segmentation features, and left and right hand interaction features; Using a feature fusion module to fuse the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features to obtain a fused feature; Based on the fusion features, the left-right-hand MANO coefficients and camera parameters are regressed using the left-right-hand MANO coefficient regression module, and the hand is reconstructed using the regressed left-right-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand; The hand reconstruction model is trained according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module.

2. The hand reconstruction model training method according to claim 1, characterized in that: The feature fusion module is used to fuse the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features to obtain fused features, including: Fusing the left and right hand positioning features with the left and right hand joint area segmentation features to obtain left and right hand features; Determining the number and positions of hands according to the left and right hand positioning features, and determining a fusion coefficient according to the number and positions of hands; According to the fusion coefficient, the left and right hand features and the left and right hand interaction features are fused to obtain the fused features.

3. The hand reconstruction model training method according to claim 1, characterized in that: The hand images in the training data set are images of the upper body region of the human body containing hands.

4. The hand reconstruction model training method according to claim 1, characterized in that: The feature extraction module is a neural network with a Transformer structure.

5. The hand reconstruction model training method according to claim 4, characterized in that: The process of obtaining the training data set includes: The training dataset is constructed according to the acquired public hand reconstruction dataset, the actually collected hand motion dataset and the hand synthesis dataset.

6. The hand reconstruction model training method according to claim 1, characterized in that: The process of acquiring the hand synthesis dataset includes: Use the actual collected limb and hand motion data, as well as the limb and hand motion data extracted from the public human motion data set to redirect and drive the 3D virtual digital human; Based on the three-dimensional virtual digital human, randomly set camera parameters, light source parameters and background, and render to generate a two-dimensional digital human video; Obtaining the three-dimensional key point coordinates of the hands of the three-dimensional virtual digital human, and calculating the MANO coefficients of the left and right hands, wherein the MANO coefficients of the left and right hands include the hand posture coefficient and the hand shape coefficient; The hand synthesis data set is obtained according to the two-dimensional digital human video, the hand posture coefficient and the hand shape coefficient.

7. The hand reconstruction model training method according to any one of claims 1 to 6, characterized in that: Training a hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model includes: Regressing the center points of the left and right hands based on the left and right hand positioning features to obtain a regression center point, and calculating a hand center point loss value based on the regression center point and the center point in the hand image; Calculating a hand joint loss value according to the left and right hand joint region segmentation features and the left and right hand joint region segmentation features in the hand image; A hand posture coefficient loss value and a hand shape coefficient loss value are calculated based on the regressed left and right hand MANO coefficients and the left and right hand MANO coefficients in the hand picture; Extracting the three-dimensional key point coordinates and the two-dimensional key point coordinates of the left and right hands from the reconstructed hand; Calculating a 3D key point loss value according to the 3D key point coordinates and the 3D key point coordinates of the hand in the hand image; Calculating a two-dimensional key point loss value according to the two-dimensional key point coordinates and the two-dimensional key point coordinates of the hand in the hand image; Calculating a reconstructed bone length according to the reconstructed hand, and calculating the reconstructed bone length with the hand bone length in the hand image to obtain a bone length loss value; A total loss value is calculated according to the hand center point loss value, the hand joint loss value, the hand posture coefficient loss value, the hand shape coefficient loss value, the three-dimensional key point loss value, the two-dimensional key point loss value and the bone length loss value; The hand reconstruction model is subjected to negative feedback training according to the total loss value, and when a training end condition is reached, the training is ended to obtain the trained hand reconstruction model.

8. A hand reconstruction method, characterized in that: include: Acquire a picture to be reconstructed, where the picture to be reconstructed contains a hand; The image to be reconstructed is input into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method described in any one of claims 1 to 7.

9. A hand reconstruction model training device, characterized in that: include: A global feature extraction module is used to extract features from hand images in a training data set using the feature extraction module to obtain global features; A fine-grained feature extraction module is used to extract the global features using the left and right hand positioning modules, the left and right hand joint area segmentation modules, and the left and right hand interaction feature extraction modules to obtain left and right hand positioning features, left and right hand joint area segmentation features, and left and right hand interaction features; A fusion module, used for fusing the left and right hand positioning features, the left and right hand joint area segmentation features and the left and right hand interaction features using a feature fusion module to obtain a fusion feature; A training reconstruction module is used to regress the left-hand MANO coefficients and camera parameters using the left-hand MANO coefficient regression module based on the fusion features, and reconstruct the hand using the regressed left-hand MANO coefficients and the regressed camera parameters to obtain a reconstructed hand; A model training module is used to train a hand reconstruction model according to the reconstructed hand and the hand in the hand picture to obtain the trained hand reconstruction model; the hand reconstruction model is composed of the feature extraction module, the left and right hand positioning module, the left and right hand joint area segmentation module, the left and right hand interaction feature extraction module, the feature fusion module and the left and right hand MANO coefficient regression module.

10. A hand reconstruction device, characterized in that: include: A module for acquiring a picture to be reconstructed, used for acquiring a picture to be reconstructed, wherein the picture to be reconstructed includes a hand; A reconstruction module is used to input the image to be reconstructed into a trained hand reconstruction model to obtain a hand reconstruction result; the trained hand reconstruction model is a model trained using the hand reconstruction model training method described in any one of claims 1 to 7.

Images