Three-dimensional human hand geometric reconstruction method and system

Abstract

The invention provides a three-dimensional human hand geometric reconstruction method and system. The method comprises the steps that a camera is used for shooting a video containing a static human hand and analyzing the video into an image sequence; inputting the image sequence and the camera internal reference into a trained neural network, and outputting human hand parameters by the neural network; inputting the human hand parameters into the parameterized human hand template to obtain grids and three-dimensional skeleton nodes of the human hand; and three-dimensional human hand geometric reconstruction is carried out based on the human hand grids and the three-dimensional skeleton nodes. The human hand geometric reconstruction method is simple and low in use threshold; by adopting multi-view data capture, the reconstructed hand geometric structure is highly vivid and is suitable for a high-precision demand scene; the precision and the cost can be flexibly adjusted according to the video shooting duration, and the expansibility is high.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional modeling technology in generative artificial intelligence and computer graphics, and specifically to a method and system for reconstructing three-dimensional human hand geometry. Background Art

[0002] Artificial Intelligence Generated Content (AIGC), one of the core drivers of current technological development, is profoundly transforming the landscape of digital content creation and becoming an indispensable element of new-quality productivity. Among AIGC's numerous applications, 3D digital human generation technology is particularly noteworthy. Its highly realistic digital human modeling demonstrates enormous application potential and market demand in diverse fields, including industrial manufacturing, gaming and film, entertainment and education, and healthcare. The realistic visual effects not only significantly enhance the credibility of content but also provide users with an unprecedented interactive experience, further promoting the integration of the virtual and real worlds.

[0003] The human hand, as the most flexible organ in the human body with the strongest information transmission ability, plays a vital role in scenarios such as virtual reality, augmented reality, and human-computer interaction. In games, realistic virtual gestures can greatly enhance the player's sense of immersion, making the gaming experience more vivid and realistic. However, traditional methods for reconstructing the three-dimensional structure of the human hand mainly rely on classical computer graphics technology. These methods require extremely high professional skills of art designers and expensive professional equipment, resulting in a time-consuming and labor-intensive process of constructing the three-dimensional geometric structure of the human hand. The subsequent optimization and rendering also require a lot of resource support, making it difficult to achieve large-scale application. In addition, when generating complex gestures, traditional methods often require designers to perform a lot of manual adjustments and optimizations, further increasing the investment in time and labor costs.

[0004] With the rapid development of AIGC technology, deep learning-based methods for generating 3D hand geometry have emerged. These methods take a single image of a hand as input and generate the corresponding 3D hand geometry through an automated or semi-automated process. Because images are the most accessible form of media and can be captured using only a standard camera or mobile phone, these methods are highly practical and widely available. Compared to traditional methods, AIGC-based methods for generating 3D hand geometry significantly reduce costs and improve efficiency, making the production of 3D hand geometry more convenient and efficient.

[0005] Despite this, 3D reconstruction of the human hand still faces numerous challenges. The human hand contains 27 bones and multiple joints, each with a significant degree of freedom, resulting in complex and varied gestures. This gives gestures rich expressive power but also increases the probability of self-occlusion, which in turn affects the accuracy of hand geometry reconstruction. To alleviate this problem, researchers have attempted to use multi-view images as input. However, the acquisition of multi-view images requires specialized equipment and facilities, and the precise calibration of each camera's intrinsic and extrinsic parameters in advance. However, existing multi-view image-based methods struggle to find an ideal balance between the number of viewpoints (i.e., the number of cameras) and reconstruction quality. Sparse view input can result in the omission of self-occluded or unseen areas, while a large number of dense view inputs requires the long-term investment and maintenance of extensive hardware equipment and resources, which undoubtedly increases the complexity and cost of producing the hand's geometric structure.

[0006] Therefore, how to ensure the quality of reconstruction while reducing dependence on professional equipment and a large amount of hardware resources and improving the efficiency and accuracy of generating three-dimensional human hand geometric structures has become a key issue that needs to be urgently addressed in the current AIGC field. Summary of the Invention

[0007] In order to solve the above problems, this application proposes a three-dimensional human hand geometry reconstruction method, which uses a video containing a static human hand captured by an ordinary camera device or a mobile phone as input, and uses a self-supervised method to train a three-dimensional human hand geometry reconstruction model, so that it can self-supervise and learn the geometric structure of the human hand and camera extrinsics from a continuous image sequence containing the human hand, thereby achieving efficient three-dimensional human hand geometry reconstruction without relying on professional equipment and a large amount of hardware resources.

[0008] In a first aspect of the present application, a method for reconstructing three-dimensional human hand geometry is provided, comprising: using a camera to shoot a video containing a static human hand and parsing it into an image sequence; inputting the image sequence and the camera intrinsic parameters into a trained neural network, and having the neural network output human hand parameters; inputting the human hand parameters into a parameterized human hand template to obtain a mesh and three-dimensional skeletal nodes of the human hand; and performing three-dimensional human hand geometry reconstruction based on the mesh and three-dimensional skeletal nodes of the human hand.

[0009] In some embodiments, the training of the neural network may include: a feature inversion stage: for an image sequence and corresponding camera intrinsic parameters serving as a training sample, obtaining the hand parameters and camera extrinsic parameters corresponding to the training sample through feature inversion to calibrate the training sample; a training stage: training the neural network based on the calibrated training sample so that it can predict the hand parameters and camera extrinsic parameters based on the input image sequence and camera intrinsic parameters.

[0010] In some embodiments, the feature inversion stage may include the following steps: S1: inputting the image sequence as training samples and its corresponding camera intrinsic parameters into the neural network to generate initial hand parameters and camera extrinsic parameters corresponding to each image; S2: inputting the hand parameters into the parameterized hand template to generate the current hand mesh and three-dimensional bone nodes; S3: using the mesh and three-dimensional bone nodes and the camera extrinsic parameters corresponding to each image to perform two-dimensional rasterization projection to obtain the current hand mask and two-dimensional bone nodes of each image; S4: inputting the image sequence into a pre-trained hand segmentation / gesture recognition module to generate the hand mask and two-dimensional bone nodes corresponding to each image as the true value of the training sample; S5: calculating the loss function based on the hand mask and two-dimensional bone nodes generated from steps S3 and S4; S6: calculating the gradient based on the loss function and inversely updating the hand parameters and camera extrinsic parameters until the loss function converges or the number of iterations reaches a preset value.

[0011] In some embodiments, the loss function can be calculated based on one or more of the following losses: position loss of mesh vertices, position loss of skeleton nodes in the world coordinate system, mask loss, position loss of skeleton nodes in the image coordinate system, image extrinsic parameter neighborhood loss, and regularization loss.

[0012] In some embodiments, the camera extrinsic parameters may include a rotation matrix and a displacement vector, and step S6 includes: reversely updating the displacement vector corresponding to each image based on the position loss of the bone nodes in the image coordinate system, the image extrinsic parameter neighborhood loss, and the gradient calculated by the regularization loss, while keeping the other parameters unchanged; reversely updating the rotation matrix and displacement vector corresponding to each image based on the position loss of the mesh vertices, the position loss of the bone nodes in the world coordinate system, the mask loss, the position loss of the bone nodes in the image coordinate system, the image extrinsic parameter neighborhood loss, and the gradient calculated by the regularization loss, while keeping the other parameters unchanged; reversely updating the human hand parameters and the camera extrinsic parameters corresponding to each image based on the mask loss, the position loss of the bone nodes in the image coordinate system, the image extrinsic parameter neighborhood loss, and the gradient calculated by the regularization loss.

[0013] In some embodiments, the parameterized human hand template may adopt a MANO human hand template, and the human hand parameters include shape parameters and posture parameters.

[0014] In some embodiments, the two-dimensional rasterization projection may adopt an affine projection method.

[0015] In some embodiments, the mask loss can be used to limit the distance between the rasterized projection of the three-dimensional mesh of the reconstructed human hand on the image plane and the human hand mask of the original image, and the position loss of the bone nodes in the image coordinate system can be used to limit the distance between the rasterized projection of the bone nodes of the reconstructed human hand on the image plane and the bone nodes of the human hand in the original image.

[0016] In another aspect of the present application, a three-dimensional human hand geometry reconstruction system is provided, which includes: a neural network that outputs human hand parameters based on an input image sequence and camera intrinsic parameters; a parameterized human hand template that obtains a human hand mesh and three-dimensional skeletal nodes based on the human hand parameters from the neural network; and a reconstruction module that performs three-dimensional human hand geometry reconstruction based on the human hand mesh and three-dimensional skeletal nodes.

[0017] In another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the method according to the first aspect of the present application, or run the system according to the second aspect of the present application.

[0018] In an embodiment of the present application, the input data is a video or a continuous image sequence, i.e., a dense perspective image. The hand geometry is regressed using self-supervised learning, and the hand posture and shape are directly inverted to obtain the hand posture and shape. The input data is then automatically labeled, and the model is trained to learn the human hand parameters from the image sequence.

[0019] In addition, the three-dimensional human hand geometry reconstruction method proposed in this application only requires ordinary camera equipment or mobile phones to complete data collection, without the need for any wearable devices. While ensuring the quality of reconstruction, it reduces dependence on professional equipment and a large amount of hardware resources, and improves the efficiency and accuracy of generating three-dimensional human hand geometry. Similar to the method based on single-view images, this method has an extremely low threshold for use, which greatly simplifies the use and deployment process. It can capture human hand movements from different perspectives, and the reconstructed human hand geometry is highly realistic. Its accuracy is comparable to that of the method based on dense perspective images, and it can meet high-precision demand scenarios. In addition, the algorithm of this application has a certain degree of interpretability. By presetting the initialization parameters of the model, it accelerates the convergence of the inversion and reduces the consumption of computing resources. By controlling the length of the video shooting, it takes into account both accuracy and cost, and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a three-dimensional human hand geometry reconstruction method according to an embodiment of the present application is shown.

[0021] Figure 2 FIG4 is a diagram showing a training process of a neural network for three-dimensional human hand geometry reconstruction according to an embodiment of the present application.

[0022] Figure 3 A flow chart showing the steps of the feature inversion stage according to one embodiment of the present application is shown.

[0023] Figure 4 A diagram showing experimental results of the InterHand2.6M (Interacting Hands, an interactive gesture dataset containing approximately 2.6 million labeled samples) dataset according to an embodiment of the present application is shown.

[0024] Figure 5 The figure shows the effect of the self-collected video data experiment according to one embodiment of the present application.

[0025] Figure 6 A diagram showing the experimental effect of self-collected multi-view image data according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application more clear, the present application will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be noted that the embodiments given in this application are only for illustration and do not limit the scope of protection of this application.

[0027] Figure 1 The flowchart of the method for reconstructing the three-dimensional geometry of a human hand according to one embodiment of the present application is shown. The method mainly includes the following steps:

[0028] First, a video containing static human hands is shot with a camera and parsed into an image sequence, and the corresponding camera intrinsic parameters are calibrated for the image sequence.

[0029] Specifically, the hand is held stationary in a fixed position, and the camera is controlled to move in a slow, smooth trajectory, continuously capturing videos of the hand from different perspectives. This process simulates the effect of dense multi-camera acquisition, replacing a multi-camera array with multi-frame motion from a single camera to reduce hardware complexity. Preferably, the captured video stream is parsed into a continuous sequence of image frames, with each frame ensuring that the hand is clearly visible and unobstructed. Optionally, operations such as denoising and contrast enhancement are performed on the image to improve the robustness of subsequent processing.

[0030] The camera intrinsic parameters (i.e., the internal parameters of the camera) corresponding to the image sequence can usually be obtained by the following methods:

[0031] (1) Directly use factory parameters: If the camera has provided accurate internal parameter data when it leaves the factory, it can be directly used to simplify the process.

[0032] (2) Zhang Zhengyou calibration method: To improve the calibration accuracy, the Zhang Zhengyou calibration method can be used. This method takes multiple checkerboard images at different angles and derives the camera's intrinsic parameters based on corner detection and homography matrix calculation.

[0033] The intrinsic parameters of a camera are used to describe its internal optical and geometric characteristics, and mainly include the following parameters: Focal Length: determines the image scaling; Principal Point: the intersection of the optical axis and the imaging plane, usually located at the center of the image; Distortion Coefficients: corrects lens distortion (such as radial distortion and tangential distortion).

[0034] In one embodiment of the present application, since the distortion of ordinary cameras is usually small and mainly affects the edges of the image, and the human hand is mostly located in the center of the image when shooting, the effect of distortion on the overall calibration result can be ignored. Based on this, the intrinsic parameter matrix of the camera is It can be simplified to a matrix form containing only the focal length and principal point coordinates:

[0035]

[0036] in and are the horizontal and vertical focal lengths of the camera, and Represents the coordinates of the camera's principal point.

[0037] Continue to refer Figure 1 After obtaining the image sequence and camera intrinsic parameters, they are input into a trained neural network, which outputs the hand parameters; then the hand parameters are input into a parameterized hand template to obtain the hand mesh and three-dimensional bone nodes; then, based on the hand mesh and three-dimensional bone nodes, three-dimensional hand geometry reconstruction can be achieved.

[0038] The hand parameters output by the neural network depend on the type of parameterized hand template subsequently used. Common hand templates include the Nimble template, the Dart template, and the MANO template. The Nimble template combines muscles and bones, and has a larger number of meshes (approximately 20 times that of the Mano template), resulting in high overhead in practical applications. The MANO template is a parameterized hand template based on a statistical shape model and joint drive, efficiently describing the 3D shape and posture of the hand in a low-dimensional parameter space. These parameterized hand templates are typically used to obtain the hand mesh and 3D skeletal nodes based on the input hand parameters. Different types of hand templates require different hand parameters. Dart adds wrist and accessories to the MANO template. For ease of description, the MANO template is used in the embodiments of this application, but this is for illustrative purposes only and is not intended to be limiting.

[0039] The hand parameters corresponding to the MANO template include shape parameters and posture parameters. The shape parameters are recorded as , used to control the overall shape and size of the human hand. The first 10 dimensions control the statistical changes in the shape of the human hand (such as finger thickness, palm width, etc.), and the 11th dimension is a global scaling factor used to adjust the overall size of the human hand. The posture parameters are recorded as , which contains the rotation angles of 15 joints (expressed in the form of axis-angle vectors), and is used to describe the relative rotation of each joint of the human hand, and thus determine the three-dimensional posture and gesture of the human hand.

[0040] Because the MANO template is used, the neural network is pre-trained to output the shape parameters of the human hand based on the input image sequence and the camera internal parameters. and posture parameters .like Figure 1 As shown, the output from the neural network is passed as input to a parameterized human hand template (such as a MANO template). The MANO template is based on the input shape parameters and posture parameters Calculate the 3D mesh and 3D bone nodes (joint3d) of the hand. Specifically, input parameters β and θ to the linear blend skinning (LBS) function of the MANO template, calculate the deformation of the mesh vertices, and output the 3D mesh vertex coordinates. , where 778 is the number of vertices and 3 is the three-dimensional space coordinate (x, y, z). The joint regressor based on the MANO template maps the deformed mesh vertices to the bone node coordinates. Output bone node coordinates , where 21 is the number of skeletal nodes (such as fingertips, joints, and other key points) and 3 is the 3D spatial coordinate. Then, based on the hand mesh and 3D skeletal nodes obtained above, 3D hand geometry reconstruction can be achieved.

[0041] The three-dimensional human hand geometry reconstruction method of the present application only requires ordinary camera equipment or mobile phones to complete data acquisition, without the need for any wearable devices. The method uses a self-supervised method to train the neural network, so that it can predict the human hand parameters corresponding to the image sequence based only on the input image sequence and the camera's internal parameters, without the need to calibrate the external parameters of the camera when shooting the image sequence. Therefore, while ensuring the quality of reconstruction, the method reduces dependence on professional equipment and a large amount of hardware resources, and improves the efficiency and accuracy of the generation of three-dimensional human hand geometry. Similar to the method based on single-view images, the method has an extremely low threshold for use, greatly simplifies the use and deployment process, and can capture human hand movements from different perspectives. The reconstructed human hand geometry is highly realistic, and its accuracy is comparable to that of the method based on dense perspective images, which can meet high-precision demand scenarios.

[0042] The following combination Figure 2Introduce the self-supervised training process of neural networks. Figure 2 As shown, the neural network training process mainly includes a feature inversion phase and a training phase. The feature inversion phase involves calibrating the image sequence and corresponding camera intrinsic parameters, and obtaining the corresponding hand parameters and camera extrinsic parameters of the training sample through feature inversion. The training phase involves training the neural network based on the calibrated training samples, enabling it to predict the hand parameters and camera extrinsic parameters based on the input image sequence and camera intrinsic parameters.

[0043] Specifically, the camera extrinsics (i.e., the external parameters of the camera) describe the position and orientation of the camera in three-dimensional space, mainly including the rotation matrix R (rotation matrix) and the translation vector T (translation vector). and displacement vector It is expressed as follows:

[0044] ,

[0045] Usually, the rotation matrix must strictly satisfy the orthogonality condition (R ⊤ R=I and det(R)=1) to ensure that the rotation transformation it describes is a rigid body transformation. In the process of training a neural network, if the rotation matrix is directly regressed, numerical errors may accumulate during gradient optimization, destroying the orthogonality constraint, and thus making the training unstable or the results invalid. In some embodiments of the present application, an axis-angle vector A is used instead of the rotation matrix R, and the Rodriguez rotation formula is used to convert the axis-angle vector A into the rotation matrix R. The axis-angle vector It is expressed as follows:

[0046]

[0047] Figure 3 FIG1 shows a flow chart of the steps of the feature inversion stage according to an embodiment of the present application. Specifically, Figure 3 As shown in Figure 2, the feature inversion stage mainly includes the following steps:

[0048] In step S1, the image sequence and its corresponding camera intrinsic parameters as training samples are input into the neural network to obtain the initial hand parameters and the camera extrinsic parameters corresponding to each image. This process can be understood as the initialization step of the feature inversion stage. The neural network is designed to predict the hand parameters corresponding to the image sequence and the camera extrinsic parameters corresponding to each image based on the input image sequence and its corresponding camera intrinsic parameters. However, in the feature inversion stage, the neural network has not yet been trained. Here, the hand parameters of the image sequence and the camera extrinsic parameters corresponding to each image, which are roughly predicted by the untrained neural network, are only used as the initialization parameters of the feature inversion stage. In the embodiments of the present application, the neural network can be any neural network architecture that can perform deep learning. In some other embodiments, the neural network can use an existing single-image-based hand reconstruction model and cascade two fully connected layers on its basis to predict the hand parameters and the camera extrinsic parameters corresponding to each image.

[0049] In step S2, the hand parameters are input into the parameterized hand template for calculation to obtain the current hand mesh and 3D skeleton nodes. The specific process can be referred to above in conjunction with the MANO template and will not be repeated here.

[0050] In step S3: the hand mesh and 3D skeleton nodes output by the MANO template and the camera extrinsics corresponding to each image are used to perform 2D rasterization projection to obtain the current hand mask and 2D skeleton nodes of each image. Figure 2 As shown, the 3D mesh and 3D skeleton nodes of the hand and the external parameters corresponding to each image are input into the projection module to generate a 2D rasterized projection, namely the hand mask and 2D skeleton nodes. , bone nodes , camera internal parameters And the camera extrinsics corresponding to each image and Passed as input to the projection module, and outputs the 2D rasterized coordinates of the mesh and 3D bone nodes in the image plane In some embodiments, the projection module may use an affine projection method when performing two-dimensional rasterization projection. The specific calculation formula is as follows:

[0051]

[0052] ,

[0053] in Represents the three-dimensional coordinates of the mesh vertex or bone node in the world coordinate system. Through external and internal references The transformed three-dimensional point coordinates (camera coordinate system), Px is the x coordinate (horizontal) in the camera coordinate system, Py is the y coordinate (vertical) in the camera coordinate system, Pz is the z coordinate (depth, that is, the distance from the point to the camera) in the camera coordinate system, the superscript Represents matrix transpose. and Represents the width and height of the image respectively.

[0054] In step S4, the image sequence in the training sample is input into the pre-trained human hand segmentation / gesture recognition module to generate the human hand mask and two-dimensional skeleton node corresponding to each image as the true value of the training sample in the feature inversion stage. Among them, the human hand segmentation module can select Segment-Anything Model (SAM) or other high-performance segmentation models (such as DeepLabV3+, U-Net) to perform pixel-level classification on the input image and generate the human hand mask ,in The gesture recognition module can use MediaPipeGHands or other lightweight gesture recognition models (such as OpenPose, HRNet), and can be based on the hand mask output by the segmentation model. Or directly perform key point detection on each image and output the skeleton nodes .in Represents the two-dimensional coordinates of the bone node in the image coordinate system.

[0055] In step S5, the loss function is calculated based on the hand mask and 2D skeleton nodes generated from steps S3 and S4. In the feature inversion stage, the current hand mask and 2D skeleton nodes for each image obtained in step S3 are used as the predicted values, while the hand mask and 2D skeleton nodes for each image generated in S4 are used as the true values to calculate the loss function.

[0056] In step S6, the gradient is calculated based on the loss function and the hand parameters and camera extrinsic parameters are updated inversely until the loss function converges or the number of iterations reaches a preset value. Thus, the hand parameters and camera extrinsic parameters corresponding to the image sequence are obtained through the above-mentioned feature inversion process. These are used as true values to calibrate the training samples in the subsequent neural network training. Thus, the training phase includes training the neural network based on the training samples calibrated with hand parameters and camera extrinsic parameters, enabling it to predict hand parameters and camera extrinsic parameters based on the input image sequence and camera intrinsic parameters. Thus, the feature inversion phase and the training phase jointly implement a self-supervised training process for the neural network.

[0057] In some embodiments, the loss function is calculated based on one or more of the following losses: mesh vertex position loss, skeleton node position loss in the world coordinate system, mask loss, skeleton node position loss in the image coordinate system, image extrinsic reference neighborhood loss, and regularization loss. Specifically:

[0058] (1) Position loss of mesh vertices

[0059] Ideally, the geometric structure of the reconstructed hand should be as close to the true value (i.e., the ground truth) as possible. However, due to the complexity and diversity of hand parameters, the numerical ranges of different components (such as shape and posture parameters) and their influence on the geometric structure of the parameterized hand template vary significantly. Directly using the output of the neural network as a constraint in the loss function may cause the hand template to over-focus on a particular hand shape or state during training, while neglecting other shapes or postures, thus affecting the comprehensiveness and accuracy of the reconstruction.

[0060] In the embodiment of the present application, in order to solve this problem, a loss function is introduced. , which is used to limit the predicted mesh vertices and the MANO mesh vertices with annotations Specifically, The formula is as follows:

[0061]

[0062] By introducing This ensures that the neural network gives equal weight to hands of all shapes and poses during training, preventing the model from falling into a local optimum. Furthermore, the use of L2 distance effectively suppresses the influence of noise and outliers, improving the stability and accuracy of the reconstruction results.

[0063] (2) Position loss of bone nodes in the world coordinate system

[0064] Preferably, in order to ensure that the reconstructed human hand has a realistic and reasonable bone structure, relying solely on the position loss of mesh vertices may not be able to fully capture the dynamic changes and mutual constraints of bone nodes. Therefore, in some embodiments of the present application, Skeleton nodes used to constrain predictions and annotated MANO skeleton nodes The second L2 distance between them ensures that the reconstructed human hand skeleton structure is highly consistent with the true value. The formula is as follows:

[0065]

[0066] (3) Mask loss

[0067] Combined with the above Figure 2 and Figure 3 In the self-supervised training process of 3D hand geometry reconstruction, the projection module is used to reconstruct the hand mask and skeleton nodes, and then the hand geometry structure is inverted. In order to further improve the accuracy and robustness of the model, As a key component, the loss function plays a vital role in ensuring the consistency between the reconstructed hand mask and the original image hand mask. In the embodiment of the present application, the hand mask and the skeleton nodes are reconstructed by the projection module to achieve the inversion of the hand geometry. In theory, the reconstructed hand mask should be close to the hand mask of the original image, so the Rasterized projection of a 3D mesh onto the image plane for constrained reconstruction of the human hand and the original image hand mask The first L1 distance between them is as follows:

[0068]

[0069] (4) Position loss of bone nodes in the image coordinate system

[0070] Preferably, in order to ensure that the reconstructed human hand has a real and reasonable two-dimensional bone structure, some embodiments of the present application also introduce Rasterized projection of the skeleton nodes on the image plane to constrain the reconstruction of the human hand and the original image human hand skeleton nodes The second L1 distance between them is as follows:

[0071]

[0072] (5) Image extrinsic reference neighborhood loss

[0073] Specifically, due to the slow motion of the camera, in some embodiments of the present application, a Used to limit the displacement vector corresponding to adjacent images and axis angle vector The L2 loss between , its formula is as follows:

[0074]

[0075] Wherein, N represents the number of frames in a video or image sequence, and i is a natural number indicating the i-th frame in the image sequence.

[0076] (6) Regularization loss

[0077] Preferably, in order to ensure smooth mesh vertices of the reconstructed human hand, in some embodiments of the present application, Used to constrain the vertex positions of the mesh To reduce noise, the formula is as follows:

[0078]

[0079] in The core idea of the Laplace filter function is to move each vertex to the average position of its neighboring vertices, thereby reducing the change of local curvature.

[0080] Preferably, by dynamically adjusting the weight coefficients of each loss function , , , , , and accumulate to generate the final loss function , achieving adaptive balance of various optimization objectives at different training stages, thereby improving the overall reconstruction effect. The formula is as follows:

[0081]

[0082] In some embodiments, in step S6 of the feature inversion stage, the gradient can be calculated based on the loss function and the hand parameters and camera extrinsic parameters can be updated in reverse order in the following manner:

[0083] 1) Based on 、 as well as The calculated gradient is used to reversely update the displacement vector corresponding to each image while keeping the other parameters unchanged

[0084] 2) Based on 、 、 as well as The calculated gradient is used to reversely update the camera extrinsic parameters corresponding to each image, while keeping the other parameters unchanged;

[0085] 3) Based on 、 as well as The calculated gradient is used to reversely update the hand parameters and the camera extrinsic parameters corresponding to each image.

[0086] The three-dimensional human hand geometry reconstruction method of this application adopts a "coarse-fine" two-stage optimization strategy, which not only utilizes the generalization ability of neural networks but also retains the physical constraints of feature inversion, ultimately achieving efficient and accurate three-dimensional human hand reconstruction.

[0087] To validate the effectiveness of the trained neural network, the inventors also conducted experiments on the InterHand2.6M.30fps dataset. "InterHand" refers to the dataset's focus on interactive hand gestures, encompassing both single-hand and two-handed interactions. "2.6M" refers to the dataset's inclusion of approximately 2.6 million frames of annotated image data. These images cover a wide range of hand gestures and interaction scenarios, including RGB images of both single-hand and two-handed interactions. "30fps" refers to the video sequences in the dataset, captured at 30 frames per second (fps). This ensures the dataset provides sufficient temporal resolution to capture detailed hand movements.

[0088] In the experiment, data containing only the right hand was selected as the training sample. During the training phase, the gradient stopped decreasing after 1024 epochs of displacement vector inversion (an epoch refers to a complete learning process of the model for the entire training dataset). The joint inversion was stopped after 308 epochs. Finally, the average position error of the mesh vertices was 1.7mm, and the average position error of the bone nodes was 0.6mm.

[0089] The 16 key images of the experimental results are as follows Figure 4 As shown, the first row is a sequence of images of static human hands, which is the original input image. The second row of green masks is the rasterized projection of the three-dimensional mesh of the real human hand on the image plane, which is generated by inputting the three-dimensional mesh and the internal and external parameters of the camera into the projection module. The third row of blue masks is the rasterized projection of the three-dimensional mesh of the reconstructed human hand on the image plane. Figure 4 The 4th to 6th lines and the 1st to 3rd lines represent the same gesture, but they cannot be displayed in one line, so they are divided into two lines. Figure 4 It can be seen that the hand parameters and camera extrinsics inverted according to the feature inversion step of this application are very accurate, the hand restoration degree is very high, and the reconstruction effect is very good.

[0090] In addition, the inventors also verified the effect on self-collected video data, a total of 1500 frames, such as Figure 5 As shown, Figure 5 The upper part is the rasterized projection effect of 9 frames sampled according to the duration. Figure 5 The lower half is a three-dimensional human hand geometric model reconstructed using the method of the present invention. It can be seen intuitively that the reconstructed human hand structure and shape have a high degree of restoration.

[0091] The inventors also verified the effect on multi-view images collected in professional venues, with a total of 62 effective viewing angles, such as Figure 6 As shown, Figure 6 The upper part is the rasterized projection effect of 12 viewing angles. Figure 6 The lower half is a three-dimensional human hand geometric model reconstructed using the method of the present invention.

[0092] In summary, the reconstruction effects of the method of the present invention under different data sources (single-view image sequence, self-collected video data, multi-view images) and different conditions have been verified, and the accuracy and robustness of the method of the present invention have been verified.

[0093] The 3D human hand geometry reconstruction method proposed in this application has the following beneficial effects:

[0094] (1) Low equipment requirements and easy to use: Data collection can be completed with only ordinary camera equipment or mobile phones, without the need for any wearable devices. Similar to the method based on single-view images, this method has a very low threshold for use, greatly simplifying the use and deployment process.

[0095] (2) Multi-view data, high reconstruction accuracy: It can capture human hand movements from different perspectives, providing comprehensive and rich data for the reconstruction process. Based on this, the reconstructed human hand geometry is highly realistic, and its accuracy is comparable to that of methods based on dense perspective images, which can meet high-precision requirements.

[0096] (3) The algorithm has strong interpretability and high computational efficiency: The algorithm has a certain degree of interpretability. By presetting the initialization parameters of the model, it can accelerate the convergence speed of the inversion process, thereby effectively reducing the consumption of computing resources.

[0097] (4) Balancing accuracy and cost, with strong scalability: By adjusting the length of the video, a flexible trade-off can be made between accuracy and cost, making this method highly scalable in different application scenarios.

[0098] In another embodiment of the present application, a 3D human hand geometry reconstruction system is also proposed. The reconstruction system includes: a neural network that outputs human hand parameters and camera extrinsic parameters based on the input image sequence and camera intrinsic parameters; a parameterized human hand template that obtains the mesh and 3D bone nodes of the human hand based on the input human hand parameters; and a reconstruction module that performs 3D human hand geometry reconstruction based on the mesh and 3D bone nodes of the human hand. The working mechanism of each module can be referred to in conjunction with the above. Figure 1-3 The introduction of , will not be repeated here.

[0099] The three-dimensional human hand geometry reconstruction method and system proposed in this application can be widely used in multiple industries and fields due to its functions such as efficient data acquisition, accurate geometry reconstruction, self-supervised learning and multi-view fusion, mainly including but not limited to high-precision hand modeling for real-time gesture interaction and virtual characters in the fields of virtual reality (VR) and augmented reality (AR); natural gesture recognition and operation of smart devices in the field of human-computer interaction; precise monitoring and evaluation of hand rehabilitation training in the field of medical rehabilitation; and capture and reconstruction of character hand motions in special effects production in the field of film and television production.

[0100] Due to the versatility and adaptability of the three-dimensional human hand geometry reconstruction method of this application, it can provide a stable and reliable solution for various scenarios that require high-precision human hand modeling and interaction, preventing the decline in user experience, interaction failure or increased production costs caused by insufficient data collection or low reconstruction accuracy, thereby improving the performance and user experience of the entire system.

[0101] In one embodiment of the present application, a computer-readable storage medium is also proposed, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above-mentioned three-dimensional human hand geometry reconstruction method, or run the above-mentioned system. The computer-readable storage medium may include a volatile memory, such as a random access memory, such as a static random access memory, a double data rate synchronous dynamic random access memory, etc.; the memory may also include a non-volatile memory, such as a flash memory. The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned method), computer instructions, etc. The above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0102] The processor is configured to execute a computer program stored on a computer-readable storage medium to implement the various steps of the methods described in the above embodiments. For details, please refer to the relevant descriptions in the previous method embodiments. The processor and memory may be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor may be coupled via a bus.

[0103] Those skilled in the art will appreciate that the embodiments of the present application can provide methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0104] Although the present application has been described through preferred embodiments, the present application is not limited to the embodiments described herein, and includes various changes and modifications that may be made without departing from the scope of the present application.

Claims

1. A method for reconstructing three-dimensional human hand geometry, comprising: Use a camera to shoot a video containing a static human hand and parse it into an image sequence; Inputting the image sequence and the camera intrinsic parameters into a trained neural network, and having the neural network output human hand parameters; Inputting the human hand parameters into a parameterized human hand template to obtain a mesh and three-dimensional skeleton nodes of the human hand; Three-dimensional human hand geometry reconstruction is performed based on the mesh and three-dimensional skeleton nodes of the human hand.

2. The method according to claim 1, wherein the training of the neural network comprises: Feature inversion stage: For the image sequence used as the training sample and the corresponding camera intrinsic parameters, the hand parameters and camera extrinsic parameters corresponding to the training sample are obtained through feature inversion to calibrate the training sample; Training phase: The neural network is trained based on the calibrated training samples so that it can predict the hand parameters and camera extrinsic parameters based on the input image sequence and camera intrinsic parameters.

3. The method according to claim 1, wherein the feature inversion stage comprises the following steps: S1: Input the image sequence as training samples and its corresponding camera intrinsic parameters into the neural network to generate the initial human hand parameters and the camera extrinsic parameters corresponding to each image; S2: Inputting the human hand parameters into a parameterized human hand template to generate a mesh and three-dimensional skeleton nodes of the current human hand; S3: Using the grid and the three-dimensional skeleton nodes and the camera extrinsic parameters corresponding to each image, perform two-dimensional rasterization projection to obtain the current human hand mask and two-dimensional skeleton nodes of each image; S4: Input the image sequence into a pre-trained human hand segmentation / gesture recognition module to generate a human hand mask and two-dimensional skeleton nodes corresponding to each image as the true value of the training sample; S5: Calculate the loss function based on the hand mask and two-dimensional skeleton nodes generated from steps S3 and S4; S6: Calculate the gradient based on the loss function and reversely update the hand parameters and camera extrinsic parameters until the loss function converges or the number of iterations reaches a preset value.

4. The method according to claim 3, wherein the loss function is calculated based on one or more of the following losses: position loss of mesh vertices, position loss of skeleton nodes in the world coordinate system, mask loss, position loss of skeleton nodes in the image coordinate system, image extrinsic parameter neighborhood loss, and regularization loss.

5. The method according to claim 4, wherein the camera extrinsic parameters include a rotation matrix and a displacement vector, and step S6 comprises: Reversely update the displacement vector corresponding to each image based on the position loss of the skeleton node in the image coordinate system, the image extrinsic reference neighborhood loss, and the gradient calculated by the regularization loss, while keeping the other parameters unchanged; Reversely update the rotation matrix and displacement vector corresponding to each image based on the gradient calculated based on the position loss of the mesh vertices, the position loss of the skeleton nodes in the world coordinate system, the mask loss, the position loss of the skeleton nodes in the image coordinate system, the image extrinsic parameter neighborhood loss, and the regularization loss, while keeping the other parameters unchanged; The hand parameters and the camera extrinsic parameters corresponding to each image are reversely updated based on the gradient calculated based on the mask loss, the position loss of the skeleton nodes in the image coordinate system, the image extrinsic parameter neighborhood loss, and the regularization loss.

6. The method according to claim 1, wherein the parameterized human hand template adopts a MANO human hand template, and the human hand parameters include shape parameters and posture parameters.

7. The method according to claim 3, wherein the two-dimensional rasterization projection adopts an affine projection method.

8. According to the method according to claim 4, the mask loss is used to limit the distance between the rasterized projection of the three-dimensional mesh of the reconstructed human hand on the image plane and the mask of the human hand in the original image, and the position loss of the bone nodes in the image coordinate system is used to limit the distance between the rasterized projection of the bone nodes of the reconstructed human hand on the image plane and the bone nodes of the human hand in the original image.

9. A three-dimensional human hand geometry reconstruction system, comprising: A neural network outputs hand parameters based on the input image sequence and camera intrinsic parameters; Parameterized human hand template, which obtains the mesh and 3D skeleton nodes of the hand according to the human hand parameters from the neural network; The reconstruction module reconstructs the three-dimensional geometry of the human hand based on the mesh and three-dimensional skeleton nodes of the human hand.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 8, or run the system according to claim 9.

Images