Hand pose estimation method and apparatus
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LION (SHENZHEN) ROBOT TECHNOLOGY CO LTD
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-21
Smart Images
Figure CN122435685A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of human-computer interaction and virtual reality technology, and in particular to a method and apparatus for estimating hand posture. Background Technology
[0002] Precise, robust, and low-latency motion capture of human hands is a key technology for realizing immersive virtual / augmented reality (VR / AR), driving dexterous robots, and creating efficient digital content. The following are current mainstream hand motion capture technologies and their inherent limitations: (1) Systems based on external optical markings: Such systems (such as Vicon and OptiTrack) achieve sub-millimeter-level "gold standard" accuracy by attaching reflective markings to the hand and tracking them with multiple high-speed infrared cameras. However, these systems are expensive, require specialized environments, and are prone to failure due to the loss of markings when fingers are crossed or gripped, thus strictly limiting their application to controlled laboratories.
[0003] (2) Label-free systems based on computer vision: These systems utilize deep learning algorithms to directly estimate hand pose from monocular or multi-view RGB or RGB-D images. Their advantages include low cost and no need for wearable devices. However, visual methods are inherently limited by occlusion problems; whether it's mutual occlusion between fingers or interactive occlusion between the hand and objects, it can lead to decreased tracking accuracy or even failure. Furthermore, the algorithms are sensitive to changes in lighting, and the high computational cost makes it difficult to meet the low latency requirements of real-time interaction while maintaining high accuracy.
[0004] (3) Wearable gloves based on inertial measurement units (IMUs): This type of system integrates multiple IMUs on the glove to directly measure joint rotation information at high frequency. It is naturally unaffected by external lighting and occlusion, and has advantages in robustness and real-time performance. However, this technical approach faces two major bottlenecks: ① Kinematic drift: IMUs estimate attitude by integrating angular velocity. During this process, tiny measurement noises accumulate, causing the attitude estimate to deviate from the true value over time, a phenomenon known as "drift." Although some corrections can be made using accelerometers and magnetometers, the problem remains severe in environments with rapid motion or magnetic field interference.
[0005] ② Lack of Personalized Adaptation and Calibration Difficulties: Most existing commercial gloves use a universal hand skeleton model, which cannot adapt to the huge differences in hand size and proportion among individual users, resulting in unnatural animation and inaccurate interaction. Recent academic research (such as FSGlove) attempts to solve for personalized model parameters through optimization, but its calibration process relies on a series of "soft constraints" and "assumptions" that require perfect user cooperation (such as assuming the user's palm is flat and the fingertip distance is zero when pinching). These assumptions are very easy to introduce errors in actual operation, making the calibration process cumbersome and the results inaccurate. Summary of the Invention
[0006] In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide a high-precision and high-robust hand posture estimation method and device.
[0007] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a hand pose estimation method, comprising the following steps: Based on real-time measurement data from an inertial measurement unit installed on the hand, a priori estimate of the hand posture is predicted. Based on the prior estimates, a differentiable, articulated 3D hand model is driven, and virtual rendering data is generated through differentiable rendering. A visual error function is constructed between the virtual rendering data and the actual observation data collected at the same time. The hand posture parameters are adjusted by an optimization algorithm to minimize the visual error function, and the corrected posterior measurement value is obtained. By combining the prior estimate and the posterior measurement, the optimal estimate of the current hand posture is obtained.
[0008] Furthermore, it also includes the following steps: A differentiable, articulated three-dimensional hand model is constructed, wherein the three-dimensional hand model is a three-dimensional Gaussian splash model.
[0009] Furthermore, the steps in constructing a differentiable, articulated 3D hand model specifically include the following sub-steps: Build a personalized hand skeleton model that matches the user's hand shape; Collect first visual data when the user wears the hand-worn device, and construct a three-dimensional Gaussian splash model based on the first visual data. The hand-worn device integrates multiple inertial measurement units and visual reference markers. The personalized hand skeleton model is skeletally bound and skinned with the three-dimensional Gaussian splash model to obtain a differentiable, articulated three-dimensional hand model that can be deformed by posture parameters.
[0010] Furthermore, the step of constructing a personalized hand skeleton model that matches the user's hand shape includes the following sub-steps: Collect second visual data when a user performs preset actions with their bare hand; The initial morphological parameters are estimated using a single-frame 3D hand reconstruction algorithm based on the second visual data. An energy function is constructed based on the initial morphological parameters. By optimizing the energy function, personalized morphological parameters specific to the user are obtained.
[0011] Furthermore, the first visual data is an RGB-D video stream, and the step of constructing the three-dimensional Gaussian splash model includes the following sub-steps: For each frame of RGB image in the RGB-D video stream, perform hand region segmentation to extract the image and corresponding depth map of the wearable device area. Estimate the camera pose of each frame in the segmented image sequence; Based on the camera pose and the segmented image, a three-dimensional Gaussian splash model containing only the hand-worn device is reconstructed.
[0012] The step of binding and skinning the personalized hand skeleton model with the three-dimensional Gaussian splash model includes the following sub-steps: The personalized hand skeleton model and the three-dimensional Gaussian splash model are initially rigidly aligned, and then precise registration is performed using registration technology. Each Gaussian splatter element in the three-dimensional Gaussian splatter model is calculated to have a skin weight relative to each joint in the personalized hand skeleton model. The skin weight is inversely proportional to the distance from the Gaussian splatter element to the joint.
[0013] Furthermore, after the step of binding the personalized hand skeleton model with the three-dimensional Gaussian splash model, the following sub-steps are also included: The spatial calibration between the inertial measurement unit and the personalized hand skeleton model is completed, including the following sub-steps: Collect inertial measurement unit data when a user performs calibration actions while wearing a wearable hand device; A joint optimization energy function is constructed based on the inertial measurement unit data. The joint optimization energy function includes the inertial measurement unit attitude loss and the geometric constraint loss based on the calibration action. The calibration parameters are solved by minimizing the joint optimization energy function.
[0014] Furthermore, the step of generating virtual rendering data through differentiable rendering by driving a differentiable, articulated 3D hand model based on the prior estimate includes the following sub-steps: Based on the prior estimates and the morphological parameters of the constructed personalized hand skeleton model, the skeletal transformation matrix of each joint in the personalized hand skeleton model is calculated. Based on the bone transformation matrix and the skin weights pre-calculated for each Gaussian element in the differentiable, articulated 3D hand model, the Gaussian elements are driven to deform, resulting in a 3D hand model that corresponds to the current prior pose after deformation. The deformed 3D hand model is input into a differentiable renderer to render all deformed Gaussian elements into virtual RGB images and depth maps, which serve as virtual rendering data.
[0015] Furthermore, the visual error function is expressed by the following formula:
[0016] in, Indicates visual error. Indicates luminosity loss, Indicates deep loss. This indicates the reprojection loss of the visual reference mark. The weighting coefficients representing photometric loss, The weighting coefficients represent the depth loss. The weighting coefficients represent the reprojection loss of the visual reference mark. This represents the hand pose parameters to be optimized; The step of adjusting the hand posture parameters through an optimization algorithm to minimize the visual error function includes: using the prior estimate as an initial value, minimizing the visual error function through gradient descent, optimizing the hand posture parameters, and obtaining the posterior measurement value.
[0017] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: to provide a hand posture estimation device, comprising: One or more inertial measurement units are mounted on a wearable hand device to collect real-time inertial data; A vision sensor for acquiring real-time visual data including the wearable hand device; A processing unit is configured to execute the hand pose estimation method described above.
[0018] The hand posture estimation method and apparatus of the present invention have at least the following beneficial effects: (i) This invention combines the high-frequency response of the inertial measurement unit with visual observation of differentiable Gaussian splash rendering. The prediction thread for prior estimation of hand posture runs at high frequency to ensure low latency response; the visual correction thread for posterior measurement of hand posture runs independently at low frequency. The two are asynchronously fused through an extended Kalman filter. Even if vision fails due to rapid movement or brief occlusion, continuous posture can still be output by inertial prediction. This effectively eliminates the integral drift inherent in the pure inertial scheme and overcomes the defect of the pure vision scheme being susceptible to occlusion, achieving high-precision and high-robust real-time hand posture estimation.
[0019] (ii) This invention constructs a user-personalized biomechanical skeletal model, introduces a three-dimensional Gaussian splash model, and constructs a differentiable, articulated three-dimensional hand model to complete high-precision personalized hand modeling; and adopts a joint optimization calibration that only requires two simple actions, "flat hand" and "pinching fingers", to solve all calibration parameters at once, thereby improving calibration efficiency and accuracy. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of one embodiment of the hand posture estimation method of the present invention.
[0021] Figure 2 for Figure 1 Flowchart of step S100.
[0022] Figure 3 for Figure 2 Flowchart of step S110.
[0023] Figure 4 for Figure 2 Flowchart of step S130.
[0024] Figure 5 for Figure 2 Flowchart of step S140. Detailed Implementation
[0025] The following disclosure provides various embodiments or examples for implementing different features of the present invention. Specific examples will be described below to simplify the invention. Of course, these are merely examples and are not intended to limit the invention.
[0026] Although the numerical ranges and parameter settings presented in this invention are approximations, the numerical settings in specific instances are reported as precisely as possible. Any numerical value inherently contains some necessary error arising from the standard deviation found in the respective test measurements. Similarly, as used herein, the term "about" generally refers to within 10%, 5%, 1%, or 0.5% of a given value or range. Alternatively, the term "about" means within an acceptable average standard error that can be conceived by one of ordinary skill in the art. Except in instances of operation / work, or unless expressly stated otherwise, all numerical ranges, totals, values, and percentages, such as those for material quantities, durations, temperatures, operating conditions, amounts, and other similarities disclosed herein, should be understood to be modified by the term "about" in all cases. Therefore, unless otherwise stated, the numerical parameter settings set forth in this invention and the appended claims are approximations that can be changed upon request. At a minimum, each numerical parameter should be interpreted based on the number of significant figures reported and the application of ordinary rounding techniques. A range herein may be expressed as from one endpoint to another or between two endpoints. All scopes disclosed herein include endpoints unless otherwise stated.
[0027] Furthermore, the technical parts described in this invention and the appended claims are primarily the improved technical parts of this invention, and do not limit the object protected by this invention to only having these technical parts. Other known essential components (structures and / or methods) and / or non-essential components of the object protected, besides the technical parts described in this invention and the appended claims, are not included in this invention and the appended claims because they do not fall within the scope of improvements of this invention; however, this does not mean that the object protected by this invention does not possess these known components.
[0028] Please see Figure 1 This is a flowchart of one embodiment of the hand pose estimation method of the present invention. This embodiment is mainly divided into an offline preparation stage (step S100) and a real-time tracking stage (steps S200 to S500). The offline preparation stage is executed once when the user first uses the system to construct a differentiable, articulated three-dimensional hand model; the real-time tracking stage estimates a smooth, drift-free real-time hand pose by combining high-frequency inertial prediction and low-frequency visual correction.
[0029] This implementation method includes the following steps: S100. Construct a differentiable, articulated 3D hand model.
[0030] This step is part of the offline preparation phase, where a 3D hand model that can be driven by pose parameters and rendered using differentiable methods is established. The 3D hand model is a 3D Gaussian splatter (3DGS) model, composed of millions of 3D Gaussian primitives. Each Gaussian primitive contains parameters such as center position, rotation, scaling, color, and opacity, capable of finely representing the texture and geometric details of the glove. The differentiability of this model allows for direct optimization of pose parameters using gradient descent in subsequent steps to minimize visual errors (see step S400 for details).
[0031] Please see Figure 2 This step S100 includes the following sub-steps: S110. Construct a personalized hand skeleton model that matches the shape of the user's hand.
[0032] Specifically, a parametric MANO hand model is used as the carrier, which generates a 3D hand mesh through morphological and posture parameters. The purpose of this step is to accurately capture the inherent biomechanical characteristics of the user's bare hand, that is, to solve for the morphological parameters specific to the user.
[0033] Please see Figure 3 This step S110 includes the following sub-steps: S111. Collect second visual data when the user performs a preset action with bare hands.
[0034] Specifically, the operation for acquiring the second visual data is as follows: the user performs a series of preset actions (such as resting, clenching a fist, bending and stretching, etc.) with their bare hand in front of a single head-mounted RGB-D camera, and RGB images and depth images are recorded simultaneously. The second visual data is the RGB image of the user's bare hand performing the preset actions.
[0035] S112. Based on the second visual data, the initial morphological parameters are estimated using a single-frame 3D hand reconstruction algorithm.
[0036] Specifically, for each frame of the RGB image, existing single-frame 3D hand reconstruction algorithms, such as HandOS, are used to directly obtain initial estimates of the MANO morphological and pose parameters. The specific methods are existing technologies and will not be elaborated here. Since the morphological parameters should theoretically remain unchanged across multiple frames, the morphological parameters of all frames are averaged to obtain the initial morphological parameters.
[0037] S113. Construct an energy function based on the initial morphological parameters, and obtain personalized morphological parameters specific to the user by optimizing the energy function.
[0038] To improve the accuracy of the personalized hand skeleton model, step S113 is performed, where an energy function is constructed and optimized using initial morphological parameters as initial values. During the optimization process, an optimizer such as Adam is used to iteratively optimize a joint optimization set consisting of multiple frames of RGB images and depth maps, minimizing the sum of the energy functions of all frames. This ultimately yields personalized morphological parameters that accurately describe the user's hand shape, ensuring that the personalized hand skeleton model is highly consistent in morphology with the user's real bare hand.
[0039] The formula for the energy function is as follows:
[0040] in, Represents the energy function; Indicates contour loss, Indicates depth map loss. The weighting coefficients represent the contour loss. This represents the weighting coefficients of the depth map loss.
[0041] The contour loss is used to penalize the difference between the rendered model contour and the real hand contour mask segmented from the RGB image. For example, a differentiable renderer (such as PyTorch3D) is used to render the contour of a personalized hand skeleton model, and the IoU loss or Chamfer distance between it and the real contour mask is calculated.
[0042] The depth map loss is used to penalize the difference between the model-rendered depth map and the real depth map captured by the camera, ensuring the geometric consistency of the model in 3D space. The depth map loss is the L1 norm or L2 norm between the model-rendered depth map and the real depth map.
[0043] S120. Collect the first visual data of the user wearing the hand-worn device, and construct a three-dimensional Gaussian splash model based on the first visual data.
[0044] In this embodiment, the wearable hand device is a glove integrating multiple inertial measurement units (IMUs) and visual reference markers. The visual reference markers are QR codes, specifically QR codes. The method for acquiring the first visual data is as follows: the user wears the glove and a head-mounted RGB-D camera, opens their palm, and keeps their hand posture unchanged. They then slowly rotate their wrist back and forth in front of the camera, recording an RGB-D video stream of approximately 30-60 seconds to capture the static geometric shape of the glove from different perspectives. The first visual data is the RGB-D video stream acquired when the user wears the wearable hand device.
[0045] The method for constructing a 3D Gaussian splash model is as follows: For each frame of the RGB-D video stream, segment the hand region to extract the image of the wearable device area and its corresponding depth map; estimate the camera pose of each frame in the segmented image sequence; based on the camera pose and the segmented images, reconstruct a 3D Gaussian splash model containing only the wearable device. For example, firstly, a hand detection and segmentation algorithm (such as YOLO-World or SegmentAnything Model) is used to accurately extract the image of the glove region and its corresponding depth map. Secondly, the segmented image sequence is input into a motion recovery structure toolchain, such as COLMAP. COLMAP accurately estimates the camera pose of each frame through feature matching and binding adjustments. Finally, using the estimated camera pose as input, a high-fidelity 3D Gaussian splash model containing only the glove is reconstructed offline using a standard 3D Gaussian splash algorithm; this 3D Gaussian splash model is the glove model.
[0046] S130. The personalized hand skeleton model is bound to the three-dimensional Gaussian splash model and skinned to obtain a differentiable, articulated three-dimensional hand model that can be deformed by posture parameters.
[0047] Please see Figure 4 This step S130 includes the following sub-steps: S131. Perform initial rigid alignment of the personalized hand skeleton model and the three-dimensional Gaussian splash model, and then perform precise registration using registration technology.
[0048] Specifically, the personalized hand skeleton model defined by personalized morphological parameters obtained in step S110 is initially rigidly aligned with the three-dimensional Gaussian splash model reconstructed in step S120, and then precise registration is performed using registration techniques such as non-rigid iterative nearest point algorithm to determine the precise position of the skeleton in the coordinate system of the three-dimensional Gaussian splash model.
[0049] S132. Calculate the skinning weight of each Gaussian element in the three-dimensional Gaussian splash model relative to each joint in the personalized hand skeleton model. The skinning weight is inversely proportional to the distance from the Gaussian element to the joint.
[0050] For example, a linear hybrid skinning algorithm is used to calculate the skinning weights of each Gaussian splatter element in the 3D Gaussian splatter model relative to all joints in the personalized hand skeleton model. The skinning weights are inversely proportional to the distance from the Gaussian splatter element to the joints. A KNN algorithm can be used to bind each Gaussian splatter element to its K nearest joints, and normalized weights can be calculated based on the distance. At this point, an articulated 3D Gaussian splatter model driven by the pose parameters of the personalized hand skeleton model is constructed. When the pose parameters change, the position and rotation of each Gaussian splatter element will change accordingly based on its skinning weights.
[0051] To improve the accuracy of hand pose estimation in subsequent steps, as a preferred implementation, the following sub-steps are included after step S130: S140. Complete the spatial calibration between the inertial measurement unit and the personalized hand skeleton model.
[0052] Please see Figure 5 This step S140 includes the following sub-steps: S141. Collect inertial measurement unit data when the user performs calibration actions while wearing the wearable device.
[0053] The calibration actions include, but are not limited to, flat hand and pinching finger. Flat hand means that the palm is open and parallel to the ground, and pinching finger means that the tips of the thumb and index finger are in contact. Inertial measurement unit data are recorded simultaneously when the calibration actions are performed.
[0054] S142. Construct a joint optimization energy function based on the inertial measurement unit data. The joint optimization energy function includes the inertial measurement unit attitude loss and the geometric constraint loss based on the calibration action.
[0055] When the calibration action is a flat hand and a pinch, the geometric loss based on the calibration action includes a flat hand pose loss and a pinch geometric loss. The formula for the joint optimization energy function is as follows:
[0056] in, This represents the joint optimization energy function. This represents the attitude loss of the inertial measurement unit. This indicates a loss of balance. Indicates the geometric loss of the pinch finger. The weighting coefficients representing the attitude loss of the inertial measurement unit. The weighting coefficients represent the loss for the even-handed posture. The weighting coefficients represent the geometric loss of the pinch gesture.
[0057] Inertial Measurement Unit (IMU) Attitude Loss: Each IMU has a measurement attitude in its own coordinate system. At the same time, the attitude of the skeletal joints to which the IMU is attached can be calculated in the model coordinate system through the forward kinematics of the personalized hand skeleton model. This loss term is used to penalize the rotational difference that exists after aligning the two to the same coordinate system.
[0058] The attitude loss of the inertial measurement unit The calculation formula is as follows:
[0059] in, This represents the global alignment rotation parameter, i.e., the rotation from the inertial measurement unit coordinate system to the personalized hand skeleton model coordinate system. This represents the measurement attitude of the i-th inertial measurement unit in its own coordinate system. This represents the installation offset rotation parameter of the i-th inertial measurement unit. This represents the pose of the skeletal joint to which the i-th inertial measurement unit is attached in the personalized hand skeleton model coordinate system. This represents the logarithmic mapping from a rotation matrix to its Lie algebra.
[0060] Flat-hand posture loss: In frames where the user performs a flat-hand action, this loss term penalizes the angular difference between the palm normal and the direction of gravity (which can be determined by the accelerometer readings of the back-of-hand inertial measurement unit), providing the main constraint for solving the global alignment rotation.
[0061] Pinch geometry loss: In frames where the user performs a pinching action, this loss term penalizes the Euclidean distance between the three-dimensional coordinates of the thumb and index fingertips calculated by the personalized hand skeleton model, providing constraints for fine-tuning the morphological parameters of the personalized hand skeleton model.
[0062] S143. Solve for the calibration parameters by minimizing the joint optimization energy function.
[0063] The calibration parameters include global alignment rotation parameters, mounting offset rotation parameters for each inertial measurement unit (IMU), and fine-tuning of the morphological parameters of the personalized hand skeleton model. Although the fine-tuning of the morphological parameters of the personalized hand skeleton model is not explicitly shown in the above formulas, it is embedded in the posture of the skeletal joints to which the IMU is attached in the coordinate system of the personalized hand skeleton model. Among them.
[0064] Regarding the global alignment rotation parameters: The measurement data of the inertial measurement unit exists in its own coordinate system, while the personalized hand skeleton model exists in the model coordinate system. Solving for the global alignment rotation parameters establishes the transformation relationship from the coordinate system of the inertial measurement unit to the coordinate system of the personalized hand skeleton model. This is the basis for linking the physical motion measured by the inertial measurement unit with the motion of the virtual hand model.
[0065] Regarding the installation offset rotation parameters: The actual installation position and orientation of each inertial measurement unit on the glove may have a small but not negligible deviation relative to the bone to which it is attached. Solving for the installation offset rotation parameters of each inertial measurement unit will enable the measurement data of the inertial measurement unit to be accurately converted to the local coordinate system of the bone to which it is attached.
[0066] Regarding the fine-tuning of morphological parameters: Although personalized morphological parameters have been obtained in step S110, slight deformations of the hand or the thickness of the glove may introduce minor deviations when the user is actually wearing gloves. The calibration process fine-tunes these morphological parameters so that the calculated geometric relationships (such as fingertip distance) of the model under calibration actions while wearing gloves are more consistent with the real situation in the physical world (such as fingertip contact when pinching fingers), thereby improving the model's adaptation accuracy.
[0067] S200: Based on real-time measurement data from an inertial measurement unit located on the hand, predict a priori estimates of the hand's posture.
[0068] Specifically, it operates at a high frequency (approximately 200Hz) to achieve low-latency attitude prediction. At each time step, the processing unit receives real-time measurement data (angular velocity and acceleration) from the inertial measurement unit on the glove and, based on the fused optimal attitude from the previous moment, quickly predicts the prior estimate of the hand attitude at the current moment using a kinematic integral model. This process involves only simple numerical integration, with extremely low computational cost, thus ensuring a very high operating frequency.
[0069] S300. Based on the prior estimate, drive a differentiable, articulated 3D hand model and generate virtual rendering data through differentiable rendering.
[0070] The specific method is as follows: Based on the prior estimate and the morphological parameters of the constructed personalized hand skeleton model, calculate the bone transformation matrix of each joint in the personalized hand skeleton model; based on the bone transformation matrix and the skin weights pre-calculated for each Gaussian unit in the differentiable, jointed 3D hand model, drive the Gaussian unit to deform, and obtain the 3D hand model corresponding to the current prior pose after deformation; input the deformed 3D hand model into a differentiable renderer to render all the deformed Gaussian units into virtual RGB images and depth maps as virtual rendering data.
[0071] For example, operating at a low frequency (approximately 30Hz–60Hz), the latest prior estimate of the hand pose is first obtained. Then, using this prior estimate and calibrated morphological parameters, the transformation matrix of all skeletal joints is calculated through forward kinematics of a personalized hand skeletal model. Next, for each Gaussian primitive in the differentiable, jointed 3D hand model, it is transformed according to its skinning weights and the aforementioned transformation matrix to obtain the deformed Gaussian primitive. Finally, a differentiable renderer (e.g., a rendering pipeline based on a rasterized 3D Gaussian splatter model) is used to render all the deformed Gaussian primitives into virtual RGB images and depth maps. The camera pose used for rendering is determined by a visual reference marker on the back of the hand (i.e., the QR code described in step S120); if the QR code is not in the camera's field of view, the estimate from the previous time step is used.
[0072] S400. Construct and minimize the visual error function to obtain the posterior measurement value. By constructing the visual error function between the virtual rendering data and the actual observation data collected at the same time, the hand posture parameters are adjusted through an optimization algorithm to minimize the visual error function, thereby obtaining the corrected posterior measurement value.
[0073] This step S400, together with step S300 above, constitutes the visual correction thread, which runs at a low frequency (e.g., 30-60Hz). It uses observation data acquired by the actual camera to correct the prior attitude estimate obtained in step S200, thereby eliminating the cumulative drift caused by inertial integration. The actual observation data consists of the RGB image and depth map actually acquired by the camera at the current moment. The visual error function is expressed by the following formula:
[0074] in, Indicates visual error. Indicates luminosity loss, Indicates deep loss. This indicates the reprojection loss of the visual reference mark. The weighting coefficients representing photometric loss, The weighting coefficients represent the depth loss. The weighting coefficients represent the reprojection loss of the visual reference mark. This represents the hand pose parameters to be optimized.
[0075] The formula for calculating the photometric loss is as follows:
[0076] in, This represents a virtual RGB image generated through rendering. This represents the actual RGB images captured by the camera at the same moment; This represents a coefficient between 0 and 1, used to balance the importance of structural similarity loss and pixel-level loss; This represents the L1 norm, used to calculate the absolute difference between corresponding pixel values in two images.
[0077] The depth loss is used to calculate the L1 norm difference between the rendered depth map and the true depth map. In a preferred embodiment, the loss is calculated only within the hand contour region to exclude background depth noise and thus enhance the constraint on the geometry.
[0078] The visual reference marker reprojection loss is activated when the visual reference marker (i.e., the QR code on the glove) appears in the camera's field of view. The formula for calculating this loss is as follows:
[0079] in, This indicates traversing all corners of the QR code. This represents the projected coordinates of the corner point on the camera image plane, calculated based on the current pose parameters. This represents the actual pixel coordinates of the corner points detected from real images using a visual library. This represents the square of the L2 norm, which is the square of the Euclidean distance.
[0080] When adjusting the hand pose parameters to minimize the visual error function using an optimization algorithm, since the entire rendering and error calculation process is differentiable, the gradient of the visual error relative to the pose parameters can be calculated. That is, the visual error function is minimized using gradient descent, thus optimizing the hand pose parameters and obtaining a posterior measurement. Specifically, using the prior estimate of the hand pose predicted in step S200 as the initial value, several steps of gradient descent optimization are performed to obtain a posterior pose measurement that minimizes the visual error. The formula for calculating the posterior measurement is as follows:
[0081] in, This represents a more accurate posterior measurement after visual correction. Represents the prior estimate. Indicates the learning rate. The visual error function is represented in The gradient at a given point indicates the direction in which the error decreases most rapidly.
[0082] S500: The prior estimate and the posterior measurement are fused to obtain the optimal estimate of the current hand posture.
[0083] To obtain the optimal attitude that combines inertial response speed with the elimination of long-term drift, this step employs an extended Kalman filter (EKF) to fuse the prior estimate obtained in step S200 with the posterior measurement obtained in step S400. The state vector of the EKF includes the hand's attitude parameters and its first-order derivative angular velocity. This allows the filter to make predictions using high-frequency angular velocity information provided by the inertial measurement unit, while simultaneously correcting the state using the posterior measurement.
[0084] Specifically, whenever an angular velocity measurement is received from the inertial measurement unit, the EKF combines the optimal estimate from the previous moment with the kinematic integral model. Perform the prediction to obtain the prior estimate and its covariance at the current time:
[0085] in, This represents the optimal estimate at the previous moment. This represents the prior estimate at the current moment. This represents the measured angular velocity value. Denotes the Jacobian of the state transition matrix. Represents the process noise covariance. and These represent the covariance matrices of the previous time step and the current prediction, respectively, used to represent the uncertainty of the estimation.
[0086] When a posterior posture measurement is received, it is used as an observation. The EKF then performs a weighted average of this measurement with the current prior estimate to obtain the optimal estimate of the current hand posture and its covariance.
[0087] in, Indicates Kalman gain, H represents the observation matrix; R represents the process noise covariance. Represents the identity matrix. This represents the optimal estimate of the current hand posture. This represents the covariance matrix at the current time.
[0088] This invention also discloses a hand pose estimation device for implementing the hand pose estimation method described in the above embodiments. Specifically, the hand pose estimation device of this embodiment includes: one or more inertial measurement units disposed on a wearable hand device for acquiring real-time inertial data, wherein the wearable hand device also integrates multiple visual reference markers; a visual sensor for acquiring real-time visual data including the wearable hand device; and a processing unit configured to execute the hand pose estimation method as described in steps S100 to S500 above.
[0089] This invention combines the high-frequency response of an inertial measurement unit (IMU) with visual observation through differentiable Gaussian splash rendering. The prediction thread for prior hand posture estimation runs at high frequency, ensuring low-latency response; the visual correction thread for posterior hand posture measurement runs independently at low frequency. The two are asynchronously fused through an extended Kalman filter. Even if vision fails due to rapid movement or brief occlusion, the system can still output continuous posture based on inertial prediction. This effectively eliminates the inherent integral drift of pure inertial schemes while overcoming the occlusion susceptibility of pure visual schemes, achieving high-precision and robust real-time hand posture estimation. By constructing a user-personalized biomechanical skeleton model and introducing a three-dimensional Gaussian splash model, a differentiable, articulated three-dimensional hand model is constructed, completing high-precision personalized hand modeling. Furthermore, a joint optimization calibration method requiring only two simple actions, "flat hand" and "pinching fingers," is used to solve all calibration parameters at once, improving calibration efficiency and accuracy.
Claims
1. A method for estimating hand pose, characterized in that, Includes the following steps: Based on real-time measurement data from an inertial measurement unit installed on the hand, a priori estimate of the hand posture is predicted. Based on the prior estimates, a differentiable, articulated 3D hand model is driven, and virtual rendering data is generated through differentiable rendering. A visual error function is constructed between the virtual rendering data and the actual observation data collected at the same time. The hand posture parameters are adjusted by an optimization algorithm to minimize the visual error function, and the corrected posterior measurement value is obtained. By combining the prior estimate and the posterior measurement, the optimal estimate of the current hand posture is obtained.
2. The hand pose estimation method as described in claim 1, characterized in that, It also includes the following steps: A differentiable, articulated three-dimensional hand model is constructed, wherein the three-dimensional hand model is a three-dimensional Gaussian splash model.
3. The hand pose estimation method as described in claim 2, characterized in that, The steps involved in constructing a differentiable, articulated 3D hand model include the following sub-steps: Build a personalized hand skeleton model that matches the user's hand shape; Collect first visual data when the user wears the hand-worn device, and construct a three-dimensional Gaussian splash model based on the first visual data. The hand-worn device integrates multiple inertial measurement units and visual reference markers. The personalized hand skeleton model is skeletally bound and skinned with the three-dimensional Gaussian splash model to obtain a differentiable, articulated three-dimensional hand model that can be deformed by posture parameters.
4. The hand pose estimation method as described in claim 3, characterized in that, The process of building a personalized hand skeleton model that matches the user's hand shape includes the following sub-steps: Collect second visual data when a user performs preset actions with their bare hand; The initial morphological parameters are estimated using a single-frame 3D hand reconstruction algorithm based on the second visual data. An energy function is constructed based on the initial morphological parameters. By optimizing the energy function, personalized morphological parameters specific to the user are obtained.
5. The hand pose estimation method as described in claim 3, characterized in that, The first visual data is an RGB-D video stream. The steps for constructing the 3D Gaussian splash model include the following sub-steps: For each frame of RGB image in the RGB-D video stream, perform hand region segmentation to extract the image and corresponding depth map of the wearable device area. Estimate the camera pose of each frame in the segmented image sequence; Based on the camera pose and the segmented image, a three-dimensional Gaussian splash model containing only the hand-worn device is reconstructed.
6. The hand pose estimation method as described in claim 4, characterized in that, The step of binding and skinning the personalized hand skeleton model with the three-dimensional Gaussian splash model includes the following sub-steps: The personalized hand skeleton model and the three-dimensional Gaussian splash model are initially rigidly aligned, and then precise registration is performed using registration technology. Each Gaussian splatter element in the three-dimensional Gaussian splatter model is calculated to have a skin weight relative to each joint in the personalized hand skeleton model. The skin weight is inversely proportional to the distance from the Gaussian splatter element to the joint.
7. The hand pose estimation method as described in claim 3, characterized in that, After the step of binding the personalized hand skeleton model with the three-dimensional Gaussian splash model, the following sub-steps are also included: The spatial calibration between the inertial measurement unit and the personalized hand skeleton model is completed, including the following sub-steps: Collect inertial measurement unit data when a user performs calibration actions while wearing a wearable hand device; A joint optimization energy function is constructed based on the inertial measurement unit data. The joint optimization energy function includes the inertial measurement unit attitude loss and the geometric constraint loss based on the calibration action. The calibration parameters are solved by minimizing the joint optimization energy function.
8. The hand pose estimation method as described in claim 6, characterized in that, The step of driving a differentiable, articulated 3D hand model based on the prior estimate and generating virtual rendering data through differentiable rendering includes the following sub-steps: Based on the prior estimates and the morphological parameters of the constructed personalized hand skeleton model, the skeletal transformation matrix of each joint in the personalized hand skeleton model is calculated. Based on the bone transformation matrix and the skin weights pre-calculated for each Gaussian element in the differentiable, articulated 3D hand model, the Gaussian elements are driven to deform, resulting in a 3D hand model that corresponds to the current prior pose after deformation. The deformed 3D hand model is input into a differentiable renderer to render all deformed Gaussian elements into virtual RGB images and depth maps, which serve as virtual rendering data.
9. The hand pose estimation method as described in claim 8, characterized in that, The visual error function is expressed by the following formula: in, Indicates visual error. Indicates luminosity loss, Indicates deep loss. This indicates the reprojection loss of the visual reference mark. The weighting coefficients representing photometric loss The weighting coefficients represent the depth loss. The weighting coefficients represent the reprojection loss of the visual reference mark. This represents the hand pose parameters to be optimized; The step of adjusting the hand posture parameters through an optimization algorithm to minimize the visual error function includes: using the prior estimate as an initial value, minimizing the visual error function through gradient descent, optimizing the hand posture parameters, and obtaining the posterior measurement value.
10. A hand posture estimation device, characterized in that, include: One or more inertial measurement units are mounted on a wearable hand device to collect real-time inertial data; A vision sensor for acquiring real-time visual data including the wearable hand device; A processing unit is configured to perform the hand pose estimation method as described in any one of claims 1 to 9.