Hand posture estimation method and device and head-mounted display equipment

Abstract

The invention provides a hand posture estimation method and device and head-mounted display equipment, and relates to the technical field of man-machine interaction, and the method comprises the steps: determining the pixel position and the relative distance of each hand joint point, and the bending degree of each finger according to a hand binocular image; according to the bending degree of each finger, an optimization variable of the hand posture is determined; according to the optimization variable, the pixel position and the relative distance of each hand joint point and the bending degree of each finger, an optimization function of the hand posture is generated; and performing iterative optimization on the hand posture according to the optimization function to obtain a hand posture estimation result. By combining the pixel position and the relative distance of each hand joint point and the bending degree of the finger, the hand posture is iteratively optimized, so that the estimation accuracy of the hand posture is improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a hand pose estimation method, apparatus, and head-mounted display device. Background Technology

[0002] 3D gesture estimation is crucial in the XR (Extended Reality) field because it provides a natural and intuitive way of interacting, greatly enhancing the user experience and immersion.

[0003] Currently, hand pose estimation primarily relies on binocular cameras on head-mounted display devices to acquire real-time images of the hand from different perspectives. The 2D (two-dimensional) hand joints extracted from these images are then projected onto the binocular cameras to convert them into 3D hand joints. However, due to distortions and other issues in the images captured by binocular cameras, the accuracy of hand pose estimation is low. Summary of the Invention

[0004] This invention provides a hand pose estimation method, apparatus, and head-mounted display device to address the shortcomings of low hand pose estimation accuracy in the prior art and improve the accuracy of hand pose estimation.

[0005] This invention provides a hand pose estimation method, comprising the following steps: Based on the binocular images of the hand, determine the pixel position and relative distance of each hand joint, as well as the curvature of each finger; Based on the degree of curvature of each finger, determine the optimization variables for hand pose; An optimization function for hand pose is generated based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger. The hand pose is iteratively optimized according to the optimization function to obtain the hand pose estimation result.

[0006] According to a hand pose estimation method provided by the present invention, generating an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger includes: Based on the pixel position and relative distance of each hand joint, the projection error and relative distance error terms are determined; Based on the optimization variables, determine the smoothness constraint terms; Based on the curvature of each finger, determine the curvature error term; The sum of the projection error and relative distance error, the smoothness constraint, and the curvature error is used as the optimization function for the hand pose.

[0007] According to a hand pose estimation method provided by the present invention, the step of determining projection error and relative distance error terms based on the pixel positions and relative distances of each hand joint includes: The projection error is determined based on the pixel position and confidence level of each hand joint in the image coordinates, as well as the three-dimensional coordinates of each hand joint. The relative distance error is determined based on the relative distance and confidence level of each hand joint in each camera coordinate, the three-dimensional coordinates of each hand joint, the palm size, and the coordinates of the proximal metacarpal joint of the middle finger. The projection error and relative distance error terms are determined based on the projection error and the relative distance error.

[0008] According to a hand pose estimation method provided by the present invention, the three-dimensional coordinates of the hand joints are determined based on the following method: Based on the rotational degrees of freedom of the hand joints, determine the rotational transformation operator of the hand joints; The pose information of the hand joints is determined based on the rotational degrees of freedom and the rotational transformation operator. The three-dimensional coordinates of the hand joints are determined based on the posture information of the hand joints and the position transformation matrix of each finger.

[0009] According to a hand pose estimation method provided by the present invention, the step of determining a smoothness constraint term based on the optimization variables includes: The smoothness constraint term is determined based on the difference between the optimization variables at two adjacent time points.

[0010] According to a hand pose estimation method provided by the present invention, determining a curvature error term based on the curvature of each finger includes: The curvature error term is determined based on the curvature and confidence level of each finger, and the angle of rotation of each hand joint of each finger about the x-axis.

[0011] According to a hand pose estimation method provided by the present invention, determining the optimization variables of the hand pose based on the curvature of each finger includes: The degree of flexure of each finger is determined based on the rotational degrees of freedom of each of the hand joints; The optimization variables for the hand pose are determined based on the curvature of each finger, the size of the palm, and the joint length coefficient of each finger.

[0012] According to a hand pose estimation method provided by the present invention, the hand pose estimation method further includes: Based on the joint length coefficients of each finger, the overall hand state parameters are determined; During the iterative optimization process, monitor the inter-frame changes in the total parameters at the finger joint scale. If the inter-frame change is less than a first preset threshold at multiple consecutive adjacent time points, the hand state is determined to be stable. If the inter-frame change exceeds a second preset threshold for multiple consecutive adjacent time points, it is determined that the user has changed.

[0013] The present invention also provides a hand pose estimation device, comprising the following modules: The hand parameter determination module is used to determine the pixel position and relative distance of each hand joint point, as well as the curvature of each finger, based on the binocular images of the hand. The optimization variable determination module is used to determine the optimization variables of the hand pose based on the curvature of each finger. An optimization function generation module is used to generate an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger. The hand pose estimation module is used to iteratively optimize the hand pose according to the optimization function to obtain the hand pose estimation result.

[0014] The present invention also provides a head-mounted display device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hand pose estimation method as described above.

[0015] The hand pose estimation method, apparatus, and head-mounted display device provided by this invention determine the pixel position and relative distance of each hand joint and the curvature of each finger based on binocular images of the hand; determine optimization variables for hand pose based on the curvature of each finger; generate an optimization function for hand pose based on the optimization variables, the pixel position and relative distance of each hand joint, and the curvature of each finger; and iteratively optimize the hand pose using the optimization function to obtain the hand pose estimation result. This invention improves the accuracy of hand pose estimation by combining the pixel position and relative distance of each hand joint and the curvature of the fingers for iterative optimization. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of the hand pose estimation method provided by the present invention.

[0018] Figure 2 This is a flowchart illustrating the hand pose estimation method provided by the present invention.

[0019] Figure 3 This is a flowchart illustrating the optimized function for generating hand poses provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the hand pose estimation device provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the head-mounted display device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following is combined with Figures 1-5 The present invention describes a hand pose estimation method, apparatus, and head-mounted display device.

[0024] The hand pose estimation method provided by this invention can be applied in human-computer interaction scenarios, such as human-computer interaction in terminals and human-computer interaction scenarios in vehicle systems. Figure 1 This is an application environment diagram of the hand pose estimation method provided by the present invention, such as... Figure 1As shown, terminal 20 communicates with server 10 via a network. The data storage system can store the data that server 10 needs to process. The data storage system can be integrated onto server 10 or placed in the cloud or on another network server. Terminal 20 can determine the pixel position and relative distance of each hand joint point, as well as the curvature of each finger, based on the binocular images of the hand. Based on the curvature of each finger, it determines the optimization variables for hand pose. Based on the optimization variables, the pixel position and relative distance of each hand joint point, and the curvature of each finger, it generates an optimization function for hand pose. Iteratively optimizing the hand pose using the optimization function yields the hand pose estimation result. Alternatively, terminal 20 can send the acquired binocular images of the hand to server 10, where server 10 determines the hand pose estimation result. Server 10 then returns the hand pose estimation result to terminal 20, and terminal 20 determines the pose of each hand joint point based on the hand pose estimation result.

[0025] The terminal 20 may include a Virtual Reality Headset (VR head-mounted display), a head-mounted display device, an electronic display screen, and a Mixed Reality (MR) device. MR devices may include MR glasses, MR helmets, MR cameras, etc., and head-mounted display devices may include Augmented Reality Glasses (AR glasses), MR glasses, etc. The in-vehicle system may include in-vehicle chips, in-vehicle devices (such as in-vehicle infotainment systems or in-vehicle computers capable of constructing augmented reality spaces), etc.

[0026] Figure 2 This is a flowchart illustrating the hand pose estimation method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following: Step 201: Based on the binocular images of the hand, determine the pixel position and relative distance of each hand joint, as well as the curvature of each finger.

[0027] Binocular hand images are images of the hand captured simultaneously from different perspectives by two cameras (binocular cameras) on a head-mounted display device. A binocular hand image includes a left-eye image and a right-eye image at the same time. The left-eye image (or right-eye image) may include both the left and right hands, or only one hand, or no hand at all.

[0028] Hand joints refer to key anatomical points that characterize the structure of the hand, such as finger joints, including the tips of each finger (such as the thumb, index finger, middle finger, ring finger, and little finger) and joints (such as the tip of the thumb, the middle joint of the index finger, and the base of the palm; usually there are 26 key joints in a hand).

[0029] The pixel position of a hand joint refers to the coordinates of a hand joint (such as the fingertip, the base of the finger, the wrist joint, etc.) in the pixel coordinate system of a two-dimensional image. For example, the pixel position of the fingertip of the right index finger in the left image is (320, 240), which means that it is located at the 320th pixel horizontally and the 240th pixel vertically in the left image.

[0030] The relative distance between hand joints refers to the three-dimensional spatial distance between two joints in the camera coordinate system. For example, in the left camera coordinate system, the 3D coordinates of the thumb tip and the index finger tip are (X1, Y1, Z1) and (X2, Y2, Z2) respectively. The Euclidean relative distance between the two points is the straight-line distance between the two points in three-dimensional space. .

[0031] Finger curvature describes the degree of bending of the finger, and is usually represented by calculating the joint angle between adjacent joints in 3D. For example, the 3D coordinates of the metacarpophalangeal joint, proximal interphalangeal joint, and distal interphalangeal joint of the index finger can form two vectors. The smaller the angle between the vectors, the more bent the finger is.

[0032] Images of the hand region are acquired using a binocular camera on a head-mounted display device. Multiple frames of binocular hand images are extracted from these images. The observation data and corresponding confidence scores for each hand joint in each frame are calculated. This observation data can include the two-dimensional position, depth information, and pose information of the hand joints. Specifically, the binocular hand images are processed to obtain the following information for each hand joint in each frame: the pixel position of each hand joint in the image coordinates and its confidence score, the relative distance of each hand joint in each camera coordinate system and its confidence score, and the curvature of each finger and its confidence score.

[0033] For example, by inputting multiple frames of binocular images of the hand into a pre-trained hand joint detection network model, the model outputs a heatmap for each frame, along with the curvature and confidence level of each finger in each frame. The heatmap may include: Keypoint image location heatmap: The size is 22×22, which means that the model predicts a 22×22 confidence map for each of the 22 hand joints. The value of each pixel in the map represents the confidence (probability) that the location is the location of the joint.

[0034] Relative distance heatmap: The size is 22×1, which means that for each of the 22 key points in the model, a scalar value is predicted to represent its depth or relative distance.

[0035] Furthermore, based on the heatmap, the pixel positions and confidence levels of each hand joint in the image coordinates, as well as the relative distances and confidence levels of each hand joint in each camera coordinate system, can be obtained. For example, this can be achieved through the following steps: (1) Traverse all the grids in the heatmap, find the position of the grid with the largest element, and take its value as the covariance, that is: ; in, Indicates the first A heatmap of a hand joint should be understood as a two-dimensional matrix, where each position... The value indicates that the location is a hand joint. The higher the confidence (probability) value of the location, the higher the confidence. This indicates taking the supremum (finding the maximum value), i.e., in the heatmap. Find the pixel with the largest value. Indicates the first The coordinates of the largest pixel at the nth hand joint are the coordinates of the nth hand joint. The approximate pixel position of each hand joint.

[0036] It should be understood that the first Covariance of individual hand joints This is equivalent to traversing the first... All pixel coordinates of the heatmap of each hand joint. Then, the maximum confidence value (i.e., the peak value of the heatmap) is found. Covariance is used to characterize the confidence level of the predicted key point location.

[0037] (2) Determine an effective calculation region around the hand joints on the heat map. (Kernel, or kernel function region), ensuring the region size is appropriate and does not exceed the heatmap boundaries. For example, constraining the maximum width of the kernel used for calculation, the kernel boundary is obtained through the following calculation: ; ; ; in, , Indicates the nucleus is in The left and right boundaries of the direction , Indicates the nucleus is in The upper and lower boundaries of the direction This indicates the maximum set core width. Indicates the width of the heatmap. Indicates the first Thermographs of individual hand joints Direction Size (i.e., safe expansion radius). Indicates the first Thermographs of individual hand joints Direction size.

[0038] It should be understood that the core is a localized treatment area surrounding the hand joints, and its maximum width needs to be limited. To avoid introducing too much noise or exceeding the range of the heatmap.

[0039] (3) The hand joint detection model initially predicts grid-level (integer) coordinates, but the actual positions of hand joints may be between pixels (sub-pixel level), thus requiring more precise calculations. For example, the precise position of pixel coordinates can be calculated in the following way: ; in, This represents the calculated, sub-pixel-level precise coordinates of the hand joints. Indicates the effective calculation region Confidence level of the internal heat map; , Indicates integer pixel coordinates When transforming to the coordinates of the pixel center, it should be understood that in the image coordinate system, the coverage area of ​​one pixel is... and Its center point coordinates are .

[0040] It should be understood that by using the confidence level of the heatmap as a weight, the coordinates of the grid center are weighted and summed, and then divided by the total weight, the coordinates of the hand joint points are improved from integer grids to sub-pixel level, thus significantly improving the positional accuracy.

[0041] (4) Extract precise depth information (relative distance) of hand joints from the output of the hand joint network detection model and evaluate the confidence level (covariance) of this depth information. For example, the relative distance ratio and its covariance can be calculated as follows: ; ; ; in, Representing the coordinates of the heatmap The value corresponding to this position can be understood as the first... A one-dimensional response distribution of the output of each hand joint; horizontal axis Representing depth values ​​(or discrete depth value encoding), such as This represents the closest distance to the camera. This represents the furthest point from the camera; This represents the set of accessible regions in the heatmap. Represents the raw estimate of depth. This represents the ratio of relative distances. This indicates the confidence level of the assessment depth information (i.e., the confidence level of the relative distance).

[0042] It should be understood that The calculation is a confidence-weighted average, which is the sum of all depth values. Use its corresponding confidence level The average of these values ​​as weights yields a continuous, sub-pixel-level depth estimate. Through the The purpose of normalization is to unify the depth values ​​of all joints into a fixed range, so that heatmaps of different sizes and distance ratios of different joints can be compared in a unified manner, which facilitates subsequent processing and optimization. The calculation process involves calculating the final depth result. Substitution In the context of the query, find the confidence value corresponding to that position.

[0043] The curvature and confidence score of each finger are directly output by the hand joint network detection model. The channel output is a 10×1 vector, where the first five dimensions are the curvature (using...). (represented), the last five dimensions are the corresponding covariances (using) express).

[0044] Step 202: Determine the optimization variables for hand pose based on the curvature of each finger.

[0045] It should be understood that the optimization variable of hand pose is a highly structured and parameterized mathematical representation vector. Its purpose is to describe the global pose and local shape of the hand in three-dimensional space completely and accurately through a set of relatively low-dimensional values.

[0046] Based on the curvature of each finger and other parameters, optimization variables for hand pose are determined. These optimization variables are multi-dimensional vectors, whose dimensions include pose parameters representing global hand rotation and rotation of each finger joint, as well as shape parameters representing palm size and finger length.

[0047] Step 203: Generate an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger.

[0048] The hand pose optimization function can be understood as an evaluation criterion or objective function used to quantify the overall difference (error) between the currently estimated hand pose and all observed sensor data.

[0049] Based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger, multiple data items are determined, and the sum of these data items is used as the optimization function for hand pose.

[0050] Step 204: Iteratively optimize the hand pose according to the optimization function to obtain the hand pose estimation result.

[0051] The goal is to minimize the optimization function, iteratively optimizing the hand pose. Iteration stops when a predetermined number of iterations (e.g., 30) is reached, or when the error cannot be reduced further. The result of this iterative optimization is used as the 3D coordinates of the hand joints in the current frame image, optimized based on the recognition results. The optimized 3D coordinates of each hand joint are more accurate, resulting in less finger tremor and better hand pose recognition.

[0052] The hand pose estimation method provided in this invention determines the pixel position and relative distance of each hand joint and the curvature of each finger based on a binocular image of the hand. Based on the curvature of each finger, optimization variables for the hand pose are determined. An optimization function for the hand pose is generated based on the optimization variables, the pixel position and relative distance of each hand joint, and the curvature of each finger. The hand pose is then iteratively optimized using the optimization function to obtain the hand pose estimation result. This invention improves the accuracy of hand pose estimation by combining the pixel position and relative distance of each hand joint and the curvature of the fingers for iterative optimization.

[0053] Based on the above embodiments, determining the optimization variables for hand pose according to the curvature of each finger includes: The degree of flexure of each finger is determined based on the rotational degrees of freedom of each of the hand joints; The optimization variables for the hand pose are determined based on the curvature of each finger, the size of the palm, and the joint length coefficient of each finger.

[0054] It should be understood that finger bending is a final overall form, which is formed by the superposition of the rotations of multiple independent joints on the finger (such as metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints). Therefore, the degree of bending of each finger is represented, realized, and controlled by combining and adjusting the rotational degrees of freedom of all joints on that finger.

[0055] For example, optimization variables for hand pose. It can be represented as: ; ; ; The state of the other fingers can be represented as follows: ; in, Indicates the global rotational posture of the wrist joint; This represents the rotational degrees of freedom of the wrist joint, which has three dimensions and can simultaneously perform swinging motion. Planar rotation and twist (around) (axis rotation) This represents the rotational degrees of freedom of the metacarpal joints, which are 3-dimensional. This represents the rotational degrees of freedom of the proximal metacarpal joint, which is two-dimensional and allows for swinging. (planar rotation), without torsion; , These represent the rotational degrees of freedom of the middle and distal ends of the finger, respectively. Each of these degrees of freedom has 1 dimension and can be curled (rotation around a single axis, such as bending the finger).

[0056] The degree of flexion of the thumb represents the rotational state of the thumb joints. It is composed of the rotational parameters of the metacarpal joints (3D), interphalangeal joints (1D), and distal joints (1D), which together control the flexion and abduction of the thumb. These represent the curvature of the index, middle, ring, and little fingers, respectively, characterizing the joint rotation state of the index, middle, ring, and little fingers. Each finger is composed of rotational parameters of the metacarpophalangeal joint (3D), proximal interphalangeal joint (2D), interphalangeal joint (1D), and distal joint (1D), which together control the flexion and abduction of each finger. This refers to the static dimensional parameters of the palm (such as width and thickness). These parameters are usually constant for the same user and are used to accommodate the differences in hand shape among different users. These represent the joint length coefficients (i.e., the relative length ratios of bones) of each finger. They are 3-dimensional and describe the static anatomical features of the fingers (such as whether the index finger is longer than the ring finger). This parameter is constant for the same user.

[0057] Therefore, the optimization variables for hand pose It is a 48-dimensional vector.

[0058] This invention compresses complex, high-dimensional hand geometry into a low-dimensional, structured, and physically realistic parameter vector, thereby transforming pose estimation into an efficient and robust mathematical optimization problem. This allows for the accurate and efficient reconstruction of the user's hand movements by optimizing this set of parameters.

[0059] Figure 3 This is a flowchart illustrating the optimization function for generating hand poses provided by the present invention, as shown below. Figure 3 As shown, the step of generating an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger includes: Step 301: Determine the projection error and relative distance error terms based on the pixel position and relative distance of each hand joint point; Step 302: Determine the smoothness constraint term based on the optimization variables; Step 303: Determine the curvature error term based on the curvature of each finger; Step 304: The sum of the projection error and relative distance error, the smoothness constraint and the curvature error is used as the optimization function for the hand pose.

[0060] The goal of the projection error and relative distance error terms is to ensure that the optimized 3D hand model matches the original image observation evidence as closely as possible. For example, the 3D joints of the current hand model can be projected onto the 2D image plane, and the difference between their positions and the observed pixel positions (such as Euclidean distance) can be calculated to determine the projection error. Simultaneously, the difference between the 3D depth values ​​of the model's joints and the observed depth values ​​can be directly calculated to determine the relative distance error.

[0061] The goal of the smoothness constraint is to ensure that the optimized hand pose is reasonable and stable. In a video sequence, the smoothness constraint penalizes overly abrupt and discontinuous changes in pose between the current frame and the previous frame, making the output hand movement smoother and more stable, reducing shakiness. Optionally, it can also penalize unnatural abrupt changes in the hand model parameters themselves, ensuring that pose changes conform to biomechanical principles.

[0062] The goal of the bend error term is to ensure that the optimized hand pose is consistent with the observed gesture at a high-level semantic level. For example, the bend state of each finger can be parsed from the current hand model parameters, and the difference between it and the observed bend can be calculated to obtain the bend error term.

[0063] For example, the optimization function can be calculated in the following way. : ; in, Indicates in The projection error and relative distance error terms at time, Indicates in Smoothness constraints at time points, Indicates in The curvature error term at time, Indicates in Optimization variables at time intervals Indicates in Optimization variables at any given time.

[0064] This invention improves the accuracy of hand pose estimation by uniformly transforming observation data and constraints from different sources and of different types into mathematical error terms, and finding a globally optimal hand pose solution by minimizing the sum of all error terms.

[0065] Based on the above embodiments, determining the projection error and relative distance error terms according to the pixel positions and relative distances of each of the hand joint points includes: The projection error is determined based on the pixel position and confidence level of each hand joint in the image coordinates, as well as the three-dimensional coordinates of each hand joint. The relative distance error is determined based on the relative distance and confidence level of each hand joint in each camera coordinate, the three-dimensional coordinates of each hand joint, the palm size, and the coordinates of the proximal metacarpal joint of the middle finger. The projection error and relative distance error terms are determined based on the projection error and the relative distance error.

[0066] Using the 2D pixel positions, confidence levels, and 3D coordinates of hand joints, a camera projection model is employed to calculate the difference between the theoretical and actual detected positions projected from the 3D coordinates onto the 2D image—the projection error. 2D points with higher confidence levels have a greater weight in constraining the error, ensuring the 3D pose projection closely matches the observed image. Furthermore, by combining the relative distance from the camera and its confidence level, 3D coordinates, hand size, and middle finger reference joint coordinates, the difference between the actual distance ratio of joints in the 3D pose and the observed / physiological distance ratio is calculated. Hand size and the middle finger joint serve as physiological benchmarks to prevent optimization that results in 3D proportions that violate human anatomy (e.g., an excessively long thumb, abnormal joint spacing). Finally, projection error and relative distance error are used together as core indicators to quantify the rationality of the 3D pose, ensuring the pose matches the 2D image while conforming to distance proportions and physiological structure, thus improving estimation accuracy.

[0067] For example, the projection error and relative distance error terms can be calculated in the following way. : ; in, This indicates the number of effective palm areas observed by the left and right cameras. Indicates the first Confidence of pixel positions of hand joints Indicates the first The pixel positions of each hand joint. Indicates the first The three-dimensional coordinates of each hand joint. Represents the camera projection model. Indicates the first Confidence level of the relative distances between individual hand joints Indicates the coordinates of the proximal metacarpal joint of the middle finger. Indicates the first The relative distance between each hand joint point Indicates the first Projection error of each hand joint point Indicates the first The relative distance error of each hand joint point.

[0068] The embodiments of the present invention can effectively improve the accuracy and robustness of hand 3D pose estimation by performing high-precision mapping and joint optimization of image pixels and three-dimensional spatial geometric information.

[0069] Based on the above embodiments, the three-dimensional coordinates of the hand joints are determined in the following way: Based on the rotational degrees of freedom of the hand joints, determine the rotational transformation operator of the hand joints; The pose information of the hand joints is determined based on the rotational degrees of freedom and the rotational transformation operator. The three-dimensional coordinates of the hand joints are determined based on the posture information of the hand joints and the position transformation matrix of each finger.

[0070] Based on the differences in rotational degrees of freedom of the hand joints, different rotational transformation operators are used. Through the layer-by-layer superposition of quaternions, the posture is transmitted from the wrist to the fingertips, ultimately obtaining the relative postures of all hand joints. It should be understood that the joint postures of the fingers (represented by quaternions) (From the wrist) Initially, the posture of the parent joint is passed down layer by layer through the rotation of the child joint itself. For example, the metacarpal joints of the index finger. attitude It is formed by the basic posture of the wrist combined with its own rotation (through...) (Transformation) to obtain; the proximal metacarpal joint of the index finger attitude This refers to the posture of the metacarpal joints. Then superimposed its own rotation (through) Transformation); intermediate joints distal joints By analogy, the posture of the previous joint plus its own rotational transformation is used to achieve a chain-like transmission of posture from the wrist to the fingertips.

[0071] It should be understood that different joints in the human body have different degrees of rotational freedom (3 degrees of freedom, 2 degrees of freedom, 1 degree of freedom), therefore different operators are used to model rotation.

[0072] Quaternions are a tool for describing 3D rotations, avoiding the Euler angle gimbal lock problem and enabling efficient superposition of rotational transformations through multiplication: if the parent joint pose is a quaternion... The sub-joint itself rotates into ( / / Then the sub-joint orientation is (Quaternion multiplication is used to superimpose the parent pose and child rotation). Finally, starting from the wrist, each joint superimposes its own rotation through quaternion multiplication, which allows the calculation of the relative pose information of all joints with respect to the wrist (i.e., the 3D rotation state of each joint).

[0073] For example, taking the index finger as an example, the posture information of the hand joints can be calculated in the following way: ; ; ; ; in, , , Let represent the rotation transformation operator, where Used to handle 3-DOF rotations Used to handle 2-DOF rotations. It is used to handle 1-DOF rotations, and uses different operators to convert rotation parameters into quaternion rotation transformations based on the differences in the joint rotation degrees of freedom. , , , These represent the quaternions of the postures of each joint of the index finger. Quaternions are mathematical tools for describing 3D rotations, which can efficiently superimpose rotational transformations through multiplication, avoiding the gimbal lock problem of Euler angles. This represents the rotational degrees of freedom of the metacarpal joints, which are 3-dimensional. The degree of rotation of the proximal metacarpal joint of the index finger is represented by two dimensions. , These represent the rotational degrees of freedom of the intermediate and distal joints of the index finger, respectively, each with a degree of freedom of 1 dimension.

[0074] For example, rotation transformation operator It can be calculated in the following ways: ; ; ; ; in, These represent the components of the swing motion, used to control the joint's movement. direction, The swaying motion belongs to the lateral swing / extension and retraction type of movement, such as the fingers swinging left and right, or forward and backward. Specifically, Indicates in Swing in direction. Indicates in Directional swing. The angle indicates the twist or curl. Twist is the twisting of a joint around its own long axis, while curl is the angle at which the fingers bend towards the palm, such as the bending angle of the fingertips. Indicates by The resulting vector is used to describe the direction and total amplitude of the Swing's swing. Representing vectors The modulus reflects the total amplitude of the swing; It means that it is by The derived angle is used to quantify the rotation angle of the Swing swing.

[0075] This represents the rotational degrees of freedom of the wrist joint, corresponding to It is the starting point for conveying posture. In Input Through trigonometric functions and vector magnitude This transforms a 3-DOF rotation into a quaternion rotation transformation.

[0076] For example, rotation transformation operator It can be calculated in the following ways: ; ; in, This represents the rotational degrees of freedom of the proximal metacarpal joint, which has two dimensions; in Input ,pass and This transforms a 2-DOF oscillation into a quaternion transformation.

[0077] For example, rotation transformation operator It can be calculated in the following ways: ; in, ,or ; , These represent the rotational degrees of freedom at the middle and distal ends of the finger, respectively, each with one degree of freedom. Input ,or ,pass and This transforms a single-axis rotation into a quaternion transformation (with all other components being 0).

[0078] To address the differences in degrees of freedom among different joints, a dedicated rotation transformation operator is used to transmit posture from the wrist to the fingertips by progressively superimposing rotations through quaternions. This ultimately provides a precise description of the 3D relative posture of all finger joints. Based on this, the motion characteristics of human joints are matched, and the problem of describing and transmitting 3D rotations is efficiently solved.

[0079] Furthermore, based on the posture information of the hand joints and the position transformation matrix of each finger, the three-dimensional coordinates of the hand joints are determined. For example, starting from the wrist, each finger has a set of basic recursive parameters that conform to the physiological structure of the human body. These basic recursive parameters exist in the form of a matrix, namely, a position transformation matrix or a basic recursive matrix.

[0080] It should be understood that the joints of the human finger form a chain structure (wrist → metacarpal bone → proximal phalanx → middle phalanx → distal phalanx → fingertip). Therefore, the calculation of joint position follows the logic of parent joint position + child joint offset under parent joint posture, which is achieved layer by layer through quaternion rotation + physiological basis matrix + size parameters. For example, child joint position = parent joint posture rotation "basic offset" + size scaling.

[0081] For example, taking the thumb as an example, the three-dimensional coordinates of the hand joints can be calculated in the following way: ； ; ; ; ; in, The matrix representing the position transformation of the thumb. , , , , They represent The first, second, third, fourth, and fifth rows. In one embodiment, It can be represented as: .

[0082] A quaternion representing the posture of the wrist joint. , , , These are quaternions representing the postures of each joint of the thumb. , , They represent The first, second, and third lines in the text. , , , , These represent the 3D position vectors of each joint of the thumb, thus obtaining the three-dimensional coordinates of each joint of the thumb.

[0083] The process of calculating the three-dimensional coordinates of the thumb's hand joints is as follows: based on the wrist posture... Based on the first row of the rotation thumb fundamental matrix (Rotate the physiologically default metacarpal offset to the direction of the current wrist posture), then multiply by the hand size. This gives the position of the thumb metacarpal joint relative to the wrist. The position of the thumb metacarpal joint is determined. Based on the second row of the rotation thumb-based matrix (The physiological offset of the proximal phalanx relative to the metacarpal bone), multiplied by the size of the hand. This yields the position of the proximal joint relative to the metacarpal joints (after superimposing the metacarpal joint positions, the global coordinates of the proximal joint are obtained). The orientation of the proximal joint of the thumb is then considered. Based on the third row of the rotation thumb base matrix Combined with palm size and the length coefficient of the first segment of the thumb (Fine-tuning the middle phalanx length) yields the position of the middle phalanx joint. The posture of the thumb's middle phalanx joint is taken into account. Based on the baseline, combined with the basic offset Palm size and length coefficient Calculate the position of the distal joint. Consider the posture of the distal joint of the thumb. Based on the baseline, combined with the basic offset Palm size and length coefficient Calculate the position of the fingertip.

[0084] Starting from the wrist, rotational information is transmitted using posture quaternions. Combined with the default offset of the physiological basis matrix, the overall scaling of the size coefficient, and the fine adjustment of the length coefficient, the 3D position of each joint is calculated layer by layer, ultimately achieving a precise and adjustable 3D model of the hand that conforms to human physiology.

[0085] This invention first matches a dedicated rotation transformation operator based on the rotational degrees of freedom of the hand joints to accurately and robustly describe the 3D rotational posture of each joint. Then, by combining posture information with a position transformation matrix that conforms to the physiological structure of the human body, it iteratively calculates the three-dimensional coordinates of the hand joints that conform to both the continuity of joint movement and the laws of human anatomy. Finally, it achieves high-precision, natural and reasonable 3D pose modeling of the hand, providing a reliable pose foundation for subsequent applications such as hand motion analysis and virtual interaction.

[0086] Based on the above embodiments, determining the smoothness constraint term according to the optimization variable includes: The smoothness constraint term is determined based on the difference between the optimization variables at two adjacent time points.

[0087] Through the design of optimization variables After recovering the joint poses that adapt to each user's different hand shape, the difference between the optimization variables at two adjacent time points is calculated, and the absolute value of the difference is used as a smoothness constraint term. The purpose is to keep the pose changes at adjacent time points smooth and continuous, because the human hand movement itself is continuous and there will be no sudden and drastic changes. This constraint can ensure that the change of pose over time conforms to the physiological movement law.

[0088] For example, the smoothness constraint term can be calculated in the following way. : ; in, Indicates in Optimization variables at time intervals Indicates in Optimization variables at any given time.

[0089] This invention calculates the difference between optimization variables at two adjacent time points and uses the absolute value of the difference as a smoothness constraint. This constrains the variation range of pose parameters at adjacent time points, making the reconstructed hand movements (such as waving and grasping) more consistent with the actual continuous transition patterns of the human body, avoiding unnatural effects such as abrupt changes and twitching. On the other hand, the smoothness constraint reduces the oscillation of optimization results between adjacent frames, helping the algorithm to converge more stably and quickly to a reasonable solution that satisfies both single-frame observation and motion continuity.

[0090] Based on the above embodiments, determining the curvature error term according to the curvature of each finger includes: The curvature error term is determined based on the curvature and confidence level of each finger, and the angle of rotation of each hand joint of each finger about the x-axis.

[0091] The curvature of each finger is calculated by rotating each joint around its [joint]. The angle of rotation of the axis, that is, the angle around which each joint rotates. The sum of the rotation angles of the axis (because finger bending mainly involves the joints rotating around). (rotational motion of the axis), and the calculation method differs for different fingers due to the different number of joints.

[0092] For example, taking the thumb as an example, the degree of bending of the thumb The calculation method is as follows: ; in, Indicates the metacarpal joint circumference The rotation angle of the axis.

[0093] The curvature of other fingers The calculation method is as follows: ; in, Indicates the pericarp joint circumference The rotation angle of the axis.

[0094] The formula for calculating the curvature error term is: ; in, This represents the model's predicted bending degree of the five fingers (one value each for the thumb, index finger, middle finger, ring finger, and little finger, forming a 5×1 vector). This indicates the observed curvature of each finger (obtained through images, sensors, etc.). It is a 5×1 vector; This represents the original error vector between the predicted and observed curvature of each finger; The covariance (i.e., confidence level) represents the observed curvature of each finger. It is a 5×1 vector; This means converting the covariance vector into a diagonal matrix (only the diagonal lines have values, and the off-diagonal lines are 0), which facilitates subsequent weighting by confidence level. This represents the inverse of the diagonal matrix (the diagonal elements are the reciprocals of the covariance). Based on this, fingers with small covariance (high confidence) have large reciprocals of their covariance, which amplifies the error weight of that finger; fingers with large covariance (low confidence) have small reciprocals, which reduces the error weight. This represents the error vector weighted by confidence level, which increases the proportion of reliable observations in the error calculation and decreases the proportion of unreliable observations, thereby improving the robustness of the error term. This represents calculating the square of the Euclidean norm over the weighted error vector, transforming the vector error into a scalar error, which facilitates minimizing the total error in optimization algorithms.

[0095] This invention, through the design of a curvature error term, ensures that the model's predicted finger curvature aligns with actual observations, while also addressing observation noise (such as a finger being obscured, resulting in low observation confidence and reducing its interference with optimization) through confidence weighting. Ultimately, when optimizing hand pose, the degree of finger curvature conforms to physiological laws and accurately matches the observation results of the actual scene.

[0096] Based on the above embodiments, the hand pose estimation method further includes: Based on the joint length coefficients of each finger, the overall hand state parameters are determined; During the iterative optimization process, monitor the inter-frame changes in the total parameters at the finger joint scale. If the inter-frame change is less than a first preset threshold at multiple consecutive adjacent time points, the hand state is determined to be stable. If the inter-frame change exceeds a second preset threshold for multiple consecutive adjacent time points, it is determined that the user has changed.

[0097] By detecting the continuity of total changes in finger joint dimensions, the system can intelligently determine whether the finger joint dimensions are stable or whether the user has changed, and then dynamically adjust and optimize the strategy.

[0098] During the iterative optimization process, the scale parameter is determined. Calculate the total parameters at the finger joint scale based on the changes in factors: ; in, Indicates in Total parameters of finger joint scale at any given time. It equals the sum of the norms of the joint length coefficients of the thumb, index finger, middle finger, ring finger, and little finger.

[0099] When continuous Secondary satisfaction At this point, it is considered that the user's finger joints have been estimated to be relatively stable, and the optimization of finger joint scales for different users has been completed. Subsequent optimization processes will then set... It is a fixed value.

[0100] When continuous Secondary satisfaction At this point, it's assumed that the current user of the device may have changed, causing a change in the dimensions of the finger joints. It should be understood that significant and continuous changes in finger joint dimensions are due to substantial differences in the geometric characteristics (length, thickness, etc.) of the fingers among different users, and this change may be caused by a user change. In this case, the optimization process will be re-optimized. To adapt to the finger size of new users and ensure the accuracy of subsequent pose estimation.

[0101] The embodiments of the present invention improve computational efficiency and ensure pose estimation accuracy by detecting the continuity of scale changes and dynamically balancing scale stability and user change adaptability.

[0102] The hand pose estimation device provided by the present invention will be described below, and the hand pose estimation device described below can be referred to in correspondence with the hand pose estimation method described above.

[0103] refer to Figure 4 The hand pose estimation device provided by the present invention includes a hand parameter determination module 401, an optimization variable determination module 402, an optimization function generation module 403, and a hand pose estimation module 404.

[0104] The hand parameter determination module 401 is used to determine the pixel position and relative distance of each hand joint point, as well as the curvature of each finger, based on the binocular images of the hand. The optimization variable determination module 402 is used to determine the optimization variables of the hand pose based on the curvature of each finger. The optimization function generation module 403 is used to generate an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger. The hand pose estimation module 404 is used to iteratively optimize the hand pose according to the optimization function to obtain the hand pose estimation result.

[0105] The hand pose estimation device provided in this invention determines the pixel position and relative distance of each hand joint and the curvature of each finger based on a binocular image of the hand. It then determines optimization variables for the hand pose based on the curvature of each finger. An optimization function for the hand pose is generated based on the optimization variables, the pixel position and relative distance of each hand joint, and the curvature of each finger. The hand pose is iteratively optimized using the optimization function to obtain the hand pose estimation result. This invention improves the accuracy of hand pose estimation by combining the pixel position and relative distance of each hand joint and the curvature of the fingers for iterative optimization.

[0106] In one embodiment, the optimization function generation module 403 is further configured to: Based on the pixel position and relative distance of each hand joint, the projection error and relative distance error terms are determined; Based on the optimization variables, determine the smoothness constraint terms; Based on the curvature of each finger, determine the curvature error term; The sum of the projection error and relative distance error, the smoothness constraint, and the curvature error is used as the optimization function for the hand pose.

[0107] In one embodiment, the optimization function generation module 403 is further configured to: The projection error is determined based on the pixel position and confidence level of each hand joint in the image coordinates, as well as the three-dimensional coordinates of each hand joint. The relative distance error is determined based on the relative distance and confidence level of each hand joint in each camera coordinate, the three-dimensional coordinates of each hand joint, the palm size, and the coordinates of the proximal metacarpal joint of the middle finger. The projection error and relative distance error terms are determined based on the projection error and the relative distance error.

[0108] In one embodiment, the optimization function generation module 403 is further configured to: Based on the rotational degrees of freedom of the hand joints, determine the rotational transformation operator of the hand joints; The pose information of the hand joints is determined based on the rotational degrees of freedom and the rotational transformation operator. The three-dimensional coordinates of the hand joints are determined based on the posture information of the hand joints and the position transformation matrix of each finger.

[0109] In one embodiment, the optimization function generation module 403 is further configured to: The smoothness constraint term is determined based on the difference between the optimization variables at two adjacent time points.

[0110] In one embodiment, the optimization function generation module 403 is further configured to: The curvature error term is determined based on the curvature and confidence level of each finger, and the angle of rotation of each hand joint of each finger about the x-axis.

[0111] In one embodiment, the optimization variable determination module 402 is further configured to: The degree of flexure of each finger is determined based on the rotational degrees of freedom of each of the hand joints; The optimization variables for the hand pose are determined based on the curvature of each finger, the size of the palm, and the joint length coefficient of each finger.

[0112] In one embodiment, the hand pose estimation module 404 is further configured to: Based on the joint length coefficients of each finger, the overall hand state parameters are determined; During the iterative optimization process, monitor the inter-frame changes in the total parameters at the finger joint scale. If the inter-frame change is less than a first preset threshold at multiple consecutive adjacent time points, the hand state is determined to be stable. If the inter-frame change exceeds a second preset threshold for multiple consecutive adjacent time points, it is determined that the user has changed.

[0113] Figure 5 An example is a schematic diagram of the physical structure of a head-mounted display device, such as... Figure 5 As shown, the head-mounted display device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a hand pose estimation method. This method includes: determining the pixel position and relative distance of each hand joint and the curvature of each finger based on a binocular image of the hand; determining optimization variables for the hand pose based on the curvature of each finger; generating an optimization function for the hand pose based on the optimization variables, the pixel position and relative distance of each hand joint, and the curvature of each finger; and iteratively optimizing the hand pose according to the optimization function to obtain a hand pose estimation result.

[0114] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hand pose estimation method provided by the above methods. The method includes: determining the pixel position and relative distance of each hand joint and the curvature of each finger based on a binocular image of the hand; determining optimization variables for the hand pose based on the curvature of each finger; generating an optimization function for the hand pose based on the optimization variables, the pixel position and relative distance of each hand joint and the curvature of each finger; and iteratively optimizing the hand pose based on the optimization function to obtain a hand pose estimation result.

[0116] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the hand pose estimation method provided by the methods described above. The method includes: determining the pixel position and relative distance of each hand joint and the curvature of each finger based on a binocular image of the hand; determining optimization variables for the hand pose based on the curvature of each finger; generating an optimization function for the hand pose based on the optimization variables, the pixel position and relative distance of each hand joint, and the curvature of each finger; and iteratively optimizing the hand pose according to the optimization function to obtain a hand pose estimation result.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating hand pose, characterized in that, include: Based on the binocular images of the hand, determine the pixel position and relative distance of each hand joint, as well as the curvature of each finger; Based on the degree of curvature of each finger, determine the optimization variables for hand pose; An optimization function for hand pose is generated based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger. The hand pose is iteratively optimized according to the optimization function to obtain the hand pose estimation result.

2. The hand pose estimation method according to claim 1, characterized in that, The step of generating an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger includes: Based on the pixel position and relative distance of each hand joint, the projection error and relative distance error terms are determined; Based on the optimization variables, determine the smoothness constraint terms; Based on the curvature of each finger, determine the curvature error term; The sum of the projection error and relative distance error, the smoothness constraint, and the curvature error is used as the optimization function for the hand pose.

3. The hand pose estimation method according to claim 2, characterized in that, The step of determining the projection error and relative distance error terms based on the pixel positions and relative distances of each of the hand joints includes: The projection error is determined based on the pixel position and confidence level of each hand joint in the image coordinates, as well as the three-dimensional coordinates of each hand joint. The relative distance error is determined based on the relative distance and confidence level of each hand joint in each camera coordinate, the three-dimensional coordinates of each hand joint, the palm size, and the coordinates of the proximal metacarpal joint of the middle finger. The projection error and relative distance error terms are determined based on the projection error and the relative distance error.

4. The hand pose estimation method according to claim 3, characterized in that, The three-dimensional coordinates of the hand joints are determined based on the following method: Based on the rotational degrees of freedom of the hand joints, determine the rotational transformation operator of the hand joints; The pose information of the hand joints is determined based on the rotational degrees of freedom and the rotational transformation operator. The three-dimensional coordinates of the hand joints are determined based on the posture information of the hand joints and the position transformation matrix of each finger.

5. The hand pose estimation method according to claim 2, characterized in that, The step of determining the smoothness constraint term based on the optimization variables includes: The smoothness constraint term is determined based on the difference between the optimization variables at two adjacent time points.

6. The hand pose estimation method according to claim 2, characterized in that, The step of determining the curvature error term based on the curvature of each finger includes: The curvature error term is determined based on the curvature and confidence level of each finger, and the angle of rotation of each hand joint of each finger about the x-axis.

7. The hand pose estimation method according to claim 1, characterized in that, The process of determining the optimization variables for hand pose based on the curvature of each finger includes: The degree of flexure of each finger is determined based on the rotational degrees of freedom of each of the hand joints; The optimization variables for the hand pose are determined based on the curvature of each finger, the size of the palm, and the joint length coefficient of each finger.

8. The hand pose estimation method according to any one of claims 1 to 7, characterized in that, The hand pose estimation method further includes: Based on the joint length coefficient of each finger, the overall hand state parameters are determined; During the iterative optimization process, monitor the inter-frame changes in the total parameters at the finger joint scale; If the inter-frame change is less than a first preset threshold at multiple consecutive adjacent time points, the hand state is determined to be stable. If the inter-frame change exceeds a second preset threshold for multiple consecutive adjacent time points, it is determined that the user has changed.

9. A hand pose estimation device, characterized in that, include: The hand parameter determination module is used to determine the pixel position and relative distance of each hand joint point, as well as the curvature of each finger, based on the binocular images of the hand. The optimization variable determination module is used to determine the optimization variables of the hand pose based on the curvature of each finger. An optimization function generation module is used to generate an optimization function for hand pose based on the optimization variables, the pixel positions and relative distances of each hand joint, and the curvature of each finger. The hand pose estimation module is used to iteratively optimize the hand pose according to the optimization function to obtain the hand pose estimation result.

10. A head-mounted display device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the hand pose estimation method as described in any one of claims 1 to 8.

Images