A hand tracking method and system based on kinematics optimization in an AR environment

Abstract

The application provides a hand tracking method and system based on kinematic optimization in an AR environment, wherein the method comprises the following steps: constructing an anatomical bone template; collecting a depth image and a color image of an operator's hand through a camera to generate a hand point cloud; forming an initial bone model by using a light convolutional neural network; constructing a residual vector based on the hand point cloud, the initial bone model, and length constraints of each bone and activity range constraints of each joint; wherein the optimization vector to be solved for the bone model comprises a root position of the bone model and rotation parameters of N joint nodes; solving an error function iteratively by using a nonlinear least square method, applying the finally solved optimization vector to corresponding joints of a virtual hand model, driving a skin mesh to update, realizing synchronization of a virtual hand animation and a real hand action, and completing augmented reality interaction; and the precision, physical rationality and reliability of the virtual hand model are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a hand tracking method and system based on kinematic optimization in an AR environment. Background Technology

[0002] In augmented reality (AR)-assisted industrial operations training, trainees often need to wear AR devices to observe and imitate standard hand operation procedures in a virtual-real integrated environment. For example, in training for assembling precision parts (such as motor assemblies), due to the large number of parts, their high cost, and their susceptibility to damage, directly performing repeated operations on actual parts would lead to high material waste and safety risks. Therefore, a common practice is to first have workers practice assembling virtual parts in an AR environment until they are proficient in the movements before proceeding to actual operations.

[0003] In this application scenario, the AR system needs to be able to track the position and posture of the worker's real hand in real time and drive the virtual hand model to make completely consistent movements. Otherwise, the trainee will clearly feel a "disconnect" between the virtual hand and their own movements, such as the virtual fingers not matching the actual fingers, abnormal joint bending direction, and virtual fingers penetrating parts. These problems not only destroy the visual realism, but more importantly, they can lead to the trainee forming incorrect muscle memory or operating habits, thereby weakening the actual effect of virtual training, and even causing assembly errors or parts damage in actual operation.

[0004] Currently, the field of hand tracking mainly employs methods such as pure deep learning regression, iterative nearest point (ICP) registration, or inverse kinematics (IK). These methods generally suffer from the following problems: either the output pose lacks biomechanical constraints such as bone length and joint range of motion, easily resulting in anatomically unreasonable twisted poses; or they only focus on surface fitting while ignoring the physical rationality of joint movements; or they rely on end-effector position input and suffer from multiple solution redundancy issues. These shortcomings make it difficult for existing technologies to simultaneously meet the requirements of AR applications for tracking accuracy, pose realism, and real-time performance. Summary of the Invention

[0005] The purpose of this invention is to address the above problems by providing a kinematically optimized hand tracking method and system in an AR environment, so as to meet the requirements of tracking accuracy and posture realism in industrial AR training such as virtual assembly of precision parts.

[0006] In a first aspect, the present invention provides a hand tracking method based on kinematic optimization in an AR environment, comprising: An anatomical skeleton template is constructed, and the length constraints of each bone and the range of motion constraints of each joint are predefined; the anatomical skeleton template has N joints based on anatomy; The depth and color images of the operator's hands are captured by a camera to generate a hand point cloud. A lightweight convolutional neural network is used to perform initial pose estimation on the color image and depth image, and outputs the initial three-dimensional position of each joint in the camera coordinate system to form an initial skeleton model. Based on the hand point cloud, the initial skeletal model, and the length constraints of each bone and the range of motion constraints of each joint, a residual vector is constructed. And the sum of squares of the residual vectors is used as the error function; in, Let be the optimization vector to be solved, used to represent the pose of the optimized skeletal model; the optimization vector includes the root position of the optimized skeletal model and . Rotation parameters of each joint; the residual vector Calculated based on the skeletal model determined by the current optimization vector; The optimization vector that minimizes the error function is obtained by iteratively solving using the nonlinear least squares method. The root position and rotation parameters of the obtained optimization vector are then applied to the corresponding joints of the virtual hand model to drive the skin mesh update, thereby synchronizing the virtual hand animation with the real hand movements and completing the augmented reality interaction.

[0007] According to the technical solution provided by the present invention, the residual vector This includes: the first error term, the second error term, and the third error term corresponding to the current skeletal model; The first error term is calculated from the distance from each point in the hand point cloud to the surface of the skeletal model; The second error term is calculated from the deviation between the current rotation parameter and the neutral rotation parameter of each joint; the neutral rotation parameter is the median rotation parameter in the range of motion constraint of the joint. The third error term is calculated from the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template.

[0008] According to the technical solution provided by the present invention, the first error term is calculated according to the following method: For each point in the hand point cloud, find the nearest triangle on the surface of the current skeleton model; The distance from the hand point cloud points to the triangular face is calculated, which constitutes the first error term.

[0009] According to the technical solution provided by the present invention, the rotation parameter includes a rotation angle, the median rotation parameter includes a neutral position rotation angle, and the second error term is calculated according to the following method: For each joint, the deviation between its current rotation angle and the neutral rotation angle is multiplied by the ROM weight corresponding to the current joint. The ROM weight is dynamically adjusted by the basic weight of the joint based on the distance between the current rotation angle and the nearest ROM boundary. The ROM boundary is the upper or lower limit of the joint's range of motion.

[0010] According to the technical solution provided by the present invention, the third error term is calculated according to the following method: The deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template is obtained by multiplying the predetermined weight coefficient of the current bone; the predetermined weight coefficient of the current bone is defined in the anatomical skeleton template, and the predetermined weight coefficient of the bone gradually decreases from the wrist joint to the fingertip joint.

[0011] According to the technical solution provided by the present invention, the anatomical skeleton template includes 27 joint points, including 1 radiocarpal joint, 7 intercarpal joints, 5 metacarpal joints, 2 joints of the thumb, and 3 joints of each of the other four fingers; wherein, the radiocarpal joint is the root joint of the anatomical skeleton model, and the root position is the three-dimensional position of the radiocarpal joint in the camera coordinate system.

[0012] According to the technical solution provided by the present invention, the anatomical skeleton template is provided with four templates according to gender and left-right direction, and each anatomical skeleton template corresponds to a lightweight convolutional neural network model. The process of using a lightweight convolutional neural network to perform initial pose estimation on the color image and depth image, and outputting the initial 3D position of each joint in the camera coordinate system to form an initial skeletal model, specifically includes the following steps: Select the corresponding convolutional neural network model based on the anatomical skeleton template chosen by the user; The initial pose estimation is performed on the color image and depth image based on the selected convolutional neural network model, and the initial three-dimensional position of each joint in the camera coordinate system is output to form an initial skeleton model.

[0013] According to the technical solution provided by the present invention, the step of iteratively solving for the optimization vector that minimizes the error function using the nonlinear least squares method specifically includes the following steps: At the optimization vector corresponding to the current iteration step, a first-order Taylor expansion is performed on the residual vector to obtain a linear approximate expression of the residual vector in the neighborhood of the optimization vector; the linear approximate expression is composed of the sum of the current residual vector value and the Jacobian matrix multiplied by the increment of the optimization vector, where the Jacobian matrix is ​​a matrix composed of the first-order partial derivatives of the residual vector with respect to each component of the optimization vector; Substituting the linear approximation into the error function yields an approximate error function for the optimized vector increment; The optimal vector increment is obtained by taking the derivative of the approximation error function with respect to the optimization vector increment and setting the derivative to zero. The optimization vector is updated incrementally based on the obtained optimization vector. Repeat the above steps until the preset convergence condition is met or the number of iterations reaches the preset upper limit, and use the optimized vector obtained from the last iteration as the solution result.

[0014] Secondly, the present invention provides a hand tracking system based on kinematic optimization in an AR environment, comprising: The acquisition module is used to acquire depth and color images of the operator's hands; Processing module: Used to generate hand point clouds based on depth and color images of the operator's hands; An anatomical skeleton template is constructed, and the length constraints of each bone and the range of motion constraints of each joint are predefined; the anatomical skeleton template has N joints based on anatomy; A lightweight convolutional neural network is used to perform initial pose estimation on the color image and depth image, and outputs the initial three-dimensional position of each joint in the camera coordinate system to form an initial skeleton model. Based on the hand point cloud, the initial skeletal model, and the length constraints of each bone and the range of motion constraints of each joint, a residual vector is constructed. And the sum of squares of the residual vectors is used as the error function; in, Let be the optimization vector to be solved, used to represent the pose of the optimized skeletal model; the optimization vector includes the root position of the optimized skeletal model and . Rotation parameters of each joint; the residual vector Calculated based on the skeletal model determined by the current optimization vector; The optimization vector that minimizes the error function is obtained by iteratively solving using the nonlinear least squares method. The root position and rotation parameters of the obtained optimization vector are then applied to the corresponding joints of the virtual hand model to drive the skin mesh update, thereby synchronizing the virtual hand animation with the real hand movements and completing the augmented reality interaction.

[0015] According to the technical solution provided by the present invention, the residual vector This includes: the first error term, the second error term, and the third error term corresponding to the current skeletal model; The first error term is calculated from the distance from each point in the hand point cloud to the surface of the skeletal model; The second error term is calculated from the deviation between the current rotation parameter and the neutral rotation parameter of each joint; the neutral rotation parameter is the median rotation parameter in the range of motion constraint of the joint. The third error term is calculated from the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention pre-constructs an anatomical skeleton template containing N joint points and constraints on the length and range of motion of each bone. During the optimization process, the distance from the point cloud to the surface of the skeleton model, the deviation of the joint angle from the neutral position, and the deviation of the bone length from the template are all incorporated into the residual vector. This ensures that the skeleton model corresponding to the final optimized vector satisfies both the matching with the real hand point cloud and biomechanical rationality. Specifically, the length constraint prevents the bones from being unreasonably stretched or compressed, and the range of motion constraint avoids anatomically impossible twisting postures of the joints. Thus, when the virtual hand animation performs fine operations such as assembly and grasping, the joint bending angle and the relative position of the fingers remain consistent with the actual human hand.

[0017] This invention uses the initial 3D joint positions output by a lightweight convolutional neural network as the starting point for optimization. It then combines this with a nonlinear least squares method to iteratively solve the residual vector. Compared to existing methods that rely solely on deep learning regression or surface registration, this approach eliminates the need for large amounts of labeled data or dense particle sampling, resulting in controllable computational overhead and enabling real-time tracking on AR devices. Furthermore, because length and range of motion constraints are directly embedded during optimization, the final virtual hand pose maintains anatomical accuracy without additional post-processing. Finally, by directly applying the root position and rotation parameters from the optimized vector to the corresponding joints of the virtual hand model to drive skin mesh updates, it avoids accuracy loss between different modules, ensuring synchronization between the virtual hand animation and real hand movements. This meets the requirements for realism and training effectiveness in scenarios such as industrial virtual assembly.

[0018] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this invention do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a hand tracking method based on kinematic optimization in an AR environment, as provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the unfolded process of step S300 in a kinematic optimization-based hand tracking method in an AR environment, as provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the unfolded process of step S500 in an AR-based hand tracking method provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. This description is merely illustrative and explanatory, and should not be construed as limiting the scope of protection of the present invention in any way. Specifically, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0023] To facilitate understanding of the present invention, a brief introduction to related technologies is given first: In augmented reality (AR) technology, the accurate tracking of the virtual hand model to the position and posture of the real hand is crucial, as it is a core prerequisite for achieving a natural, immersive, and believable virtual-real fusion experience. If there is a deviation or delay in tracking, the user will immediately perceive a disconnect between the virtual hand and their own body movements. This not only undermines the visual realism but also directly disrupts the user's trust in the operation and sense of immersion.

[0024] The following techniques are mainly used to make a virtual image's hand model follow the user's real hand movements: Pure deep learning regression methods, trained on massive amounts of data, can directly guess the coordinates of key points from images. While fast, they often lack accuracy and pose accuracy. For example, joint positions regressed by pure CNN / RNN lack skeletal connection constraints, making anatomically impossible phenomena such as bone stretching, compression, and reverse bending prone to occur; joint angle errors are greater than 5°.

[0025] Iterative Closest Point (ICP) is a fitting strategy that focuses on making the model's surface and the real-world hand point cloud (i.e., scanned point cloud data) highly coincide in space. However, this can lead to outputs that violate the range of motion of human joints, resulting in "joint distortion." It is often combined with algorithms such as deep learning and particle filtering. When ICP is fused with deep learning, a neural network is used to directly learn the alignment transformation between two point clouds or predict matching point pairs. This method relies on large-scale and diverse training datasets, limiting its generalization ability. When ICP is fused with particle filtering, a global search is performed by maintaining numerous candidate poses (particles) to provide reliable initial values ​​for ICP. This fusion requires a large number of particles to ensure accuracy, resulting in huge computational costs and poor real-time performance.

[0026] Inverse Kinematics (IK): IK is an inference strategy that infers joint angles from known end-effector positions (such as fingertips). However, its updates depend on external input; without this end-effector information, it cannot function. Therefore, IK is often combined with other algorithms. When IK is fused with deep learning, the neural network is trained to directly learn the mapping from the end-effector target pose to joint angles. Human joints have high redundancy, meaning that the same target pose corresponds to countless joint poses. This can cause contradictions in the training data, ultimately leading to an averaged and unreasonable pose output by the network. When IK is fused with particle filtering, a large number of weighted particles (i.e., possible joint pose assumptions) are used to approximate the posterior distribution of the motion state. To achieve high accuracy, a large number of particles are usually required, resulting in a huge computational load and poor real-time performance.

[0027] In view of this, embodiments of the present invention provide a hand tracking method and system based on kinematic optimization in an AR environment. The method constructs an anatomical skeleton template, uses a lightweight convolutional neural network for initial pose estimation, constructs a residual vector based on hand point cloud, skeleton length constraints and joint range of motion constraints, iteratively solves the optimization vector using nonlinear least squares method, and finally applies the optimized pose parameters to a virtual hand model to achieve high-precision, high-reasonability and low-latency hand tracking.

[0028] To make the technical solution of the present invention clearer and easier to understand, the following describes a hand tracking method based on kinematic optimization in an AR environment provided by an embodiment of the present invention.

[0029] It should be noted that this method can be applied to the processing module of AR devices (such as AR glasses and AR headsets), and is implemented by the processor executing the corresponding computer program.

[0030] like Figure 1 As shown in the figure, this is a flowchart illustrating a hand tracking method based on kinematic optimization in an AR environment provided in this embodiment. The method includes the following steps: S100. Construct an anatomical skeleton template and predefine the length constraints of each bone and the range of motion constraints of each joint; the anatomical skeleton template has N joints based on anatomy; Specifically, in this embodiment of the invention, to accommodate different users, the templates are divided into four types: male left hand, male right hand, female left hand, and female right hand, each stored in a configuration file. For example, the anatomical template contains 27 joints, namely one radiocarpal joint, seven intercarpal joints, five metacarpal joints, two joints of the thumb, and three joints each of the other four fingers; wherein, the radiocarpal joint is the root joint of the anatomical skeletal model, and the root position is the three-dimensional position of the radiocarpal joint in the camera coordinate system.

[0031] For each anatomical skeleton template, the predefined structure mainly includes three core parts: joint topology, bone length constraint, and joint range of motion constraint.

[0032] The length of each bone can be preset to a fixed value (e.g., 70mm for the metacarpal, 45mm for the proximal phalanx, 25mm for the middle phalanx, and 20mm for the distal phalanx; the specific values ​​may vary slightly depending on the gender template).

[0033] The range of motion of each joint is set according to anatomical data. For example, the flexion range of the metacarpophalangeal joint is 0° to 90°, and the neutral position is 45°; the flexion range of the interphalangeal joint is 0° to 80°, and the neutral position is 40°. Therefore, the range of motion of the joints can actually be determined by recording the upper limit angle and the lower limit angle of the range of motion of each joint. The specific values ​​are not specifically limited here.

[0034] Finally, the joint topology: Each template directly defines 27 joints and their parent-child connections based on anatomical data, forming a tree-like hierarchical distribution (see Table 1 below). The radiocarpal joint (located at the wrist) is taken as the root node, and the remaining nodes are connected step by step according to the anatomical hierarchy. In other words, the spatial position of the radiocarpal joint (three-dimensional coordinates in the camera coordinate system) is the root position in the optimization vector, and the positions of other joints are determined step by step through the root position and the rotation parameters of each level of joint.

[0035] Table 1. Examples of tree-like hierarchical distribution of joint topology

[0036] It should be noted that the 27 joint points in this embodiment do not include the fingertip point; the length of the distal phalanx of each finger is constrained by the distance between the last joint (such as the proximal joint of the distal phalanx) and the fingertip point. The fingertip point is not an independent optimization variable, but is naturally determined by the length and direction of the bone.

[0037] S200: Captures depth and color images of the operator's hands using a camera, and generates hand point clouds; Specifically, by using an RGB-D camera integrated into the AR device (such as the Intel RealSense D455), color and depth images are simultaneously acquired at a frame rate of 30fps or higher. By utilizing the intrinsic and extrinsic parameters of the depth image, depth pixels are converted into three-dimensional spatial points. After filtering out background points, a 3D point cloud of the hand area can be generated. .

[0038] S300: Use a lightweight convolutional neural network to perform initial pose estimation on color and depth images, output the initial three-dimensional position of each joint in the camera coordinate system, and form an initial skeleton model. Based on the foregoing, considering the significant geometric differences in real human hands due to gender and left / right hand orientation, the anatomical skeleton template has four templates set according to gender and left / right direction. Correspondingly, each sub-template is pre-trained with a dedicated lightweight convolutional neural network model. In other words, the system maintains a total of four lightweight CNN models, each specifically designed for initial joint position regression on hand images of a particular category (such as the male right hand).

[0039] Furthermore, the lightweight convolutional neural network used in this embodiment has the following specific structure: the input layer receives a stitched image tensor with a size of 128×128×4; it sequentially passes through a 3×3 convolutional layer (32 output channels, stride 2), a depthwise separable convolutional layer (64 output channels), an inverse residual module (128 output channels), a global average pooling layer, a fully connected layer (1024-dimensional output), and finally outputs an 81-dimensional vector through a fully connected layer without an activation function. The total number of parameters is approximately 1.2M, and the time for a single forward inference on a mobile AR device is less than 5ms.

[0040] Combination Figure 2 Step S300 specifically includes the following steps: S301. Select the corresponding convolutional neural network model based on the anatomical skeleton template selected by the user; S302. Based on the selected convolutional neural network model, perform initial pose estimation on the color image and depth image, output the initial three-dimensional position of each joint in the camera coordinate system, and form an initial skeleton model. During the initialization phase of the AR system or when a user uses it for the first time, the system prompts the operator to enter their gender through a graphical interface. Based on the user's selection, the system selects the corresponding template from four anatomical skeleton templates and reads the CNN model file associated with the selected template from local storage or network cache.

[0041] Since users typically use both hands simultaneously in real-world applications, the system automatically selects either male or female hand templates after gender selection. However, this doesn't preclude each gender from having different left and right hand templates (while the left and right hands are topologically symmetrical, they are not strictly mirror images in actual form; for example, right-handed individuals usually have slightly more developed right-hand muscles than their left-hand muscles, and there are subtle differences in skeletal parameters). By setting separate templates and CNN models for the left and right hands, the system can more accurately adapt to the actual anatomical features of the left and right hands, avoiding systematic errors introduced by using a single model or a simple mirror image relationship. In scenarios where both hands are tracked simultaneously, the independent templates and models for each hand ensure that the initial pose estimation accuracy for each hand reaches its optimal level.

[0042] Next, the system reads the CNN model file associated with the selected template from local storage or network cache (e.g., if the user selects "male right hand," then model_male_right.pth is loaded). This model is structured as a lightweight convolutional neural network (such as MobileNetV3 or ShuffleNetV2), whose input layer accepts a concatenated tensor of a color image and a depth image, and whose output layer has 81 dimensions (corresponding to the x, y, z coordinates of 27 joints). During training, the model uses only labeled data matching its own template category; therefore, its internal feature extraction layer is optimized for the proportions, shape, and joint distribution of this type of hand.

[0043] Subsequently, when processing color and depth images, the selected convolutional neural network model will use the color image I captured in the current frame. rgb With depth image I depth After preprocessing (e.g., scaling to 128×128 pixels, normalization, etc.), they are concatenated into a four-channel input tensor. Then Feeding the loaded CNN model into a forward propagation process directly yields an 81-dimensional output vector. and the output vector By sequentially dividing the coordinates into 27 three-dimensional points, the initial three-dimensional position of each joint in the camera coordinate system is obtained.

[0044] After obtaining the initial 3D positions, the initial skeletal model can be constructed. This process includes organizing the previously output 27 3D point coordinates into a tree-like initial skeletal model based on the predefined joint topology in the selected anatomical skeletal template. This initial skeletal model includes the spatial position of each joint and the initial length of each bone (which can be calculated from the difference in coordinates between adjacent joints).

[0045] It should be noted that, since the CNN model is trained on similar hand shapes, the deviation between the bone lengths and the preset template lengths in the initial skeletal model is usually small, and the joint angles are also generally within a reasonable range. This provides high-quality initial values ​​for subsequent kinematic optimization. Furthermore, after obtaining the initial skeletal model, initial optimization vectors are constructed. As follows: Root position Take the three-dimensional coordinates of the radiocarpal joint in the initial skeletal model; for the first... j For each joint, based on its positional relationship with its parent joint in the initial skeletal model, a 3D rotation vector is calculated from the parent joint coordinate system to the child joint coordinate system (e.g., Euler angles or axis angles are calculated using the angle between the vectors), which serves as the initial rotation parameter for that joint. Therefore, the initial optimization vector is obtained as the starting point for the nonlinear least squares iteration.

[0046] S400, based on hand point cloud, initial skeletal model, and length constraints of each bone and range of motion constraints of each joint, constructs residual vectors. And the sum of squares of the residual vectors is used as the error function; in, Let be the optimization vector to be solved, used to represent the pose of the optimized skeletal model; the optimization vector includes the root position of the optimized skeletal model and . Rotation parameters of each joint; residual vector Calculated based on the skeletal model determined by the current optimization vector; The core objective of step S400 is to establish a quantifiable objective function that expresses the differences between the current skeletal model and real hand observations (hand point clouds) and anatomical data as an error function about the optimization vector, and to minimize the value of this error function through subsequent optimization.

[0047] Specifically, the residual vector This includes: the first error term, the second error term, and the third error term corresponding to the current skeletal model; The first error term is calculated from the distance from each point in the hand point cloud to the surface of the skeletal model. The calculation process is as follows: for each point in the hand point cloud, find the nearest triangle on the surface of the current skeletal model; calculate the distance from the hand point cloud point to the triangle, which constitutes the first error term.

[0048] Next, we define the optimization vector to be solved. The pose of the skeletal model to be solved is represented by the following formula (1): Formula (1); In the formula, This refers to the root position of the skeletal model (i.e., the three-dimensional coordinates of the radiocarpal joint in the camera coordinate system). For the first The rotational parameters of each joint (expressed in Euler angles, representing the rotation angles about the x, y, and z axes respectively). The total number of joints; Transpose matrix.

[0049] To efficiently calculate the distance from the hand point cloud to the surface of the skeletal model, this embodiment uses a capsule to approximate the envelope surface of each bone, and discretizes the capsule surface into a triangular mesh; for example, for the bone connecting joint a and joint b, the capsule is defined as a line segment. With axis and radius A combination of a cylinder and two hemispheres at either end. Where the radius... The radius is taken as the average anatomical radius corresponding to the bone (e.g., the radius of the metacarpal is about 8 mm, and the radius of the phalanges is about 5 mm). The capsules of adjacent bones connect naturally at the joints to form a continuous, simplified skin surface, i.e., the surface of the bone model. This surface relies solely on the skeleton pose and a predefined radius, offering advantages such as fast computation speed and differentiability. Accordingly, by dividing the surface into a triangular mesh with a preset precision (e.g., 8 subdivision points on each circumference), the triangular surfaces required for calculating the first error term can be obtained.

[0050] Below, the first error term measures the surface of the current skeletal model. The degree of fit between the point cloud and the actual hand; for each point in the point cloud Calculate its value onto the surface of the bone model. The shortest Euclidean distance; in practical calculations, because Composed of capsules, this minimum distance can be obtained by calculating the distance from a point to a line segment and then subtracting the radius of the capsule. Specifically, let a point... The projected distance to the skeletal line segment is The distance to the surface of the skeletal capsule is then... (If the projection falls outside the endpoints of the line segment, additional processing is required.) Finally, the minimum distance of all skeletal capsule bodies is taken as the final distance. .

[0051] Based on the above calculations, the first error term is finally obtained as formula (2): Formula (2); In the formula, This is the first error term; These are the final distances from the 1st to the Mth points in the hand point cloud to the surface of the current skeletal model.

[0052] Here, the second error term is calculated from the deviation between the current rotation parameter and the neutral rotation parameter of each joint; the neutral rotation parameter is the median rotation parameter in the range of motion constraint of the joint; since the rotation parameter includes the rotation angle and the median rotation parameter includes the neutral rotation angle, the calculation process of the second error term is as follows: for each joint, the deviation between its current rotation angle and the neutral rotation angle is multiplied by the ROM weight corresponding to the current joint; the ROM weight is dynamically adjusted by the basic weight of the joint according to the distance between the current rotation angle and the nearest ROM boundary, and the ROM boundary is the upper or lower limit of the range of motion of the joint.

[0053] In this embodiment of the invention, the second error term is used to constrain the rotation angle of each joint to be close to the anatomically neutral position, and automatically increases the penalty weight when approaching the boundary of the range of motion.

[0054] For the There are 1 joint, and let its current rotation angle vector be... The neutral position rotation angle vector is The lower limit vector of the activity range is The upper limit vector is Typically, the three degrees of freedom of the ROM are independent of each other, so they can be processed separately.

[0055] To simplify the calculation process, the following explanation uses a single degree of freedom as an example. In actual implementation, the three degrees of freedom are calculated independently and then summed, but the calculation principle is the same.

[0056] Define angular deviation The formula is as follows (3): Formula (3); Define the distance from the current angle to the nearest ROM boundary. The following formula (4): ; Dynamic ROM weight The calculation is shown in the following formula (5): Formula (5); In the formula, For the example of a single degree of freedom, the first The rotation angle of each joint, For the example of a single degree of freedom, the first The neutral rotation angle of each joint; and The lower and upper bounds of the activity range for a single degree of freedom example; The basic weights preset for this joint in the anatomical template (e.g., the wrist joint has a higher weight, and the interphalangeal joint has a lower weight). To adjust the coefficient and control the intensity of the boundary penalty, this embodiment takes... ; For a very small positive number (such as 10) -6 ), to prevent division by zero.

[0057] Therefore, when the joint angle is far from the boundary, Larger When the joint angle approaches the boundary, The weight increases dramatically, thus forcing the optimization result to keep the angle within the ROM.

[0058] For a joint with three rotational degrees of freedom, the total second error is calculated on the three degrees of freedom. To integrate the second error into the nonlinear least squares framework, a second error term is defined. The weighted angle deviations are such that their sum of squares is exactly equal to the total second error. See formula (6) below for details: Formula (6); In the formula, ~ It represents the rotation angle deviation from the 1st to the 3Nth degree of freedom (i.e., the difference between the current rotation angle and the neutral rotation angle). ~ The corresponding dynamic ROM weights; N=27 is the total number of joints, and 3N is the total number of rotational degrees of freedom; This is the second error term.

[0059] Finally, the third error term is calculated from the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template. The calculation process is as follows: the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template is multiplied by the predetermined weight coefficient of the current bone. The predetermined weight coefficient of the current bone is defined in the anatomical skeleton template. The predetermined weight coefficient of the bone gradually decreases from the wrist joint to the fingertip joint.

[0060] Specifically, the third error term is used to constrain bone length to maintain consistency with the anatomical template, for the... ( , (Total number of bones) Root bone (connecting joint a and joint b), current length: ;in, and These are the joint positions calculated using forward kinematics based on the current optimized vector; the preset fixed values ​​in the template are... The length deviation is the difference between the current length and the fixed value.

[0061] Furthermore, each bone corresponds to a predetermined weighting coefficient. The predefined weighting coefficient is defined in the template and decreases from the wrist to the fingertip, as shown in the example below: Setting 1: Radiocarpal joint to intercarpal joint: ; Setting 2: From the intercarpal joints to the metacarpals: ; Setting 3: Metacarpals to proximal phalanges: ; Setting 4: Proximal phalanx to middle phalanx: ; Setting 5: Middle phalanx to distal phalanx: .

[0062] Combine the square roots of all bones (keeping the sign) into the third error term. For details, please refer to the following formula (7): Formula (7); In the formula, This is the difference between the current length of the k-th bone and the template length. For the corresponding bone weight coefficient, This represents the total number of bones.

[0063] After obtaining the first, second, and third error terms as described above, concatenating these three sub-vectors vertically will form the complete residual vector. For details, please refer to the following formula (8): Formula (8); Therefore, the error function is defined as follows (9): Formula (9); S500 uses a nonlinear least squares method to iteratively solve for the optimization vector that minimizes the error function. The root position and rotation parameters of the obtained optimization vector are then applied to the corresponding joints of the virtual hand model to drive the skin mesh update, thereby synchronizing the virtual hand animation with the real hand movements and completing augmented reality interaction.

[0064] Combination Figure 3 The S500 unfolding process includes the following steps: S501. At the optimization vector corresponding to the current iteration step, perform a first-order Taylor expansion on the residual vector to obtain a linear approximate expression of the residual vector in the neighborhood of the optimization vector. The linear approximate expression is composed of the sum of the current residual vector value and the Jacobian matrix multiplied by the increment of the optimization vector, where the Jacobian matrix is ​​a matrix composed of the first-order partial derivatives of the residual vector with respect to each component of the optimization vector. S502. Substitute the linear approximation into the error function to obtain the approximate error function for the increment of the optimization vector; S503. Take the derivative of the approximation error function with respect to the optimization vector increment and set the derivative to zero to obtain the optimization vector increment; S504. Update the optimization vector incrementally based on the optimization vector obtained from the solution; S505. Repeat the above steps until the preset convergence condition is met or the number of iterations reaches the preset upper limit. Use the optimized vector obtained in the last iteration as the solution result.

[0065] The goal of this step is to solve for the error function. Minimize the optimization vector And the solution will yield the minimized optimization vector. The corresponding skeletal pose parameters are applied to the virtual hand model, driving its skinned mesh to achieve synchronization with the real hand movements.

[0066] Next, we solve for the error function. Minimize the optimization vector The process is as follows: Let the current iteration number be... The corresponding optimization vector is In order to obtain The approximate expression of the error function in the vicinity is first applied to the residual vector. exist Performing a first-order Taylor expansion at this point, ignoring higher-order terms, yields the following formula (10): Formula (10); in, The increment of the optimization vector to be solved. For residual vectors The Jacobian matrix of the optimization vector is defined by the following formula (11): Formula (11); Right now The element in the i-th row and j-th column is exist The value at point M is such that the dimension of the Jacobian matrix is ​​(M+3N+Nb)×(3+3N), where M is the number of points in the point cloud, N=27 is the number of joints, and N... b =26 represents the total number of bones.

[0067] Next, we construct an approximate error function for the optimization vector increment: Substituting the linear approximation of formula (10) into the error function , get about For an approximate expression, please refer to the following formula (12): Formula (12); remember, = , = The above formula can then be expanded into the following formula (13): Formula (13); Regarding the approximation error Taking the derivative and setting it to zero, we obtain the linear normal equation and solve for the optimal vector increment. Update the optimization vector for the next iteration according to formula (14): Formula (14); Finally, repeat the above steps until the preset convergence condition is met or the number of iterations reaches the preset upper limit. Specifically, the convergence judgment and termination condition can be selected from one of the following methods: Method 1: Incremental modulus is less than a preset threshold; Method 2: The relative change in the error function is less than a preset threshold; Method 3: Reach the maximum number of iterations; Ultimately, what was obtained That is, the optimal vector that minimizes the error function is taken as the optimal skeleton pose parameter, and the root position of this parameter is... and rotational parameters of all joints The transformation matrix of the corresponding joint in the virtual hand model is directly assigned. Through the linear blend skinning algorithm, the world coordinates of all vertices are recalculated according to the joint index and weight bound to each mesh vertex. This drives the skinned mesh to update to a pose that is completely consistent with the real hand, thus completing the real-time synchronous rendering of the virtual hand animation.

[0068] Considering that when workers practice tightening screws, they virtually grasp the parts and rotate their wrists, but feel no resistance, which is very different from actual operation, the embodiments of the present invention can estimate the virtual contact force between the fingers and the parts by using data such as joint rotation angles and bone lengths that have already been obtained.

[0069] In a preferred embodiment, the method further includes: based on the optimized skeletal posture parameters and the geometric model of the virtual part, the system further calculates the virtual interaction force between the corresponding joints of the virtual hand model and the part for tactile feedback; Specifically, when the shortest distance between the surface of the skeletal capsule or the surface of the skin mesh of the virtual hand model and the surface of the virtual part is less than a preset threshold (e.g., 1 mm, which can be adjusted according to the part's precision and feedback sensitivity), it is determined to be in a contact state. In the contact state, the contact force is estimated using a spring-damping model based on the penetration depth and relative velocity of the current contact point (which can be derived from the joint angular velocity). The contact force is equal to the elastic coefficient multiplied by the penetration depth plus the damping coefficient multiplied by the relative velocity, and the direction is perpendicular to the contact surface. This contact force value is fed back to the operator through the vibration handle or data glove that comes with the AR device, allowing the operator to obtain a realistic tactile perception during the virtual assembly process. This simulates the actual feel of tightening screws, engaging parts, and other operations, further improving the effectiveness of virtual training.

[0070] The penetration depth is calculated as follows: the system calculates the shortest distance between the surface of the skeletal capsule of the virtual hand model and the surface of the virtual part in real time; if the shortest distance is negative, it indicates that penetration has occurred, and the absolute value of this distance is taken as the penetration depth; if there is no penetration, the penetration depth is zero. Relative velocity is defined as the velocity of the contact point on the virtual hand model relative to the contact point on the virtual part; in this embodiment, the part is considered stationary, so the relative velocity can be simplified to the spatial motion velocity of the contact point on the virtual hand. This velocity can be calculated by the positional change of the contact point: the system records the three-dimensional coordinates of the same contact point in the previous and current frames, thus obtaining the average linear velocity of that point, which is used as an approximation of the relative velocity.

[0071] Based on the above description, this invention proposes a hand tracking method based on kinematic optimization in an AR environment, comprising: First, constructing an anatomical skeleton template containing 27 joints, bone length constraints, joint range of motion constraints, and neutral rotation parameters, wherein the radiocarpal joint is defined as the root joint. Color and depth images of the hand are acquired using a depth camera to generate a hand point cloud, and a lightweight convolutional neural network (with four independent templates and corresponding models for gender and left / right hands) is used to output the initial three-dimensional position of each joint in the camera coordinate system, forming an initial skeleton model. Then, an optimization vector (containing the root position and all joint rotation parameters) is used to represent the skeleton pose to be solved, and a residual vector consisting of three error terms is constructed: the first error term is the distance from each point in the hand point cloud to the surface of the current skeleton model capsule; the second error term is the deviation between the current rotation angle of each joint and the neutral angle, constrained by dynamic ROM weights (the closer the angle is to the activity boundary, the greater the weight); the third error term is the deviation between the distance between adjacent joints and the template bone length, with the weight decreasing progressively from the wrist to the fingertip. Using the sum of squares of the residual vectors as the error function, a nonlinear least squares method (such as Levenberg) is employed. The Marquardt algorithm iteratively solves for the optimal vector that minimizes the error function. Finally, the obtained root position and rotation parameters are applied to the corresponding joints of the virtual hand model. By driving mesh updates through linear blending skinning, real-time synchronization between virtual hand animation and real hand movements is achieved, thus completing augmented reality interaction.

[0072] Beneficial Effects: This invention effectively avoids anatomical inconsistencies such as bone stretching and joint reversal bending commonly found in pure deep learning methods by directly embedding length constraints and joint range of motion constraints from anatomical skeleton templates into a nonlinear least squares optimization framework. A lightweight convolutional neural network is used to provide initial estimates, combined with dynamic ROM weights and decreasing bone weights, significantly reducing the number of iterations (convergence typically occurs in 3-5 iterations) while maintaining pose realism. Real-time tracking at over 60fps is achieved on mobile AR devices. Furthermore, four independent templates and corresponding CNN models are set for gender and left / right hand, further improving the individual adaptation accuracy of the initial pose estimation. In industrial training scenarios such as virtual assembly of precision parts, this invention enables virtual hands to grasp and manipulate virtual parts with natural and accurate movements, without penetration or distortion, effectively avoiding erroneous muscle memory caused by pose distortion. This reduces the risk of part damage during physical training and improves the reliability and practicality of AR-assisted operation training.

[0073] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be noted that due to the limitations of textual expression, and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A hand tracking method based on kinematic optimization in an AR environment, characterized in that, include: Construct an anatomical skeleton template and predefine the length constraints of each bone and the range of motion constraints of each joint; The anatomical skeleton template has N joints based on anatomy; The depth and color images of the operator's hands are captured by a camera to generate a hand point cloud. A lightweight convolutional neural network is used to perform initial pose estimation on the color image and depth image, and outputs the initial three-dimensional position of each joint in the camera coordinate system to form an initial skeleton model. Based on the hand point cloud, the initial skeletal model, and the length constraints of each bone and the range of motion constraints of each joint, a residual vector is constructed. And the sum of squares of the residual vectors is used as the error function; in, Let be the optimization vector to be solved, used to represent the pose of the optimized skeletal model; the optimization vector includes the root position of the optimized skeletal model and . Rotation parameters of each joint; the residual vector Calculated based on the skeletal model determined by the current optimization vector; The optimization vector that minimizes the error function is obtained by iteratively solving using the nonlinear least squares method. The root position and rotation parameters of the obtained optimization vector are then applied to the corresponding joints of the virtual hand model to drive the skin mesh update, thereby achieving synchronization between the virtual hand animation and the real hand movement and completing augmented reality interaction.

2. The method according to claim 1, characterized in that, The residual vector This includes: the first error term, the second error term, and the third error term corresponding to the current skeletal model; The first error term is calculated from the distance from each point in the hand point cloud to the surface of the skeletal model; The second error term is calculated from the deviation between the current rotation parameter and the neutral rotation parameter of each joint; the neutral rotation parameter is the median rotation parameter in the range of motion constraint of the joint. The third error term is calculated from the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template.

3. The method according to claim 2, characterized in that, The first error term is calculated using the following method: For each point in the hand point cloud, find the nearest triangle on the surface of the current skeleton model; The distance from the hand point cloud points to the triangular face is calculated, which constitutes the first error term.

4. The method according to claim 2, characterized in that, The rotation parameters include rotation angles, the median rotation parameters include neutral rotation angles, and the second error term is calculated using the following method: For each joint, the deviation between its current rotation angle and the neutral rotation angle is multiplied by the ROM weight corresponding to the current joint. The ROM weight is dynamically adjusted by the basic weight of the joint based on the distance between the current rotation angle and the nearest ROM boundary. The ROM boundary is the upper or lower limit of the joint's range of motion.

5. The method according to claim 2, characterized in that, The third error term is calculated using the following method: The deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template is obtained by multiplying the predetermined weight coefficient of the current bone; the predetermined weight coefficient of the current bone is defined in the anatomical skeleton template, and the predetermined weight coefficient of the bone gradually decreases from the wrist joint to the fingertip joint.

6. The method according to claim 1, characterized in that, The anatomical skeleton template contains 27 joints, including 1 radiocarpal joint, 7 intercarpal joints, 5 metacarpal joints, 2 joints of the thumb, and 3 joints of each of the other four fingers; wherein, the radiocarpal joint is the root joint of the anatomical skeleton model, and the root position is the three-dimensional position of the radiocarpal joint in the camera coordinate system.

7. The method according to any one of claims 1-6, characterized in that, The anatomical skeleton template is set with four templates according to gender and left-right direction, and each anatomical skeleton template corresponds to a lightweight convolutional neural network model. The process of using a lightweight convolutional neural network to perform initial pose estimation on the color image and depth image, and outputting the initial 3D position of each joint in the camera coordinate system to form an initial skeletal model, specifically includes the following steps: Select the corresponding convolutional neural network model based on the anatomical skeleton template chosen by the user; The initial pose estimation is performed on the color image and depth image based on the selected convolutional neural network model, and the initial three-dimensional position of each joint in the camera coordinate system is output to form an initial skeleton model.

8. The method according to any one of claims 1-6, characterized in that, The process of iteratively finding the optimization vector that minimizes the error function using the nonlinear least squares method specifically includes the following steps: At the optimization vector corresponding to the current iteration step, a first-order Taylor expansion is performed on the residual vector to obtain a linear approximate expression of the residual vector in the neighborhood of the optimization vector; the linear approximate expression is composed of the sum of the current residual vector value and the Jacobian matrix multiplied by the increment of the optimization vector, where the Jacobian matrix is ​​a matrix composed of the first-order partial derivatives of the residual vector with respect to each component of the optimization vector; Substituting the linear approximation into the error function yields an approximate error function for the optimized vector increment; The optimal vector increment is obtained by taking the derivative of the approximation error function with respect to the optimization vector increment and setting the derivative to zero. The optimization vector is updated incrementally based on the obtained optimization vector. Repeat the above steps until the preset convergence condition is met or the number of iterations reaches the preset upper limit, and use the optimized vector obtained from the last iteration as the solution result.

9. A hand tracking system based on kinematic optimization in an AR environment, characterized in that, include: The acquisition module is used to acquire depth and color images of the operator's hands; Processing module: Used to generate hand point clouds based on depth and color images of the operator's hands; An anatomical skeleton template is constructed, and the length constraints of each bone and the range of motion constraints of each joint are predefined; the anatomical skeleton template has N joints based on anatomy; A lightweight convolutional neural network is used to perform initial pose estimation on the color image and depth image, and outputs the initial three-dimensional position of each joint in the camera coordinate system to form an initial skeleton model. Based on the hand point cloud, the initial skeletal model, and the length constraints of each bone and the range of motion constraints of each joint, a residual vector is constructed. And the sum of squares of the residual vectors is used as the error function; in, Let be the optimization vector to be solved, used to represent the pose of the optimized skeletal model; the optimization vector includes the root position of the optimized skeletal model and . Rotation parameters of each joint; the residual vector Calculated based on the skeletal model determined by the current optimization vector; The optimization vector that minimizes the error function is obtained by iteratively solving using the nonlinear least squares method. The root position and rotation parameters of the obtained optimization vector are then applied to the corresponding joints of the virtual hand model to drive the skin mesh update, thereby achieving synchronization between the virtual hand animation and the real hand movement and completing augmented reality interaction.

10. The system according to claim 9, characterized in that, The residual vector This includes: the first error term, the second error term, and the third error term corresponding to the current skeletal model; The first error term is calculated from the distance from each point in the hand point cloud to the surface of the skeletal model; The second error term is calculated from the deviation between the current rotation parameter and the neutral rotation parameter of each joint; the neutral rotation parameter is the median rotation parameter in the range of motion constraint of the joint. The third error term is calculated from the deviation between the distance between adjacent joints and the corresponding distance between joints in the anatomical skeleton template.

Images