Hand tracking method for enhancing hand-hand interaction posture precision, hardware and application

Abstract

The invention provides a hand tracking method, hardware and application for enhancing hand-hand interaction posture precision, and the method comprises the steps: building an end-to-end deep learning network, receiving an image sequence, and outputting a hand three-dimensional posture; collecting the situation that both hands of a user exist in a scene at the same time, screening effective sample data, and inputting the effective sample data into an end-to-end deep learning network; establishing an anti-mode-penetrating loss function, and training the network; hand tracking is completed through the trained end-to-end deep learning network; the method realizes a computer readable storage medium, a computer device and a system, and is applied to hand tracking processing when the positions of the two hands of a user are overlapped in a VR and / or AR scene. The method is easy to implement, a core network architecture does not need to be modified, the method can be integrated into most of existing end-to-end hand tracking frameworks only by adding a loss function in a training stage, the sense of reality is enhanced, the immersion of VR experience is remarkably improved, posture accuracy and physical authenticity can be jointly optimized, and a better overall effect is obtained.

Description

Technical Field

[0001] This invention relates to the technical field of calculation, estimation or counting, and in particular to a hand tracking method, hardware and application for enhancing the accuracy of hand-to-hand interaction posture. Background Technology

[0002] In virtual reality / augmented reality (VR / AR), high-precision real-time hand tracking is crucial for achieving natural interaction. In recent years, end-to-end deep learning solutions, such as Meta's Umetrack solution, have greatly promoted the development of hand tracking technology by directly predicting the absolute 3D pose of the hand in the world coordinate system from images through unified multi-view and temporal processing.

[0003] However, existing technologies still have certain limitations when dealing with two-handed interaction, especially when the user's hands touch or overlap. Existing tracking solutions may predict postures that do not conform to the laws of physics, such as the fingers of one hand penetrating the palm of the other hand, i.e., inter-pernetration. This inter-pernetration problem seriously undermines the immersion and realism of the VR experience. Therefore, a simple and effective method is urgently needed to solve this problem. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a hand tracking method, hardware, and application for enhancing the accuracy of hand-to-hand interaction postures, thereby improving the physical realism of posture prediction and the stability of the interactive experience.

[0005] The technical solution adopted in this invention is a hand tracking method for enhancing the accuracy of hand-to-hand interaction posture, which establishes an end-to-end deep learning network for receiving image sequences and outputting three-dimensional hand posture.

[0006] The system collects data on scenarios where a user's hands are present simultaneously, filters out valid sample data, and inputs it into the end-to-end deep learning network.

[0007] Establish an anti-penetration loss function to train the end-to-end deep learning network;

[0008] Hand tracking is accomplished using a trained end-to-end deep learning network.

[0009] Preferably, the effective sample data is an image of the user's hands overlapping.

[0010] Preferably, the anti-penetration loss function is related to L pose L temp Associated with hinge loss, where L pose L is the L1 norm between the 3D keypoints output by the network and the ground truth values. temp This is due to timing loss.

[0011] Preferably, the hinge loss satisfies,

[0012] L inter = max(0, ε-d min )

[0013] Where ε is the safety threshold, d min The minimum Euclidean distance d between the left-hand keypoint set and the right-hand keypoint set. min .

[0014] Preferably, the set of key points for the left hand and the set of key points for the right hand are all the joints of the corresponding hand.

[0015] Preferably, 0 < ε ≤ 5 mm.

[0016] A computer-readable storage medium storing a hand tracking program for enhancing the accuracy of hand-to-hand interaction postures, the program implementing the hand tracking method for enhancing the accuracy of hand-to-hand interaction postures when executed by a processor.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the hand tracking method for enhancing the accuracy of hand-to-hand interaction postures.

[0018] A hand tracking system, comprising:

[0019] An image acquisition unit is used to acquire images of the hand;

[0020] A configuration unit is used to configure the parameters of the image acquisition unit;

[0021] A control terminal is used to acquire hand images and parameters of the image acquisition unit, and output corresponding hand posture data in accordance with the hand tracking method for enhancing the accuracy of hand-to-hand interaction posture.

[0022] One output unit is used to output hand posture data.

[0023] An application of the hand tracking method described above for enhancing the accuracy of hand-to-hand interaction posture is applied to the processing of hand tracking when the user's hands overlap in VR and / or AR scenarios.

[0024] This invention provides a hand tracking method, hardware, and application for enhancing the accuracy of hand-to-hand interaction postures. The method establishes an end-to-end deep learning network to receive image sequences and output three-dimensional hand postures; it collects data on situations where a user's two hands are present simultaneously in a scene, filters out valid sample data, and inputs it into the end-to-end deep learning network; it establishes an anti-mold-breakage loss function to train the end-to-end deep learning network; and it uses the trained end-to-end deep learning network to complete hand tracking. The method is implemented in a computer-readable storage medium, computer device, and system, and is applied to the processing of hand tracking when the user's hands overlap in VR and / or AR scenarios.

[0025] The beneficial effects of this invention are as follows:

[0026] (1) Simple to implement; Based on the existing deep neural network training framework that can perform pose regression for one or both hands, a two-hand interaction anti-mold loss function is introduced. No core network architecture needs to be modified. Only a loss function needs to be added during the training stage to integrate it into most existing end-to-end hand tracking frameworks.

[0027] (2) Enhance realism; By penalizing clipping during training, the network learns to generate physically more reasonable hand postures, effectively alleviating clipping during hand interaction, thereby significantly improving the immersiveness of the VR experience.

[0028] (3) End-to-end optimization; anti-mold constraint is integrated into an end-to-end differentiable framework (umetrack framework), so that attitude accuracy and physical realism can be jointly optimized, rather than as post-processing, thereby obtaining better overall results. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention;

[0030] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0032] This invention relates to a hand tracking method for enhancing the accuracy of hand-to-hand interaction posture, which establishes an end-to-end deep learning network for receiving image sequences and outputting three-dimensional hand posture;

[0033] The system collects data on scenarios where a user's hands are present simultaneously, filters out valid sample data, and inputs it into the end-to-end deep learning network.

[0034] Establish an anti-penetration loss function to train the end-to-end deep learning network;

[0035] Hand tracking is accomplished using a trained end-to-end deep learning network.

[0036] In this invention, an end-to-end deep learning network is used as the basic model, which can accept multi-view image sequences from a VR headset as input and directly regress the three-dimensional pose parameters of the hands, such as joint angles θ and root node world transformation matrix Th. When the system detects that there are two hands in the scene at the same time, the basic model will simultaneously predict the 3D pose parameters of the left and right hands, and then obtain the 3D world coordinates of all joints of each hand. During the training phase of the network, an anti-clipping loss function is introduced, which is specifically used to penalize the clipping that occurs between the two hand models.

[0037] The valid sample data consists of images of the user's hands overlapping.

[0038] In this invention, the sample uses a mark point optical tracking system to capture the 3D pose of the human hand in real time and simultaneously record the corresponding image information. The images that overlap during the user's hand movements are taken as valid samples (data).

[0039] The anti-diffusive loss function and L pose L temp Associated with hinge loss, where L pose L is the L1 norm between the 3D keypoints output by the network and the ground truth values. temp This is due to timing loss.

[0040] In this invention, the total loss function satisfies,

[0041] L total = L pose +λ t *L temp +λ i *L inter

[0042] Where, λ i The weight hyperparameter is used to balance the impact of the anti-penetration loss term. The loss function is used to train the entire neural network end-to-end using this parameter. The weights in the network are optimized by using a grid search method.

[0043] L pose satisfy,

[0044]

[0045] in, Let J be the 3D true coordinates of the j-th key point. Let j be the 3D predicted coordinates of the j-th key point. Let J be the L1 norm, and J be the total number of keypoints. This is the true attitude parameter vector. To predict the attitude parameter vector, These are the weighting coefficients. The predicted values ​​used to constrain attitude parameters are consistent with the actual values.

[0046] L temp satisfy,

[0047]

[0048] in, T is the total number of sequences. Let L be the attitude reference vector for frame t. temp Punishing drastic changes in posture sequences over time encourages smoother motion.

[0049] Hinge loss is satisfied.

[0050] L inter = max(0, ε-d min )

[0051] Where ε is the safety threshold, d min The minimum Euclidean distance d between the left-hand keypoint set and the right-hand keypoint set. min .

[0052] The left-hand key point set and the right-hand key point set are all the joints of the corresponding hands.

[0053] 0 < ε ≤ 5 mm, where ε is a small positive number representing the critical value at which the hands almost touch but do not penetrate.

[0054] If and only if d min When the distance between the two hands is less than ε, a penalty loss is incurred. This loss is only positive when the distance between the two hands is less than the safety threshold. The closer the distance, the deeper the clipping, and the greater the loss value.

[0055] By training on a dataset containing a large number of hand-eye interaction actions, the model will automatically learn how to avoid collisions and penetrations between the two hands while maintaining accurate pose.

[0056] The present invention also relates to a computer-readable storage medium storing a hand tracking program for enhancing the accuracy of hand-to-hand interaction postures, which, when executed by a processor, implements the hand tracking method for enhancing the accuracy of hand-to-hand interaction postures.

[0057] The present invention also relates to a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the hand tracking method for enhancing the accuracy of hand-to-hand interaction postures.

[0058] The present invention also relates to a hand tracking system, comprising:

[0059] An image acquisition unit is used to acquire images of the hand;

[0060] A configuration unit is used to configure the parameters of the image acquisition unit;

[0061] A control terminal is used to acquire hand images and parameters of the image acquisition unit, and output corresponding hand posture data in accordance with the hand tracking method for enhancing the accuracy of hand-to-hand interaction posture.

[0062] One output unit is used to output hand posture data.

[0063] The present invention also relates to an application of the hand tracking method described above for enhancing the accuracy of hand-to-hand interaction postures, applied to the processing of hand tracking when the user's hands overlap in VR and / or AR scenarios.

[0064] Specifically, it is loaded into the hardware through a corresponding program and installed in devices such as VR all-in-one headsets. It collects multi-view image sequences and camera parameter matrices as inputs, and outputs the three-dimensional pose parameters of the hand through a trained end-to-end deep learning network. The three-dimensional pose parameters of the hand are used to drive the hand model in the VR / AR scene, so that interaction in the scene can be completed, such as pinching, hand raycasting, and tapping.

[0065] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0070] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A hand tracking method for enhancing the accuracy of hand-to-hand interaction posture, characterized in that: An end-to-end deep learning network was established to receive image sequences and output the three-dimensional pose of the hand. The system collects data on scenarios where a user's hands are present simultaneously, filters out valid sample data, and inputs it into the end-to-end deep learning network. Establish an anti-penetration loss function to train the end-to-end deep learning network; Hand tracking is accomplished using a trained end-to-end deep learning network.

2. The hand tracking method for enhancing hand-to-hand interaction posture accuracy according to claim 1, characterized in that: The valid sample data consists of images of the user's hands overlapping.

3. The hand tracking method for enhancing hand-to-hand interaction posture accuracy according to claim 1, characterized in that: The anti-diffusive loss function and L pose L temp Associated with hinge loss, where L pose L is the L1 norm between the 3D keypoints output by the network and the ground truth values. temp This is due to timing loss.

4. A hand tracking method for enhancing hand-to-hand interaction posture accuracy according to claim 3, characterized in that: Hinge loss is satisfied. L inter = max(0, ε-d min ) Where ε is the safety threshold, d min The minimum Euclidean distance d between the left-hand keypoint set and the right-hand keypoint set. min .

5. A hand tracking method for enhancing hand-to-hand interaction posture accuracy according to claim 4, characterized in that: The left-hand key point set and the right-hand key point set are all the joints of the corresponding hands.

6. A hand tracking method for enhancing hand-to-hand interaction posture accuracy according to claim 4, characterized in that: 0 < ε ≤ 5 mm.

7. A computer-readable storage medium, characterized in that: It stores a hand tracking program for enhancing the accuracy of hand-to-hand interaction postures, which, when executed by a processor, implements the hand tracking method for enhancing the accuracy of hand-to-hand interaction postures as described in any one of claims 1 to 6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the hand tracking method for enhancing hand-to-hand interaction posture accuracy as described in any one of claims 1 to 6.

9. A hand tracking system, characterized in that: include: An image acquisition unit is used to acquire images of the hand; A configuration unit is used to configure the parameters of the image acquisition unit; A control terminal is used to acquire hand images and parameters of the image acquisition unit, and output corresponding hand posture data in accordance with the hand tracking method for enhancing hand-to-hand interaction posture accuracy as described in any one of claims 1 to 6. One output unit is used to output hand posture data.

10. An application of the hand tracking method for enhancing hand-to-hand interaction posture accuracy as described in any one of claims 1 to 6, characterized in that: It is used for hand tracking in VR and / or AR scenarios when the user's hands overlap.

Images