Hand-object interaction method for providing tactile feedback

Abstract

A hand-object interaction method for providing tactile feedback uses an object 3D model to generate a sparse view of an object for object tracking, is not limited to objects in a training set, and can be generalized to various daily objects; real-time hand and object posture tracking is achieved, each frame processing comprises hand tracking and object tracking, a mapping mechanism of physical hand and object interaction and virtual hand and object interaction is achieved, the frame rate of 26 frames per second is finally achieved, and the real-time experience requirement of a user is met.

Description

Technical Field

[0001] This invention relates to a hand-object interaction method that provides tactile feedback, belonging to the field of information technology, and particularly to the field of hand-object interaction technology. Background Technology

[0002] Hand-object interaction technology is one of the important directions of future interaction technology and is currently widely used in wearable devices such as smart head-mounted displays. However, current hand-object interaction is often limited to the interaction between the hand and virtual objects, lacking the haptic feedback experience and reducing immersion. Due to challenges such as real-time performance, generalization ability, and occlusion problems, developing hand-object interaction applications that provide haptic feedback faces great difficulties. To solve these problems, the academic community has proposed a number of methods to improve algorithm accuracy. For example, a method that uses the MegaPose model and the FrankMocap model to simultaneously track the hand and object poses and performs joint optimization through interaction loss and collision loss can only maintain a frame rate of 5 frames per second. Another method uses pose estimation based on infinitesimal planes combined with an independent hand tracker to achieve real-time pose calculation, but this method still has insufficient tracking stability of point correspondence in scenes with object occlusion.

[0003] Currently, six-DOF tracking of everyday objects remains a technical challenge in the field of computer vision. Existing feature-point tracking methods require pre-collecting and storing feature points; however, these feature points are not robust to occlusion and are difficult to use in hand-object interaction applications. Deep learning methods require training with labeled object pose data, lack good generalization ability, and have high computational costs, making them difficult to meet real-time requirements.

[0004] Gesture pose estimation and reconstruction commonly employs the MANO model to estimate finger joint poses and reconstruct hand meshes, demonstrating strong robustness even in occlusion and fast-moving scenarios. However, due to mutual occlusion issues caused by hand-object interaction, hand-object pose estimation results often fail to meet the requirements of interactive applications, and estimating hand and object poses separately often results in unrealistic interactions. To address these issues, researchers have proposed various optimization schemes, such as combining 3D contact point priors, reconstructing 3D hand-object models using 2D image cues, and introducing interaction and collision losses during the process; and reconstructing hand and object models using point clouds and optimizing hand poses using energy terms based on object poses. However, these optimization methods are costly and cannot meet real-time requirements.

[0005] In summary, achieving a real-time, low-cost, robust hand-object interaction method with haptic feedback has become a pressing technical challenge in the field of hand-object interaction technology. Summary of the Invention

[0006] In view of this, the purpose of this invention is to invent a method that uses a 3D model of an object to construct a sparse view and perform six-degree-of-freedom tracking of the object, combines a gesture algorithm to estimate the spatial pose of finger joints, and empowers downstream hand-object interaction applications based on the spatial pose of the object and the gesture, thereby solving the problems of generalization, real-time performance, strong robustness, and haptic feedback in hand-object interaction methods.

[0007] To achieve the above objectives, the present invention proposes a hand-object interaction method that provides tactile feedback, the method comprising the following steps:

[0008] (1) Using the initial CAD model of the object as input, create a sparse viewpoint model to represent the three-dimensional geometry of the object; render from multiple viewpoints around the object, placing a virtual camera on the vertices of a geodesic mesh surrounding the object, which is created by subdividing icosahedral triangles into fourth subdivisions; calculate model points for all viewpoints. and normal vector ;Utilizing the homogeneous transformation between the camera reference coordinate system and the model reference coordinate system Rotation transformation and projection transformation To obtain the 2D-3D correspondence; model points obtained from the nearest viewpoint of the sparse viewpoint model. and normal vector The corresponding line model is defined by projecting it onto the image using the following equation:

[0009]

[0010]

[0011] Where i represents the current line index, C represents the camera coordinate system, and M represents the model coordinate system. Indicates the center point of the corresponding line. This represents the normal vector of the corresponding line. Indicates projection transformation;

[0012] (2) Render the virtual model of the target object in the center of the display screen. When the user manually aligns its outline and reaches the corresponding line threshold, start object tracking; extend the region algorithm SRT3D to high-speed object tracking in hand-object interaction scenarios. That is, the SRT3D algorithm achieves fast tracking through the corresponding lines of the region edge. During the pose update process, the SRT3D algorithm only calculates the pixel-level probability distribution noise of the foreground and background along the corresponding lines and approximates it as a normal distribution; the position of the distribution mean represents the maximum probability of the foreground outline; assuming that the calculated object pose value is close enough to the actual situation, the optimal distribution of each corresponding line It should approximate a normal distribution with a preset minimum variance. The Kullback-Leibler divergence is used to measure the similarity between the current distribution and the optimal distribution of the corresponding line, and this is used as the confidence level for estimating the tracking results. The calculation formula is as follows:

[0013]

[0014] In the above formula, N represents the number of corresponding lines (in the embodiment, the value is 200). The threshold value is 1.0 in this embodiment. The confidence level is used to determine whether an object is lost during tracking and to restart the initialization process in a timely manner when the confidence level is lower than the set threshold.

[0015] (3) The MediaPipe-Hands model with GPU acceleration is used as a fast hand tracking module to detect the two-dimensional bounding box of the hand, which limits the computation to the hand area and improves efficiency; the HaMeR model is used for three-dimensional gesture estimation and hand mesh reconstruction. This vision-based attention mechanism model can accurately recover hand movements in complex gestures and interactive occlusions, and reconstruct the fitted hand mesh.

[0016] (4) The effect of hand occlusion on object tracking is eliminated to the greatest extent by filtering method. The specific content of the filtering method is: first, a hand mask is generated by rendering the three-dimensional hand model of the current image frame in real time; then, before normal optimization, the pixel center of each corresponding line is determined, the corresponding lines whose center is located in the hand mask are filtered, and finally, the pose calculation is performed based on the remaining corresponding lines.

[0017] When the occlusion area of ​​the hand is large, too many corresponding lines will be filtered out. If corresponding lines exceeding a set threshold are filtered out, the pose reconstructed from the hand is used to optimize object tracking. The pose transformation of the held object is performed based on the transformation matrix of the hand root node, that is, the coordinate transformation of the root node of the MANO hand model between the current frame and the previous frame is calculated. Then it is applied to transition prediction under heavy occlusion of the object's hand;

[0018] (5) Based on the hand-object spatial pose estimated by the visual algorithm, define the process of real and virtual hand-object interaction mapping, and activate the virtual-real hand-object interaction through the defined gestures; the user determines how to grasp real and virtual objects according to the required interaction method, captures and records the current hand pose and position relative to the object, and visualizes the defined gestures on the grid of real and virtual objects; allows the user to customize the grasping position of virtual objects to ensure that the grasping action between physical objects and virtual objects is consistent with the expected interaction; the user achieves consistent mapping by defining the hand-object interaction input and output, generating a pair of triplets (real object, gesture, relative position) and (virtual object, gesture, relative position), which are stored to facilitate subsequent grasping matching and mapping, and realize multiple virtual object mappings according to different ways of grasping objects;

[0019] (6) Traverse all gestures associated with real objects, calculate the Cartesian distance between the current gesture and the defined gesture, and for each grasping action within the set threshold range, calculate the matching score according to the following formula. :

[0020]

[0021] In the above formula, Joint represents the number of joints in the hand. This represents the forward difference of the j-th hand joint. Let represent the upward difference of the j-th hand joint. This indicates the position of the j-th hand joint. This indicates the position of the j-th hand joint.

[0022] ·

[0023] ·

[0024] In the above formula, This represents the rotation of the j-th hand joint. This represents the rotation of the defined j-th hand joint. This represents the forward vector in the Unity engine. This represents the upward vector of the Unity engine; the gripping action with the highest matching score will be selected as the final gesture in the mapping process.

[0025] The beneficial effects of this invention are that it uses a 3D model of an object to generate a sparse view of the object for object tracking, which is not limited to objects in the training set and can be generalized to various everyday objects; real-time hand and object posture tracking, each frame processing includes hand tracking and object tracking, realizes a mapping mechanism between physical hand and object interaction and virtual hand and object interaction, and finally achieves a frame rate of 26 frames per second to meet the user's real-time experience needs. Attached Figure Description

[0026] Figure 1 This is a flowchart of a hand-object interaction method that provides tactile feedback, as proposed in this invention.

[0027] Figure 2 Here are the experimental results of this invention: a qualitative result diagram of object tracking in the HO3D dataset;

[0028] Figure 3 This is an experimental result of an embodiment of the present invention: a qualitative result diagram of object tracking in outdoor video streams. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] See Figure 1 This invention introduces a hand-object interaction method that provides haptic feedback, the method comprising the following steps:

[0031] (1) Using the initial CAD model of the object as input, create a sparse viewpoint model to represent the three-dimensional geometry of the object; render from multiple viewpoints around the object, and place a virtual camera on the vertices of a geodesic grid surrounding the object, which is created by subdividing the triangles of an icosahedron into four subdivisions.

[0032] In this embodiment, a sparse viewpoint model is created based on the mesh reconstructed by 3D Gaussian sputtering to represent the three-dimensional geometry of the object from multiple viewpoints around the object. ) is then rendered.

[0033] For all viewpoints, calculate the model points. and normal vector ;Utilizing the homogeneous transformation between the camera reference coordinate system and the model reference coordinate system Rotation transformation and projection transformation To obtain the 2D-3D correspondence; model points obtained from the nearest viewpoint of the sparse viewpoint model. and normal vector The corresponding line model is defined by projecting it onto the image using the following equation:

[0034]

[0035]

[0036] Where i represents the current line index, C represents the camera coordinate system, and M represents the model coordinate system. Indicates the center point of the corresponding line. This represents the normal vector of the corresponding line. Indicates projection transformation;

[0037] (2) Render a virtual model of the target object in the center of the display screen. When the user manually aligns its outline and reaches the corresponding line threshold (in this embodiment, the threshold is set to 0.8), object tracking is started. Extend the region algorithm SRT3D (https: / / link.springer.com / article / 10.1007 / s11263-022-01579-8) to high-speed object tracking in hand-object interaction scenarios. That is, the SRT3D algorithm achieves fast tracking through the corresponding lines of the region edge. During the pose update process, the SRT3D algorithm only calculates the pixel-level probability distribution noise of the foreground and background along the corresponding lines and approximates it as a normal distribution. The position of the distribution mean represents the maximum probability of the foreground outline. Assuming that the calculated object pose value is close enough to the actual situation, the optimal distribution of each corresponding line is... It should approximate a normal distribution with a preset minimum variance. The Kullback-Leibler divergence is used to measure the similarity between the current distribution and the optimal distribution of the corresponding line, and this is used as the confidence level for estimating the tracking results. The calculation formula is as follows:

[0038]

[0039] In the above formula, N represents the number of corresponding lines. The threshold is indicated; the confidence level is used to determine whether an object is lost during tracking, and to promptly restart initialization when the score confidence level falls below the set threshold.

[0040] (3) The MediaPipe-Hands model with GPU acceleration (https: / / mediapipe.readthedocs.io / en / latest / solutions / hands.html) is used as a fast hand tracking module to detect the 2D bounding box of the hand, which restricts the computation to the hand region and improves efficiency; the HaMeR model (https: / / github.com / geopavlakos / hamer) is used for 3D gesture estimation and hand mesh reconstruction. This vision-based attention mechanism model can accurately recover hand movements in complex gestures and interactive occlusions, and reconstruct the fitted hand mesh.

[0041] (4) The effect of hand occlusion on object tracking is eliminated to the greatest extent by filtering method. The specific content of the filtering method is: first, a hand mask is generated by rendering the three-dimensional hand model of the current image frame in real time; then, before normal optimization, the pixel center of each corresponding line is determined, the corresponding lines whose center is located in the hand mask are filtered, and finally, the pose calculation is performed based on the remaining corresponding lines.

[0042] When the hand occlusion area is large, too many corresponding lines will be filtered. If corresponding lines exceeding a set threshold are filtered out, the pose reconstructed from the hand is used to optimize object tracking. The pose transformation of the held object is performed according to the transformation matrix of the hand root node, that is, the coordinate transformation of the root node of the MANO hand (https: / / mano.is.tue.mpg.de / ) between the current frame and the previous frame is calculated. Then it is applied to transition prediction under heavy occlusion of the object's hand;

[0043] (5) Based on the hand-object spatial pose estimated by the visual algorithm, define the process of real and virtual hand-object interaction mapping, and activate the virtual-real hand-object interaction through the defined gestures; the user determines how to grasp real and virtual objects according to the required interaction method, captures and records the current hand pose and position relative to the object, and visualizes the defined gestures on the grid of real and virtual objects; allows the user to customize the grasping position of virtual objects to ensure that the grasping action between physical objects and virtual objects is consistent with the expected interaction; the user achieves consistent mapping by defining the hand-object interaction input and output, generating a pair of triplets (real object, gesture, relative position) and (virtual object, gesture, relative position), which are stored to facilitate subsequent grasping matching and mapping, and realize multiple virtual object mappings according to different ways of grasping objects;

[0044] (6) Traverse all gestures associated with real objects, calculate the Cartesian distance between the current gesture and the defined gesture, and for each grasping action within a set threshold (in this embodiment, the threshold is set to 0.05 meters), calculate the matching score according to the following formula. :

[0045]

[0046] In the above formula, Joint represents the number of hand joints. In this example, the Joint value is 21. This represents the forward difference of the j-th hand joint. Let represent the upward difference of the j-th hand joint. This indicates the position of the j-th hand joint. This indicates the position of the j-th hand joint.

[0047] ·

[0048] ·

[0049] In the above formula, This represents the rotation of the j-th hand joint. This represents the rotation of the defined j-th hand joint. This represents the forward vector of the Unity rendering engine (https: / / unity.com / cn). This represents the upward vector of the Unity rendering engine; the gripping action with the highest matching score will be selected as the final gesture in the mapping process.

[0050] The inventors conducted numerous experiments on the method proposed in this invention, and the experimental results are as follows:

[0051] The Meta Quest Pro mixed reality headset development system is used, utilizing the headset's pass-through function to render virtual objects superimposed on real-world objects. An additional camera is fixed to the top of the headset and wired to a local computer. The additional camera provides 1280×720 resolution RGB image input to the visual algorithm, and the actual depth is calibrated using the headset's depth calibration. The coordinate systems of the additional camera and the headset camera are calibrated using the Zhang Zhengyou camera calibration method. This invention is implemented on an Nvidia GeForce RTX 3090 GPU, using SocketIO to achieve data communication between the headset and the computer within a local area network. Application development is based on the Unity engine, and the application is deployed to the headset.

[0052] Object reconstruction and hand / object pose tracking were performed using a PC (Intel Core i9-12900KF CPU, 5.2GHz, 128GB RAM, Nvidia GeForce RTX 3090 GPU). A sparse view model of the object was generated every 20-60 seconds using its 3D model. RGB color images with a resolution of 1280 x 720 were used for both hand and object tracking channels. Processing per frame included hand tracking (36 ms, including bounding box, hand mask, and hand model generation) and object tracking (2 ms), ultimately achieving a stable frame rate of 26 frames per second to meet the user's real-time experience requirements.

[0053] Experiments were conducted using six object frame sequences from the HO3D dataset and eight object frame sequences from the H2O dataset, and the tracking success rate was compared with the baseline SRT3D. Required translation error and rotational error The calculation formula is:

[0054]

[0055]

[0056] Indicates the current object coordinates. Represents the true value of the object's coordinates. This represents the transpose of the rotation matrix of the current object. This represents the actual value of the object's rotation matrix. The current frame satisfies... <5cm and When the angle is less than 10°, tracking is considered successful. As shown in Table 1, the method of this invention performs excellently in most sequences.

[0057] Table 1

[0058]

[0059] To verify the stability and robustness of the method of the present invention, the inventors evaluated the experimental results without pose reset (dataset) and reinitialization (outdoor video stream). Figure 2 and Figure 3 Some qualitative results are presented. The method of this invention exhibits stronger robustness and stability in scenarios involving large-scale hand rotation and prolonged hand occlusion.

[0060] Experimental results demonstrate that the method proposed in this invention is feasible and effective.

[0061] The present invention has been described in detail above with reference to the embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention. The scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A hand-object interaction method providing haptic feedback, characterized in that: The method includes the following steps: (1) Using the initial CAD model of the object as input, create a sparse viewpoint model to represent the three-dimensional geometry of the object; render from multiple viewpoints around the object, and place a virtual camera on the vertices of a geodesic grid surrounding the object, which is created by subdividing the triangles of an icosahedron into four subdivisions. For all viewpoints, calculate the model points. and normal vector ; Homogeneous transformation of camera reference coordinate system and model reference coordinate system Rotation transformation and projection transformation To obtain the 2D-3D correspondence; Model points obtained from the nearest viewpoint of the sparse viewpoint model and normal vector The corresponding line model is defined by projecting it onto the image using the following equation: Where i represents the current line index, C represents the camera coordinate system, and M represents the model coordinate system. Indicates the center point of the corresponding line. This represents the normal vector of the corresponding line. Indicates projection transformation; (2) Render the virtual model of the target object in the center of the display screen. When the user manually aligns its outline and reaches the corresponding line threshold, start object tracking; extend the region algorithm SRT3D to high-speed object tracking in hand-object interaction scenarios. That is, the SRT3D algorithm achieves fast tracking through the corresponding lines of the region edge. During the pose update process, the SRT3D algorithm only calculates the pixel-level probability distribution noise of the foreground and background along the corresponding lines and approximates it as a normal distribution; the position of the distribution mean represents the maximum probability of the foreground outline; assuming that the calculated object pose value is close enough to the actual situation, the optimal distribution of each corresponding line It should approximate a normal distribution with a preset minimum variance. The Kullback-Leibler divergence is used to measure the similarity between the current distribution and the optimal distribution of the corresponding line, and this is used as the confidence level for estimating the tracking results. The calculation formula is as follows: In the above formula, N represents the number of corresponding lines. The threshold is indicated; the confidence level is used to determine whether an object is lost during tracking, and to promptly restart initialization when the score confidence level falls below the set threshold. (3) The MediaPipe-Hands model with GPU acceleration is used as a fast hand tracking module to detect the two-dimensional bounding box of the hand, which limits the computation to the hand area and improves efficiency; the HaMeR model is used for three-dimensional gesture estimation and hand mesh reconstruction. This vision-based attention mechanism model can accurately recover hand movements in complex gestures and interactive occlusions, and reconstruct the fitted hand mesh. (4) The effect of hand occlusion on object tracking is eliminated to the greatest extent by filtering method. The specific content of the filtering method is: first, a hand mask is generated by rendering the three-dimensional hand model of the current image frame in real time; then, before normal optimization, the pixel center of each corresponding line is determined, the corresponding lines whose center is located in the hand mask are filtered, and finally, the pose calculation is performed based on the remaining corresponding lines. When the occlusion area of ​​the hand is large, too many corresponding lines will be filtered out. If corresponding lines exceeding a set threshold are filtered out, the pose reconstructed from the hand is used to optimize object tracking. The pose transformation of the held object is performed based on the transformation matrix of the hand root node, that is, the coordinate transformation of the root node of the MANO hand model between the current frame and the previous frame is calculated. Then it is applied to transition prediction under heavy occlusion of the object's hand; (5) Based on the hand-object spatial pose estimated by the visual algorithm, define the process of real and virtual hand-object interaction mapping, and activate the virtual-real hand-object interaction through the defined gestures; the user determines how to grasp real and virtual objects according to the required interaction method, captures and records the current hand pose and position relative to the object, and visualizes the defined gestures on the grid of real and virtual objects; allows the user to customize the grasping position of virtual objects to ensure that the grasping action between physical objects and virtual objects is consistent with the expected interaction; the user achieves consistent mapping by defining the hand-object interaction input and output, generating a pair of triplets (real object, gesture, relative position) and (virtual object, gesture, relative position), which are stored to facilitate subsequent grasping matching and mapping, and realize multiple virtual object mappings according to different ways of grasping objects; (6) Traverse all gestures associated with real objects, calculate the Cartesian distance between the current gesture and the defined gesture, and for each grasping action within the set threshold range, calculate the matching score according to the following formula. : In the above formula, Joint represents the number of joints in the hand. This represents the forward difference of the j-th hand joint. Let represent the upward difference of the j-th hand joint. This indicates the position of the j-th hand joint. This indicates the position of the j-th hand joint. · · In the above formula, This represents the rotation of the j-th hand joint. This represents the rotation of the defined j-th hand joint. This represents the forward vector in the Unity engine. This represents the upward vector of the Unity engine; the gripping action with the highest matching score will be selected as the final gesture in the mapping process.

Images