Virtual hand grabbing method and system for mixed reality interaction

Abstract

The invention discloses a virtual hand grabbing method and system for mixed reality interaction, which is based on a hinge proxy hand model, utilizes a hand tracking device and a physical engine, and fuses hand key point tracking data and a force closing algorithm, and is a grabbing method which is based on physical simulation, is real-time and efficient and does not need to predefine object information. High-precision and real-time hand-object interaction is realized, a virtual hand observed by a user in a mixed reality picture is effectively prevented from penetrating through a virtual object, and the interaction experience in VR / AR equipment is remarkably improved. Moreover, the method supports the real-time adjustment of the dynamic grabbing gesture based on the surface of the object, and is suitable for various VR / AR devices.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a virtual hand grasping method and system for mixed reality interaction. Background Technology

[0002] In virtual reality (VR) and augmented reality (AR) applications, the natural interaction of users' hands with virtual objects is a key technology for enhancing immersion and operational accuracy.

[0003] Currently, mainstream hand interaction technologies mainly include grasping methods based on gesture recognition and raycasting. These methods typically rely on predefined gestures or use ray-based clicks to select targets, and are widely used in current VR / AR products due to their simplicity and high stability. However, these methods often lack realistic tactile and visual feedback, causing users to not perceive the physical constraints of real grasping behavior during interaction, thus weakening immersion and naturalness. Summary of the Invention

[0004] This application provides a virtual hand grasping method and system for mixed reality interaction, which can meet the dual requirements of immersive applications for natural interaction and performance.

[0005] This invention provides a virtual hand grasping method for mixed reality interaction, comprising: Based on real-time collection of joint data of the user's real hand, a hinge proxy hand model with a preset number of degrees of freedom is constructed. The driving hinge proxy hand model moves realistically and naturally within the physics engine; Real-time detection of the contact between the driven hinged proxy hand model and the virtual object is performed to determine the contact point information between the fingertip and the object surface. Using the defined contact point information, the generalized force vector of each contact point is calculated, and a closed space of the grasping force is constructed to evaluate the stability of the grasping posture. Based on the contact point information between the user's real hand posture and the virtual object, the finger joint angles of the hinge proxy hand model are updated in real time to maintain a natural grasping posture and avoid penetration.

[0006] In one exemplary instance, constructing a hinged proxy hand model with a preset number of degrees of freedom includes: Acquire the real-time three-dimensional joint data of the user's actual hand; Based on the collected joint data and hand skeletal structure, a virtual proxy hand model with the preset number of degrees of freedom corresponding to the real hand is constructed; Based on the established virtual proxy hand model, a corresponding rigid body hand model is constructed in the physics engine, and the rigid body components are combined into a movable hinged proxy hand model through hinge connections.

[0007] In one exemplary instance, the construction of the corresponding rigid body hand model in the physics engine, and the assembly of the rigid body components into a movable hinged proxy hand model via hinge connections, includes: Iterate through each joint node defined in the virtual agent hand model and generate a corresponding physical rigid body component for each joint node; Set physical properties for each rigid body component; Based on the connection relationship between nodes in the virtual proxy hand model, hinge connections are added between adjacent rigid body components, and a one-dimensional spring-damping actuator is bound to each hinge joint to combine the rigid body components into a movable hinge proxy hand model.

[0008] In one exemplary instance, the driving hinge proxy hand model performs realistic and natural motion within a physics engine, including: Based on the collected motion data of the user's real hand, the spatial position of each joint of the hand in the virtual physics engine is determined; Based on the determined spatial position, the external force and torque required to drive the hinge proxy hand model to move are calculated and applied to the corresponding joints in the hinge proxy hand model; Based on wrist-driven motion, the collected target joint angles are used to drive the rotation of each joint, generating the natural finger movements.

[0009] In one exemplary instance, generating the natural finger movement includes: During the wrist movement control process, the position and posture of the user's real wrist are used as a reference to drive the overall translation and rotation of the palm part in the virtual proxy hand model in the physics engine; During the finger joint driving process, the hinge joint of each finger joint is controlled to rotate according to the collected actual joint angle changes.

[0010] In one exemplary instance, determining the contact point information between the fingertip and the object surface includes: A double-layer structure is provided at each fingertip of the hinged proxy hand model. The outer layer is a lightweight detection layer for contact sensing, and the inner layer is a rigid collision body for physical simulation. When the hinged proxy hand model is driven close to the surface of a virtual object in the physics engine, the outer detection layer triggers an event, identifies the contact triangle, and records its surface normal vector as the contact direction reference in the contact point information; when the inner rigid collider collides with the virtual object in a real rigid collision, the intersection point from the fingertip center to the collider surface is traced in reverse according to the detected normal vector direction to obtain the contact point coordinates in the contact point information.

[0011] In one exemplary instance, the calculation of the generalized force vector at each contact point and the construction of a gripping force closed space to evaluate the stability of the gripping posture includes: Based on the position and normal vector of the contact point, calculate the contact force vector of each contact point; The contact force vector is converted into an equivalent generalized force acting on the centroid of the target object, and a grasping mapping matrix for each contact point is constructed based on the equivalent generalized force. Combine the grasping mapping matrices of all contact points to form a global grasping mapping matrix, and determine that the force closure condition is satisfied when the rank of the global grasping mapping matrix is ​​6.

[0012] In one exemplary instance, the real-time updating of the finger joint angles of the hinged proxy hand model to maintain a natural grasping posture and avoid penetration includes: Based on the joint positions of the user's actual hand, the target contact points are generated by projecting them onto the surface of a virtual object. Define the rotation transformation matrix of the hinge connection and calculate the current fingertip position vector; Calculate the error vector between the target fingertip position vector and the current fingertip position vector; construct the Jacobian matrix of the position vector with respect to the angles of each joint; map the error vector to the angle update of each joint by taking the pseudo-inverse of the Jacobian matrix; Based on the actual movement of the user's hand, the joint angles of the hinge proxy hand model are updated in real time using the angle update amount of each joint to ensure that the grasping posture is natural and without penetration.

[0013] This application also provides a computer-readable storage medium storing computer-executable instructions for executing the virtual hand grasping method for mixed reality interaction described above.

[0014] This application embodiment further provides a virtual hand grasping system for mixed reality interaction, including: a hinge proxy hand module, a contact point detection module, a grasping judgment module, and a posture update module; wherein, The hinge proxy hand module is used to construct a hinge proxy hand model with a preset number of degrees of freedom based on real-time collected joint data of the user's real hand; and drive the hinge proxy hand model to perform realistic and natural movements in the physics engine. The contact point detection module is used to detect the contact between the driven hinged proxy hand model and the virtual object in real time, and determine the contact point information between the fingertip and the object surface. The grasping judgment module is used to calculate the generalized force vector of each contact point using the determined contact point information, and construct the grasping force closed space to evaluate the stability of the grasping posture. The posture update module is used to update the finger joint angles of the hinge proxy hand model in real time based on the contact point information between the user's real hand posture and the virtual object, so as to maintain a natural grasping posture and avoid penetration.

[0015] The virtual hand grasping method for mixed reality interaction provided in this application is based on a hinged proxy hand model. Utilizing a hand tracking device and a physics engine, and by fusing hand keypoint tracking data and a force closure algorithm, it is a physical simulation-based, real-time, efficient grasping method that does not require predefined object information. This achieves high-precision, real-time hand-object interaction, effectively preventing the virtual hand from penetrating virtual objects as observed by the user in mixed reality scenes, significantly improving the interactive experience in VR / AR devices. Furthermore, this application supports real-time adjustment of dynamic grasping gestures based on the object surface, adapting to various VR / AR devices.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0017] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0018] Figure 1 This is a flowchart illustrating the virtual hand grasping method for mixed reality interaction in this application embodiment; Figure 2 This is a schematic diagram illustrating the process of the virtual hand grasping method for mixed reality interaction in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the construction process of the hinge proxy hand model in an embodiment of this application; Figure 4 This is a schematic diagram of an embodiment of contact point detection and non-penetrating grasping in this application. Figure 5 This is a schematic diagram of another embodiment of contact point detection and non-penetrating grasping in this application; Figure 6 This is a schematic diagram of the composition structure of the virtual hand grasping system for mixed reality interaction in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of a virtual hand grasping system for mixed reality interaction in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be arbitrarily combined with each other.

[0020] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0022] It is understood that the terms "first" and "second" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0023] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0024] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0025] The steps illustrated in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in a different order than that presented here.

[0026] To overcome the limitations of traditional methods, recent research has focused on physics-based hand-object interaction simulation. These methods attempt to achieve more realistic grasping and manipulation behaviors by constructing collision detection, rigid body dynamics calculations, and contact force modeling between the hand and virtual objects. However, these simulation methods face several technical challenges: First, human hand movements have high degrees of freedom, making it difficult to accurately control the motion mapping of the virtual hand, often resulting in distorted or unnatural surrogate hand postures. Second, the lack of realistic haptic feedback mechanisms allows the user's real hand to easily penetrate objects in the virtual scene, disrupting physical consistency. Third, many methods still rely on predefined grasping posture templates, making it difficult to adapt to new objects of varying shapes or unknowns, lacking versatility and adaptability. Finally, complex physics simulations and high-precision collision detection computationally expensive, making it difficult to achieve high frame rates and low latency real-time interactive experiences on computationally limited head-mounted VR / AR devices.

[0027] Therefore, the technical problems that urgently need to be solved include: how to achieve high-precision and natural hand-object contact modeling; how to design a dynamic grasping judgment mechanism with adaptive capabilities; and how to improve the computational efficiency and system stability of physical simulation while ensuring real-time performance, so as to meet the dual requirements of immersive applications for natural interaction and performance.

[0028] To address at least one of the aforementioned problems, embodiments of this application provide a virtual hand grasping method for mixed reality interaction, applicable to VR or AR environments, and meeting the dual requirements of immersive applications for natural interaction and performance. Combined with... Figure 2 The embodiments of this application provide a virtual hand grasping method for mixed reality interaction, as follows: Figure 1 As shown, it may include: Step 100: Based on real-time collected joint data of the user's actual hand, construct a hinge proxy hand model with a preset number of degrees of freedom.

[0029] In one exemplary instance, step 100 may include: To acquire real-time 3D spatial joint data of the user's actual hand, for example, the position and angle of each joint can be collected by a hand tracking device and transmitted in real time via network protocols (such as TCP / IP). In other words, the joint data collected by the hand tracking device can be transmitted in real time to the main system running virtual agent hand modeling and physical simulation via TCP / IP protocol, ensuring stable and efficient data and supporting decoupled module operation. Alternatively, the joint data of the hand can also be captured by the camera built into the AR / VR device.

[0030] A virtual proxy hand model corresponding to the real hand is pre-constructed based on the hand's skeletal structure; the movement of the virtual proxy hand model is driven by the collected joint data. Based on the established virtual proxy hand model, a corresponding rigid body hand model is constructed in the physics engine, and the rigid body components are combined into a movable hinged proxy hand model through hinge connections.

[0031] In one exemplary instance, during system initialization, to ensure that hand data can be transmitted to the main processing system in real time and stably, the spatial position and angle information of each joint can be directly acquired through a hand tracking device and input into the host (such as a PC or AR / VR system) for subsequent processing. This communication mechanism allows the hand tracking device to continuously send the collected hand joint position and posture data at a high frequency (such as 60Hz or higher), while the host acts as the receiving end, receiving and processing this data in real time to drive the virtual proxy hand model. The use of the TCP / IP protocol offers advantages in versatility and module decoupling, enabling stable operation of data acquisition and physical modeling across different hardware platforms or processes, which is beneficial for system expansion and remote collaboration. The system performs coordinate system initialization based on the first frame of input data after communication is established. Specifically, the position and posture of the user's real hand at startup are set as the origin and reference direction of the system space, and the initial pose of the virtual proxy hand model is aligned accordingly. This calibration process ensures that the real hand and the virtual hand maintain consistency in spatial position and orientation, thereby avoiding problems such as hand drift and posture misalignment caused by initial deviations. This mechanism enables an effective mapping between virtual and real spaces, providing an accurate coordinate basis and initial reference for subsequent gesture-driven, interaction collision, and grasping judgments. In other words, the initialization step, through the joint processing of communication link establishment and posture alignment, creates a stable, efficient, and accurate foundational environment for the subsequent motion control and interactive response of the hinged proxy hand model.

[0032] In one embodiment, such as Figure 3 As shown in Part A, hand-tracking devices (such as VR headset cameras, Leap Motion, or inertial-optical fusion data gloves) can acquire real-time joint data in three-dimensional space, including the wrist, metacarpophalangeal joints (MCP), proximal interphalangeal joints (PIP), and distal interphalangeal joints (DIP), totaling a preset number of 26 degrees of freedom. This data includes three-dimensional spatial coordinates and orientation information. This joint data is typically represented by the pose (position + rotation angle) of the joint points and is uniformly transformed into a system-defined coordinate system to ensure coordinate consistency during subsequent virtual agent hand-driven processes. Figure 3 The examples shown in Part A illustrate how each joint supports a different number of degrees of freedom (DOF). The MCP typically supports two rotational degrees of freedom, corresponding to rotations around the Z and Y axes, representing flexion movements of the finger root joints in two directions. The PIP and DIP each support one rotational degree of freedom, typically rotating around the Y axis. The wrist joint can support six degrees of freedom, including position (X / Y / Z) and planar pose (rotation around the X / Y / Z axes). This information provides a physiological basis for the subsequent construction of virtual skeleton models. Figure 3 The example shown in Part B of the diagram shows a scene captured with a bare hand on the left and a tracking scene captured with gloves on the right. The local coordinate axes for each joint are (x, y, z). The small arrows indicate the spatial position and orientation information of the joints acquired by the hand tracking device, showing that this data is the input source for building the virtual proxy hand model.

[0033] In one embodiment, a virtual proxy hand model with a preset number of degrees of freedom (DOF), such as 26, can be constructed based on the human hand's anatomical structure, i.e., the hand's skeletal structure, and the collected joint data. This can be understood as 26 DDF for a single hand and 52 DDF for both hands. This virtual proxy hand model uses the wrist as a reference point, connecting joints such as the metacarpophalangeal joints and knuckles according to a parent-child hierarchy to form a complete skeletal topology. By constructing a virtual proxy hand model corresponding to the real hand based on the collected joint data and hand skeletal structure, a skeleton control model with the correct topology and degrees of freedom is created to express "how the hand should move." The virtual proxy hand model is a logical structure representing a 26-DOF hand motion framework, without physical attributes (such as mass or collision volume), i.e., a virtual skeleton (joint nodes + connection relationships + posture targets), serving as a control reference for constructing a realistic physical model.

[0034] Each finger is composed of multiple joints connected together. In one embodiment, a hinge joint can be used to restrict the joints to move only on physiologically permissible axes of rotation, thereby improving the naturalness and controllability of the movement.

[0035] In one embodiment, the virtual hand model can adopt a simplified collision model, with the palm as a cube and the finger joints as capsules, and rigid body collision and detection with virtual objects can be achieved through the PhysX physics engine.

[0036] After constructing the virtual proxy hand model (i.e., the skeleton model), a rigid body hand model with the same structure can be generated in the physics engine to achieve interactive physical simulation behavior. This process can include several key steps such as rigid body generation, property setting, constraint connection, and drive control, and may include: First, iterate through each joint node defined in the skeleton model (such as the palm, finger roots, knuckles, etc.) and generate a corresponding physical rigid body component for each joint node. For example... Figure 3 As shown in Section C, these rigid body components can be cubes (for the palm) or capsules (for the finger joints) in shape to simplify collision calculations while preserving near-realistic geometry. The initial spatial position and orientation of the rigid body components are derived from the node coordinates in the skeleton model, ensuring consistent alignment of the overall structure.

[0037] Subsequently, physical properties are assigned to each rigid body component, including the dimensions of the collider (e.g., a capsule for the finger, a cube for the palm), its mass, moment of inertia, coefficient of friction, joint motion damping, and joint stiffness. In one embodiment, the mass and moment of inertia can be proportionally distributed according to the dimensions of the rigid body component to ensure that the overall hand has reasonable dynamic response characteristics in the physical simulation. The collider parameters determine the accuracy and effect of the rigid body component when in contact with other objects, and are the foundation for subsequent grasping, collision, and reaction force feedback.

[0038] After creating the rigid body components, you can add hinge joints between adjacent rigid body components based on the connection relationships between nodes in the skeleton model, such as... Figure 3 The example shown in Part D illustrates a complete rigid body surrogate hand structure, with hinges connecting each phalanx. The rotational axes are labeled with triaxial arrows. White curved arrows indicate rotational degrees of freedom (e.g., uniaxial flexion and extension). These hinge constraints limit relative rotation between two rigid body components to specific axes (e.g., phalanges can only move in the flexion and extension directions). Furthermore, angle limits, damping coefficients, and elastic parameters can be set to match the actual physiological movement characteristics of the human hand, preventing excessive rotation or structural misalignment. Figure 3 The example shown in section E illustrates the hinge model used in the physics engine for the knuckle joint, clarifying the single-axis rotation range and angle control method, demonstrating that the model simulates the flexion and extension behavior of a real joint using one-dimensional rotation plus constrained angles.

[0039] To drive rigid body components to move in a physics engine according to realistic hand movements, a one-dimensional spring-damped actuator can be attached to each hinge joint. This actuator calculates the difference between the current posture of the rigid body component and the target angle of the skeleton model in real time, and generates a driving torque based on the set stiffness and damping coefficients. This allows the rigid body component to automatically approach the target posture, achieving natural and physically consistent rotational control. The soft constraint method provided in this embodiment maintains the smoothness of hand movements while preventing abrupt jumps or mechanical conflicts.

[0040] In this embodiment, the rigid body hand model structure is integrated and runs in a physics engine such as NVIDIA PhysX, ensuring that all rigid body motion, collision, and contact feedback are uniformly calculated and processed by the underlying engine. Under the architecture of this embodiment, the hinged proxy hand model can realistically collide with objects in the virtual scene, receive reaction forces, and achieve natural grasping, pressing, or blocking behaviors.

[0041] By constructing a corresponding rigid hand structure in the physics engine based on the established virtual proxy hand model, and combining the rigid bodies into a movable hinge proxy hand model through hinge connections, the virtual hand can not only move, but also move reasonably and collide in the physical world. This completes the mapping from a pure logic control structure (skeleton) to a real physical interactive body (rigid hand structure), generating a physical version of the virtual hand, namely the hinge proxy hand model, that can participate in the interaction of the virtual physical world and has natural dynamic response. This provides technical support for subsequent highly immersive gesture interaction, controllerless grasping, and anti-penetration processing.

[0042] Step 101: Drive the hinge proxy hand model to move realistically and naturally in the physics engine.

[0043] In one exemplary instance, step 101 may include: Based on the collected motion data of the user's real hand, the spatial position of each joint in the virtual physics engine is determined; based on the determined spatial position, the external force and torque required to drive the hinge proxy hand model to move are calculated and applied to the corresponding joints in the hinge proxy hand model; based on the wrist drive, the collected target joint angles are used to drive the rotation of each joint to generate natural finger movements.

[0044] To drive the hinged proxy hand model to move realistically and naturally within the physics engine, ensuring its global position and local joint movements closely match the user's actual hand, and providing a controllable gesture state for subsequent contact detection and grasping judgment, in one embodiment, the external forces and torques driving the hinged proxy hand model's movement within the physics engine are calculated. Based on wrist-driven motion, the collected target joint angles are used to drive the rotation of each joint, generating natural finger movements, which may include: First, during wrist motion control, the position and posture of the user's actual wrist are used as a reference to drive the overall translation and rotation of the palm in the virtual proxy hand model within the physics engine. Since directly forcing the rigid body position would disrupt physical consistency, a method based on external force and torque can be used for driving. Specifically, the spatial deviation vector between the actual wrist position and the virtual wrist is calculated. And generate the translational driving force according to formula (1): (1) In formula (1), Indicates the applied linear force. Indicates the mass of the rigid body at the palm. This represents the first adjustment coefficient. This indicates the simulation time step.

[0045] For attitude control, the system calculates the angle difference. And based on the moment of inertia of the rigid body The required rotational torque is calculated as shown in formula (2): (2) In formula (2), This represents the second adjustment coefficient.

[0046] In this embodiment, based on the collected motion data of the user's real hand, by continuously applying the translational driving force and rotational torque shown in formulas (1) and (2) to the physics engine, the virtual proxy hand can smoothly track the overall motion trajectory of the user's real hand, while retaining the collision response and dynamic inertial characteristics, thereby ensuring that the interaction process is natural and not abrupt, and realizing physical realism and operational consistency.

[0047] Then, during finger joint actuation, the system controls the rotation of each finger joint's hinge joint based on the collected changes in actual joint angles. To simulate a dynamic process consistent with human movement characteristics, the system uses a one-dimensional spring-damping model to calculate the driving torque. In each frame, the system acquires the current target joint angle. Angle from the previous frame Current target joint angular velocity Angular velocity compared to the previous frame And calculate the joint drive torque according to formula (3). (3) In formula (4), where, Indicates the joint drive torque. Indicates the spring stiffness coefficient. The damping coefficient is represented. In this embodiment, the joint target angle is calculated using the actual joint position vector, enabling the fingers to perform bending, extending, and pressing movements naturally and smoothly without overshooting or oscillation, while realistically responding to the user's hand intentions. This ensures that the hinge proxy hand model moves naturally and in accordance with physical rules.

[0048] Through the coordinated use of the wrist and finger two-level control mechanisms in this embodiment, the hinged proxy hand is ensured to have high synchronization and high physical consistency in the physics engine, providing technical support for achieving natural gesture interaction and rigid body collision.

[0049] Through steps 100 and 101, a virtual hand model with realistic physical behavior is constructed in a VR or AR environment. This virtual hand model can be driven in real time based on the user's real hand movements to interact naturally, accurately, and controllably with virtual objects (such as grasping, colliding, and manipulating), without the need for a traditional controller. In other words, a control and simulation system is realized that allows the virtual hand to move naturally, grasp reasonably, and collide realistically in the physical world like a real hand, providing a physical basis for achieving natural hand interaction.

[0050] Step 102: Real-time detection of the contact between the driven hinge proxy hand model and the virtual object to determine the contact point information between the fingertip and the object surface.

[0051] In one exemplary instance, step 102 may include: A double-layer structure is set at each fingertip of the hinged proxy hand model. The outer layer is a lightweight detection layer for contact sensing, and the inner layer is a rigid collider for physical simulation. When the hinged proxy hand model is driven close to the surface of a virtual object in the physics engine, the outer detection layer first triggers an event, identifies the contact triangle, and records its surface normal vector as a contact direction reference in the contact point information; when the inner rigid body collider collides with the virtual object in a real rigid body collision, it traces the intersection point from the fingertip center to the collider surface in reverse according to the detected normal vector direction, and obtains the contact point coordinates in the contact point information.

[0052] In this step, the contact point information includes the three-dimensional position coordinates of the contact point and its corresponding normal vector. In one embodiment, this information can be extracted in real time through the physics query interface in the Unity engine and transmitted to the host system via TCP / IP communication or an in-engine script. The extracted contact point information can be used for subsequent grasping state determination, hand posture adjustment, collision force feedback, or anti-penetration control, etc.

[0053] like Figure 4As shown in Part A, during user operation, the real hand is captured by a tracking device, and its position may penetrate virtual objects (such as...) due to the lack of realistic collision feedback. Figure 4 The hinged proxy hand model is driven by a real hand in the physics engine and is subject to collision limitations imposed by the virtual object's surface, thus automatically stopping outside the contact surface when approaching the virtual object. To accurately estimate this contact behavior, in this embodiment, a double-layer structure including the aforementioned outer and inner layers is provided at each fingertip of the hinged proxy hand model. In one embodiment, the outer lightweight detection layer can be a mesh contact sensing layer conforming to the fingertip contour, used to identify whether it is in contact with a triangular mesh facet of the virtual object and to obtain the normal vector direction of the mesh facet. The rigid collider can be designed as a capsule and nested inside the detection layer for subsequent inverse kinematics of the actual collision response and contact point position.

[0054] like Figure 4 Part B illustrates how the contact point detection method handles the evolution of contact states during dynamic processes from the perspective of motion trajectory. The figure shows the process of a finger gradually moving from state 1 to state 4, with the actual finger joint position (denoted as...) , (representing different states) continuously penetrates the object, while the proxy hand, due to physical constraints, has its joint positions (denoted as...) The contact point formed on the surface of the object will then remain. Nearby. In each state, there is a spatial offset between the real finger and the proxy hand. However, the contact point of the proxy hand is always used as the reference, avoiding erroneous interactive judgments caused by the penetration of the real hand. This application's embodiments support multi-finger contact and simultaneous multi-point detection, making it particularly suitable for target objects with complex concave and convex shapes.

[0055] In one embodiment, in order to improve the overall system operating efficiency, a simplified collision model and mesh filtering strategy can be adopted in the contact point detection process to avoid overcomputation or delay problems in complex mesh scenarios, thereby ensuring that the system maintains real-time interactive capabilities.

[0056] The contact point detection method in step 102 effectively solves the common problems of penetration and misjudgment in traditional natural hand interaction by modeling the difference between physically penetrable real hand input and contact with a proxy hand with a rigid structure. Through step 102, high-precision contact point positioning between the hinged proxy hand and any virtual object is achieved, supporting interactive operations on irregular, concave and convex surfaces or non-predefined objects, enhancing the system's versatility and natural interactive experience.

[0057] Step 103: Using the determined contact point information, calculate the generalized force vector for each contact point and construct the gripping force closed space to evaluate the stability of the gripping posture.

[0058] In one exemplary instance, a fast and stable real-time assessment of the grasping posture can be achieved based on force closure theory to determine whether the current multi-finger contact can stably control the target object. Using determined contact point information such as contact point position and surface normal vector, the generalized force vector (including normal force, frictional force, and frictional torque) at each contact point is calculated, and a closed space of grasping force is constructed based on Coulomb friction cone constraints. Grasping stability can be determined by judging the rank of the grasping mapping matrix (e.g., rank(G) = 6), without requiring pre-configured object information. This supports real-time evaluation of the grasping posture stability and further allows estimation of the grasping force magnitude through virtual-to-real penetration depth.

[0059] In one exemplary instance, step 103 may include: Calculate the contact force vector at each contact point based on its location and normal vector. The contact force vector is converted into an equivalent generalized force acting on the center of mass of the target object. And based on equivalent generalized force Construct a crawling mapping matrix for each contact point ; Capture mapping matrix of all contact points Combined to form a global crawling mapping matrix And determine the global crawling mapping matrix. When the rank is 6, the force closure condition is satisfied.

[0060] In one exemplary instance, if the global crawling mapping matrix is ​​determined... If the rank is less than 6, it means that the contact point cannot exert sufficient constraints on the object in six degrees of freedom (3 translations and 3 rotations), which means the force closure condition is not satisfied. Other possibilities include: Instruct users to adjust gestures or add effective contact points to enhance the spatial coverage of grasping control.

[0061] If the global crawling mapping matrix is ​​determined If the rank is 6, the stability condition is satisfied. Then the current grasping posture is maintained (e.g., the gesture will be locked) and grasping state feedback is triggered (e.g., grasping animation, vibration feedback, object attached to finger, etc.).

[0062] In one embodiment, calculating the contact force vector at each contact point may include: Calculate the generalized force at each contact point based on its spatial location and the corresponding surface normal vector. Generalized force Including normal force, frictional force and frictional torque, satisfying the Coulomb friction cone constraint shown in formula (4), that is, its linear part should be inside the friction cone, and the rotating part is subject to torsional friction constraint, as shown in formula (4): (4) In formula (4), The spinor basis matrix represents the contact point. Represents the contact force vector. The friction constraint is expressed as shown in formula (5): (5) In formula (5), and These represent the linear friction coefficient and the torsional friction coefficient, respectively. Represents linear force components. This represents the torque component.

[0063] In one embodiment, the contact force vector is converted into an equivalent generalized force acting on the center of mass of the target object. And based on equivalent generalized force Construct a crawling mapping matrix for each contact point This can include: The generalized forces at each contact point are uniformly mapped to the center of mass of the target object, thus constructing the equivalent force exerted by the force on the overall rigid body state, i.e., the equivalent generalized force. And construct the corresponding crawling mapping matrix. This mapping can be achieved through the position vector of the contact point relative to the centroid of the target object. Local rotation matrix The transformation relationship is constructed as shown in formula (6): (6) In one embodiment, the mapping result of all contact points is the grasping mapping matrix. They can be combined into a global crawling mapping matrix. This is used to assess overall crawling capabilities. It involves judging the global crawling mapping matrix. The rank of the object is used to assess whether the current grasp satisfies the force closure condition (i.e., whether it is sufficient to impose a full-space 6-DOF constraint on the target object). If... If the current contact configuration has theoretical grasping stability, it indicates that the grasping is unstable or that no effective closure has been formed.

[0064] like Figure 5 As shown in section A, multiple contact points (small black dots) and their friction cones (the range of frictional force directions) are displayed. Each contact point corresponds to a force vector. This is mapped to a generalized force on the target object through spinor transformation. Multiple contact points combined form a global grasping mapping matrix. This is used to evaluate whether grasping provides sufficient spatial control. For example... Figure 5 As shown in Section B, this illustrates the contact point and force direction (white arrows) estimated by the system when a user grasps an object using a gesture. This reflects how the force direction and magnitude at each contact point are calculated / displayed under the grasping posture. The bottom right corner displays only the visual contact point model. Figure 5 As shown in section C, four small images illustrate the actual interaction results of a user attempting to grasp a cube or water bottle. Images (I) and (II) show failed grasps, marked as having unreasonable force distribution, while images (III) and (IV) show successful grasps, marked as having reasonable force distribution. Each small image overlays the contact point, the direction of the contact force, and whether the object remains stable, reflecting whether the actual grasp meets the requirements. Direct impact on crawling stability.

[0065] In one exemplary instance, to estimate the actual gripping strength, the depth to which a real hand penetrates a virtual object can be used as an indirect measure of contact force. The greater the penetration depth, the greater the gripping force applied by the user is presumed. This value can be recorded as an interaction parameter and further optimized in conjunction with haptic feedback or force field simulation.

[0066] In one embodiment, the grasping judgment algorithm can be implemented using C# scripts in the Unity environment. Real-time performance is achieved simply by embedding the calculation process into each frame's physical update. The judgment result can be displayed in real-time through the user interface (UI), providing feedback to the user on whether the current grasping state is stable and whether the force closure condition is met, enhancing the transparency and interpretability of the system interaction. Furthermore, if the force closure condition is not met, the UI can instruct the user to adjust gestures or add effective contact points to enhance the spatial coverage of the grasping control.

[0067] Step 103, based on geometric contact information and tribophysical modeling, constructs a grasping mechanics judgment process that can run in real time without the need for a pre-set object model, supporting the feedback decision mechanism for multi-finger fine grasping in natural hand interaction scenarios.

[0068] Step 104: Based on the contact point information between the user's real hand posture and the virtual object, update the finger joint angles of the hinge proxy hand model in real time to maintain a natural grasping posture and avoid penetration.

[0069] It should be noted that there is no strict time order between steps 104 and 103. Step 104 can be performed as soon as the contact point information is obtained in step 102. Step 104 can run independently before or in parallel with the grasping judgment, always maintaining the visual naturalness and non-penetrating nature of the virtual agent hand, and the posture update is continuously executed regardless of whether the grasping is successful or not.

[0070] Step 104 can project the contact point onto the surface of the virtual object based on the real joint pose. That is, based on the penetration state between the real hand and the virtual object, the inverse kinematics algorithm is used to solve the joint angle change required for the target fingertip position, and the finger joint angle of the hinge proxy hand model is updated using the Jacobian matrix pseudo-inverse calculator to achieve continuous posture adjustment and natural motion feedback.

[0071] In one exemplary instance, step 104 may include: Based on the actual joint positions of the user's hand, the data is projected onto the surface of a virtual object to generate target contact points. ; Define the rotational transformation matrix of the hinge connection (such as the Z-axis and Y-axis rotation of the MCP joint), and calculate the fingertip position vector. ; Calculate the target fingertip position vector With the current fingertip position vector The error vector between the positions is calculated; the Jacobian matrix of the position vector with respect to the angles of each joint is constructed; the error vector is mapped to the angle update of each joint by taking the pseudo-inverse of the Jacobian matrix. Based on the user's actual hand movements, the joint angles of the hinge proxy hand model are updated in real time using the angle update values ​​of each joint to ensure a natural grasping posture without penetration.

[0072] In one embodiment, generating target contact points by projecting them onto the surface of a virtual object based on the actual hand joint position may include: Based on the spatial position of the joints of a real hand, the actual coordinates of the fingertips in three-dimensional space are projected onto the surface of a virtual object to obtain a target contact point. The target contact point, as the location the proxy hand needs to reach, is used to drive the inverse kinematics solution.

[0073] In one embodiment, based on the skeletal structure of each finger, a positive kinematic model consisting of multiple joint rotations is defined, i.e., a rotational transformation matrix for hinge connections is defined. For example, for the index finger, multiple rotational transformation matrices can be constructed, including the Z-axis and Y-axis rotations of the MCP joint and the Z-axis rotations of the PIP and DIP joints, and these can be combined into a total rotation-translation transformation. As shown in formula (7), the fingertip spatial position vector The coordinates can be calculated by applying this rotation-translation transformation to the base point coordinates: (7) In one embodiment, the error vector between the target fingertip position and the current fingertip position is... The partial derivatives of the position vector with respect to each joint angle can be described by the Jacobian matrix as shown in formula (8): (8) By analyzing the Jacobian matrix Seeking pseudo-inverse This allows us to obtain the angle update for each joint. Adjust your hand posture accordingly.

[0074] In this embodiment, during the dynamic adjustment phase—that is, when the joint angles of the hinge proxy hand model are updated in real time based on the movement of the real hand—inverse kinematics calculations and angle updates are continuously performed based on the continuous movement of the real hand. This ensures that during the grasping process, the virtual proxy fingers always maintain a natural curve, a smooth transition, and remain in contact with the virtual object surface. Thus, even if the user's real hand penetrates the object, the virtual proxy hand remains on the surface, forming a visual barrier, enhancing the realism and immersion of the interaction.

[0075] In one embodiment, the pose update algorithm can be implemented using C# scripts in the Unity engine and combined with the PhysX physics engine to complete the synchronous pose update. The calculated joint angle changes can be transmitted to the host system in real time via TCP / IP protocol for further analysis, physical feedback modeling, or rendering visualization and other subsequent module processing.

[0076] The pose update process in step 104 uses inverse kinematics and Jacobi solution strategies to map the penetrating input of the user's real hand to the non-penetrating, natural grasping state of the virtual proxy hand in real time, ensuring visual and physical consistency in mixed reality.

[0077] The virtual hand grasping method for mixed reality interaction provided in this application is a physical simulation-based, real-time, efficient grasping method that does not require predefined object information. Based on a hinged proxy hand model, this application utilizes a hand tracking device and a physics engine, and achieves high-precision, real-time hand-object interaction by fusing hand key point tracking data and force closure algorithms. This effectively prevents the virtual hand from penetrating virtual objects as observed by the user in mixed reality scenes, significantly improving the interactive experience in VR / AR devices. Furthermore, this application supports real-time adjustment of dynamic grasping gestures based on the object surface, adapting to various VR / AR devices.

[0078] This application also provides a computer-readable storage medium storing computer-executable instructions for performing the virtual hand grasping method for mixed reality interaction described in any of the preceding claims.

[0079] This application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: steps for performing the virtual hand grasping method for mixed reality interaction described in any of the preceding claims.

[0080] This application also provides a virtual hand grasping system for mixed reality interaction, combined with Figure 6 and Figure 7 It may include: a hinge proxy hand module, a contact point detection module, a grasping judgment module, and a posture update module; among which, The hinge proxy hand module is used to construct a hinge proxy hand model with a preset number of degrees of freedom based on real-time collected joint data of the user's real hand; and drive the hinge proxy hand model to perform realistic and natural movements in the physics engine. The contact point detection module is used to detect the contact between the driven hinged proxy hand model and the virtual object in real time, and determine the contact point information between the fingertip and the object surface. The grasping judgment module is used to calculate the generalized force vector of each contact point using the determined contact point information, and construct the grasping force closed space to evaluate the stability of the grasping posture. The posture update module is used to update the finger joint angles of the hinge proxy hand model in real time based on the contact point information between the user's real hand posture and the virtual object, so as to maintain a natural grasping posture and avoid penetration.

[0081] In one exemplary instance, the hinge proxy module may include: a joint data input submodule, a rigid body setting submodule, and a motion control submodule; wherein... The joint data input submodule is used to acquire real-time three-dimensional joint data of the user's actual hand. The Rigid Body Setup submodule is used to construct a virtual proxy hand model corresponding to the real hand based on the collected joint data and hand skeletal structure; based on the established virtual proxy hand model, it constructs the corresponding rigid body hand model in the physics engine, and combines the rigid body components into a movable hinged proxy hand model through hinge connections. The motion control submodule is used to calculate the external force and torque that drive the hinge proxy hand model to move in the physics engine based on the collected motion data of the user's real hand; on the basis of wrist drive, it uses the collected target joint angles to drive the rotation of each joint to generate natural finger movements.

[0082] In one exemplary instance, the contact point detection module may include: a collision detection submodule and a contact point generation submodule, wherein, The collision detection submodule is used to set up a double-layer structure at each fingertip of the hinged proxy hand model. The outer layer is a lightweight detection layer for contact sensing, and the inner layer is a rigid collider for physical simulation. The contact point generation submodule is used when the hinge proxy hand model is driven close to the surface of a virtual object in the physics engine. The outer detection layer first triggers an event, identifies the contact triangle, and records its surface normal vector as the contact direction reference in the contact point information. When the inner rigid body collider collides with the virtual object in a real rigid body collision, it traces the intersection point from the fingertip center to the collider surface in reverse according to the detected normal vector direction to obtain the contact point coordinates in the contact point information.

[0083] In one exemplary instance, the capture and judgment module is specifically used for: Based on the location and normal vector of the contact point, calculate the contact force vector at each contact point; then convert the contact force vector into an equivalent generalized force acting on the center of mass of the target object. And based on equivalent generalized force Construct a crawling mapping matrix for each contact point ; capture mapping matrix of all contact points Combined to form a global crawling mapping matrix And determine the global crawling mapping matrix. When the rank is 6, the force closure condition is satisfied.

[0084] In one exemplary instance, the crawling judgment module is further configured to: if it is determined that the global crawling mapping matrix... If the rank is less than 6, it instructs the user to adjust the gesture or add effective contact points to enhance the spatial coverage of the grasp control.

[0085] In one exemplary instance, the pose update module may include: an inverse kinematics solution submodule and a joint angle update submodule, wherein, The inverse kinematics solution submodule is used to project the actual hand joint positions onto the surface of a virtual object to generate the target contact points. Define the rotational transformation matrix of the hinge connection (such as the Z-axis and Y-axis rotation of the MCP joint), and calculate the fingertip position vector. ; Calculate the target fingertip position With the current fingertip position The error vector between the positions is calculated; the Jacobian matrix of the position vector with respect to the angles of each joint is constructed; the error vector is mapped to the angle update of each joint by taking the pseudo-inverse of the Jacobian matrix. The joint angle update submodule is used to update the joint angles of the hinge proxy hand model in real time based on the actual hand movement, using the angle update amount of each joint, to ensure that the grasping posture is natural and without penetration.

[0086] The virtual hand grasping system for mixed reality interaction provided in this application is based on a hinged proxy hand model. Utilizing a hand tracking device and a physics engine, and by fusing hand keypoint tracking data and a force closure algorithm, it is a physically-simulated, real-time, efficient grasping system that does not require predefined object information. This achieves high-precision, real-time hand-object interaction, effectively preventing the virtual hand from penetrating virtual objects as observed by the user in mixed reality scenes, significantly improving the interactive experience in VR / AR devices. Furthermore, this application supports real-time adjustment of dynamic grasping gestures based on the object surface, adapting to various VR / AR devices.

[0087] Figure 7 This application illustrates the working process of a virtual hand grasping system for mixed reality interaction provided in its embodiments, such as... Figure 7 As shown, the leftmost input section uses the previous frame (frame t−1) as the basis for inputting both left and right hand data. The user wears a hand tracking device (such as VR gloves or a camera) to collect real-time spatial pose and joint data of the real hand. Adjacent to the leftmost input section is the hinged proxy hand module, which consists of two subsystems. The real hand modeling submodule acquires the key joint positions of the real hand (as shown in the figure) and constructs a real hand skeleton model (real hand model). The hinged proxy hand modeling submodule, based on the real hand data, constructs a virtual proxy hand model with a rigid body structure and hinge connections. The virtual proxy hand consists of a rigid palm body and a capsule-shaped phalanx body. Through one-dimensional rotation control, it achieves a high degree of freedom representation and driving of the hand structure in the physics engine. Figure 7 The middle section showcases the physics engine calculations, including: a contact point detection module, which detects fingertip contact points based on the collision relationship between the proxy hand and the object; and outputs information such as the position and normal vector of the contact point; and a grasping judgment module, which calculates the global grasping mapping matrix based on the contact points. Determine whether the condition is met. This is to achieve the determination of gripping force closure. Figure 7 The top right corner is the posture update module. The real hand may pass through the virtual object, so the system performs posture constraints, including: displaying the real hand's penetration state; generating the target position by projecting the contact point; updating the posture of the hinge proxy hand through inverse kinematics; and ensuring that the proxy hand remains in contact with the object surface to maintain a natural gesture. Figure 7The bottom right corner contains the proxy hand grasping and output module. The updated proxy hand performs the actual grasping action; virtual objects can be attached to the proxy fingertips to achieve interactive control; the state is synchronously updated to frame t for output, and then the loop continues to the next frame. The system logic loop includes: inputting real hand data (t-1); driving the proxy hand model; performing contact detection and grasping judgment; if penetration occurs, performing posture update; executing the grasping response → outputting the proxy hand + virtual object state (t); feeding back to the next frame (t+1) to continue the loop processing.

[0088] Figure 7 This showcases a complete, multi-module coupled virtual grasping interaction system that integrates hand tracking, hinge proxy hand modeling, physical collision detection, grasping force closure judgment, inverse kinematics, and real-time updates. The system is particularly suitable for scenarios involving natural hand grasping interaction, effectively solving core issues such as hand penetration, unstable grasping, and feedback delay, thereby enhancing the immersiveness and physical realism of virtual interaction.

[0089] Although the embodiments disclosed in this application are as described above, the content described is merely for the purpose of understanding this application and is not intended to limit this application. Any person skilled in the art to which this application pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application; however, the scope of patent protection of this application shall still be determined by the scope defined in the appended claims.

Claims

1. A virtual hand grasping method for mixed reality interaction, characterized in that, include: Based on real-time collection of joint data of the user's real hand, a hinge proxy hand model with a preset number of degrees of freedom is constructed. The driving hinge proxy hand model moves realistically and naturally within the physics engine; Real-time detection of the contact between the driven hinged proxy hand model and the virtual object is performed to determine the contact point information between the fingertip and the object surface. Using the defined contact point information, the generalized force vector of each contact point is calculated, and a closed space of the grasping force is constructed to evaluate the stability of the grasping posture. Based on the contact point information between the user's real hand posture and the virtual object, the finger joint angles of the hinge proxy hand model are updated in real time to maintain a natural grasping posture and avoid penetration.

2. The virtual hand grasping method according to claim 1, wherein, The construction of the hinge proxy hand model with a preset number of degrees of freedom includes: Acquire the real-time three-dimensional joint data of the user's actual hand; Based on the collected joint data and hand skeletal structure, a virtual proxy hand model with the preset number of degrees of freedom corresponding to the real hand is constructed; Based on the established virtual proxy hand model, a corresponding rigid body hand model is constructed in the physics engine, and the rigid body components are combined into a movable hinged proxy hand model through hinge connections.

3. The virtual hand grasping method according to claim 2, wherein, The construction of the rigid body hand model corresponding to the physical engine, and the combination of the rigid body components into a movable hinged proxy hand model through hinge connections, includes: Iterate through each joint node defined in the virtual agent hand model and generate a corresponding physical rigid body component for each joint node; Set physical properties for each rigid body component; Based on the connection relationship between nodes in the virtual proxy hand model, hinge connections are added between adjacent rigid body components, and a one-dimensional spring-damping actuator is bound to each hinge joint to combine the rigid body components into a movable hinge proxy hand model.

4. The virtual hand grasping method according to claim 1, wherein, The driving hinge proxy hand model performs realistic and natural movements in the physics engine, including: Based on the collected motion data of the user's real hand, the spatial position of each joint of the hand in the virtual physics engine is determined; Based on the determined spatial position, the external force and torque required to drive the hinge proxy hand model to move are calculated and applied to the corresponding joints in the hinge proxy hand model; Based on wrist-driven motion, the collected target joint angles are used to drive the rotation of each joint, generating the natural finger movements.

5. The virtual hand grasping method according to claim 4, wherein, The generation of the natural finger movements includes: During the wrist movement control process, the position and posture of the user's real wrist are used as a reference to drive the overall translation and rotation of the palm part in the virtual proxy hand model in the physics engine; During the finger joint driving process, the hinge joint of each finger joint is controlled to rotate according to the collected actual joint angle changes.

6. The virtual hand grasping method according to claim 1, wherein, The determination of the contact point information between the fingertip and the object surface includes: A double-layer structure is provided at each fingertip of the hinged proxy hand model. The outer layer is a lightweight detection layer for contact sensing, and the inner layer is a rigid collision body for physical simulation. When the hinged proxy hand model is driven close to the surface of a virtual object in the physics engine, the outer detection layer triggers an event, identifies the contact triangle, and records its surface normal vector as the contact direction reference in the contact point information; when the inner rigid collider collides with the virtual object in a real rigid body collision, the intersection point from the fingertip center to the collider surface is traced in reverse according to the detected normal vector direction to obtain the contact point coordinates in the contact point information.

7. The virtual hand grasping method according to claim 1, wherein, The calculation of the generalized force vector at each contact point and the construction of a closed space for the gripping force to evaluate the stability of the gripping posture include: Based on the position and normal vector of the contact point, calculate the contact force vector of each contact point; The contact force vector is converted into an equivalent generalized force acting on the centroid of the target object, and a grasping mapping matrix for each contact point is constructed based on the equivalent generalized force. Combine the grasping mapping matrices of all contact points to form a global grasping mapping matrix, and determine that the force closure condition is satisfied when the rank of the global grasping mapping matrix is ​​6.

8. The virtual hand grasping method according to claim 1, wherein, The real-time updating of the finger joint angles of the hinged proxy hand model to maintain a natural grasping posture and avoid penetration includes: Based on the joint positions of the user's actual hand, the target contact points are generated by projecting them onto the surface of a virtual object. Define the rotation transformation matrix of the hinge connection and calculate the current fingertip position vector; Calculate the error vector between the target fingertip position vector and the current fingertip position vector; construct the Jacobian matrix of the position vector with respect to the angles of each joint; map the error vector to the angle update of each joint by taking the pseudo-inverse of the Jacobian matrix; Based on the actual movement of the user's hand, the joint angles of the hinge proxy hand model are updated in real time using the angle update amount of each joint to ensure that the grasping posture is natural and without penetration.

9. A computer-readable storage medium storing computer-executable instructions for performing the virtual hand grasping method for mixed reality interaction as described in any one of claims 1-8.

10. A virtual hand grasping system for mixed reality interaction, characterized in that, include: The system includes a hinge agent module, a contact point detection module, a grasping judgment module, and a posture update module; among them, The hinge proxy hand module is used to construct a hinge proxy hand model with a preset number of degrees of freedom based on real-time collected joint data of the user's real hand; and drive the hinge proxy hand model to perform realistic and natural movements in the physics engine. The contact point detection module is used to detect the contact between the driven hinged proxy hand model and the virtual object in real time, and determine the contact point information between the fingertip and the object surface. The grasping judgment module is used to calculate the generalized force vector of each contact point using the determined contact point information, and construct the grasping force closed space to evaluate the stability of the grasping posture. The posture update module is used to update the finger joint angles of the hinge proxy hand model in real time based on the contact point information between the user's real hand posture and the virtual object, so as to maintain a natural grasping posture and avoid penetration.

Images