Mixed reality physical entity interaction method and system based on hand skeleton tracking

Abstract

The invention discloses a mixed reality physical entity interaction method and system based on hand skeleton tracking, and belongs to the technical field of human-computer interaction. The problems that in existing mixed reality interaction, in-air gestures are low in precision and prone to fatigue, and a special controller is not portable and high in cost are solved. The method comprises the following steps: acquiring hand three-dimensional skeleton node data when a user grabs a passive physical entity through head-mounted computing equipment; at least two key skeleton nodes are selected, and control parameters representing the real-time relative space relation of the key skeleton nodes are calculated; and establishing mapping between the parameter and the target attribute of the virtual interaction element so as to drive virtual interaction in real time. According to the method, any common physical entity is temporarily converted into a high-precision interaction medium, and the method is mainly used for realizing natural and low-cost fine control with physical feedback in mixed reality application.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method and system for manipulating ordinary physical entities by tracking the user's hand bones in a mixed reality (MR), augmented reality (AR), or virtual reality (VR) environment, so as to achieve high-precision interaction with virtual information. Background Technology

[0002] With the rapid development of Mixed Reality (MR) technology, it is gradually becoming the next generation of mainstream computing platforms. However, current mainstream MR interaction methods still have many limitations, affecting the depth and breadth of the user experience.

[0003] In-Air Gestures: Represented by the "gaze-based pinch" gesture supported by Apple Vision Pro. When performing fine-grained operations such as dragging a progress bar or adjusting parameters, this type of interaction often results in low operational precision and easy user arm fatigue (the so-called "gorilla arm effect") due to the lack of stable physical feedback and effective support, thus creating an "execution gap" between the user's intention and the actual execution effect.

[0004] Dedicated Controllers: These are exemplified by the hand controllers included with mainstream MR devices such as the Meta Quest or Pico. While these controllers offer accurate six degrees of freedom (6DoF) tracking and physical feedback, their drawbacks are also significant: users need to carry the controllers at all times, resulting in poor portability; and when the controllers are needed, users must interrupt their current activity to find and pick them up, severely disrupting the immersive and smooth experience.

[0005] Other tracking solutions: For example, tracking solutions based on specific visual markers can interfere with the user's normal vision and have a significant delay; while solutions based on dedicated trackers, although providing stable tracking results, significantly increase the user's additional economic costs and equipment burden.

[0006] In existing technologies, there are also some interaction methods that use cameras to recognize gestures or object outlines. For example, Chinese patent CN105528082B discloses a three-dimensional space and gesture recognition and tracking interaction method, which achieves scene registration and gesture recognition by acquiring images, generating point clouds, and matching feature points. Another Chinese patent CN105681859A discloses a method for controlling a smart TV based on human skeleton tracking, which uses depth sensors such as Kinect to track the position of the elbow and palm to control cursor movement and function operation. These solutions have their application value in specific scenarios, but the former is complex and dependent on environmental features, while the latter is mainly for large-scale, low-precision cursor movement and is difficult to meet the needs of high-precision, continuous, and fine-tuned control in MR environments.

[0007] Therefore, there is an urgent need in this field for a new interaction solution that can provide real tactile feedback and stable support from physical entities, without relying on any dedicated controllers or visual markers, thereby achieving a more natural, convenient, low-cost, and high-precision refined human-computer interaction. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] This invention aims to solve the problems of insufficient accuracy and fatigue in air gesture interaction in existing mixed reality interaction technologies, as well as the poor convenience and high cost of dedicated controllers.

[0010] (II) Technical Solution

[0011] To address the aforementioned technical problems, this invention provides a mixed reality physical entity interaction method based on hand skeleton tracking. This method is executed on a head-mounted computing device and includes the following steps:

[0012] 1. Acquiring Hand Skeletal Data: The image sensor mounted on the head-mounted computing device acquires real-time images of a user's hand grasping or touching a physical entity, and uses the device's built-in hand tracking algorithm to obtain the three-dimensional skeletal node data of the hand. The physical entity is a passive, unmarked, ordinary object.

[0013] 2. Define control parameters: Select at least two key bone nodes from the three-dimensional bone node data, and calculate one or more control parameters for human-computer interaction based on the real-time three-dimensional spatial relationship of the at least two key bone nodes.

[0014] 3. Establish mapping relationship: Establish a preset mapping relationship between the calculated control parameters and the target attributes of one or more interactive elements in the virtual user interface (UI).

[0015] 4. Driving Virtual Interaction: Based on the mapping relationship, the changes in the control parameters are converted into changes in the target attributes of the virtual interaction elements in real time, and the display content of the head-mounted computing device is updated, thereby enabling users to perform fine-grained manipulation of virtual information by fine-tuning the finger posture of grasping physical entities.

[0016] Furthermore, the control parameters include, but are not limited to: the angle between the vector composed of at least two key skeleton nodes and the preset reference coordinate system, the magnitude or rate of change of the magnitude of the vector, the projection of the vector onto a specific plane, or the area or shape change of the geometry composed of multiple key skeleton nodes.

[0017] Furthermore, the present invention also provides a dynamic binding and unbinding mechanism, including:

[0018] By establishing a binding relationship between virtual interactive elements and physical entities through a first specific gesture (such as "pinch in the air - move closer - release"), the fine control mode is activated;

[0019] The binding relationship can be released by a second specific gesture (such as "quick swipe" or "double tap"), restoring the normal interaction mode.

[0020] As another aspect of the present invention, the present invention also provides a mixed reality physical entity interaction system based on hand skeleton tracking, which is deployed in a head-mounted computing device and includes:

[0021] Hand skeleton data acquisition module: Used to acquire three-dimensional skeleton node data of the user's hand in real time through image sensors and hand tracking algorithms.

[0022] Control parameter calculation module: Used to select key bone nodes from the three-dimensional bone node data and calculate control parameters that characterize their relative spatial relationships.

[0023] Interactive mapping and driving module: Used to convert the changes in the control parameters into changes in the target attributes of virtual interactive elements according to a preset mapping relationship, and drive the display to update.

[0024] (III) Beneficial Effects

[0025] Compared with the prior art, the present invention has the following significant advantages:

[0026] 1. Universality and Low Cost of Interaction: Any everyday object around the user (such as a water cup, pen, remote control, etc.) can be temporarily transformed into a high-precision interaction medium without the need for any built-in electronic components, external feature markers, or dedicated trackers. This greatly reduces the user's barrier to entry and economic costs, enhancing the universality of the technology.

[0027] 2. Natural physical feedback: Users interact by grasping real objects, receiving stable and intuitive physical support and tactile feedback. This effectively solves the problems of muscle fatigue caused by hovering gestures in mid-air and the inaccuracy caused by the lack of a frame of reference.

[0028] 3. Precision of Interaction: This invention uses the relative spatial relationships (such as angles and distances) between key skeletal nodes of the finger as control variables, transforming the user's minute, stable finger movements into high-precision control inputs. This is particularly suitable for interactive scenarios requiring continuous and fine-tuning, such as video editing, volume adjustment, and fine-tuning of 3D models.

[0029] 4. System Compatibility and Deployability: This method can be implemented on existing mixed reality all-in-one machines that support standard protocols such as OpenXR, without relying on any additional hardware. The algorithm logic can run efficiently on the device, exhibiting strong compatibility and practical deployability. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0031] Figure 1 This is a schematic diagram of the overall flow of the interaction method provided by the present invention.

[0032] Figure 2 This is a functional module structure diagram of the interactive system provided by the present invention.

[0033] Figure 3 This is a schematic diagram illustrating the principle of fine adjustment of the virtual slider based on the single-hand thumb rotation angle in Embodiment 1 of the present invention.

[0034] Figure 4 This is a schematic diagram illustrating the principle of virtual canvas movement and scaling based on the relative motion of two fingers in Embodiment 2 of the present invention.

[0035] Figure 5 This is a schematic diagram illustrating the principle of inferring the 6DoF pose of a handheld physical entity based on multiple bone nodes in Embodiment 3 of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0037] Example 1: Fine adjustment of virtual slider based on single-hand thumb rotation angle

[0038] This embodiment details a method for achieving high-precision, continuous, and jitter-free adjustment of slider controls in a virtual user interface (UI) by grasping a passive physical entity with one hand and utilizing the slight, stable rotation of the thumb.

[0039] I. System and Environment Setup

[0040] This embodiment is executed in a typical mixed reality (MR) environment. This environment includes:

[0041] Head-mounted computing device: A MR headset that integrates binocular or multi-camera setups, an inertial measurement unit (IMU), a processor, and a high refresh rate display (e.g., 90Hz).

[0042] Physical entity (307): A plain cylindrical water cup placed within the user’s reach, which does not have any electronic, communication or tracking functions.

[0043] Virtual object (306): A virtual volume control slider rendered on the display of the MR headset, with a value ranging from 0.0 to 100.0.

[0044] Hand tracking system: A software algorithm built into the head-mounted device that can output real-time 3D skeletal node data of the user's hands at a high frequency (e.g., above 60Hz). Each node data output by the algorithm includes at least its 3D position (x, y, z) in the world coordinate system.

[0045] II. Detailed Execution Steps

[0046] S101: Dynamic Binding and Mode Activation

[0047] S101a - Intent recognition and selection: The user (300) first performs a "pinch" gesture (the tips of the index finger and thumb touch) on the virtual slider (306) in the air. After the system recognizes the gesture, it copies a virtual slider (306) and shrinks it to a specific ratio, attaching it to the user's "pinch" gesture.

[0048] S101b - Binding Trigger: The user maintains a pinch gesture and moves their hand near the physical entity (307). The system defines an invisible spherical "binding area" around the cup, with a radius R_bind of, for example, 5 cm. When the user's hand enters this area and releases the pinch gesture, the system determines that the user's intention is to "place" the virtual slider on the physical entity.

[0049] S101c - Mode Switching: After successful binding, the system enters "Fine Control Mode". The virtual slider (306) may visually appear as a halo or establish a faint virtual connection line with the user's hand to provide status feedback. At this time, conventional air gesture interaction is suspended, and control is completely handed over to the fine control algorithm of this embodiment.

[0050] S102: Establishment of Control Reference Frame and Parameter Initialization

[0051] In the first frame of entering the fine control mode, the system immediately performs a one-time initialization operation to establish a baseline for subsequent real-time calculations.

[0052] S102a - Critical Node Locking: The system locks and continuously tracks the 3D world coordinates of the following four critical nodes from the hand skeleton data stream:

[0053] Thumb tip node P1 (301): HandThumbTip

[0054] Thumb root joint P2 (302): HandThumb3 or HandThumbMetacarpal

[0055] HandWristRoot (P_wrist)

[0056] The second joint node of the middle finger, P_mid: HandMiddle2

[0057] S102b - Dynamic Hand Coordinate System Construction: To eliminate the influence of overall hand movement and rotation on thumb fine-tuning measurements, the system constructs a local coordinate system Σ (304) that follows the palm in real time:

[0058] Define the "hand forward" vector: F_hand = (P_mid - P_wrist).normalized

[0059] Obtain the "world up" vector: U_world, usually (0, 1, 0), which can be calibrated by the gravity direction provided by the IMU.

[0060] Calculate the "right-hand" vector: R_hand = Vector3.Cross(F_hand, U_world).normalized. This vector is the X-axis of the coordinate system Σ. To prevent gimbal lock (when the palm is vertically upward or downward), if the magnitude of R_hand is too small (e.g., less than 0.1), the "world forward" vector (0, 0, 1) is used instead for cross product calculation.

[0061] Calculate the "hand upward" vector: U_hand = Vector3.Cross(R_hand, F_hand).normalized. This is the Y-axis. F_hand is the Z-axis.

[0062] S102c - Control Vector and Ratchet Anchor Point Initialization:

[0063] Construct a dynamic vector V (303) representing the thumb pose: V = P1 - P2.

[0064] Calculate the projection length of V onto the rightward vector R_hand of the hand; this is the core control signal. This projection length p is obtained through a dot product operation: p = Vector3.Dot(V, R_hand).

[0065] The system stores the calculated projection length p as the initial ratchet anchor point p_anchor. At the same time, it initializes an accumulated rotation value θ_accumulated to the angle corresponding to the current slider value (for example, if the slider is at 50%, then θ_accumulated is initialized to half of the total adjustment range).

[0066] S103: Real-time control parameter calculation and "ratchet mechanism" state management

[0067] After initialization, the system executes the following logic in a loop in each frame. This logic simulates a "ratchet" device, which only accumulates the maximum displacement in a single direction and ignores jitter and retraction during the process.

[0068] S103a - Real-time displacement calculation: In the current frame, the system recalculates the current thumb projection length p_current. Then it calculates its displacement relative to the anchor point p_anchor: Δp = p_current - p_anchor.

[0069] S103b - State Machine Management: The system maintains an internal state machine containing two states: "Idle" and "Rotating".

[0070] From Idle to Rotating: When the system is in the Idle state, if it detects that |Δp| ​​exceeds a preset activation threshold ε_act (e.g., 0.002 meters, or 2 millimeters), it considers that the user has started a new fine-tuning gesture. The system immediately switches to the Rotating state, records the current Δp as the peak displacement p_peak of this gesture, and resets a "still timer" T_still to 0.

[0071] In Rotating state:

[0072] The system continuously monitors Δp. If the direction of Δp is the same as p_peak, and |Δp| ​​> |p_peak|, it indicates that the user is continuing to fine-tune their thumb in the same direction. The system then updates p_peak = Δp and resets T_still to 0. This ensures that p_peak always records the farthest distance from the anchor point during this gesture.

[0073] If the user's thumb retracts, trembles, or pauses, |Δp| ​​will no longer exceed |p_peak|, at which point T_still begins to accumulate Time.deltaTime.

[0074] From Rotating to Idle: When T_still accumulates to more than a preset stillness duration T_duration (e.g., 0.3 seconds), the system determines that the fine-tuning gesture has ended.

[0075] S103c - Cumulative Value Update: The system performs a settlement the moment the gesture ends, i.e., the state switches from Rotating back to Idle.

[0076] Calculate the adjustment amount contributed by this gesture: Δθ = p_peak * S, where S is a sensitivity coefficient (e.g., 5000) used to amplify small physical displacements into significant virtual adjustment amounts.

[0077] Update the total cumulative adjustment value: θ_accumulated = θ_accumulated + Δθ. Also, limit the range of θ_accumulated to prevent it from exceeding the 0-100% range.

[0078] Key step: Reset the ratchet mechanism. Set the current projection p_current to the new anchor p_anchor and clear p_peak to zero. This way, the next fine-tuning gesture will restart the displacement calculation from the current thumb position.

[0079] S104: Mapping and Driving Virtual Interaction

[0080] The displayed value of the virtual slider (306) is updated in real time and smoothly, and its value consists of two parts: the settled cumulative value and the real-time displacement of the currently performing gesture.

[0081] Real-time target value calculation:

[0082] θ_target = θ_accumulated + (p_peak * S) (when in the Rotating state)

[0083] θ_target = θ_accumulated (when in the Idle state)

[0084] Mapping to UI value: The system converts θ_target into a 0-100 value Value_UI for the slider using a linear mapping function f(θ). For example: Value_UI = (θ_target / θ_total_range) * 100, where θ_total_range is a preset total adjustment required for the slider to accumulate from 0 to 100 (e.g., 1440 degrees).

[0085] Visual update: The system applies the calculated Value_UI to the rendering properties of the virtual slider (306), and the user can see the real-time, smooth change in the slider position.

[0086] S105: Unbinding and Mode Deactivation

[0087] When the user finishes adjusting and wants to unlock, a predefined "unbinding gesture" is executed. For example, the user quickly waves their index finger (or their entire hand) back and forth twice in the space between the virtual slider and the physical cup. The system's built-in spatiotemporal gesture recognizer detects this trajectory pattern (e.g., crossing a virtual "tripwire" plane twice within one second). Once the recognition is successful, the system immediately unbinds the user, the value of the virtual slider is fixed, and the system returns to the normal interaction mode.

[0088] Through the above steps, this embodiment achieves an extremely precise, stable, and intuitive fine control method, effectively solving the problems of fatigue and low precision in aerial gestures in the prior art.

[0089] Example 2: Virtual canvas movement and scaling based on the relative distance between two fingers

[0090] This embodiment details a method for performing intuitive and high-precision two-dimensional panning and zooming operations on virtual objects (such as pictures, maps, documents, etc.) by placing a hand on a physical surface, defining a dynamic two-dimensional control plane using the fingers of that hand, and then using the relative movement of one or two fingers.

[0091] I. System and Environment Setup

[0092] The execution environment of this embodiment is similar to that of Embodiment 1, including:

[0093] Head-mounted computing device: Functionally the same as the MR head-mounted display described in Example 1.

[0094] Physical surface (400): A flat, unactive, unmarked surface, such as an office desktop.

[0095] Virtual Object (404): An interactive virtual image rendered in the MR headset that the user wishes to pan and zoom.

[0096] Hand tracking system: The function is the same as described in Example 1, and it can output high-precision three-dimensional hand skeleton node data in real time.

[0097] II. Detailed Execution Steps

[0098] S201: Definition and Dynamic Binding of Control Regions

[0099] S201a - Control Gesture Recognition and Region Initialization:

[0100] The user places their left hand (407) naturally open and on the physical surface (400) with the palm facing down.

[0101] The system continuously monitors hand skeletal data. When it detects that the angle between the left palm normal vector and the world gravity vector is less than a preset threshold (e.g., 20 degrees), and at least four fingertips are on the same plane (allowing for a certain error) for more than a time window (e.g., 0.5 seconds), the system determines that the user has made a "ready to control" gesture.

[0102] Once recognition is successful, the system begins high-frequency tracking of the three-dimensional coordinates of the left thumb tip P_thumb (401) and the index finger tip P_index (402).

[0103] S201b - Visual Feedback and Region Rendering:

[0104] The system uses the real-time tracked P_thumb and P_index as diagonal vertices to render a semi-transparent virtual control area (403) on the physical surface that dynamically changes with finger movement. The color of this area, for example, light blue, indicates that it is in a "to be bound" state.

[0105] S201c - Binding Operations and Mode Activation:

[0106] The user uses their other hand (right hand) to grab the target virtual image (404) using a standard "pinch-drag" air gesture.

[0107] The user drags the virtual image above the blue virtual control area (403) and then releases the pinch gesture.

[0108] The system detects that the virtual image has been released within the Z-axis (perpendicular to the desktop) projection range of the control area, thus determining that the binding is successful.

[0109] After successful binding, the system enters "canvas control mode". As visual feedback, the color of the control area (403) changes from light blue to the active green, and the virtual image (404) will slightly "suck" to the center of the area.

[0110] S202: Real-time Data Acquisition and Preprocessing

[0111] In canvas control mode, the system continuously processes key node data to prepare for subsequent intent judgment.

[0112] S202a - High-Frequency Data Tracking: The system synchronously tracks the three-dimensional positions of P_thumb and P_index in the world coordinate system at the display refresh rate (e.g., 90Hz).

[0113] To eliminate sensor noise and minor jitter, the system applies a low-pass filter (e.g., the moving average of the past 5 frames) to the acquired raw coordinates to obtain smoothed coordinates P'_thumb and P'_index.

[0114] The system projects the smoothed 3D coordinates onto a 2D plane defined by the control region (403), resulting in 2D coordinates p_thumb = (x_t, y_t) and p_index = (x_i, y_i). All subsequent calculations are based on these 2D coordinates.

[0115] S202c - Velocity and Distance Calculation: In each frame, the system calculates:

[0116] Instantaneous speed: v_thumb = (p_thumb_current - p_thumb_previous) / Δt and v_index = (p_index_current - p_index_previous) / Δt.

[0117] Finger distance: D_current = distance(p_thumb, p_index).

[0118] S203: User Intent Arbitration and Pan Operation Execution

[0119] The system determines the user's intention by analyzing the movement of two fingers.

[0120] S203a - Intent Arbitration Logic: The system presets a speed threshold ε_v (e.g., 5 cm / s).

[0121] Translation intention: If |v_thumb| < ε_v and |v_index| > ε_v (i.e., the thumb is approximately stationary and the index finger is moving), then the user intention is determined to be translation (405).

[0122] Zooming intent: If |v_thumb| > ε_v and |v_index| > ε_v (i.e. both fingers are moving), then the user intent is determined to be zooming (406).

[0123] Idle state: Other situations (such as both fingers being still) are judged as having no intention to operate.

[0124] S203b - Translation operation execution:

[0125] When the system determines that the intention is to translate, it uses the two-dimensional displacement vector of the index finger in the current frame, Δp_index = p_index_current - p_index_previous, as the control input.

[0126] The system multiplies the two-dimensional displacement vector Δp_index by a sensitivity coefficient K_pan (which can be set by the user and defaults to 1.0) and then applies it to the three-dimensional world coordinates of the virtual image (404) (usually the XY plane of its local coordinate system).

[0127] Position_new = Position_old + K_pan * Δp_index.

[0128] This achieves the intuitive effect of "pressing down" the thumb as an anchor point and "dragging" the canvas with the index finger.

[0129] S204: Zoom operation executed

[0130] S204a - Initial state recording of zoom gesture: When the system state switches from "idle" or "pan" to "zoom" for the first time, the system will immediately record the finger distance D_current as the initial reference distance D_gesture_start for this continuous zoom gesture, and record the initial zoom ratio Scale_gesture_start of the virtual image at this moment.

[0131] S204b - Real-time scaling factor calculation: In each subsequent frame, as long as the system still determines a scaling intention:

[0132] The system continuously calculates the current finger distance D_current.

[0133] Calculate the scaling factor relative to the start of the current gesture: S_gesture = D_current / D_gesture_start.

[0134] S204c - Applying scaling and pivot point determination:

[0135] The calculated scaling factor is applied to the image scaling ratio at the start of the gesture to obtain the target scaling ratio: Scale_target = Scale_gesture_start * S_gesture.

[0136] To make the zooming operation feel natural (i.e., zooming around the center of the user's finger), the system dynamically sets the pivot point of the zooming operation at the midpoint between p_thumb and p_index.

[0137] The system updates the scaling property of the virtual image (404) to Scale_target and transforms it around the calculated pivot point to achieve a smooth and intuitive pinch scaling effect.

[0138] S205: Unbinding and Mode Deactivation

[0139] S205a - Unbinding Gesture Recognition: When the user wishes to end the operation, the entire left palm is swept horizontally across the virtual control area (403) at a relatively fast speed (e.g., the speed of the center point of the wrist |v_wrist| exceeds 1 m / s).

[0140] S205b - System Status Reset:

[0141] Once the system recognizes the "erase" gesture, it immediately unbinds the virtual image from the control area.

[0142] The position and scale of the virtual image (404) are ultimately fixed in the current state.

[0143] The rendered green control area (403) has disappeared.

[0144] The system exits "Canvas Control Mode" and returns to the normal interactive state, waiting for the next control gesture to activate.

[0145] Example 3: 6DoF Pose Inversion of Handheld Physical Entities Based on Multi-Skeleton Nodes

[0146] This embodiment details a method for accurately tracking multiple key skeletal nodes of a user's holding hand and using advanced estimation algorithms to infer in real time and accurately the complete six-degrees-of-freedom (6DoF) pose (i.e., position and rotation in three-dimensional space) of a physical entity grasped by the hand that has no tracking function, and then mapping it 1:1 onto any virtual three-dimensional model.

[0147] I. System and Environment Setup

[0148] The execution environment for this embodiment includes:

[0149] Head-mounted computing device: Functionally the same as the MR head-mounted display described in Example 1, the key difference being that its hand tracking system can stably output the three-dimensional coordinates of at least 20 hand skeletal nodes.

[0150] Physical Entity (501): A small, irregularly shaped rigid object, such as a Bluetooth earphone charging case. The object is a passive, unmarked, ordinary item.

[0151] Virtual Object (505): A complex three-dimensional virtual model, such as a spaceship, that the user wants to intuitively manipulate with 6DoF by using a physical entity in their hands.

[0152] II. Detailed Execution Steps

[0153] S301: Dynamic Binding and Reference System Establishment

[0154] S301a - Binding Operation: (This step is the same as the initial version) The user holds the physical entity (501) with their left hand (500). The user uses their right hand to "pinch-drag-release" the virtual model (505) onto the object held in their left hand to complete the binding.

[0155] S301b - Establishing a Local Hand Reference Frame: After successful rigging, the system immediately enters "6DoF Control Mode". In this mode, the system uses the real-time position of the left hand's wrist center node (P_wrist) as the origin O_hand, and constructs a dynamically following local coordinate system Σ_hand based on the wrist, index finger base, and middle finger base nodes. The positions of all subsequent bone nodes will be converted to coordinates relative to this local coordinate system to eliminate the effects of translation and rotation of the hand as a whole in world space.

[0156] S302: Selection of Feature Skeleton Node Set and Construction of Input Vector

[0157] To ensure the robustness and efficiency of the algorithm, the system does not use all the skeletal nodes, but instead selects a set of feature skeletal nodes that can most stably reflect the object's contour and posture under the grasping posture (502).

[0158] S302a - Node Filtering: A preferred feature set includes:

[0159] Palm Structure Nodes: Wrist center (P_wrist), and all metacarpal root nodes (P_mcp). These nodes form a stable palm base.

[0160] Proximal contact points: The proximal interphalangeal joints (P_pip) of all fingers. These joints are typically where the fingers make primary contact with objects and apply gripping force.

[0161] Distal Constraint Points: Thumb tip (P_thumb_tip) and pinky tip (P_pinky_tip). These two opposing fingertips provide important constraints for the rotation of the object.

[0162] S302b - Input Vector Construction: The system concatenates the local coordinates (i.e., coordinates in the Σ_hand coordinate system) of the N selected feature skeletal nodes in a predetermined order into a 3N-dimensional input feature vector F_input. For example, if 12 nodes are selected, F_input is a 36-dimensional vector.

[0163] S303: Pose Estimation Algorithm

[0164] The system employs one of the following two alternative schemes to compute the input feature vector F_input into the 6DoF pose of the physical entity in real time. This pose is typically represented as a 7-dimensional vector Pose_obj = [px, py, pz, qx, qy, qz, qw], where p is the position and q is a quaternion representing rotation.

[0165] Solution A: End-to-end learning solution based on graph neural networks (GNN) (S303-A)

[0166] This approach uses a deep learning model to directly learn the complex mapping from hand skeletal morphology to object pose.

[0167] S303-A-1: Model Structure and Training (Offline)

[0168] Data Acquisition and Processing: As described in the previous version, a large number of pairs of {(F_input_i, Pose_gt_i)} data were acquired using a high-precision motion capture system. Before training, all F_input_i were normalized (e.g., by subtracting the mean and dividing by the standard deviation) to accelerate model convergence.

[0169] Graph Construction: The natural topological structure of the hand skeleton is defined as a graph G = (V, E), where V is the set of feature bone nodes and E is the bone edges connecting them.

[0170] GNN model design:

[0171] Input layer: The initial features of each node v ∈ V are its normalized local coordinates.

[0172] Graph Convolution Layers: These employ multi-layered (e.g., 3-5 layers) graph convolutional networks. In each layer, each node aggregates information from its neighbors to learn higher-order structural features. For example, the first layer might learn the bending of a single finger, while higher layers might learn the "cup" or "pinch" configuration of the entire hand. A feasible graph convolution operation is h_v' = σ( Σ_{u∈N(v)} W * h_u ), where h_v is the feature of node v, N(v) are its neighbors, W is the learnable weight matrix, and σ is the activation function (e.g., ReLU).

[0173] Readout Layer: After the last graph convolution, a global pooling operation (such as mean pooling or max pooling) is used to aggregate the features of all nodes into a single graph-level feature vector h_G.

[0174] Output layer (MLP Head): h_G is input into a multilayer perceptron (MLP), and the output layer of the MLP is designed with two branches: a 3D output for predicting the position p, and a 4D output for predicting the quaternion q (normalized before output to ensure that it is a unit quaternion).

[0175] Loss function: Loss = λ_pos * ||p_pred - p_gt||² + λ_rot * (1 - |dot(q_pred, q_gt)|). Where λ_pos and λ_rot are weight hyperparameters used to balance position and rotation errors.

[0176] S303-A-2: Real-time Inference (Online)

[0177] In 6DoF control mode, the system normalizes the input vector F_input, which is constructed in real time in S302, and feeds it into the pre-trained GNN model. The model performs one forward propagation and can output the predicted object pose Pose_obj within milliseconds.

[0178] Scheme B: Geometric solution scheme based on Iterative Closest Point (ICP) variant (S303-B)

[0179] This scheme relies on geometric registration. Its core idea is that the grasping hand forms a "skeleton template", and the relative pose between this template and the grasped object is approximately unchanged during a single grasp.

[0180] S303-B-1: Online Template Registration (One-time Use)

[0181] Trigger registration: In the first few frames (e.g., 10 frames) after successful binding, the system prompts the user to maintain a stable grip posture.

[0182] Constructing a reference template: The system collects the local coordinates of the feature bone nodes in these 10 frames and calculates their average value to obtain a stable, denoised reference bone template T_ref (an N x 3 point cloud matrix). This step is equivalent to "scanning" and recording the unique "imprint" formed by the hand bones relative to the object under the current grasping posture.

[0183] Initialize object pose: In this step, the system assumes that the initial pose of the object is aligned with the local coordinate system Σ_hand of the hand, that is, the initial Pose_obj is a unit transformation.

[0184] S303-B-2: Real-time Iterative Pose Optimization

[0185] In every subsequent frame:

[0186] Obtain the current point cloud: The system obtains the local coordinates of the feature skeleton nodes of the current frame to form the current point cloud P_current.

[0187] Applying the ICP algorithm: The system's goal is to find an optimal rigid body transformation (R, t) (rotation matrix and translation vector) such that the reference template T_ref, after this transformation, is closest to the current point cloud P_current. That is, to solve for: argmin_{R,t} Σ || (R * T_ref_i + t) - P_current_i ||².

[0188] Algorithm implementation:

[0189] Find the corresponding point: For each point in T_ref, find its nearest neighbor in P_current.

[0190] Calculate the transformation: Based on these point pairs, use Singular Value Decomposition (SVD) or the Kabsch algorithm to calculate (R, t) that minimizes the distance between the point pairs.

[0191] Apply the transformation and iterate: Apply the calculated (R, t) to T_ref to obtain the new template position. Repeat steps 1 and 2 until the transformation converges (e.g., the transformation matrix changes less than a threshold) or the maximum number of iterations is reached (e.g., 5-10 times). Since the hand pose changes very little between adjacent frames, the ICP algorithm typically converges quickly.

[0192] Update object pose: The (R, t) obtained after each frame iteration represents the pose change of the object relative to the hand local coordinate system since the registration time. The system accumulates this transformation to the object's total pose to obtain its final pose Pose_obj in the hand local coordinate system.

[0193] S304: Coordinate System Transformation and 1:1 Pose Mapping

[0194] S304a - Coordinate System Transformation: The object pose Pose_obj calculated in S303 is relative to the local hand coordinate system Σ_hand. To correctly display the virtual model in world space, the system needs to transform it to the world coordinate system. This is accomplished through a simple matrix multiplication: Pose_world = Pose_Σ_hand_to_world * Pose_obj, where Pose_Σ_hand_to_world is the transformation matrix of the hand's local coordinate system relative to the world coordinate system, determined in real-time by the world position of P_wrist and the basis vectors of Σ_hand.

[0195] S304b - Mapping and Rendering: The system applies the final calculated world pose Pose_world to the Transform component of the virtual model (505). Because the entire process runs in real time at a high frequency, when the user moves and rotates the physical entity in their hand in real space, the virtual model will make fully synchronized and smooth 6DoF motion in the MR view, thus successfully transforming any ordinary small object into a high-precision spatial controller.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mixed reality physical entity interaction method based on hand skeleton tracking, characterized in that, Includes the following steps: The three-dimensional skeletal node data of a user's hand that is grasping or touching a physical entity is acquired through a head-mounted computing device. The physical entity is a passive, unmarked object. From the three-dimensional skeletal node data, at least two key skeletal nodes are selected, and one or more control parameters are calculated based on the real-time relative spatial relationship of the at least two key skeletal nodes. Establish a preset mapping relationship between the one or more control parameters and the target attributes of one or more interactive elements in the virtual user interface; Based on the mapping relationship, the changes in the control parameters are converted into changes in the target attributes of the virtual interactive elements in real time, and the display content of the head-mounted computing device is updated.

2. The method according to claim 1, characterized in that, The control parameters include: the angle between a vector formed by the at least two key skeleton nodes and a reference coordinate system, the magnitude or rate of change of the magnitude of the vector, the distance or rate of change of the distance between the at least two key skeleton nodes, or the attribute changes of the geometry formed by multiple key skeleton nodes.

3. The method according to claim 1, characterized in that, The at least two key skeletal nodes are the fingertip node and the root joint node of the user's thumb; the control parameter is the angle between the vector formed by the two nodes and a preset axis of a local hand coordinate system; the virtual interactive element is a slider control.

4. The method according to claim 1, characterized in that, The at least two key skeletal nodes are the fingertip nodes of the user's thumb and index finger, and the user's hand is placed on a physical surface; The method further includes: distinguishing whether the user's operation intention is translation or scaling based on the motion state of the two nodes, and using the relative displacement or relative distance change of the two nodes as a control parameter according to the intention.

5. The method according to claim 1, characterized in that, The step of selecting at least two key skeletal nodes specifically involves selecting multiple nodes of a hand bone to form a set of feature skeletal nodes; the step of calculating control parameters specifically involves using a pose estimation algorithm model to solve the real-time spatial configuration of the set of feature skeletal nodes into a six-degree-of-freedom pose of the physical entity, and using the six-degree-of-freedom pose as the control parameters.

6. The method according to claim 5, characterized in that, The pose estimation algorithm model is either a pre-trained neural network model or a solution model based on geometric constraints and point set registration algorithms.

7. The method according to claim 1, characterized in that, Also includes: The interaction method is activated by establishing a binding relationship between the virtual interactive element and the physical entity through a first specific gesture. as well as The binding relationship is released by a second specific gesture.

8. A mixed reality physical entity interaction system based on hand skeleton tracking, characterized in that, include: A hand skeleton data acquisition module is used to acquire three-dimensional skeleton node data of a user's hand that is grasping or touching a physical entity through an image sensor of a head-mounted computing device. The physical entity is a passive, unmarked object. A control parameter calculation module is used to select at least two key bone nodes from the three-dimensional bone node data and calculate one or more control parameters based on the real-time relative spatial relationship of the at least two key bone nodes. An interactive mapping and driving module is used to convert changes in one or more control parameters into changes in target attributes of one or more interactive elements in the virtual user interface in real time, according to a preset mapping relationship, and drive the display content of the head-mounted computing device to be updated.

9. The system according to claim 8, characterized in that, The control parameter calculation module is specifically used to: select multiple nodes of a hand bone to form a feature bone node set, and use a pose estimation algorithm model to solve the real-time spatial configuration of the feature bone node set into the six-degree-of-freedom pose of the physical entity, which is used as control parameters.

10. A head-mounted computing device, characterized in that, It includes a processor, a memory, a display screen, an image sensor, and a program stored in the memory and executed by the processor, which, when executed, implements the method as described in any one of claims 1 to 7.

Images