A vr virtual-real fusion interaction method based on multi-source sensing and hand redirection

Abstract

The application discloses a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection. The method comprises the following steps: establishing a mapping relationship between a real interactive object (RIO) and a virtual interactive object (VIO); performing initial coordinate registration on the RIO, the VIO and a head-mounted display tracking space; collecting sensor data of the RIO; updating a state variable of the VIO according to the sensor data, so that the virtual object is consistent with real operation; collecting a user hand posture, and judging whether the user enters an interaction state based on a distance threshold and a gesture template; when the user enters the interaction state, performing virtual hand redirection; continuously calculating visual touch consistency error, and performing re-registration, sensor-driven updating or redirection compensation according to the error size; and rendering the updated virtual object state, virtual hand state and experimental data to a VR scene in real time. The method solves the problems of inconsistency between vision and touch, mis-triggering of an interaction event, discontinuous virtual object movement and the like in the prior art.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically, to a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection. Background Technology

[0002] Virtual reality systems typically acquire user movements through head-mounted displays, hand tracking, controllers, or external sensors, and render virtual hands and objects in real-time within the virtual scene. For general menu selection, simple grasping, or positional movement tasks, these interaction methods are sufficient for basic usage. However, in interaction scenarios involving real contact, mechanical constraints, and continuous changes in physical state, relying solely on visual rendering and air gesture input remains significantly insufficient. While the user's hand movements visually occur in virtual space, tactile perception originates from real space. If there are no tangible objects in reality, the user can only rely on air gestures to complete the operation; if physical props are present in reality, issues of spatial registration, state synchronization, and hand display consistency between real and virtual objects need to be addressed. These problems cannot be solved simply by adding a physical prop.

[0003] Existing active force feedback devices can apply external force to a user's hand, but these devices typically include motors, linkages, pneumatic or hydraulic mechanisms, resulting in high costs, complex installation, and limited workspace. These devices also require continuous tracking of the hand and actuator states, and system latency, mechanical inertia, and safety constraints all affect interaction stability. For classrooms, training rooms, or general consumer VR environments, this type of solution is difficult to deploy at low cost. Passive haptic solutions use real objects as haptic proxies for virtual objects, allowing users to feel the physical structure when touching virtual objects. However, this type of solution still faces three technical challenges in practical use. First, the coordinate systems of real and virtual objects need to remain consistently consistent. Any deviation between the headset tracking coordinates, real-world physical coordinates, and virtual scene coordinates will cause inconsistencies between the user's perceived and tactile positions. Second, when a user's hand touches a real object, it can easily obscure the object itself, leading to unstable object tracking that relies on vision, and potentially causing jumps, delays, or loss of virtual object states. Third, virtual hand models are usually driven directly by head-mounted display hand tracking, but when a real hand comes into contact with a physical object, there are collision constraints, joint posture constraints, and grip posture constraints. If the virtual hand is not redirected, it may penetrate the virtual object or not coincide with the real tactile contact point.

[0004] Existing methods typically address only one aspect. For example, some methods focus on object localization, others on hand tracking, some on the force feedback device itself, and still others on virtual scene rendering. These methods lack a unified, synchronized mechanism that connects real-world interactive objects, virtual interactive objects, real hands, virtual hands, and physical simulation. Especially when users continuously interact with real objects, the system needs to simultaneously handle changes in object pose, sensor states, virtual physical parameters, and virtual hand posture. If these data sources are updated independently, problems such as inconsistencies between visual and tactile feedback, accidental triggering of interactive events, and discontinuous virtual object movement can easily arise.

[0005] Therefore, a new VR virtual-real fusion interaction method is needed. This method should be able to establish coordinate and state mappings between real and virtual interactive objects under low-cost hardware conditions; it should still be able to drive virtual object updates using built-in sensors when external visual tracking is obstructed; it should be able to adjust the virtual hand posture according to the interaction type and gesture template when the user's hand touches a real object; and during system operation, it should continuously detect consistency errors between real objects, virtual objects, real hands, and virtual hands, and perform compensation when the error exceeds a threshold. Only in this way can the motion state and hand posture seen by the user in VR correspond to the tactile feedback perceived in reality. Summary of the Invention

[0006] The purpose of this disclosure is to provide a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection, which aims to solve problems such as inconsistency between vision and touch, accidental triggering of interactive events, and discontinuous movement of virtual objects in the prior art.

[0007] In general, a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection is provided, including steps S1-S8: Step S1: Determine the target VR interaction task and establish a mapping relationship between the real interactive object RIO and the virtual interactive object VIO. Step S2: Initial coordinate registration is performed on RIO, VIO, and head-mounted display tracking space using visual markers, depth information, and hand contact constraints through a coordinate registration method. Step S3: Collect sensor data from RIO and perform filtering, unit conversion, threshold judgment, and outlier removal on the sensor data; Step S4: Update the VIO state variables based on sensor data. Use the sensor-driven virtual interactive object state update method to make the movement, parameter changes or triggered events of the virtual object consistent with the real operation. Step S5: Collect the user's hand posture and determine whether the user has entered the interactive state based on the distance threshold and gesture template; Step S6: When the user enters the interactive state, the virtual hand is redirected through the interaction intent prediction and virtual hand redirection method to make the virtual hand contact position consistent with the real hand tactile contact position. Step S7: Continuously calculate the visual-touch consistency error, and perform re-registration, sensor drive update or redirection compensation according to the error magnitude through visual-touch consistency error monitoring and closed-loop compensation method; Step S8: Render the updated virtual object state, virtual hand state, and experimental data to the VR scene in real time.

[0008] The coordinate registration method is implemented as follows: First, the pose of the real interactive target object RIO in the head-mounted display camera coordinate system is obtained through visual markers, depth information, or hand contact constraints. And obtain the pose of the head-mounted display camera in the world coordinate system. Then, by transforming the coordinates and mapping them onto the virtual scene, the pose of the target object in the world coordinate system is obtained. for: in, , and All are 4×4 homogeneous transformation matrices; By rotation matrix and displacement vector composition; By rotation matrix and displacement vector composition; The pose of a real interactive object after transformation to the world coordinate system is represented by a rotation matrix. and displacement vector composition.

[0009] The sensor-driven virtual interactive object state update method is as follows: 1) Collect sensor data from real interactive objects, including force sensors, angle encoders, pressure sensors, and displacement sensors; 2) Sensor data preprocessing; 3) Data type identification and mapping rule selection: tension sensor data is mapped to spring tension or virtual slider initial velocity; angle encoder data is mapped to knob angle, time axis position or experimental parameters; pressure sensor data is mapped to button trigger event; displacement sensor data is mapped to piston compression or slider position. 4) Updating the state variables of virtual interactive objects; 5) Physical constraints are applied to collision, limit, and rebound events, and parameter changes are triggered by these events; in, Indicates time Module type identifier; Indicates valid status data; Indicates the interaction state determination quantity; This represents the interaction state determination function, which is jointly determined by the module type and state data. Indicates continuous weighted candidate values; This represents the external data-driven weights after double-threshold hysteresis updates; This indicates the weights driven by external data from the previous moment. Represents the Sigmoid function; Indicates the current data link confidence level; This represents the time interval from the most recent valid data to the current moment; This represents the total error in visual-tactile consistency. , , and Indicates the adjustment coefficient; and Let these represent the upper and lower thresholds for driving state switching, respectively, and satisfy the following conditions: . and This represents the state transition matrix and input mapping matrix related to the module type; Represents a state data mapping function; Represents the virtual physics driving function; Indicates the current virtual physics drive control input; Represents the set of constraints related to the module type; Indicates the weight of the driving switching smoothing term; and These represent the state variables of the virtual interactive object before and after the update, respectively.

[0010] 6) Real-time rendering and display of the motion, experimental curves, and status feedback of virtual objects in the VR scene.

[0011] The implementation method of the interaction intent prediction and virtual hand redirection method is as follows: 1) Track hand movements, including hand position. Joint rotation Palm facing Information on the degree of finger bending ; in, Indicates time A set of hand skeletal features; Indicates the first The three-dimensional position of each key point on the hand; This indicates the output of the hand tracking module. Each joint rotates; Indicates the first Tracking confidence of key points or joints; Indicates the total number of key points or joints involved in the calculation; , and These indicate the locations of key points on the wrist, the base of the index finger, and the base of the little finger, respectively. This represents the palm orientation vector calculated based on the geometric relationships of key hand points; , and Indicates the formation of the first The positions of three adjacent key points of each joint angle; Indicates the first The joint angle characteristics of each joint.

[0012] 2) Target object neighborhood detection: Determine whether the hand is within the neighborhood of the target object; 3) Current gesture extraction: extract joint angle features, palm orientation vector, and finger bending information; in, Indicates time A set of hand skeletal features; Indicates the first The three-dimensional position of each key point on the hand; This indicates the output of the hand tracking module. Each joint rotates; Indicates the first Tracking confidence of key points or joints; Indicates the total number of key points or joints involved in the calculation; , and These indicate the locations of key points on the wrist, the base of the index finger, and the base of the little finger, respectively. This represents the palm orientation vector calculated based on the geometric relationships of key hand points; , and Indicates the formation of the first The positions of three adjacent key points of each joint angle; Indicates the first Joint angle characteristics of each joint; and They represent the first Reference angles of each joint in the open and bent states; This represents the degree of bending characteristic obtained by normalizing the joint angle features. This represents the hand pose rotation matrix constructed from the hand orientation vector; This represents the local rotation matrix constructed based on the joint rotation output by the hand tracking module; This represents the compensation rotation matrix constructed based on the degree of finger bending; Indicates the first The comprehensive rotation matrix of each joint; The first value obtained by the transformation of the comprehensive rotation matrix is ​​represented as... A joint rotation quaternion.

[0013] 4) Pre-establish a gesture template library corresponding to different real-world interactive objects, and pre-set the gesture templates in the library. Each gesture template is represented as The current gesture is represented as .

[0014] 5) The gesture template matching is performed in the following manner: in, Indicates time The Each joint adaptive matching weight; Indicates joint significance; Indicates the confidence level of joint tracking; Indicates the module type for the first The importance modulation terms for each joint; , and Indicates the adjustment coefficient; Indicates the current gesture and the first The overall matching distance between gesture templates; Indicates the current gesture number The quaternion of the joint rotation and the first joint rotation The first template The four-dimensional vector inner product of the quaternions of joint rotations; and Indicates the weights of the positional consistency term and the normal consistency term; Indicates the target contact normal; Indicates the first Normalized matching confidence of each template; Indicates the matching distance compression factor; This represents the target template number with the smallest matching distance or the highest matching confidence. This indicates the template number with the second smallest matching distance; This represents the separation degree between the best and second-best templates.

[0015] 6) The interaction intent recognition method is as follows: in, and Let represent the confidence thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; and Let represent the distance thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; Indicates the template separation threshold; This indicates the result of the interaction intent determination.

[0016] 7) Virtual hand repositioning: Smoothly adjust the virtual hand from its current posture to the target contact posture to compensate for errors and offsets. in, Indicates time Redirect weight; Indicates the base redirection weight; This represents the total error in visual-tactile consistency. Indicates the confidence level of the target template match; This represents the distance from the hand to the target interaction point; Indicates a reference distance; Indicates the confidence level of hand tracking; , , and Indicates the adjustment coefficient; This represents the intermediate pose after linear interpolation of the sphere.

[0017] 8) Render and output the virtual hand: in, This indicates the target rendering pose after constraint projection; Indicates the first one used for rendering A joint rotation quaternion; This represents the projection onto the set of kinematic constraints associated with the current module type. Constrained projection operators on; This represents the redirection continuous smoothing factor, with a value range of [value missing]. .

[0018] The gesture template library includes grasp templates, press templates, pinch templates, rotate templates, and pull templates.

[0019] The specific process of the visual-tactile consistency error monitoring and closed-loop compensation method is as follows: in, and These represent the normal position error and the tangential position error, respectively. , , and These represent attitude error, state error, synchronization delay error, and feedback response error, respectively. and These represent the actual contact point position vector and the virtual contact point position vector, respectively. and These represent the real contact attitude quaternion and the virtual contact attitude quaternion, respectively; and These represent the real interaction state vector and the virtual interaction state vector, respectively. The normal vector representing the virtual contact surface; and These represent the real feedback state vector and the virtual feedback state vector, respectively. This indicates a delay in the synchronization between the virtual and real states; and These represent the normalized scales for the normal position error and the tangential position error, respectively. Indicates the normalization scale of the state error; This represents the normalized scale of the synchronization delay error; Indicates the normalized scale of the feedback response; Indicates the error type index, with values ​​taken from the set. ; Indicates the first Time-varying weights for error types; This represents the error sensitivity coefficient; Indicates the module type for the first Weighted modulation terms for error types; Indicates the corresponding modulation coefficient; Indicates time Total error in visual-tactile consistency; Indicates the compensation control amount; , and These represent the proportional, integral, and differential coefficients, respectively. Indicates the mode adjustment coefficient; Indicates the target-driven weights; This represents the consistency adjustment coefficient for interactive locking; This indicates the target interaction state locked by the system.

[0020] The visual-touch consistency error includes position error, posture error, state error, and synchronization delay.

[0021] The closed-loop compensation strategy for smoothly adjusting the virtual hand from its current posture to the target contact posture includes: sensor priority update, visual reregistration, virtual hand redirection weight adjustment, and interaction lock release and recovery.

[0022] In another general aspect, a VR virtual-real fusion interaction system based on multi-source sensing and hand redirection is provided, including: The real interactive object module is equipped with one or more of the following: pressure sensor, tension sensor, angle encoder, IMU, displacement sensor or visual marker, to collect position, attitude, force, displacement, angle or trigger state caused by user operation; The virtual interactive object module includes geometric models, colliders, motion constraints, physical parameters, and state variables. Virtual interactive objects can be sliders, springs, buttons, knobs, pistons, liquid containers, or buoyant objects. The multi-source sensor registration module establishes the transformation relationship between the real physical coordinate system, the head-mounted display tracking coordinate system, and the virtual scene coordinate system; The sensor-driven synchronization module uses sensor data from the interior or contact points of the real interactive object to drive the state update of the virtual interactive object. The interaction intent prediction module obtains the user's hand position, joint rotation, palm orientation, finger bending degree, and distance between the hand and the virtual interactive object in real time. When the user's hand enters the neighborhood of the target object and the current gesture matches the preset interaction template, it determines that the user has entered a grasping, pressing, pinching, rotating, or pulling state. The hand redirection module compensates for offsets between real hands, virtual hands, and physical props caused by tracking errors, latency, or model differences. The visual-touch consistency closed-loop control module continuously calculates the position error, posture error, state error, and synchronization delay between real and virtual interactive objects. When the error exceeds a preset threshold, it performs sensor priority update, visual reregistration, virtual hand redirection weight adjustment, or interaction lock release to restore the visual-touch consistency state.

[0023] The technical effects to be achieved by the embodiments of the present invention are as follows: 1. By initial coordinate registration and sensor-driven synchronization, real interactive objects and virtual interactive objects maintain a stable correspondence, reducing the misalignment between the real and virtual objects caused by occlusion, tracking drift, and visual detection failure.

[0024] 2. By using a gesture template library and virtual hand redirection, the virtual hand contact position seen by the user is consistent with the tactile contact position of the real hand, thus improving the realism of the interaction.

[0025] 3. By implementing closed-loop control of visual-tactile consistency error, compensation is automatically performed when the error exceeds the threshold, thus preventing the system from being in a state of visual and tactile inconsistency for a long time.

[0026] 4. By providing feedback such as elasticity, damping, rebound, and limit through real physical structures, the uncertainty of purely aerial gesture interaction is reduced.

[0027] 5. The system can be deployed using low-cost sensors and 3D-printed structures, avoiding reliance on expensive commercial force feedback equipment.

[0028] 6. By designing modular, realistic interactive objects, the same approach can be extended to multiple types of virtual experiments. Attached Figure Description

[0029] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0030] Figure 1 This is a schematic flowchart illustrating a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection according to an embodiment of the present disclosure; Figure 2This is a schematic diagram of the system structure of a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating a coordinate registration method according to an embodiment of the present disclosure; Figure 4 This is a schematic flowchart illustrating sensor-driven virtual interactive object state updates according to embodiments of the present disclosure; Figure 5 This is a schematic flowchart illustrating the visual-touch consistency error monitoring and closed-loop compensation process according to an embodiment of the present disclosure. Detailed Implementation

[0031] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0032] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0033] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0034] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0035] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0036] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0038] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0039] Figure 1 This is a schematic diagram illustrating a VR virtual-real fusion interaction method based on multi-source sensing and hand redirection according to an embodiment of the present disclosure.

[0040] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.

[0041] A VR vision-touch consistency virtual-real fusion interaction method based on multi-source sensing and hand redirection. This method establishes a mapping relationship between real interaction objects (RIOs) and virtual interaction objects (VIOs). Through multi-source pose estimation, built-in sensor state driving, coordinate system registration, interaction intent prediction, virtual hand redirection, and consistency error closed-loop compensation, it achieves consistency in position, state, interaction semantics, and haptic feedback between real and virtual objects.

[0042] The method of this invention includes the following 8 steps, and a detailed structural diagram is shown below. Figure 1 : Step S1: Determine the target VR interaction task and establish a mapping relationship between the real interactive object RIO and the virtual interactive object VIO.

[0043] Step S2 involves initial coordinate registration of the RIO, VIO, and head-mounted display tracking spaces using visual markers, depth information, and hand contact constraints. Figure 3 As shown.

[0044] Step S3: Collect sensor data from RIO and perform filtering, unit conversion, threshold judgment, and outlier removal on the sensor data.

[0045] Step S4: Update the VIO state variables based on sensor data to ensure that the virtual object's motion, parameter changes, or triggered events are consistent with real operations, such as... Figure 4 As shown.

[0046] Step S5: Collect the user's hand posture and determine whether the user has entered the interactive state based on the distance threshold and gesture template.

[0047] Step S6: When the user enters the interactive state, the virtual hand is redirected to make the virtual hand contact position consistent with the real hand tactile contact position.

[0048] Step S7: Continuously calculate the visual-touch consistency error, and perform re-registration, sensor driver update, or redirection compensation based on the error magnitude, such as... Figure 5 As shown.

[0049] Step S8: Render the updated virtual object state, virtual hand state, and experimental data to the VR scene in real time.

[0050] The interactive objects employ a modular design. Each interactive object consists of a tactile body and a sensing / feedback core. The tactile body provides information on shape, size, grip, and surface contact; the sensing / feedback core provides data on displacement, angle, pressure, tension, or vibration. The two are connected via a standard interface, allowing the same sensing / feedback core to be replaced with different tactile bodies to adapt to various virtual experiments.

[0051] The specific structure of the system of this invention is as follows: Figure 2 As shown, it includes the following modules: Realistic interactive object module. Realistic interactive objects include physical structures used to generate haptic feedback, such as sliders, springs, buttons, knobs, pistons, grips, rods, or experimental blocks. One or more of the following are mounted on the realistic interactive object: pressure sensor, tension sensor, angle encoder, IMU, displacement sensor, or visual marker, to collect position, posture, force, displacement, angle, or trigger state resulting from user operations.

[0052] Virtual Interactive Object Module. A virtual interactive object is a digital object in a VR environment that corresponds to a real interactive object. It includes geometric models, colliders, motion constraints, physical parameters, and state variables. Virtual interactive objects can be sliders, springs, buttons, knobs, pistons, liquid containers, buoyant objects, or other experimental components.

[0053] Multi-source sensor registration module. This module establishes the transformation relationship between the real physical coordinate system, the head-mounted display tracking coordinate system, and the virtual scene coordinate system. The system first obtains the initial pose of the real interactive object through visual markers, depth information, or hand contact constraints, and then maps it to the virtual scene through coordinate transformation.

[0054] Sensor-driven synchronization module. After initial registration, the system does not rely entirely on external vision for continuous tracking. Instead, it uses sensor data from the interior or contact points of the real interactive object to drive the state updates of the virtual interactive object. For example, tension sensor data is mapped to spring tension or the initial velocity of the virtual slider; angle encoder data is mapped to knob angle, time axis position, or experimental parameters; pressure sensor data is mapped to button trigger events; and displacement sensor data is mapped to piston compression or slider position.

[0055] Interaction Intent Prediction Module. The system acquires the user's hand position, joint rotation, palm orientation, finger flexion, and distance between the hand and the virtual interactive object in real time. When the user's hand enters the neighborhood of the target object and the current gesture matches the preset interaction template, the system determines that the user has entered a grasping, pressing, pinching, rotating, or pulling state.

[0056] Hand redirection module. This module pre-builds a gesture template library corresponding to different real-world interactive objects. The first gesture in the preset gesture template library... Each gesture template is represented as The current gesture is represented as .

[0057] The system employs a closed-loop control module for visual-touch consistency. It continuously calculates the positional error, posture error, state error, and synchronization delay between real and virtual interactive objects. When the error exceeds a preset threshold, the system performs sensor priority updates, visual reregistration, virtual hand redirection weight adjustment, or interaction lock release to restore visual-touch consistency.

[0058] The coordinate registration method in the system is as follows: First, the initial pose of the real interactive object is obtained through visual markers, depth information, or hand contact constraints. Then, it is mapped onto the virtual scene through coordinate transformation. Let the pose of the target object relative to the camera be... The camera's pose relative to the world coordinate system is Then the pose of the target object in the world coordinate system is: .

[0059] in, , and All are 4×4 homogeneous transformation matrices. The pose of a real interactive object in the head-mounted display camera coordinate system is represented by a rotation matrix. and displacement vector composition; The pose of the head-mounted display camera in the world coordinate system is represented by a rotation matrix. and displacement vector composition; The pose of a real interactive object after transformation to the world coordinate system is represented by a rotation matrix. and displacement vector composition.

[0060] The sensor-driven method for updating the state of virtual interactive objects is as follows: 1) Collect sensor data from real interactive objects, including force sensors, angle encoders, pressure sensors, and displacement sensors; 2) Sensor data preprocessing; 3) Data type identification and mapping rule selection: tension sensor data is mapped to spring tension or virtual slider initial velocity; angle encoder data is mapped to knob angle, time axis position or experimental parameters; pressure sensor data is mapped to button trigger event; displacement sensor data is mapped to piston compression or slider position. 4) Updating the state variables of virtual interactive objects; 5) Apply physical constraints to collision, limit, and rebound events, and change parameters based on these events: in, Indicates time Module type identifier; Indicates valid status data; Indicates the interaction state determination quantity; This represents the interaction state determination function, which is jointly determined by the module type and state data. Indicates continuous weighted candidate values; This represents the external data-driven weights after double-threshold hysteresis updates; Represents the Sigmoid function; Indicates the current data link confidence level; This represents the time interval from the most recent valid data to the current moment; This represents the total error in visual-tactile consistency. , , and Indicates the adjustment coefficient; and Let these represent the upper and lower thresholds for driving state switching, respectively, and satisfy the following conditions: . and This represents the state transition matrix and input mapping matrix related to the module type; Represents a state data mapping function; Represents the virtual physics driving function; Represents the set of constraints related to the module type; Indicates the weight of the driving switching smoothing term; and These represent the state variables of the virtual interactive object before and after the update, respectively.

[0061] 6) Real-time rendering and display of the motion, experimental curves, and status feedback of virtual objects in the VR scene.

[0062] The implementation method of interactive intent prediction and virtual hand redirection is as follows: 1) Track hand movements, including hand position. Joint rotation Palm facing Information on the degree of finger bending ; in, Indicates time A set of hand skeletal features; Indicates the first The three-dimensional position of each key point on the hand; This indicates the output of the hand tracking module. Each joint rotates; Indicates the first Tracking confidence of key points or joints; Indicates the total number of key points or joints involved in the calculation; , and These indicate the locations of key points on the wrist, the base of the index finger, and the base of the little finger, respectively. This represents the palm orientation vector calculated based on the geometric relationships of key hand points; , and Indicates the formation of the first The positions of three adjacent key points of each joint angle; Indicates the first The joint angle characteristics of each joint.

[0063] 2) Target object neighborhood detection: Determine whether the hand is within the neighborhood of the target object; 3) Current gesture extraction: extract joint angle features, palm orientation vector, and finger bending information; in, This indicates the pinching distance between the tips of the thumb and index finger; and These indicate the positions of the tips of the thumb and index finger, respectively. Indicates the first The degree of bending characteristic corresponding to the root finger; This represents the hand pose rotation matrix constructed from the hand orientation vector; This represents a local rotation matrix constructed from joint angles; This represents the compensation rotation matrix constructed based on the degree of finger bending; Indicates the first The comprehensive rotation matrix of each joint; The first value obtained by the transformation of the comprehensive rotation matrix is ​​represented as... A joint rotation quaternion.

[0064] 4) Pre-establish a gesture template library corresponding to different real-world interactive objects. Each gesture template is represented as The current gesture is represented as .

[0065] 5) The gesture template matching is performed in the following manner: in, Indicates time The Each joint adaptive matching weight; Indicates joint significance; Indicates the confidence level of joint tracking; Indicates the module type for the first The importance modulation terms for each joint; , and Indicates the adjustment coefficient; Indicates the current gesture and the first The overall matching distance between gesture templates; Indicates the current gesture number The quaternion of the joint rotation and the first joint rotation The first template The four-dimensional vector inner product of the quaternions of joint rotations; and Indicates the weights of the positional consistency term and the normal consistency term; Indicates the target contact normal; Indicates the first Normalized matching confidence of each template; Indicates the matching distance compression factor; This represents the target template number with the smallest matching distance or the highest matching confidence. This indicates the template number with the second smallest matching distance; This represents the separation degree between the best and second-best templates.

[0066] 6) The interaction intent recognition method is as follows: in, and Let represent the confidence thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; and Let represent the distance thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; Indicates the template separation threshold; This indicates the result of the interaction intent determination. 7) Virtual hand redirection: Smoothly adjust the virtual hand from its current posture to the target contact posture to compensate for errors and offsets. in, Indicates time Redirect weight; Indicates the base redirection weight; This represents the total error in visual-tactile consistency. Indicates the confidence level of the target template match; This represents the distance from the hand to the target interaction point; Indicates a reference distance; Indicates the confidence level of hand tracking; , , and Indicates the adjustment coefficient; This represents the intermediate pose after linear interpolation of the sphere.

[0067] 8) Render and output the virtual hand: in, This indicates the target rendering pose after constraint projection; Indicates the first one used for rendering A joint rotation quaternion; This represents the projection onto the set of kinematic constraints associated with the current module type. Constrained projection operators on; This represents the redirection continuous smoothing factor, with a value range of [value missing]. .

[0068] The gesture template library includes grasp templates, press templates, pinch templates, rotate templates, and pull templates.

[0069] The specific process of the visual-tactile consistency error monitoring and closed-loop compensation method is as follows: in, and These represent the normal position error and the tangential position error, respectively. , , and These represent attitude error, state error, synchronization delay error, and feedback response error, respectively. The normal vector representing the virtual contact surface; and These represent the real feedback state vector and the virtual feedback state vector, respectively. and These represent the normalized scales for the normal position error and the tangential position error, respectively. Indicates the normalized scale of the feedback response; Indicates the first Time-varying weights for error types; This represents the error sensitivity coefficient; Indicates the module type for the first Weighted modulation terms for error types; Indicates the corresponding modulation coefficient; Indicates time Total error in visual-tactile consistency; Indicates the compensation control amount; , and These represent the proportional, integral, and differential coefficients, respectively. Indicates the mode adjustment coefficient; Indicates the target-driven weights; This represents the consistency adjustment coefficient for interactive locking; This indicates the target interaction state locked by the system.

[0070] The visual-touch consistency error includes position error, posture error, state error, and synchronization delay.

[0071] The closed-loop compensation strategy for smoothly adjusting the virtual hand from its current posture to the target contact posture includes: sensor priority update, visual reregistration, virtual hand redirection weight adjustment, and interaction lock release and recovery.

[0072] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A VR virtual-real fusion interaction method based on multi-source sensing and hand redirection, characterized in that, Including steps S1-S8: Step S1: Determine the target VR interaction task and establish a mapping relationship between the real interactive object RIO and the virtual interactive object VIO. Step S2: Initial coordinate registration is performed on RIO, VIO, and head-mounted display tracking space using visual markers, depth information, and hand contact constraints through a coordinate registration method. Step S3: Collect sensor data from RIO and perform filtering, unit conversion, threshold judgment, and outlier removal on the sensor data; Step S4: Update the VIO state variables based on sensor data. Use the sensor-driven virtual interactive object state update method to make the movement, parameter changes or triggered events of the virtual object consistent with the real operation. Step S5: Collect the user's hand posture and determine whether the user has entered the interactive state based on the distance threshold and gesture template; Step S6: When the user enters the interactive state, the virtual hand is redirected through the interaction intent prediction and virtual hand redirection method to make the virtual hand contact position consistent with the real hand tactile contact position. Step S7: Continuously calculate the visual-touch consistency error, and perform re-registration, sensor drive update or redirection compensation according to the error magnitude through visual-touch consistency error monitoring and closed-loop compensation method; Step S8: Render the updated virtual object state, virtual hand state, and experimental data to the VR scene in real time.

2. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 1, characterized in that, The coordinate registration method is implemented as follows: First, the pose of the real interactive object RIO in the head-mounted display camera coordinate system is obtained through visual markers, depth information, or hand contact constraints. And obtain the pose of the head-mounted display camera in the world coordinate system. Then, by transforming the coordinates and mapping them onto the virtual scene, the pose of the target object in the world coordinate system is obtained. for: in, , and All are 4×4 homogeneous transformation matrices; By rotation matrix and displacement vector composition; By rotation matrix and displacement vector composition; The pose of a real interactive object after transformation to the world coordinate system is represented by a rotation matrix. and displacement vector composition.

3. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 1, characterized in that, The sensor-driven virtual interactive object state update method is as follows: 1) Collect sensor data from real interactive objects, including force sensors, angle encoders, pressure sensors, and displacement sensors; 2) Sensor data preprocessing; 3) Data type identification and mapping rule selection: tension sensor data is mapped to spring tension or virtual slider initial velocity; angle encoder data is mapped to knob angle, time axis position or experimental parameters; pressure sensor data is mapped to button trigger event; displacement sensor data is mapped to piston compression or slider position. 4) Updating the state variables of virtual interactive objects; 5) Apply physical constraints to collision, limit, and rebound events, and change parameters based on these events: in, Indicates time Module type identifier; Indicates valid status data; Indicates the interaction state determination quantity; This represents the interaction state determination function, which is jointly determined by the module type and state data. Indicates continuous weighted candidate values; This represents the external data-driven weights after double-threshold hysteresis updates; This indicates the weights driven by external data from the previous moment. Represents the Sigmoid function; Indicates the current data link confidence level; This represents the time interval from the most recent valid data to the current moment; This represents the total error in visual-tactile consistency. , , and Indicates the adjustment coefficient; and Let these represent the upper and lower thresholds for driving state switching, respectively, and satisfy the following conditions: and This represents the state transition matrix and input mapping matrix related to the module type; Represents a state data mapping function; Represents the virtual physics driving function; Indicates the current virtual physics drive control input; Represents the set of constraints associated with the module type; Indicates the weight of the driving switching smoothing term; and These represent the state variables of the virtual interactive object before and after the update, respectively; 6) Real-time rendering and display of the motion, experimental curves, and status feedback of virtual objects in the VR scene.

4. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 1, characterized in that, The implementation method of the interaction intent prediction and virtual hand redirection method is as follows: 1) Track hand movements, including hand position. Joint rotation Palm facing Information on the degree of finger bending ; in, Indicates time A set of hand skeletal features; Indicates the first The three-dimensional position of each key point on the hand; This indicates the output of the hand tracking module. Each joint rotates; Indicates the first Tracking confidence of key points or joints; Indicates the total number of key points or joints involved in the calculation; , and These indicate the locations of key points on the wrist, the base of the index finger, and the base of the little finger, respectively. This represents the palm orientation vector calculated based on the geometric relationships of key hand points; , and Indicates the composition of the first The positions of three adjacent key points of each joint angle; Indicates the first Joint angle characteristics of each joint; 2) Target object neighborhood detection: Determine whether the hand is within the neighborhood of the target object; 3) Current gesture extraction: Extract joint angle features, palm orientation vector, and finger bending information. in, Indicates the first Joint angle characteristics of each joint; and They represent the first Reference angles of each joint in the open and bent states; This represents the degree of bending characteristic obtained by normalizing the joint angle features. This represents the hand pose rotation matrix constructed from the hand orientation vector; This represents the local rotation matrix constructed based on the joint rotation output by the hand tracking module; This represents the compensation rotation matrix constructed based on the degree of finger bending; Indicates the first The comprehensive rotation matrix of each joint; The first value obtained by the transformation of the comprehensive rotation matrix is ​​represented as... A joint rotation quaternion; 4) Pre-establish a gesture template library corresponding to different real-world interactive objects, and pre-set the gesture templates in the library. Each gesture template is represented as The current gesture is represented as ; 5) The gesture template matching is performed in the following manner: in, Indicates time The Each joint adaptive matching weight; Indicates joint significance; Indicates the confidence level of joint tracking; Indicates the module type for the first The importance modulation terms for each joint; , and Indicates the adjustment coefficient; Indicates the current gesture and the first The overall matching distance between gesture templates; Indicates the current gesture number The quaternion of the joint rotation and the first joint rotation The first template The four-dimensional vector inner product of the quaternions of joint rotations; and Indicates the weights of the positional consistency term and the normal consistency term; Indicates the target contact normal; Indicates the first Normalized matching confidence of each template; Indicates the matching distance compression factor; This represents the target template number with the smallest matching distance or the highest matching confidence. Indicates the template number with the second smallest matching distance; This represents the separation degree between the best and second-best templates; 6) The interaction intent recognition method is as follows: in, and Let represent the confidence thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; and Let represent the distance thresholds for entering the interactive state and maintaining the interactive state, respectively, and satisfy . ; Indicates the template separation threshold; This indicates the result of the interaction intent determination; 7) Virtual hand repositioning: Smoothly adjust the virtual hand from its current posture to the target contact posture to compensate for errors and offsets. in, Indicates time Redirect weight; Indicates the base redirection weight; This represents the total error in visual-tactile consistency. Indicates the confidence level of the target template match; This represents the distance from the hand to the target interaction point; Indicates a reference distance; This indicates the current overall confidence level of hand tracking; , , and Indicates the adjustment coefficient; Indicates the current time. The rotational quaternion of each joint; Indicates the first in the target template The rotational quaternion of each joint; This represents the intermediate pose after spherical linear interpolation; 8) Render and output the virtual hand: in, Indicates the target rendering pose after constraint projection; Indicates the first one used for rendering A joint rotation quaternion; This represents the projection onto the set of kinematic constraints associated with the current module type. Constrained projection operators on; This represents the redirection continuous smoothing factor, with a value range of [value range missing]. .

5. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 4, characterized in that, The gesture template library includes grasp templates, press templates, pinch templates, rotate templates, and pull templates.

6. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 1, characterized in that, The specific process of the visual-tactile consistency error monitoring and closed-loop compensation method is as follows: in, and These represent the normal position error and the tangential position error, respectively. , , and These represent attitude error, state error, synchronization delay error, and feedback response error, respectively. and These represent the actual contact point position vector and the virtual contact point position vector, respectively. and These represent the real contact attitude quaternion and the virtual contact attitude quaternion, respectively; and These represent the real interaction state vector and the virtual interaction state vector, respectively. The normal vector representing the virtual contact surface; and These represent the real feedback state vector and the virtual feedback state vector, respectively. This indicates a delay in the synchronization between the virtual and real states; and These represent the normalized scales for the normal position error and the tangential position error, respectively. Indicates the normalization scale of the state error; This represents the normalized scale of the synchronization delay error; Indicates the normalized scale of the feedback response; Indicates the error type index, with values ​​taken from the set. ; Indicates the first Time-varying weights for error types; This represents the error sensitivity coefficient; Indicates the module type for the first Weighted modulation terms for error types; Indicates the corresponding modulation coefficient; Indicates time Total error in visual-tactile consistency; Indicates the compensation control amount; , and These represent the proportional, integral, and differential coefficients, respectively. Indicates the mode adjustment coefficient; Indicates the target-driven weights; This represents the consistency adjustment coefficient for interactive locking; This indicates the target interaction state locked by the system.

7. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 6, characterized in that, The visual-touch consistency error includes position error, posture error, state error, and synchronization delay.

8. The VR virtual-real fusion interaction method based on multi-source sensing and hand redirection as described in claim 4, characterized in that, The closed-loop compensation strategy for smoothly adjusting the virtual hand from its current posture to the target contact posture includes: sensor priority update, visual reregistration, virtual hand redirection weight adjustment, and interaction lock release and recovery.

9. A VR virtual-real fusion interaction system based on multi-source sensing and hand redirection, employing the method described in any one of claims 1-8, characterized in that, include: The real interactive object module is equipped with one or more of the following: pressure sensor, tension sensor, angle encoder, IMU, displacement sensor or visual marker, to collect position, attitude, force, displacement, angle or trigger state caused by user operation; The virtual interactive object module includes geometric models, colliders, motion constraints, physical parameters, and state variables. Virtual interactive objects can be sliders, springs, buttons, knobs, pistons, liquid containers, or buoyant objects. The multi-source sensor registration module establishes the transformation relationship between the real physical coordinate system, the head-mounted display tracking coordinate system, and the virtual scene coordinate system; The sensor-driven synchronization module uses sensor data from the interior or contact points of the real interactive object to drive the state update of the virtual interactive object. The interaction intent prediction module obtains the user's hand position, joint rotation, palm orientation, finger bending degree, and distance between the hand and the virtual interactive object in real time. When the user's hand enters the neighborhood of the target object and the current gesture matches the preset interaction template, it determines that the user has entered a grasping, pressing, pinching, rotating, or pulling state. The hand redirection module compensates for offsets between real hands, virtual hands, and physical props caused by tracking errors, latency, or model differences. The visual-touch consistency closed-loop control module continuously calculates the position error, posture error, state error, and synchronization delay between real and virtual interactive objects. When the error exceeds a preset threshold, it performs sensor priority update, visual reregistration, virtual hand redirection weight adjustment, or interaction lock release to restore the visual-touch consistency state.

Images