Adaptive anti-jitter cross-space teleoperation collaborative interaction method and system

Abstract

The application discloses a kind of self-adaptive anti-jitter across-space teleoperation collaborative interaction method and system.It aims to solve the problem of how to fuse adaptive dynamic damping adjustment of user real-time jitter state and visual-haptic multi-modal collaborative feedback in existing portal mechanism, to bridge the gap between wide range of rapid access and local high-precision stable operation.The application dynamically adjusts operation damping based on portal technology, real-time analysis of user hand jitter characteristics, and constructs adaptive anti-jitter collaborative interaction framework combined with visual magnifier and tactile feedback.The application has the following beneficial effects: significantly improve the success rate of high-precision docking task, better than traditional Go-Go technology without sacrificing completion speed;By optimizing visual and proprioceptive mapping, reduce user cognitive and physical load, NASA-TLX evaluation verifies that its workload is the lowest;Visual magnifier promotes operation strategy from meeting requirements to precision optimization, effectively suppresses the error amplification caused by hand jitter;Based on physiological jitter analysis and dynamic adaptive module, get rid of the limitation of static parameters, provide personalized anti-jitter support for users with different proficiency, fatigue state and psychological stress level, greatly improve system robustness and adaptability.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality (VR) and human-computer interaction (HCI) technology, specifically relating to an interactive control method and system for high-risk, high-precision long-distance operations (such as bomb disposal and remote surgery). Background Technology

[0002] In high-risk remote tasks using virtual reality (such as EOD bomb disposal and telemedicine), users require both wide reach and extremely high precision. Current long-distance interaction technologies can be broadly categorized into three types:

[0003] Arm extension technology (such as Go-Go technology): uses non-linear mapping to lengthen the user's virtual arm, enabling the user to reach distant objects beyond the limitations of physical space.

[0004] Standard Wormhole technology generates a pair of portals in front of the user and next to a distant target. The user's physical hand passes through the nearby entrance portal to control a virtual hand at the distant exit portal.

[0005] Hand Redirection & C / D Ratio Adaptation: This type of technology alters the trajectory or speed of the virtual hand by introducing spatial offset or dynamically adjusting the control-display ratio (C / D gain) between the physical and virtual hands (e.g., semantic pointing). Typically, the virtual hand's movement speed is reduced as it approaches the target to improve local operational accuracy.

[0006] The aforementioned existing technologies have the following significant drawbacks when dealing with "high-risk, long-distance precision operations": Arm extension technologies such as Go-Go: When performing long-distance operations, the mapping algorithm will severely amplify the user's natural physiological hand tremors, making it impossible to complete high-precision and fine docking tasks.

[0007] Standard Wormhole teleportation technology: While it solves the problem of arm extension, it loses binocular disparity due to the limitation of the viewing angle, resulting in blurred depth perception at a distance; and users are prone to arm fatigue when operating in the air for a long time without physical support.

[0008] Traditional hand repositioning technology: While existing single hand repositioning technologies can improve accuracy in local space, they cannot independently solve the problem of "wide-range reach" over ultra-long distances. If they are directly and crudely superimposed on long-distance technologies, the lack of reasonable visual feedback compensation can easily cause spatial cognitive dissonance in users.

[0009] Traditional fixed-threshold damping interaction technology: Existing long-distance damping interaction technologies (such as fixed-threshold deceleration mechanisms) often employ one-size-fits-all static parameters (such as a fixed 5cm buffer distance and a 0.6 attenuation ratio). However, in actual high-risk tasks, different users have varying levels of skill, fatigue, and real-time psychological tension, resulting in dynamic changes in the frequency and amplitude of their hand tremors. Static parameters cannot be personalized and adaptively adjusted to the user's real-time state, easily leading to dilemmas such as "excessive damping affecting efficiency" or "insufficient damping failing to stabilize." Summary of the Invention

[0010] In view of this, the purpose of this invention is to provide an adaptive anti-jitter cross-space teleoperation collaborative interaction method and system, which can solve the following problems: how to introduce adaptive dynamic damping adjustment based on the user's real-time jitter state and visual-tactile multimodal collaborative feedback on the basis of the portal mechanism, so as to bridge the gap between large-scale reach and local high-precision stable operation.

[0011] An adaptive, jitter-resistant, cross-space teleoperation collaborative interaction method includes the following steps: Step S1: Establish a portal mapping from the user's local operating space to the remote target space, enabling the user's virtual hand to reach the remote target across spatial distances; Step S2: Perform context-aware damping adjustment, calculate the distance between the remote interactive execution end and the remote target docking point in real time, and combine the user's real-time hand tremor characteristics to dynamically adjust the control-display ratio CD Gain to suppress hand tremors during fine operations; Step S3: Perform visual magnification assistance based on over-the-shoulder perspective. Generate a virtual camera behind the remote exit portal and render the captured magnified image of the remote area onto the floating lens panel in real time to restore the depth clues and spatial details of the remote operation.

[0012] Preferably, step S1 specifically includes: S11: Render the remote target interactive object in the virtual space, and emit rays to the remote target area through the user's head view or hand controller to lock the remote docking point that needs to be operated. S12: In response to the user's selected operation, instantiate an entrance portal in the local space in front of the user and an exit portal near the remote target; S13: Establish a two-handed mapping relationship. When the user's physical hand moves behind the entrance portal, map its movement displacement to the virtual hand in front of the exit portal at an initial ratio of 1:1 to achieve long-distance reach.

[0013] Preferably, the step S2 specifically includes: S21: Calculate the Euclidean distance D between the remote interaction execution end and the target docking point in real time frame by frame; S22: Preset the buffer area range B; S23: Establish a dynamic damping calculation model based on the coupling of the jitter value J and the distance D. In this model, the real-time output value of the control-display ratio CD Gain is driven by the distance D and is constrained by a parameter framework dynamically set by the real-time jitter eigenvalue J. The parameter framework includes the buffer area range B and the gain lower limit G{min} of the control-display ratio; S24: Calculate the jitter eigenvalue J in real time and compare it with the steady threshold K; if J < K, it is determined to be in a steady state and the basic parameter framework is adopted; if J ≥ K, it is determined to be in a high-frequency jitter state and automatically switched to an enhanced parameter framework, that is, the buffer area B is adaptively enlarged and the gain lower limit G{min} is pressed down; within the selected parameter framework, calculate and output the control-display ratio CD Gain value of the current frame in real time according to the change of the distance D.

[0014] Preferably, in the basic parameter framework, the buffer area range B is 5 cm and the gain lower limit G{min} is 0.6; in the enhanced parameter framework, the buffer area range B is enlarged to 8 cm and the gain lower limit G{min} is pressed down to 0.3.

[0015] Preferably, the step S3 specifically includes: S31: Instantiate a virtual camera at a position offset by a preset distance above the rear of the remote exit portal; S32: Align the orientation of the virtual camera with the remote interaction target to form a third-person over-the-shoulder view; S33: Generate a floating 2D lens panel at a preset position above the exit portal in the remote operation space; S34: Render the real-time magnified local view of the remote end captured by the virtual camera onto the floating lens panel.

[0016] Preferably, the magnification or transparency state of the floating lens is dynamically adjusted according to the user's gaze data: when it is detected that the user's gaze is fixed on the center of the lens for a long time, the magnification is smoothly increased; when the gaze moves away, the lens automatically reduces transparency or folds.

[0017] Furthermore, it also includes: when the virtual hand enters the buffer distance threshold range, the linear motor of the physical handle generates a viscous tactile feedback that decreases in frequency and increases in intensity as the distance decreases.

[0018] An adaptive anti-jitter cross-space teleoperation collaborative interaction system, characterized in that it includes: The tracking and input module is used to acquire six-DOF spatial pose data of the user's head and hands in real time; The portal mapping module is used to establish a spatial mapping between the local operating space and the remote target space; Context-aware damping module is used to dynamically adjust the damping parameters of the control-display ratio CD Gain based on real-time jitter characteristics; The visual magnification rendering module is used to generate and provide over-the-shoulder perspective images from suspended lenses. The physiological tremor analysis and dynamic adaptive module is used to calculate the user's hand tremor characteristics in real time and trigger adaptive adjustment of the damping parameters; The visual-haptic multimodal feedback module combines eye tracking and handheld vibration feedback to achieve gaze-driven lens adjustment and spatial distance haptic rendering.

[0019] Preferably, in the visual magnification rendering module, the spatial position, magnification, or offset of the suspended lens can be customized and adjusted by the user.

[0020] The present invention has the following beneficial effects: 1. Significantly improves task success rate Based on the portal mechanism, this invention introduces adaptive dynamic damping adjustment based on the user's real-time jitter status and visual-tactile multimodal collaborative feedback, effectively bridging the gap between large-scale reach and local high-precision stable operation. Compared with traditional Go-Go technology, this invention significantly improves the success rate of high-precision docking without sacrificing task completion speed.

[0021] 2. Effectively reduces cognitive and physical workload. This invention optimizes the mapping and matching between vision and proprioception, effectively reducing the cognitive and physical burden on users, while giving users a greater sense of embodiment. Objectively verified by NASA-TLX scale assessment, this technology generates the lowest workload among similar methods.

[0022] 3. Strategy Optimization and Jitter Suppression The visual magnifying glass mechanism in this invention effectively improves placement accuracy, prompting users to shift their operating strategy from "meeting basic requirements" to "optimizing accuracy," thereby significantly suppressing the amplification of errors caused by hand tremors during fine operations.

[0023] 4. Extremely high system robustness and personalized adaptability This invention completely overcomes the limitations of static parameters by introducing physiological tremor analysis and dynamic adaptive modules; the system can provide personalized stabilization support for users with different levels of proficiency, fatigue, and psychological tension, greatly improving the system's robustness under complex working conditions. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall architecture and module composition of the adaptive anti-jitter cross-space teleoperation collaborative interaction system provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the control-display ratio (CD Gain) of the context-aware adaptive damping mechanism as a function of distance in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the floating lens and over-the-shoulder view of the visual magnification rendering module in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] This invention provides an adaptive anti-shake cross-space teleoperation collaborative interaction method and system (i.e., a Precision-Wormhole system including adaptive dynamic damping and multimodal feedback), which is mainly used to solve the problems of dynamic amplification of arm shake and lack of depth perception in high-risk virtual reality long-distance operation.

[0029] (1) System overall architecture and module composition like Figure 1 As shown, the system of the present invention mainly includes the following modules: Tracking and Input Module: Used to acquire six-degrees-of-freedom (6DoF) spatial pose data of the user's head-mounted display (HMD) and hand controllers in real time.

[0030] Wormhole module: Used to establish a spatial mapping between the user's local operating space and the remote target space, so that the user's virtual controller can reach the remote target across spatial distances.

[0031] Context-Aware Damping: The core control module, used to calculate the distance between the virtual hand and the distant target in real time and dynamically adjust the control-display gain (CD gain) to suppress hand tremors during fine operations.

[0032] Visual Magnifier: This module generates a floating lens in the user's field of view that provides an "over-the-shoulder view," restoring high-frequency spatial details and depth cues for remote operations.

[0033] Physiological tremor analysis and dynamic adaptation module: used to collect high-frequency trajectory data of the user's handheld device in real time, calculate the user's current "real-time hand tremor frequency and amplitude", and dynamically generate adaptive thresholds for context-aware damping based on this.

[0034] Vision-Haptic Multimodal Feedback Module: Combining eye-tracking with a linear motor in the controller, it enables intelligent lens adjustment based on gaze and haptic force feedback rendering based on spatial distance.

[0035] (2) Specific implementation steps: The specific implementation process of this invention includes the following key steps: Step S1: Establish a long-distance portal mapping (Wormhole Initialization) S11: The system renders a remote target interactive object in virtual space. The user fires a ray into the remote target area through head vision or hand controllers to lock onto the remote docking point where the operation needs to be performed.

[0036] S12: In response to the user's selected operation, the system instantiates an "entry portal" in front of the user (local space) and an "exit portal" near the remote target.

[0037] S13: Establish hand-to-hand mapping. When the user's physical hand moves behind the entrance portal, the system maps its motion displacement to the virtual hand in front of the exit portal at an initial 1:1 ratio, achieving long-distance reach.

[0038] Step S2: Context-Aware Motion Damping S21: The system calculates the Euclidean distance D between the remote 'interactive execution end' (such as a virtual hand, a grasped virtual object, or the end of a virtual tool) and the 'target docking point' in real time frame by frame.

[0039] S22: The system presets a high-precision buffer zone.

[0040] S23: Establish a dynamic damping calculation model based on the coupling of the jitter value J and the distance D. In this model, the real-time output value of the control-display ratio CD Gain is driven by the distance D and is constrained by a parameter framework (including the buffer area range B and the gain lower limit G{min} of the control-display ratio) dynamically set by the real-time jitter eigenvalue J.

[0041] S24: Perform adaptive parameter configuration and real-time gain mapping. The system calculates J in real time and compares it with the steady threshold K to dynamically configure the parameter path for the model in S23: Parameter configuration stage (J determines "intensity"): If it is determined to be in a steady state (J < K), the basic parameter framework is adopted (e.g., B = 5 cm, G{min} = 0.6); if it is determined to be in a high-frequency jitter state (J ≥ K), the system automatically switches to an enhanced parameter framework, that is, the buffer area B is adaptively enlarged (e.g., to 8 cm) and the gain lower limit G{min} is depressed (e.g., to 0.3).

[0042] Numerical mapping stage (D determines "value"): Within the selected parameter framework above, the system obtains the change of the distance D in real time, calculates and outputs the specific control-display ratio CD Gain value of the current frame.

[0043] Step S3: Visual Magnifier Rendering based on the over-the-shoulder view S31: The system instantiates a virtual camera at a position offset upward by a preset distance (an offset of 40 cm in this embodiment) behind the distal exit portal (i.e., behind the virtual hand).

[0044] S32: The orientation of the virtual camera is aligned with the distal interaction target, forming a third-person "over-the-shoulder view".

[0045] S33: Generate a floating 2D lens panel in the distal operation space (at a preset position above the exit portal, ensuring it is within the user's main field of view).

[0046] S34: Render the real-time magnified local view of the distal end captured by the above virtual camera onto the floating lens panel in real time (Render Texture). This design compensates for the binocular disparity lost in the standard portal technology during distal operation, helping the user obtain clear depth cues and spatial details when performing the final high-precision docking, such as Figure 3 As shown.

[0047] Step S4: Multimodal Feedback Rendering S41: Intelligent Eye Tracking: Combining eye-tracking data from HMD, when the system detects that the user's gaze (Gaze) is fixed on the center of the "floating lens" generated in S3 for an extended period of time, the magnification of the lens increases smoothly; when the gaze moves away, the lens automatically becomes semi-transparent or folds to prevent obstruction of the main field of vision.

[0048] S42: Spatial Distance Force Feedback: When the virtual hand enters the target buffer area, the system drives the linear motor of the physical handle to generate micro-vibrations. As the distance D between the virtual hand and the docking point continues to decrease, the vibration frequency decreases but the intensity of a single vibration increases, simulating "viscosity resistance," further stabilizing the user's muscle control through a visual-tactile closed loop.

[0049] (3) Working principle When a user performs a high-risk, long-distance operation, the S1 portal is used to quickly traverse the spatial distance. In the final stage of approaching the target, the S2 physiological tremor analysis module calculates the hand variance in real time, automatically judges and switches between a smooth mode (such as 5cm buffer / 0.6 gain) or a high-frequency tremor mode (such as 8cm buffer / 0.3 gain) to achieve adaptive and precise damping stabilization. At the same time, the S4 multimodal mechanism works in concert, allowing the user to obtain clear depth feedback not only by looking at the over-the-shoulder magnifying glass provided by the S3, but also by obtaining spatial distance force feedback through the viscous micro-vibration of the handle. This achieves a fully closed-loop collaborative operation that balances wide-range reach and extremely high local stability.

[0050] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive anti-jitter cross-space teleoperation collaborative interaction method, characterized in that, It includes the following steps: Step S1: Establish a portal mapping from the user's local operation space to the remote target space, enabling the user's virtual hand to reach the remote target across the spatial distance; Step S2: Perform context-aware damping adjustment, calculate the distance between the remote interaction execution end and the docking point of the remote target in real time, and dynamically adjust the control-display ratio CD Gain in combination with the user's real-time hand jitter eigenvalue to suppress hand jitter during fine operations; Step S3: Perform visual magnification assistance based on the over-the-shoulder perspective, generate a virtual camera behind the remote exit portal, and render the captured magnified local image of the remote end onto the floating lens panel in real time to restore the depth cues and spatial details of the remote operation.

2. The method according to claim 1, characterized in that, The specific steps of Step S1 include: S11: Render the remote target interaction object in the virtual space, emit a ray from the user's head field of view or the hand controller to the remote target area, and lock the remote docking point that needs to be operated; S12: Respond to the user's selection operation, instantiate an entrance portal in the local space in front of the user, and instantiate an exit portal near the remote target; ​ 3. The method according to claim 1, characterized in that, ​ ​ ​ ​ ​ 4. The method according to claim 3, characterized in that, ​ 5. The method according to claim 1, characterized in that, ​ ​ ​ ​ ​ 6. The method according to claim 5, characterized in that, The magnification or transparency of the suspended lens is dynamically adjusted based on the user's gaze data: when the user's gaze is detected to be fixed on the center of the lens for an extended period of time, the magnification is smoothly increased; when the gaze is moved away, the lens automatically reduces its transparency or folds.

7. The method according to claim 1, characterized in that, Also includes: When the virtual hand enters the buffer distance threshold range, the linear motor of the physical handle generates a viscous tactile feedback that decreases in frequency and increases in intensity as the distance decreases.

8. An adaptive anti-jitter cross-space teleoperation cooperative interactive system for implementing the method of any one of claims 1 to 7, characterized in that, include: The tracking and input module is used to acquire six-DOF spatial pose data of the user's head and hands in real time; The portal mapping module is used to establish a spatial mapping between the local operating space and the remote target space; Context-aware damping module is used to dynamically adjust the damping parameters of the control-display ratio CD Gain based on real-time jitter characteristics; The visual magnification rendering module is used to generate and provide over-the-shoulder perspective images from suspended lenses. The physiological tremor analysis and dynamic adaptive module is used to calculate the user's hand tremor characteristics in real time and trigger adaptive adjustment of the damping parameters; The visual-haptic multimodal feedback module combines eye tracking and handheld vibration feedback to achieve gaze-driven lens adjustment and spatial distance haptic rendering.

9. The system according to claim 8, characterized in that, In the visual magnification rendering module, the spatial position, magnification, or offset of the suspended lens can be customized and adjusted by the user.

Images