VR / AR (Virtual Reality / Augmented Reality) interaction method based on multi-granularity micro gestures
By introducing a gesture hierarchy structure and task hierarchy model combining thumb and multiple fingers into VR/AR, the memory burden and fatigue problems of existing interaction methods are solved, achieving high-precision, low-latency micro-gesture interaction and improving interaction efficiency and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-24
- Publication Date
- 2026-05-12
AI Technical Summary
Existing VR/AR interaction methods suffer from problems such as heavy memory burden on controllers, fatigue from bare-hand interaction, and limited space for micro-gestures when switching between high-frequency and high-precision operations in complex scenarios.
This paper introduces the combination of thumb and multiple fingers into micro-gesture interaction, constructs a multi-granular gesture hierarchy, and achieves high-precision, low-latency, and low-fatigue interaction through personalized calibration and task-level model. Graph convolutional networks are used for gesture feature recognition and intent abstraction.
It enhances the expressive power of micro-gestures in complex AR/VR interactive tasks, reduces the cognitive and memory costs for users, and provides a dexterous and efficient input method.
Smart Images
Figure CN122018698A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a VR / AR interaction method based on multi-granularity micro-gestures, belonging to the field of information technology, and particularly to the field of VR / AR interaction technology. Background Technology
[0002] With the rapid development of Virtual Reality (VR) and Augmented Reality (AR), three-dimensional user interfaces (3D UIs) are not only used to carry and present multi-source information, but also serve as the core interactive medium for users to perform operations, manage applications, and complete tasks. As the number of applications, interface layers, and window layouts continue to expand, users need to frequently switch between multiple 3D UIs to browse content, adjust system states, or manipulate virtual objects. This complex interaction scenario places higher demands on the efficiency, stability, lightweight design, and ease of use of interaction methods. Currently, the main input methods in AR / VR systems include button interaction based on controllers, bare-hand interaction based on visual hand tracking, and microgestures interaction based on finger micro-movements, which have developed in recent years. These together constitute the mainstream interaction methods in AR / VR.
[0003] Mainstream interaction methods each have their own characteristics: Gamepad controller interaction relies on physical buttons and high-precision tracking to achieve stable and continuous input. Bare-hand interaction avoids device dependence through visual hand tracking technology, making the interaction more natural and smooth. It mainly includes three types: First, long-range ray interaction, which emits a virtual ray from the palm or fingers, and completes selection and confirmation through the cooperation of the thumb and forefinger; second, short-range touch interaction, which instantiates the user's hand as a virtual hand to achieve touch screen-like clicking, swiping, and dragging; and third, micro-gesture interaction, which uses subtle finger movements as input, with small movements, strong concealment, and suitable for long-term use.
[0004] However, in increasingly complex interactive scenarios, high-frequency switching and high-precision operation are required, and the above methods all have limitations: the function button mapping of the game controller increases the user's memory burden, and holding the physical device limits portability and flexibility; bare-hand ray and touch interaction require a large range of arm movements, which can easily cause fatigue after long-term use and is not stable enough in fine tasks; most existing micro gestures use a single combination of thumb and index finger for interaction, and the gesture space is limited, making it difficult to cover increasingly complex interactive tasks.
[0005] Overcoming the shortcomings of current mainstream interaction methods and designing a high-precision, low-latency, and low-fatigue micro-gesture interaction system to more effectively meet the needs of multi-interface and multi-task parallel operation in AR / VR environments has become a technical challenge that urgently needs to be solved in the field of VR / AR interaction technology. Summary of the Invention
[0006] In view of this, the purpose of this invention is to develop a method that systematically introduces the combination of the thumb and multiple fingers into micro-gesture interaction in AR / VR scenarios, so as to overcome the problem of limited micro-gesture space caused by the traditional reliance on only the thumb-index finger combination.
[0007] To achieve the above objectives, this invention proposes a VR / AR interaction method based on multi-granularity micro-gestures, the method comprising the following steps:
[0008] (1) The traditional and single combination of thumb and index finger is expanded into a multi-finger and flexible combination of thumb and middle finger, ring finger and little finger. On this basis, unified, easy-to-operate and efficient micro-gesture actions are introduced, including fingertip tap, fingertip double tap and thumb sliding up, down and left and right along the side edge of other fingers. The thumb-index finger combination is the most common and the easiest to operate, so the gesture granularity is the highest. The thumb-little finger combination is the least common and the most difficult to operate, so the gesture granularity is the lowest. A micro-gesture granularity hierarchy structure from high to low is constructed to give the micro-gesture space a natural hierarchical attribute, reduce the user's cognitive and memory costs in actual use, and enable it to effectively distinguish a large number of potential and available micro-gestures. The gesture granularity refers to the dexterity used to describe the thumb and different finger combinations when performing micro-gestures.
[0009] (2) Before the user officially starts the interaction, the system guides the user to perform all micro gestures at a natural speed and collects the hand bone sequence data; then the system calculates the user's gesture features through graph convolutional network, constructs the gesture distribution code, and injects this code as the user feature label into the micro gesture recognition network to achieve personalized micro gesture recognition.
[0010] (3) Abstract and classify user operations from the perspective of interaction intent, and construct a task hierarchy model. This task hierarchy model extracts the intent attributes in the interaction operations, namely: taking indivisible atomic operations such as clicking and swiping as basic tasks; taking goal-oriented operations such as switching to other pages or finding specific information as navigation tasks; taking system configuration operations such as ambient light or screen size as system settings tasks; taking global perception operations such as user perspective or interaction layout as global view tasks; through the above abstraction of interaction intent, the complex AR / VR operations are summarized into intent tasks with clear structure and clear boundaries, and a highly generalized task hierarchy model is constructed.
[0011] (4) Based on the task hierarchy model, the execution frequency of different task types was further analyzed, and it was found that they have a stable and consistent distribution characteristics: that is, basic tasks are used most frequently, followed by navigation tasks, system setting tasks and global view tasks, showing a clear decreasing trend. Based on this characteristic, a task hierarchy structure arranged according to execution frequency was constructed.
[0012] (5) The task hierarchy structure and the micro-gesture granularity hierarchy structure are mapped together so that high-granularity micro-gestures correspond to high-frequency tasks and low-granularity micro-gestures correspond to low-frequency tasks, thereby forming an interactive system that conforms to ergonomics and user cognitive patterns. This interactive system is used to achieve a high-precision, low-latency, and low-fatigue VR / AR interactive mode.
[0013] The beneficial effects of this invention are as follows: By extending the traditional thumb-index finger micro-gesture to combinations of thumb and multiple fingers and introducing a gesture granularity hierarchy, this invention effectively enhances the expressive power of micro-gestures in complex AR / VR interactive tasks. The gesture granularity structure allows the micro-gesture space to grow from a finite space into a hierarchical and scalable action system, which not only enhances the dexterity of multi-finger collaborative input but also avoids the limitations of traditional methods that cannot cover complex tasks due to the limited number of gestures, thus providing a richer and more efficient input method. Attached Figure Description
[0014] Figure 1 This is a flowchart of a VR / AR interaction method based on multi-granularity micro-gestures proposed in this invention.
[0015] Figure 2 This is a schematic diagram illustrating the multi-granularity micro-gesture space in an embodiment of the present invention.
[0016] Figure 3 This is a schematic diagram of a zero-sample migration micro-gesture recognition network architecture system in an embodiment of the present invention.
[0017] Figure 4 This is a schematic diagram of the personalized calibration process in an embodiment of the present invention.
[0018] Figure 5 This is a schematic diagram of a frequency-based task hierarchy structure in an embodiment of the present invention.
[0019] Figure 6 This is a schematic diagram illustrating the collaborative mapping between the task hierarchy structure and the micro-gesture granularity hierarchy structure in an embodiment of the present invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.
[0021] See Figure 1 This invention introduces a VR / AR interaction method based on multi-granularity micro-gestures, which includes the following steps:
[0022] (1) See Figure 2 This paper expands the traditional, single combination of thumb and index finger into a multi-finger, flexible combination of thumb and middle, ring, and little fingers. Based on this, it introduces unified, easy-to-operate, and highly efficient micro-gesture actions, including single-tap, double-tap, and thumb sliding along the sides of other fingers. The thumb-index finger combination is the most common and easiest to operate, therefore having the highest gesture granularity; the thumb-little finger combination is the least common and most difficult to operate, therefore having the lowest gesture granularity. A hierarchical structure of micro-gesture granularity is constructed from high to low, giving the micro-gesture space a natural hierarchical attribute, reducing the user's cognitive and memory costs in actual use, and enabling them to effectively distinguish a large number of potential and usable micro-gestures. The gesture granularity refers to the dexterity used to describe the thumb's dexterity when performing micro-gestures in different finger combinations.
[0023] (3) In real-world application scenarios involving multiple devices and users, existing micro-gesture recognition methods suffer from performance instability. This is because different devices have inconsistent capture accuracy and noise characteristics for hand skeletal sequences, resulting in significant differences in gesture data distribution. Simultaneously, different users exhibit significant individual differences in their movement habits, such as amplitude, speed, and inertial path, when performing the same micro-gesture, causing the micro-gesture recognition network to show significant deviations when facing new users or devices. To address this issue, this invention achieves zero-shot transfer micro-gesture recognition through personalized calibration.
[0024] In the embodiments, the following are adopted: Figure 3 The zero-sample transfer micro-gesture recognition network architecture system shown adopts Figure 4 The personalized calibration process is shown below. Before the user officially begins the interaction, the system guides the user to perform all micro-gestures at a natural speed to collect their hand skeletal sequence data; then, the system calculates the user's gesture features through a graph convolutional network, constructs their gesture distribution code, and injects this code as a user feature label into the micro-gesture recognition network to achieve personalized micro-gesture recognition;
[0025] (4) Current technical solutions for micro-gesture interaction adopt a one-to-one direct mapping strategy between micro-gestures and operations, such as assigning up and down swipe gestures to content scrolling or page turning respectively. However, with the rapid increase in the number of interactive interfaces and interaction types in AR / VR scenarios, the limited number of micro-gestures is difficult to meet the infinitely growing diverse interaction needs. To this end, this invention abstracts and classifies user operations from the perspective of interaction intent, and constructs a task hierarchy model. This model extracts the intent attributes in the interaction operations, namely: taking indivisible atomic operations such as clicking and swiping as basic tasks; taking goal-oriented operations such as switching to other pages or finding specific information as navigation tasks; taking system configuration operations such as ambient light or screen size as system setting tasks; taking global perception operations such as user perspective or interaction layout as global view tasks; through the above abstraction of interaction intent, complex AR / VR operations are summarized into intent tasks with clear structure and well-defined boundaries, and a highly generalized task hierarchy model is constructed.
[0026] See Figure 5 Based on the task hierarchy model, the execution frequency of different task types was further analyzed, and it was found that they have a stable and consistent distribution characteristics: that is, basic tasks are used most frequently, followed by navigation tasks, system setting tasks and global view tasks, showing a clear decreasing trend. Based on this characteristic, a task hierarchy structure arranged according to execution frequency was constructed.
[0027] (5) See Figure 6 The task hierarchy structure and the micro-gesture granularity hierarchy structure are mapped together so that high-granularity micro-gestures correspond to high-frequency tasks and low-granularity micro-gestures correspond to low-frequency tasks, thereby forming an interaction system that conforms to ergonomics and user cognitive patterns. This interaction system is used to achieve a high-precision, low-latency, and low-fatigue VR / AR interaction method.
[0028] The inventors conducted numerous experiments on the method proposed in this invention, and the experimental results are as follows:
[0029] Twenty-five participants were recruited and, under the same VR interaction task, their methods—including controller interaction, direct bare-hand interaction, indirect bare-hand interaction, thumb-index finger micro-gesture interaction, and the method proposed in this invention—were compared. The main metrics were task completion time, arm movement distance required to complete the task, and the RULA score, which reflects ergonomic risks. The experimental task and results are described below:
[0030] Experimental Task: Participants assume a comfortable seated position and are allowed to rest at any time. Participants first freely explore the virtual reality (VR) environment to familiarize themselves with the overall layout and various 3D user interfaces. Subsequently, participants are given ample time to practice controller use, direct bare hand use, indirect bare hand use, thumb-index finger micro-gestures, and the methods proposed in this invention. The inventors designed a workflow task simulating real VR usage, covering all major interaction categories: opening the news app, selecting the first article, and browsing its details (basic task). Then switching to the library, launching the music app, playing a song, and performing a "next track" switch (navigation task). Next, entering the system settings panel, adjusting the volume, and switching Bluetooth status (system control task). Then, optimizing the global viewpoint by zooming or moving the entire VR interface panel (global viewpoint adjustment task). Finally, returning to the news app to continue browsing other articles.
[0031] The experimental results are as follows: Table 1 summarizes the task completion time for all users under different interaction methods; Table 2 summarizes the arm movement distance; and Table 3 summarizes the RULA scores. RULA (Rapid Upper Limb Assessment) is a standardized assessment tool used to evaluate the risk of musculoskeletal injury caused by upper limb, neck, and trunk posture during task performance. (Range 1-7, lower scores indicate better ergonomics).
[0032] Table 1 (Unit: seconds)
[0033] User ID / Interaction Technology handle Bare-hand indirect interaction direct interaction with bare hands Thumb-index finger micro-gesture This invention 1 37.64 37.94 40.00 32.82 26.82 2 38.31 36.20 40.17 34.40 27.70 3 37.07 36.09 43.97 37.65 30.43 4 35.54 38.61 42.25 42.05 27.76 5 41.57 40.60 45.60 36.97 29.71 6 42.79 43.53 47.30 38.89 31.08 7 39.14 40.07 43.12 37.09 30.90 8 39.15 41.23 43.18 37.85 30.18 9 38.89 34.48 37.72 32.61 22.73 10 38.91 36.89 42.29 36.77 30.04 11 41.22 37.22 42.80 36.49 28.47 12 39.96 39.31 43.16 36.82 30.87 13 39.06 38.20 42.89 37.04 26.63 14 39.71 44.43 40.57 41.21 28.98 15 38.90 35.32 38.55 35.35 23.42 16 36.81 35.60 46.31 37.73 29.32 17 40.41 36.21 43.46 35.24 26.90 18 38.64 35.56 37.63 38.13 24.35 19 38.54 38.39 42.13 39.71 27.22 20 40.39 38.80 40.32 39.21 28.65 21 39.82 37.41 38.64 36.92 29.00 22 39.32 36.40 44.20 35.51 24.58 23 39.25 37.30 42.23 35.82 29.49 24 39.79 34.21 41.29 29.91 21.10
[0034] Table 2 (Unit: meters)
[0035] User ID / Interaction Technology handle Bare-hand indirect interaction direct interaction with bare hands Thumb-index finger micro-gesture This invention 1 6.55 7.20 5.51 2.37 0.90 2 6.27 7.90 4.56 3.87 0.87 3 3.60 7.40 4.67 2.57 0.77 4 4.26 8.60 4.85 2.68 0.60 5 4.28 8.13 4.78 3.17 0.72 6 4.85 8.90 4.80 3.01 0.83 7 5.86 7.80 5.04 3.62 2.05 8 5.33 8.70 5.82 4.30 1.77 9 5.22 7.60 4.86 2.50 0.89 10 5.52 6.40 4.29 3.83 3.11 11 4.80 8.50 6.80 3.89 0.96 12 6.52 8.30 6.50 4.07 0.93 13 4.73 7.50 4.93 2.94 0.71 14 4.92 7.70 3.97 2.17 0.65 15 6.24 8.10 4.65 2.94 1.08 16 4.70 7.30 4.64 3.73 0.67 17 5.96 8.00 5.30 2.75 1.08 18 6.61 7.90 5.71 2.87 0.72 19 5.98 8.20 5.15 3.53 1.08 20 6.90 8.60 5.69 3.85 1.09 21 5.42 8.50 4.52 3.57 1.14 22 6.11 7.80 5.61 3.21 0.66 23 6.38 8.40 5.09 2.99 0.68 24 4.51 7.60 4.22 2.20 0.96
[0036] Table 3
[0037] User ID / Interaction Technology handle Bare-hand indirect interaction direct interaction with bare hands Thumb-index finger micro-gesture This invention 1 4 5 5 3 2 2 3 3 5 3 1 3 3 3 3 2 1 4 3 4 6 3 1 5 5 4 6 3 1 6 3 3 5 3 1 7 3 4 3 3 2 8 3 5 6 3 2 9 3 3 5 2 1 10 5 4 6 3 2 11 3 5 3 3 1 12 4 6 6 3 1 13 3 3 5 3 2 14 3 4 3 2 1 15 3 4 6 3 1 16 3 3 4 3 1 17 3 5 6 3 1 18 4 4 5 3 1 19 3 4 6 2 1 20 4 5 5 3 2 21 3 3 3 2 1 22 3 4 6 3 1 23 4 5 5 3 1 24 3 3 3 2 1
[0038] The above experimental results prove that the method proposed in this invention is feasible and effective.
[0039] The present invention has been described in detail above with reference to the embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention. The scope of protection of the present invention shall be defined by the appended claims.
Claims
1. A VR / AR interaction method based on multi-granularity micro-gestures, characterized in that: The method includes the following steps: (1) The traditional and single combination of thumb and index finger is expanded into a multi-finger and flexible combination of thumb and middle finger, ring finger and little finger. On this basis, unified, easy-to-operate and efficient micro-gesture actions are introduced, including fingertip tap, fingertip double tap and thumb sliding up, down and left and right along the side edge of other fingers. The thumb-index finger combination is the most common and the easiest to operate, so the gesture granularity is the highest. The thumb-little finger combination is the least common and the most difficult to operate, so the gesture granularity is the lowest. A micro-gesture granularity hierarchy structure from high to low is constructed to give the micro-gesture space a natural hierarchical attribute, reduce the user's cognitive and memory costs in actual use, and enable it to effectively distinguish a large number of potential and available micro-gestures. The gesture granularity refers to the dexterity used to describe the thumb and different finger combinations when performing micro-gestures. (2) Before the user officially starts the interaction, the system guides the user to perform all micro gestures at a natural speed and collects the hand bone sequence data; then the system calculates the user's gesture features through graph convolutional network, constructs the gesture distribution code, and injects this code as the user feature label into the micro gesture recognition network to achieve personalized micro gesture recognition. (3) Abstract and classify user operations from the perspective of interaction intent, and construct a task hierarchy model. This task hierarchy model extracts the intent attributes in the interaction operations, namely: taking indivisible atomic operations such as clicking and swiping as basic tasks; taking goal-oriented operations such as switching to other pages or finding specific information as navigation tasks; and taking system configuration operations such as ambient light or screen size as system setting tasks. Global perception operations such as user perspective or interactive layout are treated as global view tasks. Through the above abstraction of interactive intent, complex AR / VR operations are summarized into intent tasks with clear structure and well-defined boundaries, and a highly generalized task hierarchy model is constructed. (4) Based on the task hierarchy model, the execution frequency of different task types was further analyzed, and it was found that they have a stable and consistent distribution characteristics: that is, basic tasks are used most frequently, followed by navigation tasks, system setting tasks and global view tasks, showing a clear decreasing trend. Based on this characteristic, a task hierarchy structure arranged according to execution frequency was constructed. (5) The task hierarchy structure and the micro-gesture granularity hierarchy structure are mapped together so that high-granularity micro-gestures correspond to high-frequency tasks and low-granularity micro-gestures correspond to low-frequency tasks, thereby forming an interactive system that conforms to ergonomics and user cognitive patterns. This interactive system is used to achieve a high-precision, low-latency, and low-fatigue VR / AR interactive mode.