Display method for air gesture rotation operation in virtual reality environment

Abstract

The invention discloses a display method for an air gesture rotation operation in a virtual reality environment, relates to the technical field of gesture interaction, and solves the technical problem that the rotation of a virtual object cannot be accurately controlled according to gesture motion data input by a user in real time. According to the gesture position data, calculating a normal vector and an included angle of a plane formed by the two-hand connecting line vector at the fixed time, and converting the normal vector and the included angle into an Euler angle coordinate of a gesture rotation direction and a gesture rotation angular velocity; inputting the Euler angle coordinate of the gesture rotation direction and the gesture rotation angular velocity into a display control mapping model to obtain the rotation direction and the rotation velocity of the virtual object; and displaying the rotation of the virtual object on the virtual reality equipment according to the rotation direction and the rotation speed of the virtual object. According to the method, rotation control of reachable out-of-domain objects can be completed by adopting a bare hand in virtual reality, interaction is natural and smooth, and the method has explainability in behavior; a machine learning method is introduced to intelligently recognize and judge the gesture rotation direction of the user, so that the interaction accuracy is improved; the dynamic non-isomorphic mapping mechanism is applied to the air gesture rotation control operation, and the accuracy of gesture control is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of gesture interaction technology, and in particular to a display method for mid-air gesture rotation operations in a virtual reality environment. Background Art

[0002] Mid-air gesture interaction can broaden the scope of human-computer interaction and enhance the immersion and naturalness of interaction. It is one of the commonly used interaction methods in virtual environments (VR) (Erra U, Malandrino D, Pepe L. Virtual reality interfaces for interacting with three-dimensional graphs [J]. International Journal of Human–Computer Interaction, 2019, 35 (1): 75-88.). Mid-air gestures rely on the movement or semantic consistency between hand limbs and virtual objects to achieve intuitive and natural control of objects (LaViola Jr JJ, Kruijff E, McMahan RP, et al. 3D user interfaces: theory and practice [M]. Addison-Wesley Professional, 2017. and Qi J, Ma L, Cui Z, et al. Computer vision-based hand gesture recognition for human-robot interaction: a review [J]. Complex & Intelligent Systems, 2024, 10 (1): 1581-1606.). Aerial gesture rotation operation is a common task in virtual reality aerial gesture interaction. It relies on the consistency of the rotational motion of the hand and the virtual object to achieve rotational manipulation of objects outside the reachable domain (Mendes D, Caputo FM, Giachetti A, et al. A survey on 3d virtual object manipulation: From the desktop to immersive virtual environments [C] / / Computer graphics forum. 2019, 38 (1): 21-45.).

[0003] Aerial gestures rely on the consistency of movement or semantics between the hand and the virtual object to achieve intuitive and natural control of the object. Aerial gesture rotation operation is a common task in virtual reality aerial gesture interaction. It relies on the consistency of the rotation movement of the hand and the virtual object to achieve the rotational control of the object outside the reachable domain. However, when we use aerial gestures to rotate 3D objects in an immersive environment, we often cannot achieve the accuracy and control we have in the real world. This reduction in accuracy stems from the high degree of freedom and instability of hand movement, which is manifested as the inconsistency between the movement display of the virtual object and the psychological expected trajectory (Frees S, Kessler GD, Kay E. PRISM interaction forenhancing control in immersive virtual environments [J]. ACM Transactions on Computer-Human Interaction (TOCHI), 2007, 14 (1): 2-es.).

[0004] The mid-air gesture rotation operation has control uncertainty problems caused by the high freedom of gesture movement and motion instability problems caused by the physiological characteristics of gesture movement. Therefore, directly mapping the user's gesture motion data to the transformation of virtual objects will result in low interaction efficiency and accuracy. Existing research has also proposed different solutions to related problems.

[0005] To address the problems caused by high-degree-of-freedom gesture motion, most existing studies use methods that convert multi-degree-of-freedom gesture motion into single-degree-of-freedom rotation transformation of virtual objects, such as 2D plane gesture control mapping (VeitM, CapobiancoA, BechmannD. Influence of degrees of freedom's manipulation on performances during orientation tasks in virtual reality environments[C] / / Proceedingsofthe 16th ACM Symposium onVirtualRealitySoftwareandTechnology.2009:51-58.), direct mapping of contact gesture interaction agent models (MendesD, LopesP, FerreiraA. Hands-oninteractive tabletop lego application[C] / / proceedings of the 8thinternational conference on advances in computer entertainmenttechnology.2011:1-8.) and aerial gesture motion mapping (SturmanDJ, ZeltzerDA surveyofglove-based input[J].IEEE Computer graphics and Applications,1994,14(1):30-39.). Although these mapping technologies improve the accuracy of gesture interaction, they violate the original intention of naturalness and intuitiveness of gesture interaction. The interaction method using gesture control in a 2D plane is similar to the mouse control method in the real world, but in a virtual environment, this control method has a 2D to 3D conversion difference (Veit M, Capobianco A, Bechmann D. Influence of degrees of freedom's manipulation on performances during orientation tasks in virtual reality environments [C] / / Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology. 2009: 51-58.), and users cannot intuitively control the rotation operation.There are also studies that use indirect interaction methods of proxy models to achieve rotation control of virtual objects. This method has the problem that the user's attention switches back and forth between an operated object and an operating widget.

[0006] In response to the problem caused by the instability of gesture movement, some studies have used motion-display gain (Detection of Scaled Hand Interactions in Virtual Reality: The Effects of Motion Direction and Task Complexity) to avoid the noise of gesture movement, so as to improve the authenticity and naturalness of gesture interaction. For example, the PRISM technology commonly used in pointing selection tasks reduces the misoperation caused by hand instability by reducing the object conversion speed (Frees S, Kessler GD, Kay E. PRISM interaction for enhancing control inimmersive virtual environments [J]. ACM Transactions on Computer-Human Interaction (TOCHI), 2007, 14 (1): 2-es.). It can be seen that existing studies have made some attempts to control mid-air gestures, but there is no rotation task control method that can balance naturalness and stability.

[0007] Aerial gestures achieve control of virtual objects outside the reachable domain through the psychological cognitive consistency of the corresponding spatial changes between the user's gestures and the manipulated objects. It is a relatively intuitive human-computer interaction method. The human-computer interaction process can be regarded as a perception-cognition-action cycle, and action is closely related to perception and cognition. Among them, the consistency between the user's hand (effector) and eye (receptor) perception and user cognition is the key to the user's sense of rotation operation display and control space coupling (Ahmad A, Migniot C, Dipanda A. Hand pose estimation and tracking in real and virtual interaction: A review [J]. Image and Vision Computing, 2019, 89: 35-49.), and the error range that human cognition can tolerate is very large (Steinicke F, Bruder G, Jerald J, et al. Analyses of humans ensitivity to redirected walking [C] / / Proceedings of the 2008 ACM symposium on Virtual reality software and technology. 2008: 149-156.), so the gesture rotation operation is actually a fuzzy control. During the rotation operation, the user first perceives the spatial position and orientation of the virtual object through vision, and then rotates the virtual object through gesture interaction. The computer outputs the rotation direction and angle of the virtual object, and the user continuously adjusts the gesture movement direction and speed according to the display of the virtual object, forming a closed loop.

[0008] Therefore, in the field of mid-air gesture-controlled rotation operation in a virtual environment, in order to achieve natural conversion of rotation direction and precise control of rotation angle, it is necessary to establish a display-control mapping model with psychological cognitive consistency between display and control to improve the naturalness and ease of use of gesture rotation interaction. The technical status of the display-control mapping model mainly includes the following three aspects:

[0009] (1) User perception and cognitive characteristics

[0010] In the gesture interaction rotation operation task in the VR environment, the cognitive consistency between the rotation movement of the human hand and the rotation movement of the controlled virtual object is the goal of achieving natural interaction. Therefore, correctly understanding the rotation movement of the virtual object space is the basis for accurately performing the rotation operation.

[0011] Proprioceptive sensation is inherent in humans and plays a key role in virtual reality gesture interaction (Fogtmann MH, Fritsch J, Kortbek K J. Kinesthetic interaction: revealing the bodily potential in interaction design [C] / / Proceedings of the 20th Australasian conference on computer-human interaction: designing for habitus and habitat. 2008: 89-96.). For gesture interaction within the reachable domain, users improve kinesthetic control during the interaction process by utilizing proprioception (Boeck JD, Raymaekers C, Coninx K. Exploiting proprioception to improve haptic interaction in a virtual environment [J]. Presence: Teleoperators and Virtual Environments, 2006, 15 (6): 627-636.), but for interaction outside the reachable domain, the influence of proprioception on control ability is limited and may be further affected by other sensory perceptions such as vision, hearing, touch, etc. (Boeck JD, Raymaekers C, Coninx K. Exploiting proprioception to improve haptic interaction in a virtual environment [J]. Presence: Teleoperators and Virtual Environments, 2006, 15 (6): 627-636.). Therefore, in the absence of touch, visual perception is particularly important for kinesthetic control in mid-air gesture interaction.

[0012] People's ability to recognize the rotation of virtual objects directly affects the effectiveness of rotation operations (Klatzky RL. Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections [M] / / Spatial cognition: An interdisciplinary approach to representing and processing spatial knowledge. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998: 1-17. and Pietropaolo S, Crusio W E. Learning spatial orientation [J]. Encyclopedia of the Sciences of Learning, 2012: 1969-1971.). However, people's ability to recognize spatial rotation is limited, and it is difficult to imagine the rotation transformation of objects in three-dimensional space in any direction (Parsons L M. Inability to reason about an object's orientation using an axis and angle of rotation [J]. Journal of experimental psychology: Human perception and performance, 1995, 21 (6): 1259.). Some studies have suggested that in order to make the user's spatial rotation transformation cognition more accurate during rotation operations, two angles must remain unchanged during the rotation process, namely the angle between the operated object and the rotation axis and the angle between the rotation axis and the operator's spatial reference system. In particular, the angle between the operated object and the rotation axis has a greater impact on the user's operation performance. When these two angles are simple or standardized (0° or 90°), the accuracy of rotation cognition will be higher.

[0013] In summary, human proprioception and spatial rotation cognition are cognitive factors that affect rotation operations in virtual reality. For mid-air gesture rotation operations outside the reachable domain, the user's proprioception is limited, and visual perception is particularly important for improving control ability. Since human spatial rotation cognition is limited, setting the manipulated object to an orthogonal direction with the rotation axis and the spatial reference system helps users make rotation judgments.

[0014] (2) Characteristics of mid-air gesture movements

[0015] In virtual reality, the speed and direction of user gesture movement are the main parameters that describe its motion characteristics. Understanding the user gesture movement characteristics is the basis for establishing the display and control space mapping relationship between the gesture and the operated object.

[0016] Gestures have dynamic movement characteristics. Woodworth proposed the movement phase model (Woodworth RS. Accuracy of voluntary movement [J]. The Psychological Review: Monograph Supplements, 1899, 3 (3): i.) to explain the dynamic movement timing characteristics. This model was first used to describe pointing movements, but some scholars have found that this model can be applied to goal-oriented virtual hand movement tasks (Tang M, Liu X, Dong Y, et al. Absence of inertial load on hand decreases task performance in virtual reality interaction [J]. International Journal of Human–Computer Interaction, 2024, 40 (12): 3219-3233.). According to the model, the hand movement process consists of two parts: the ballistic phase and the adjustment phase. These two components are considered to be two different motor control strategies: impulse control and limb-goal control. Impulse control refers to rapid adjustments in early motor planning, while limb-goal control involves discrete corrections to reduce the difference between later limb and goal positions (Elliott D, Lyons J, Hayes SJ, et al. The multiple process model of goal-directed reaching revisited [J]. Neuroscience & Biobehavioral Reviews, 2017, 72: 95-110.).Existing studies often judge the gesture movement stage by the discontinuity of the movement stage. They judge the speed decrease point (close to 0) or the speed minimum point (acceleration passes the zero point and the acceleration changes from negative to positive) as the dividing point between the two movement stages (Tang M, Liu X, Dong Y, et al. Absence of inertial load on hand decreases task performance in virtual reality interaction[J]. International Journal of Human–Computer Interaction, 2024, 40(12): 3219-3233. and Deng CL, Geng P, Hu YF, et al. Beyond Fitts's law: a three-phase model predicts movement time to position an object in an immersive 3D virtual environment[J]. Human factors, 2019, 61(6): 879-894.).

[0017] The direction of gesture movement is mainly affected by physiological characteristics. The constraints of the arm motion chain and the role of biomechanics make the gesture movement driven by the user's arm have obvious spatial characteristics. The arm motion chain is a series of upper limb joints connecting the shoulder joint to the fingertips. Among them, the upper limb joints mainly include the shoulder joint, elbow joint, wrist joint and finger joint, and each joint has different degrees of freedom and mobility. The arm motion chain limits the arm movement space and posture comfort of the human body. The cooperation of the entire arm motion chain helps users complete interactive operations and interactive gestures in various spatial positions. Due to the limitations of the arm motion chain, the direction of hand movement in gesture interaction is often inconsistent with the intended direction, and the gesture movement performance is different for different movement directions. It is generally believed that the posture of bending the arm is more comfortable than the posture of stretching the arm. When the arm kinematic chain is longer, it results in more weight and inertia, and requires more muscle strength, resulting in reduced accuracy of interaction (Balakrishnan R, MacKenzie I S. Performance differences in the fingers, wrist, and forearm computer input control [C] / / Proceedings of the ACM SIGCHIConference on Human factors in computing systems. 1997: 303-310.).

[0018] Due to the limitations of the arm motion chain and the range of motion of the joints, the direction of the gesture movement driven by the arm movement may be inconsistent with the direction of the person's intention, which also leads to differences in the performance of different directions of the gesture movement. The division of gesture movement stages can better describe the gesture movement in the rotation operation and lay the foundation for establishing a suitable rotation angle control display mapping mechanism.

[0019] (3) Rotational angular velocity control display mapping mechanism

[0020] The existing mapping mechanisms of rotation operation speed and rotation angle include isomorphic mapping (Hinckley K, Tullio J, Pausch R, et al. Usability analysis of 3D rotation techniques [C] / / Proceedings of the 10th annual ACM symposium on User interface software and technology. 1997: 1-10. and Ware C, Rose J. Rotating virtual objects with real handles [J]. ACM Transactions on Computer-Human Interaction (TOCHI), 1999, 6 (2): 162-180.), static non-isomorphic mapping (Poupyrev I, Weghorst S, Fels S. Non-isomorphic 3D rotational techniques [C] / / Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 2000: 540-547.) and dynamic non-isomorphic mapping (Frees S, Kessler GD, Kay E. PRISM interaction for enhancing control in immersive virtual environments[J]. ACM Transactions on Computer-Human Interaction(TOCHI), 2007, 14(1): 2-es.). Isomorphic mapping is a 1:1 mapping between gesture movement and the rotation speed of the operated object, static non-isomorphic mapping is a 1:N mapping between gesture movement and the rotation speed of the operated object, and dynamic non-isomorphic mapping is a dynamic selection of the rotation gain ratio based on the gesture movement speed. Among them, dynamic non-isomorphic mapping is associated with the gesture movement stage. Reasonably adjusting the rotation speed of the operated object based on the gesture movement speed can effectively reduce gesture movement noise, reflect people's interaction intentions, and thus improve user operation accuracy and naturalness (Bowman DA, McMahan RP, Ragan E D. Questioning naturalism in 3D user interfaces[J]. Communications of the ACM, 2012, 55(9): 78-88.).The effectiveness of the dynamic non-isomorphic mechanism has been verified in pointing tasks, contact movement, and contact rotation tasks.

[0021] PRISM technology is a typical dynamic non-isomorphic mapping technology that dynamically adjusts the control display ratio between the hand and the controlled object through real-time motion speed. Movements below the minimum speed are regarded as noise and do not produce interaction. When the user's gesture rotation speed is lower than the maximum speed threshold, the object rotation speed will increase linearly with the gesture rotation speed. This method can effectively avoid the gesture motion noise caused by the user's aimless motion and reduce the rotation speed of the object when the gesture moves at a low speed, thereby improving the operation efficiency and accuracy. Some studies have modified the mapping function of PRISM technology to obtain a non-isomorphic mapping technology suitable for contact gestures and handle control rotation operations to meet the needs of fast coarse adjustment (Gao Z, Wang H, Lv H, et al. Evaluating the effects of non-isomorphic rotation on 3d manipulation tasks in mixed reality simulation [J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 28 (2): 1261-1273.). The applicability of non-isomorphic mapping technology in air gestures has not yet been verified.

[0022] Based on the stage characteristics of gesture movement, dynamic non-isomorphic rotation mapping can provide a suitable gain ratio based on the gesture movement speed to meet the efficiency and accuracy requirements of different movement stages. However, the existing dynamic non-isomorphic mapping mechanism is based on the contact gesture interaction rotation operation task, and its application in the mid-air gesture rotation operation task remains to be explored.

[0023] How to accurately control the rotation of virtual objects based on the gesture motion data input by the user in real time to improve the naturalness and ease of use of existing mid-air gesture rotation interactions remains to be solved. Summary of the invention

[0024] The present application provides a display method for mid-air gesture rotation operation in a virtual reality environment. Its technical purpose is to enable a computer to accurately control the rotation of a virtual object based on gesture motion data input by a user in real time, to ensure the psychological consistency between the control motion of the hand and the display motion of the virtual object in the mid-air gesture rotation operation, and to improve the naturalness and ease of use of existing mid-air gesture rotation interactions.

[0025] The above technical objectives of this application are achieved through the following technical solutions:

[0026] A display method for an aerial gesture rotation operation in a virtual reality environment, the display method being applicable to a rotation task controlled by both hands, comprising:

[0027] Collecting gesture position data;

[0028] The normal vector and angle of the plane formed by the line vectors connecting the two hands at a fixed time are calculated according to the gesture position data, and the normal vector and angle are converted into the Euler angle coordinates of the gesture rotation direction and the gesture rotation angular velocity;

[0029] The Euler angle coordinates of the gesture rotation direction and the gesture rotation angular velocity are input into the display control mapping model to obtain the rotation direction and rotation velocity of the virtual object;

[0030] The rotation of the virtual object is displayed on the VR device according to the rotation direction and rotation speed of the virtual object.

[0031] Furthermore, the display and control mapping model includes a random forest classification model and a dynamic non-isomorphic mapping mechanism. The random forest classification model outputs the rotation direction of the virtual object according to the Euler angle coordinates of the input gesture rotation direction; the dynamic non-isomorphic mapping mechanism obtains the rotation speed of the virtual object according to the input gesture rotation angular velocity.

[0032] Furthermore, the random forest classification model converts the Euler angle coordinates of the gesture rotation direction into four orthogonal rotation directions, where the four orthogonal rotation directions include up, down, front, and back.

[0033] Furthermore, the dynamic non-isomorphic mapping mechanism is used to correct the angular velocity of gesture rotation, which divides the gesture rotation operation into a start phase, a ballistic phase, an adjustment phase and an end phase. In the ballistic phase and the adjustment phase of the gesture rotation operation, the rotational motion of the hand and the rotational motion of the manipulated virtual object are mapped in terms of speed.

[0034] Furthermore, the dynamic non-isomorphic mapping mechanism is expressed as:

[0035]

[0036] Among them, K represents the magnification factor, CD represents the control-display ratio, S represents the gesture rotation angular velocity, MinS represents the minimum speed for judging the start of movement, and SC represents the proportional constant.

[0037] Furthermore, MinS is 4.5° / s to 5° / s, and SC is 90° / s.

[0038] The beneficial effects of this application are:

[0039] (1) The display method for mid-air gesture rotation operation in a virtual reality environment described in this application supports the use of bare hands to complete the rotation control of objects outside the reachable domain in VR, and the interaction is natural and smooth and has behavioral explainability.

[0040] (2) This method introduces a machine learning method to intelligently identify and judge the rotation direction of the user's gesture by using a random forest classification model. After being trained based on a large amount of user gesture data, the model can accurately identify the user's intention and accurately map the gesture action to the rotation direction of the virtual object in real time. This intelligent judgment mechanism not only improves the accuracy of the interaction, but also allows users to perform gesture operations more freely without worrying about the accuracy of the operation.

[0041] (3) This method applies a dynamic non-isomorphic mapping mechanism to the mid-air gesture rotation control operation, significantly improving the accuracy of gesture control. The dynamic non-isomorphic mapping mechanism effectively filters out random noise and unintentional jitter in gesture motion by dynamically adjusting the mapping ratio, ensuring accurate rotation control even when there are unstable factors in the user's gesture. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a VR display flow chart of the mid-air gesture rotation operation in the embodiment of the present application;

[0043] Figure 2 Schematic diagram of the dynamic non-isomorphic mapping mechanism in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0045] like Figure 1 As shown, the display method for mid-air gesture rotation operation in a virtual reality environment described in the present application is mainly implemented through a random forest classification model of the rotation direction and a dynamic non-isomorphic mapping mechanism of the rotation control angular velocity. After the computer obtains the gesture position data of the person, it calculates the normal vector and angle of the plane formed by the vectors connecting the two hands at a fixed time, and then converts it into the Euler angle coordinates of the rotation direction and the rotation angular velocity. After that, the gesture motion input data will reach the display control mapping model, the gesture motion direction will be judged by the random forest classification model, and the rotation angle of the virtual object will be output based on the gesture motion speed through the dynamic non-isomorphic mapping mechanism, and finally converted into a display output.

[0046] In the display-control mapping model, the hand movement direction is first converted into four orthogonal rotation directions through the random forest classification model. The random forest classification model has the advantages of simple logic and fast operation. In the hand rotation time series features, the four orthogonal rotation directions (up, down, front, and back) have consistency, understandability, and distinguishability. Therefore, this paper establishes a random forest classification model to determine the ownership of the rotation axis in these four rotation directions.

[0047] After acquiring the gesture data of the VR scene, a minimum speed threshold (MinS) is first set to identify and eliminate motion data with a speed lower than the threshold. Next, the rotational motion data points of the gesture are matched with the rotation axis direction of the virtual object (such as up, down, front, and back), and a large amount of data is used for training and verification. Finally, a 10-fold cross-validation method can be used to comprehensively consider classification performance indicators such as accuracy, precision, recall, and F1 score to conduct a detailed performance evaluation to evaluate the robustness of the model.

[0048] Taking a virtual reality model application as an example, when a cube with a size of 0.3m appears 2m in front of the subject's binocular center, the data of the rotation gesture performed by the Wizard of Oz method of 20 users are collected through experiments. With a time window of 0.5s, the unit vector (x, y, z) of the gesture rotation direction and the angular velocity (° / s) of the gesture rotation are recorded. In the calculation, the line connecting the positions of the hands recorded in the previous record and the line connecting the positions of the hands recorded in the next record (0.5s apart) are first converted into vector coordinates, and then their cross product is calculated to obtain the normal vector of the rotation axis and convert it into a unit vector. The angular velocity of the gesture rotation is calculated based on the ratio of the angle formed by the lines connecting the hands every 0.5s to the movement time.

[0049] In order to eliminate the interference of transitional motion and gesture rotation jitter on the research data, the behavioral data with a rotation angular velocity below 5° / s was eliminated, and the Euler angle coordinates of the gesture rotation motion direction recorded every 0.5s during the rotation operation of 20 subjects were used as input data, with a total of 24,831 data points. The direction of the virtual object rotation axis corresponding to each data point (Upward, Downward, Forward, Backward) was used as the output data. To verify the accuracy of the random forest classification model, 70% of the data was used as a training set and 30% as a test set. At the same time, the random forest classification model used a ten-fold cross-validation method to evaluate the robustness of the model, and a detailed evaluation of the detailed classification performance indicators including accuracy, precision, recall and F1 score was performed. The evaluation results are shown in Table 1. Finally, the prediction accuracy of the training model was 0.96, which shows that the random forest classification model has high accuracy in predicting the user's gesture rotation direction.

[0050] Table 1 Classification model performance evaluation

[0051]

[0052]

[0053] In the display-control mapping model, the rotation angular velocity is corrected through a dynamic non-isomorphic mapping mechanism. In the hand rotation timing characteristics, the rotation movement of the hand is divided into four stages: opening, ballistic, adjustment, and ending. In the opening and ending stages of the rotation operation, the hand movement speed is low and the movement jitter is large, which is the start and end of the rotation operation. The ballistic and adjustment stages are rotation operations on the object, and it is necessary to map the rotation movement of the hand and the rotation movement of the manipulated virtual object in terms of speed. The hand movement speed is fast in the ballistic stage, and the movement speed is slow and the time is long in the adjustment stage, which is a fine adjustment of the rotation of the virtual object. Therefore, the dynamic non-isomorphic rotation mapping mechanism based on PRISM proposed by predecessors can be used to establish a graded function of the user's gesture rotation angular velocity to improve the accuracy of the rotation operation.

[0054] like Figure 2 As shown, the magnification factor K is the linear ratio between the virtual object and the hand movement. Referring to the results of the gesture movement behavior data analysis, the minimum speed (MinS) and the proportional constant (SC) for judging the start of the movement are determined. The minimum speed MinS can eliminate the hand movement data in the start and end stages. In the rotation angle mapping mechanism proposed by the research, as the user's operation speed increases, especially when the speed is between the minimum speed MinS and a specific threshold speed SC, the CD ratio is linearly adjusted to gradually increase, as shown in formula (1).

[0055]

[0056] Wherein, CD represents the control-display ratio, S represents the angular velocity of the gesture rotation, MinS represents the minimum velocity for judging the start of the motion, and SC represents the proportional constant. Both MinS and SC need to be set according to the specific scenario. In this embodiment, according to the user's behavioral characteristics, MinS can be set to 4.5° / s to 5° / s, and SC is set to 90° / s.

[0057] From the above description, it can be seen that the display method for mid-air gesture rotation operation in a virtual reality environment described in the present application can effectively realize the VR display of mid-air gesture rotation, support the use of bare hands to complete the rotation control of objects outside the reachable domain in VR, and the interaction is natural and smooth and has behavioral explainability.

[0058] The above are exemplary embodiments of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A display method for mid-air gesture rotation operation in a virtual reality environment, the display method is suitable for rotation tasks controlled by both hands, characterized in that: include: Collecting gesture position data; The normal vector and angle of the plane formed by the line vectors connecting the two hands at a fixed time are calculated according to the gesture position data, and the normal vector and angle are converted into the Euler angle coordinates of the gesture rotation direction and the gesture rotation angular velocity; The Euler angle coordinates of the gesture rotation direction and the gesture rotation angular velocity are input into the display control mapping model to obtain the rotation direction and rotation velocity of the virtual object; The rotation of the virtual object is displayed on the VR device according to the rotation direction and rotation speed of the virtual object.

2. The display method according to claim 1, characterized in that: The display-control mapping model includes a random forest classification model and a dynamic non-isomorphic mapping mechanism. The random forest classification model outputs the rotation direction of the virtual object according to the Euler angle coordinates of the input gesture rotation direction; the dynamic non-isomorphic mapping mechanism obtains the rotation speed of the virtual object according to the input gesture rotation angular velocity.

3. The display method according to claim 2, characterized in that: The random forest classification model converts the Euler angle coordinates of the gesture rotation direction into four orthogonal rotation directions, and the four orthogonal rotation directions include up, down, front, and back directions.

4. The display method according to claim 2, characterized in that: The dynamic non-isomorphic mapping mechanism is used to correct the angular velocity of gesture rotation. It divides the gesture rotation operation into a start phase, a ballistic phase, an adjustment phase and an end phase. In the ballistic phase and the adjustment phase of the gesture rotation operation, the rotational motion of the hand and the rotational motion of the manipulated virtual object are mapped in terms of speed.

5. The display method according to claim 4, characterized in that: The dynamic non-isomorphic mapping mechanism is expressed as: Among them, K represents the magnification factor, CD represents the control-display ratio, S represents the gesture rotation angular velocity, MinS represents the minimum speed for judging the start of movement, and SC represents the proportional constant.

6. The display method according to claim 5, characterized in that: MinS is 4.5° / s~5° / s, and SC is 90° / s.

Images