Virtual reality object manipulation method based on adaptive manipulation gain of visual feedback

Abstract

The present invention proposes a virtual reality object manipulation method based on visual feedback and adaptive manipulation gain based on visual feedback, based on the visual metric of the user during the object manipulation process in virtual reality. When the user manipulates the object in the virtual reality environment, the method can improve the efficiency and accuracy of object manipulation in virtual reality applications. First, the concept of manipulation gain is introduced, and then the maximum and minimum manipulation gains that the user can tolerate in virtual reality are obtained. Finally, by sampling the most suitable manipulation gain of the user in virtual reality and the corresponding object-target visual metric, the displacement, rotation and scaling manipulation gain functions are established. The method can achieve real-time manipulation efficiency and significantly improve the efficiency and accuracy of object manipulation in virtual reality applications.

Description

Technical Field

[0001] The present invention belongs to the field of virtual reality technology, and in particular relates to a virtual reality object manipulation method based on adaptive manipulation gain of visual feedback. Background Art

[0002] Object manipulation (translation, rotation, and scaling) is one of the most basic operations in 3D user interaction and can be used in many virtual reality (VR) applications, such as product design, 3D object modeling, and virtual object assembly. The efficiency and accuracy of the manipulation directly affect the effectiveness of the application. Existing methods mainly adopt two ideas to improve the accuracy and efficiency of object manipulation.

[0003] The first type of method is to manipulate objects through multiple operating points. These multi-point based manipulation methods require users to manually input multiple points on the surface of the object or in 3D space, and then constrain the manipulation of the object based on these points or the axes formed by these points, which improves the accuracy of object manipulation. They are more suitable for direct manipulation of virtual objects by hand at close range, but are difficult to manipulate when the manipulated object is far away from the user.

[0004] The second method is to use the mapping between the physical movement of the user's hand and the virtual object to improve accuracy and efficiency. These methods calculate the control / display ratio according to the speed of the user's hand movement and manipulate objects in the virtual environment (VE) by adjusting the user's hand movement using this ratio. However, due to the inaccurate perception of the user's hand movement speed, the efficiency and accuracy of the operation are affected. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the insufficient virtual object manipulation accuracy of the prior art, and to provide a virtual reality object manipulation method with adaptive manipulation gain based on visual feedback, which can achieve real-time manipulation efficiency and significantly improve the efficiency and accuracy of object manipulation in virtual reality applications.

[0006] The technical solution adopted by the present invention is: a virtual reality object manipulation method based on adaptive manipulation gain of visual feedback, comprising the following steps:

[0007] The first step is to collect the maximum and minimum manipulation gains that the user can accept in virtual reality manipulation according to the dichotomy method, and average the maximum and minimum manipulation gains collected by the dichotomy method to obtain the maximum and minimum manipulation gains that the user can accept in virtual reality object manipulation; the manipulation gains include translation manipulation gain, rotation manipulation gain and scaling manipulation gain;

[0008] The second step is to randomly place objects in the virtual environment, calculate the object-target visual feedback metric of the user under the viewpoint and record the most appropriate manipulation gain of the user's manipulation object selection under the sampled viewpoint;

[0009] The third step is to fit the object-target visual feedback measurement collected in the second step and the manipulation gain selected by the user for manipulating the object under the sampling viewpoint to obtain a manipulation gain fitting function, and obtain a manipulation gain function based on the maximum and minimum manipulation gains acceptable to the user's object manipulation obtained in the first step.

[0010] The first step is specifically implemented as follows:

[0011] (1) In the virtual reality scene, the ranges of translation control gain, rotation control gain and zoom control gain are set to [1,300], [1,50] and [1,30] respectively. The maximum translation control gain, rotation control gain and zoom control gain that the user can accept during virtual reality operation are collected through the dichotomy method;

[0012] (2) In the virtual reality scene, the translation control gain, rotation control gain, and zoom control gain are set to a range of [0, 1], and the maximum translation control gain, rotation control gain, and zoom control gain that the user can accept during virtual reality operation are collected through dichotomy;

[0013] (3) Calculate the average of the maximum and minimum control gain data that the user can accept during virtual reality operation collected in step (1) and step (2) to obtain the maximum and minimum translation control gain, rotation control gain and zoom control gain that the user can accept in virtual reality.

[0014] The second step is specifically implemented as follows:

[0015] (1) Construct the left camera and the right camera related to the user's viewpoint, obtain the OBB bounding box of the target and the object under each camera, calculate the distance d between the center points of the OBB bounding boxes of the target and the object under each camera, and the angle difference θ between the long axes of the OBB bounding boxes of the target and the object, and calculate the ratio p of the projection area of ​​the object manipulated by the user's viewpoint and the projection area of ​​the target;

[0016] (2) Multiple users adjust translation manipulation gain, rotation manipulation gain and scaling manipulation gain through sliders in a virtual reality environment until the manipulation gain is appropriate for the user, and then use step (1) to calculate the corresponding object-target visual feedback metric (d, θ, p), and save the manipulation gain corresponding to the last slider.

[0017] The third step is specifically implemented as follows:

[0018] The maximum and minimum values ​​are limited to the maximum and minimum control gains that the user can accept, and the control gain function is obtained as shown in the following formula:

[0019]

[0020]

[0021]

[0022] Formula (1) is the translation control gain g t , Formula 2 is the rotation control gain g r , Formula 3 is the scaling and translation gain g s ;

[0023] d is the distance between the center points of the OBB bounding boxes of the target and the object under each camera, θ is the angle difference between the long axes of the OBB bounding boxes of the target and the object, and p is the ratio of the projection area of ​​the object manipulated by the user's viewpoint to the projection area of ​​the target.

[0024] The manipulation gain of the prior art is to calculate the control / display ratio by the speed of hand movement, and use the ratio to adjust the user's hand movement to manipulate the object in the virtual environment (VE), such as the PRISM method and the nonlinear mapping method. However, in this type of method, during the operation, the user not only needs to pay attention to the position and posture of the observed object and target in the view, but also needs to pay attention to the speed of hand movement, so the user's task load is very high, which reduces the efficiency and accuracy of the operation. The advantage of the present invention compared with the prior art is that the present invention proposes a virtual reality object manipulation method with adaptive manipulation gain of visual feedback, and the present invention measures the target-object visual measurement of the user's view in each frame, and adaptively adjusts the object manipulation gain of each frame through the manipulation gain measurement function, so as to achieve the most suitable manipulation gain for each frame. The user only needs to pay attention to the view. Compared with the existing method, the present invention does not need the position and posture of the object and target observed by the user in the view to control the manipulation speed of the hand. The present invention measures the time to complete the task, position error, rotation error and measurement error through user research. Compared with the previous method, the method of the present invention improves the efficiency and accuracy of user manipulation. Four methods are compared in three scenarios: CC1 is a traditional method, in which the manipulation gain is 1; CC2 is a PRISM method; CC3 is a nonlinear mapping method based on hand speed and acceleration; CC4 is a dynamic gain method based on hand speed; EC is the method of the present invention. Table 1 shows the time to complete the task. The user can manipulate the object to the target faster using the method of the present invention. Tables 2, 3, and 4 are position error, rotation error, and measurement error, respectively. The user can manipulate the object more accurately using the method of the present invention. The first column S1, S2, and S3 represent three scenarios; the second column EC, CC1, CC2, CC3, and CC4 represent the corresponding methods respectively; the third column represents the mean and standard deviation; the fourth column represents the relative increase value, the fifth column represents the significance, and the sixth column is cohen'd. The larger the value, the better the method of the present invention. The seventh column represents the effect of the method, which has six levels: very small, small, Medium, large, very larger, and huge. Very small means no effect, and huge means very good effect.

[0025] Table 1: Task completion time, in seconds

[0026]

[0027]

[0028] Table 2: Position error in mm

[0029]

[0030] Table 3: Rotational errors in degrees

[0031]

[0032]

[0033] Table 4: Scaling error

[0034]

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a component operation view during the assembly of an aircraft engine of the present invention; in a and b, the user is performing translation manipulation of the object. In c, the user is performing zoom manipulation of the object;

[0037] Figure 2 A flow chart for realizing the method of the present invention;

[0038] Figure 3 This is a conceptual diagram of the manipulation gain introduced in the present invention;

[0039] Figure 4 The maximum and minimum manipulation gains that can be tolerated by the user collected for the present invention; (a) and (d) are the maximum and minimum translation gains; (b) and (e) are the maximum and minimum rotation gains; (c) and (f) are the maximum and minimum proportional gains;

[0040] Figure 5 A view for obtaining the most suitable control gain in the present invention;

[0041] Figure 6 It is the fitting diagram of the manipulation gain function in the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with other drawings and specific embodiments.

[0043] like Figure 1As shown, 1 represents the target, 4 represents the original position of the component to be operated, 3 represents the operation result with a gain of 1, and 2 represents the result of the method of the present invention. In a and b, the user translates the object to the target. In a, the original component is far away from the target, and the gain calculated using the method of the present invention is greater than 1, so the translation speed is faster than the operation using a gain of 1. In b, the original translation is already very close to the target, and the adaptive gain is less than 1, so finer adjustments can be made to avoid the problem of over-manipulation caused by a large gain. In c, the user scales the object to overlap with the target. There is still a small distance between the original component and the target, and the gain calculated by the method of the present invention is greater than 1, which can help the user quickly scale the component to a size similar to the target. When the two are closer, the gain of the method of the present invention becomes smaller, thereby helping the user to fine-tune the scaling.

[0044] like Figure 2 As shown, the method of the present invention is specifically implemented as follows:

[0045] 1. Obtain the maximum and minimum control gains

[0046] It includes 2 tasks: the maximum gain task TMAX and the minimum gain task TMIN. Each manipulation (translation, rotation and scaling) requires a separate TMAX and TMIN for translation manipulation. The target is fixed in the scene. Each time the virtual object is randomly translated from the target, the distance from TMAX to the target is in the range of (1,6), and TMIN is in the range of (0.1,1.5). For rotation manipulation, the target is also fixed in the scene, and the virtual object is randomly rotated from the target. The angle to the target TMAX is in the range of (25,180), and TMIN is in the range of (0.1,40). For scaling manipulation, the virtual object is randomly scaled from the target, and the scale to the target TMAX is in the range of (0.4,5), and TMIN is in the range of (0.6,1.6). In TMAX, the translation gain is set in the range of 1 to 300, the rotation gain is set in the range of 1 to 50, and the proportional gain is set in the range of 1 to 30. The initial translation gain, rotation gain and proportional gain are set to 150.5, 25.5, and 15.5, respectively. In TMIN, the range of the manipulation gain is set between 0 and 1. The initial gain is set to 0.5. The maximum and minimum values ​​collected are averaged as the maximum and minimum manipulation gains. Figure 4 As shown, (a) and (d) are the maximum and minimum translational gains. (b) and (e) are the maximum and minimum rotational gains. (c) and (f) are the maximum and minimum proportional gains. The straight lines represent the average of the maximum or minimum steering gains. Figure 4 Obtaining acceptable maximum and minimum manipulation gains is a feature of the present invention that ensures normal use by users.

[0047] 2. Obtain the most appropriate control gain

[0048] The user's task is to require participants to adjust the most appropriate manipulation gain and manipulate the objects in the virtual scene to their corresponding targets. The concept of manipulation gain is introduced. Figure 3 As shown, the present invention defines manipulation gain as some adjustment factors for mapping hand control motion to the motion of the manipulated virtual object. Inspired by the translation, rotation and curvature gains in redirected walking, the manipulation gain is also divided into three parts: translation gain, rotation gain and scaling gain. These three gains do not affect each other during the manipulation process. An experiment is designed to collect user behavior data during the manipulation process and determine the fitting function for adaptively calculating the manipulation gain during the object manipulation process. The method constructs two auxiliary cameras related to viewpoints. The three cameras can automatically detect the relationship between the object and the target, and the fitting function for adaptively calculating the manipulation gain during the object manipulation process is established through this relationship. First, the projection area of ​​the user's viewpoint manipulated object and target and the ratio p of the projection area are calculated. Then, two auxiliary viewpoints are constructed according to the current viewpoint and the position of the manipulated object: Vl is on the left side of the object and Vt is directly above the object. The distances from these two auxiliary viewpoints to the manipulated object are the same as the distance from the current viewpoint to the manipulated object. After that, the object and target are projected from V, Vl and Vt respectively, and an oriented bounding box (OBB) is constructed for them, and the center and major axis of the OBB are further obtained. Calculate the distance and angle difference between the object and the target for each viewpoint. Obtain at the final distance d and the final angle difference θ between the object and the target, and return the projection area ratio p. The user needs to complete the task in any of the 3 scenes. After placing the object in the scene, the participant can manipulate the object using the initial gain. If the participant thinks the current gain is not appropriate, he / she can press the reset button to restore the object to the starting state (starting position, rotation, and scale), increase or decrease the gain using the up and down buttons on the handle, and then manipulate the object to the target. Participants can see the adjustment of their manipulation gain through the slider in the virtual scene, such as Figure 5As shown, Translation Gain corresponds to translation manipulation gain, Rotation Gain is rotation manipulation gain, and Scale Gain is scaling manipulation gain. This process is repeated until the participant believes that the current gain is appropriate. The participant presses the reset button and then the save button to save the current gain and the object target visual metrics (d, θ, p) corresponding to these gains. Then, the object to be randomly manipulated is automatically reset to shorten the distance between the object and the target. The user then repeats the above process to select an appropriate gain, and the virtual reality object manipulation system records the gain and the corresponding (d, θ, p). The virtual reality object manipulation system resets the object to be manipulated four times. For each user, the virtual reality object manipulation system records five sets of data. The main innovation of the present invention is: a method for adaptive manipulation gain of virtual reality object manipulation is implemented, replacing the traditional method based on hand speed.

[0049] 3. Construction of Manipulation Gain Function

[0050] The collected translation manipulation gains and the corresponding object-target visual metrics are fitted, as Figure 6 As shown, it is the best fitting function diagram of the manipulation gain, the left figure is translation, the middle figure is rotation, and the right figure is scaling. The collected translation manipulation gain and the corresponding object-target visual metric distance d are fitted to construct a translation manipulation gain function as shown in formula (1); the collected rotation manipulation gain and the corresponding object-target visual metric angle difference θ are fitted to construct a rotation manipulation gain function as shown in formula (2). The collected scaling manipulation gain and the corresponding object-target visual metric projection ratio p are fitted to construct a scaling manipulation gain function as shown in formula (3).

[0051]

[0052]

[0053]

[0054] The hardware configuration used in the experiment is Intel i7-9700K processor, Nvidia Geforce 2080Ti graphics card, 16G memory. It runs on Windows 10 64-bit system. All programs involved in the efficiency test are 64-bit programs compiled with O2 level optimization.

[0055] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0056] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A virtual reality object manipulation method based on adaptive manipulation gain of visual feedback, characterized in that The following steps are involved: In the first step, the maximum and minimum manipulation gains acceptable to the user in virtual reality manipulation are collected according to the dichotomy method, and the maximum and minimum manipulation gains collected by the dichotomy method are averaged to obtain the maximum and minimum manipulation gains acceptable to the user in virtual reality object manipulation; the manipulation gains include translation manipulation gain, rotation manipulation gain and scaling manipulation gain; In the second step, objects are randomly placed in the virtual environment, the object-target visual feedback metric under the user's viewpoint is calculated, and the most appropriate manipulation gain selected by the user for manipulating the object under the user's viewpoint is recorded; The third step is to fit the object-target visual feedback metric collected in the second step and the manipulation gain of the user's manipulation object selection under the user's viewpoint to obtain a manipulation gain fitting function, and obtain a manipulation gain function based on the maximum and minimum manipulation gains acceptable to the user's object manipulation obtained in the first step; The second step is specifically implemented as follows: (1) Construct the left camera and the right camera related to the user's viewpoint, obtain the OBB bounding box of the target and object under each camera, calculate the distance d between the center points of the OBB bounding boxes of the target and object under each camera, and the angle difference θ between the long axes of the OBB bounding boxes of the target and object. At the same time, calculate the ratio p of the projection area of ​​the object manipulated by the user's viewpoint and the projection area of ​​the target. (2) Multiple users adjust translation manipulation gain, rotation manipulation gain and scaling manipulation gain through sliders in a virtual reality environment until the manipulation gain is appropriate for the user, and then use step (1) to calculate the corresponding object-target visual feedback metric (d, θ, p), and save the manipulation gain corresponding to the last slider.

2. The method for manipulating a virtual reality object with adaptive manipulation gain based on visual feedback according to claim 1, characterized in that: The first step is specifically implemented as follows: (1) In the virtual reality scene, the ranges of translation control gain, rotation control gain and zoom control gain are set to [1,300], [1,50] and [1,30] respectively. The maximum translation control gain, rotation control gain and zoom control gain that the user can accept during virtual reality operation are collected through the dichotomy method. (2) In the virtual reality scene, the translation control gain, rotation control gain, and zoom control gain are set to the range of [0, 1], and the maximum translation control gain, rotation control gain, and zoom control gain that the user can accept during virtual reality operation are collected through dichotomy; (3) The maximum and minimum manipulation gain data that the user can accept when manipulating virtual reality collected in step (1) and step (2) are averaged to obtain the maximum and minimum translation manipulation gain, rotation manipulation gain, and zoom manipulation gain that the user can accept in virtual reality.

3. The method for manipulating a virtual reality object with adaptive manipulation gain based on visual feedback according to claim 1, characterized in that: The third step is specifically implemented as follows: The maximum and minimum values ​​are limited to the maximum and minimum control gains that the user can accept, and the control gain function is obtained as shown in the following formula: （1） （2） （3） Formula (1) is the translation control gain , Formula 2 is the rotation control gain , Formula 3 is the scaling and translation gain ; d is the distance between the center points of the OBB bounding boxes of the target and the object under each camera, θ is the angle difference between the long axes of the OBB bounding boxes of the target and the object, and p is the ratio of the projection area of ​​the object manipulated by the user's viewpoint to the projection area of ​​the target.

Images