Virtual reality object manipulation method and system based on variable virtual interaction area

Abstract

The present invention relates to a virtual reality object manipulation method and system based on a variable virtual interaction area, and the method comprises: S1: constructing a hand interaction hemisphere area H, which is an effective area for a user to control a handle controller T; S2: constructing a variable virtual interaction area V of a virtual hand O: the variable virtual interaction area is an active area of ​​a user's virtual hand, and the user performs remote operations on a virtual object through the virtual hand; S3: constructing an interaction heat body: the interaction heat body is a three-dimensional voxel array, which is used to record the interaction information of the user in a virtual space in real time; S4: determining whether the operation stage needs to be converted according to the interaction heat body, and updating the parameters of the variable virtual interaction area; S5: obtaining the relative position of the handle controller T in H, and then mapping the relative position to V, and calculating the world position of the virtual hand O.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a virtual reality object manipulation method and system based on a variable virtual interaction area. Background Art

[0002] Virtual object manipulation is one of the basic interactions in virtual reality applications. Many previous studies have been devoted to designing efficient and accurate object manipulation methods.

[0003] Techniques that focus on operation efficiency usually adjust the control-to-display ratio based on hand motion parameters, allowing users to achieve large object movements with small hand movements. Some researchers use ray casting to select objects. Once the object is selected, a virtual hand is attached to the object, and a mapping is established based on the distance between the user's body to the real hand and the user's body to the virtual object. Other researchers have proposed a nonlinear mapping method based on this, dynamically determining the mapping ratio based on the hand's speed and acceleration, and not setting an upper limit on the mapping ratio, thereby further improving the operation speed. However, it is inaccurate to estimate the user's operation intention based solely on hand motion parameters. In addition, the dynamically changing mapping ratio will affect the stability of the operation, thereby reducing the accuracy and comfort of the operation.

[0004] Technologies that emphasize operation accuracy, such as multi-point interaction, allow users to specify operation points to constrain the operation axis, thereby improving accuracy. However, these technologies are often inefficient and unsuitable for the operation of distant objects. Some researchers have also tried to improve the accuracy of long-distance operations by reducing the mapping ratio according to hand speed and reducing the sensitivity of virtual objects to hand movements.

[0005] Technologies that prioritize operational comfort mainly include extending the user's arms and using virtual agents: although extending the user's arms is in line with the user's intuition, it will cause fatigue during long-term operation; while virtual agents mainly include object agents and spatial agents, which require a lot of time to select agents and obviously destroy the user's immersion.

[0006] In terms of the interactive area, there have been studies on arm fatigue and related manipulation techniques. In terms of arm fatigue, some researchers proposed a posture scoring system, pointing out that the most relaxed posture is to relax the upper arm and slightly bend the elbow. Other researchers use electromyography for evaluation and recommend short- to medium-distance movements in the accessible space. There are fewer manipulation techniques related to the interactive area. Some researchers reduce arm fatigue by repositioning the virtual hand, thereby solving the fatigue problem caused by large-scale limb movements. However, these technologies are only applicable to close-range object manipulation. Therefore, how to perform efficient, high-precision and low-fatigue long-distance virtual object manipulation has become an urgent problem to be solved. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a virtual reality object manipulation method and system based on a variable virtual interaction area.

[0008] The technical solution of the present invention is: a virtual reality object manipulation method based on a variable virtual interaction area, comprising:

[0009] Step S1: constructing a hand interaction hemispherical area H: the hand interaction hemispherical area is a ball table, and the single-hand comfortable interaction area and the single-hand maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated according to the single-hand comfortable interaction area and the single-hand maximum interaction area; the hand interaction hemispherical area is the effective area for the user to control the handle controller T;

[0010] Step S2: construct a variable virtual interaction area V of the virtual hand O: the variable virtual interaction area is a frustum, the size of which changes with the user's operation; first, determine the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area according to the user's field of view parameters, then create a bounding box in a virtual scene, the bounding box contains possible interaction areas in the virtual scene, and then emit a ray from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user performs remote operations on virtual objects through the virtual hand;

[0011] Step S3: constructing an interactive heat volume: the interactive heat volume is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxel are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range;

[0012] Step S4: Determine whether the operation stage needs to be converted according to the interaction heat body, and update the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained and enhanced fine-grained: In the coarse-grained stage, determine whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, enter the fine-grained stage, and the interaction heat body is updated by the user's hand interaction. When the interaction is stable, switch from fine-grained to enhanced fine-grained, and update the anchor point and depth of the variable virtual interaction area again;

[0013] Step S5: Obtain the relative position of the handle controller T in H, then map the relative position to V, and calculate the world position of the virtual hand O.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. The present invention discloses a virtual reality object manipulation method based on a variable virtual interaction area, which introduces and constructs a hand interaction hemisphere area in real space. Compared with the existing method, the user can always maintain a comfortable operating posture during the entire operation process, while the traditional method requires the user to stretch his arms unnaturally or requires the user to frequently move the handle to operate the object, causing the user to quickly feel tired during the operation.

[0016] 2. The present invention introduces and constructs a variable virtual interaction area in the virtual space. Compared with the existing methods, the operation accuracy can be automatically changed according to the needs of users at different operation stages. In the coarse-grained stage, the user can move the virtual object to the vicinity of the target position with a continuous handle movement; in the fine-grained stage, the variable virtual interaction area is reduced, so that the movement range of the virtual object is greatly reduced. This solves the problem that the traditional method cannot take into account both efficiency and accuracy in the operation of long-distance objects, and greatly reduces the total movement distance of the handle during the operation, reducing the burden on users.

[0017] 3. The interactive heat body proposed in the present invention obtains the input of multiple user interaction modes in real time, and reflects the user's interaction status in real time, thereby supporting the algorithm to analyze the user's operation status. This makes the update of the variable virtual interaction area consistent with the user's operation intention, making the operation process natural and simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a virtual reality object manipulation method based on a variable virtual interaction area in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the hand interaction hemisphere area in an embodiment of the present invention;

[0020] Figure 3 Schematic diagram of a variable virtual interaction area of ​​a virtual hand in an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of the coarse-grained stage, fine-grained stage and enhanced fine-grained stage in the operation process of an embodiment of the present invention;

[0022] Figure 5 The structure block diagram of a virtual reality object manipulation system based on a variable virtual interaction area in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention provides a virtual reality object manipulation method based on a variable virtual interaction area, which realizes high-efficiency, high-precision and low-fatigue operation on a virtual object.

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0025] In order to better understand the embodiments of the present invention, the basic principles on which the present invention is based are first described:

[0026] (1) When the user moves the handle controller in real space, the most comfortable posture is for the upper arm to hang naturally close to the waist and the forearm to be naturally bent. Collecting the range of motion of this posture and constructing the hemispherical area of ​​the hand interaction area can enable the user to maintain comfort and low fatigue during operation.

[0027] (2) The size of the virtual interaction area required for the user to operate the virtual hand is constantly changing. Initially, it is the entire visible scene space under the user's viewport. As the operation becomes more refined, the virtual interaction area required to operate the virtual object continues to shrink. Adjusting the virtual interaction area according to the user's operation intention can support users to operate quickly in the coarse-grained stage and support users to make fine adjustments in the fine-grained stage.

[0028] (3) In order to accurately and appropriately update the variable virtual interaction area, an interaction heat body is constructed to record the user's interaction input, including eye movement interaction and manual interaction. The user's operation intention is analyzed based on the interaction heat body to determine the update conditions, and the appropriate parameters of the variable virtual interaction area at different stages are calculated.

[0029] (4) By calculating the relative position of the handle controller in the hand interaction hemisphere area, the handle controller can be mapped to another space with known parameters to achieve the association between the two spaces.

[0030] Embodiment 1

[0031] like Figure 1 As shown, a virtual reality object manipulation method based on a variable virtual interaction area provided by an embodiment of the present invention includes the following steps:

[0032] Step S1: constructing a hand interaction hemispherical area H: the hand interaction hemispherical area is a ball table. The single-handed comfortable interaction area and the single-handed maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated according to the single-handed comfortable interaction area and the single-handed maximum interaction area; the hand interaction hemispherical area is the effective area for the user to control the handle controller T;

[0033] Step S2: construct a variable virtual interaction area V of the virtual hand O: the variable virtual interaction area is a frustum, the size of which changes with the user's operation; first, the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area are determined according to the user's field of view parameters, and then a bounding box in a virtual scene is created, the bounding box contains possible interaction areas in the virtual scene, and then a ray is emitted from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user uses the virtual hand to perform remote operations on virtual objects;

[0034] Step S3: Constructing an interactive heat volume: The interactive heat volume is a three-dimensional voxel array, which is used to record the user's interactive information in the virtual space in real time. First, the parameters of the voxel are determined, including the edge length, heat value, state and life span. All parameters except the edge length will be updated during the operation. The virtual space is discretized into a tightly arranged voxel array with the bounding box as the range.

[0035] Step S4: Determine whether the operation stage needs to be converted according to the interaction heat body, and update the parameters of the variable virtual interaction area. The operation stage is divided into three stages: coarse-grained, fine-grained, and enhanced fine-grained. In the coarse-grained stage, determine whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, enter the fine-grained stage. The interaction heat body is updated by the user's hand interaction. When the interaction is stable, switch from fine-grained to enhanced fine-grained, and update the anchor point and depth of the variable virtual interaction area again.

[0036] Step S5: Get the relative position of the handle controller T in H, then map the relative position to V, and calculate the world position of the virtual hand O.

[0037] In one embodiment, the above step S1: constructing the hand interaction hemispherical area H: the hand interaction hemispherical area is a ball table, and the single-handed comfortable interaction area and the single-handed maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated according to the single-handed comfortable interaction area and the single-handed maximum interaction area; the hand interaction hemispherical area is the effective area for the user to control the handle controller T, specifically including:

[0038] Step S11: define a one-handed comfortable interaction area HC (o', r', dir'), where HC is a hemisphere, o' is the center of HC, which is determined by the user's elbow position, r' is the radius of HC, and dir' is the direction of HC, that is, the direction of the user's body;

[0039] Step S12: define the maximum single-hand interaction area HS (o", r"), where HS is a hemisphere, o" is the center of HS, which is determined by the user's shoulder position, and r" is the radius of HS;

[0040] Step S13: Determine the hand interaction hemisphere area H (a1, o, r, d1) through HS and HS, where H is a flat truncated hemisphere, a1 is the anchor point, that is, the center o' of HC; d1 is the depth of the flat truncated hemisphere H, that is, the radius r' of HC; o represents the center, which is calculated based on a1, d1 and the direction dir' of HC; r represents the radius, which is obtained by constructing a plane perpendicular to dir', passing through the center o of H and intersecting with HS, and the distance from the intersection point to the center o is taken as r.

[0041] The hand interaction hemisphere area in the embodiment of the present invention refers to a comfortable single-hand activity area when the user remains still, that is, the position and direction of the person's body remain unchanged. The present invention defines the hand interaction hemisphere area as a flat truncated hemisphere with four parameters H(a1, o, r, d1), such as Figure 2 , where a1 is the anchor point, o represents the center, r represents the radius, and d1 is the depth of the flat truncated hemisphere. To construct H(a1,o,r,d1), first construct two hemispheres. The first hemisphere HC(o',r',dir') is the one-handed comfortable interaction area ( Figure 2 The second hemisphere HC(o”,r”) is the largest interaction area of ​​one hand ( Figure 2 The light-colored larger semicircle). o' and o" are the centers, r' and r" are the radii of HC and HS. dir' is the direction of HC. o' is determined by the user's elbow position, and o" is determined by the user's shoulder position. These arm parameters are obtained by asking the user to keep the arm still and touching the shoulder, elbow, wrist and fingers with the handle controller in turn. dir' is the direction of the user's body. r' and r" of HC and HS in the embodiment of the present invention are 1.2 times the length of the lower arm and 1.2 times the length of the arm, respectively.

[0042] Afterwards, H(a1,o,r,d1) is determined by HC and HS. a1 is the center o' of HC, d1 is the radius r' of HC, and o is calculated based on a1, d1 and the direction dir' of HC. Then, a plane perpendicular to dir' is constructed, passing through the center o of HC and intersecting with HS, and the distance from the intersection point to the center o is taken as r. The embodiment of the present invention uses an inverse frustum as the hand interaction hemisphere area, rather than the two hemispheres HC and HS mentioned above. The reason is that the camera perspective is a frustum, and a hand interaction hemisphere area of ​​this shape can reduce the distortion of the mapping of the two areas. The embodiment of the present invention also introduces a reset area ( Figure 2The dark sector ring area is established based on two radii r0 and r1 and an angle θ. r0 is the length from the elbow to the wrist, and r1 is the length from the elbow to the finger. θ is a constant, and in the embodiment of the present invention, θ=30°.

[0043] In one embodiment, the above step S2: constructs a variable virtual interaction area V of the virtual hand O: the variable virtual interaction area is a frustum, the size of which changes with the user's operation; first, the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area are determined according to the user's field of view parameters, and then a bounding box in a virtual scene is created, the bounding box contains possible interaction areas in the virtual scene, and then a ray is emitted from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user performs remote operations on virtual objects through the virtual hand, specifically including:

[0044] Define the variable virtual interaction area as a frustum V(a2, α, t, dir, d2) with 5 parameters; where a2 is the anchor point; α is the angle in the vertical direction; t is the ratio between the width and height of the frustum; dir is the direction; d2 is the depth of the frustum: create a bounding box B for the virtual scene, the type of bounding box B is AABB box, use the center line of the variable virtual interaction area to intersect with the bounding box B to obtain the intersection C, then d2 is the distance from the anchor point a2 to C.

[0045] The variable virtual interaction area in the embodiment of the present invention refers to the area in the immersive virtual environment where the user can interact with the virtual object, and the size of the area changes with the user's operation. The embodiment of the present invention defines the variable virtual interaction area as a frustum with five parameters V(a2, α, t, dir, d2), such as Figure 3 As shown in the solid area in the triangle, a2 is the anchor point, α is the angle in the vertical direction, t is the ratio between the width and height of the frustum, dir is the direction, and d2 is the depth of the frustum. The embodiment of the present invention uses the parameters of the frustum of the user's viewport to initialize a2, α, t and dir of the variable virtual interaction area. For the depth d2 of the variable virtual interaction area, the embodiment of the present invention does not use the far plane depth of the frustum of the user's viewport, because the depth is usually much larger than the visible scene, and object operations usually occur within the visible scene. In order to obtain d2, the embodiment of the present invention creates an AABB-type bounding box B for the virtual scene, such as Figure 3 As shown in the dark rectangular frame in FIG. 1 , the intersection point C is obtained by intersecting the center line of the variable virtual interaction area with the bounding box B, and the distance from the anchor point a2 to H is defined as d2. Thereafter, the embodiment of the present invention truncates the constructed frustum with B, and obtains the final variable virtual interaction area V.

[0046] In one embodiment, the above step S3: constructing an interactive heat body: the interactive heat body is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxel are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range, specifically including:

[0047] Step S31: define the interactive heat volume as a three-dimensional voxel array, which is used to dynamically record the interactive input of the user in the virtual space; wherein the voxel is a cube, which can be expressed as vox(s,h,st,f), wherein s represents the side length of vox, h is the heat value, st is the state, and f is the lifespan;

[0048] The embodiment of the present invention sets s to 0.02 meters and initializes h to 0. st is a Boolean value, 1 indicates activity and 0 indicates inactivity, and is initialized to 0. f is the lifespan, and when st is set to 1, the lifespan will be reset to 1 second. During the operation, the parameters of vox are updated. At each frame during the operation, the voxel vox in the virtual space where the user interaction position is located can be determined, the state st is set to 1, the heat value h is increased by 1, and the lifespan f is reset to obtain a lifespan of 1 second. The lifespan of the voxel decreases over time. When the lifespan is 0, the heat value h of the voxel begins to decrease at a rate of 1 per frame. When the heat value h is 0, the voxel returns to an inactive state;

[0049] Step S32: The interactive heat body is updated in the following two user interaction modes according to different application scenarios:

[0050] a. When the interactive heat volume is applied to gaze, for each frame, the gaze point is first obtained, then the voxel to which the gaze point belongs is calculated, and the voxel is updated;

[0051] b. When the interaction heat volume is applied to the hand movement, the point of the virtual hand in the variable virtual interaction area is obtained every frame, and then the voxel where the virtual hand is located is updated in the same way as the gaze pattern.

[0052] In an embodiment of the present invention, the interactive heat body can be updated by two user interaction modes, namely gaze or hand. When the interactive heat body is applied to gaze, for each frame, the gaze point is first obtained, and then the voxel to which the gaze point belongs is calculated, and the voxel is updated. When a user gazes at an object, due to the unstable acquisition of the gaze point, the gaze point often falls on the background behind the object, resulting in depth error. Therefore, the present invention introduces a ghost plane, which is an invisible circular plane centered on the center of the object with a viewing angle of 10°, perpendicular to the viewing direction. When a user looks at an object, the corresponding voxels can be updated more accurately by using a ghost plane to capture gaze points at similar depths around the object. When the interactive heat body is applied to hand movement, the point of the virtual hand in the variable virtual interaction area is obtained for each frame, and then the voxel where the virtual hand is located is updated in the same way as the gaze mode.

[0053] In one embodiment, the above step S4: determines whether the operation stage needs to be converted according to the interaction heat body, and updates the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained and enhanced fine-grained: in the coarse-grained stage, it is determined whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, it enters the fine-grained stage, and the interaction heat body is updated by the user's hand interaction. When the interaction is stable, it switches from fine-grained to enhanced fine-grained, and the anchor point and depth of the variable virtual interaction area are updated again, which specifically includes:

[0054] Step S41: When the operation starts, the user's operation is in the coarse-grained stage, and the interaction heat body is updated according to the user's gaze situation;

[0055] First, condition 1 is judged: the interaction center and interaction dispersion of the current frame of the interaction heat body are calculated; secondly, the distance between the interaction center and the center of the virtual hand is calculated as the interaction deviation distance; when the interaction deviation distance is less than the threshold, condition 1 is met;

[0056] The threshold in the embodiment of the present invention is set to 0.2, that is, when the interaction deviation distance is less than 0.2 meters, a flag is set to indicate that condition 1 is met;

[0057] In this step, the interaction heat body I is input, and the interaction center P and interaction dispersion D of the current frame of the interaction heat body are calculated. The specific steps are:

[0058] First, get the voxels that have been accessed and activated by the user in the past 20 frames and store them in the list VL. At the same time, create an empty list W. Then, traverse each voxel in VL, add the voxel to W, and use the value of vox.h as the number of times it has been added. Next, create a zero vector P s , traverse each vox in W and add the position of vox to P sThen calculate the average position of the voxels in W as the interaction center P of the current frame. After that, create D s Set it to 0, iterate over each vox in W, calculate the square of the distance between the vox position and P, and accumulate it in D s Finally, according to D s Calculate the standard deviation as the interaction dispersion D of the current frame;

[0059] Next, condition 2 is determined: an interaction discreteness sequence is constructed based on the interaction discreteness of the most recent 20 frames, and the Augmented Dickey-Fuller test is used to determine whether the user's interaction has entered a stable state in the current frame. If so, condition 2 is satisfied.

[0060] When both condition 1 and condition 2 are satisfied, the variable virtual interaction area is updated, and the operation phase is changed from the coarse-grained phase to the fine-grained phase, that is, the process proceeds to step S42;

[0061] Among them, updating the variable virtual interaction area mainly refers to updating the parameter anchor point a2 and depth d2. The specific steps are as follows:

[0062] When the operation stage switches from the coarse-grained stage to the fine-grained stage, the variable virtual interaction area V, the hand interaction hemisphere area H, the interaction heat body I, the handle controller T, and the virtual hand O are input. First, the depth is updated, and the depth d2 of V is updated to the depth d1 of H; then the anchor point is updated. In order to avoid a sudden change in the position of the virtual hand when updating the anchor point a2 of V, the position of the controller T in the local coordinate system of H is normalized to obtain the normalized vector R, and then according to the normalized vector R, the offset of the anchor point a2 of V relative to the virtual hand O on the three axes is used as the offset vector F. Finally, the algorithm subtracts the position of O from the offset vector F to obtain the new anchor point position of the variable virtual interaction area;

[0063] Step S42: In the fine-grained stage, the interactive heat body is updated by the user's hand interaction; use condition 2 to determine whether it has entered a stable state. If so, enter the enhanced fine-grained stage, that is, go to step S43;

[0064] Step S43: updating the anchor point and depth of the variable virtual interaction area based on the interaction heat volume.

[0065] In this step, when the operation enters the enhanced fine-grained stage from the fine-grained stage, the variable virtual interaction area V, the hand interaction hemisphere area H, the interaction heat body I, the handle controller T, and the virtual hand O are input. First, the depth is updated to obtain the list VL of voxels that have been visited and are active in the past 20 frames. Then, according to the area where the virtual hand O has been active in the recent period of time, the user's interaction area can be further narrowed. The algorithm uses the least squares method to fit the voxels in VL and calculate the smallest enclosing sphere, and sets the depth d2 of V to the diameter of this sphere. Then, the anchor point is updated, and the anchor point update method is the same as in step S41.

[0066] like Figure 4 As shown, from left to right, the operation schematics of the coarse-grained stage, the fine-grained stage, and the enhanced fine-grained stage are respectively shown.

[0067] After the above steps, the construction of the user's hand interaction hemisphere area and the variable virtual interaction area of ​​the virtual hand is completed. At this time, a mapping algorithm is needed to link the movement of the handle controller with the movement of the virtual hand, so as to support the user to operate the object through the virtual hand. The mapping problem is solved by the following steps.

[0068] In one embodiment, the above step S5: obtaining the relative position of the handle controller T in H, then mapping the relative position to V, and calculating the world position of the virtual hand O, specifically includes:

[0069] Step S51: normalize the position of the handle controller T in the local coordinate system of H to obtain a normalized vector R, and then according to the normalized vector R, take the offset of the anchor point a2 of V relative to the virtual hand O on the three axes as the offset vector F;

[0070] Step S52: Add the offset vector F to the anchor point a2 of the variable virtual interaction area V to obtain the original mapping position P of the virtual hand O f ; Then, detect the anchor points a2 and P of V f Does the line between intersect with the bounding box B? If so, the intersection point P i The world position P of the virtual hand O o , otherwise P f The world position P of the virtual hand o .

[0071] In addition, the present invention also designs a handle reset method to reduce user operation fatigue. When the user places a virtual object near the target location, their hand may have been at the edge of the hand interaction hemisphere area. In this case, the actual available space for hand movement is limited, and the arm may have deviated from a comfortable posture. To solve this problem, the present invention proposes a handle controller reset method. When the operation phase switches, check whether the handle controller needs to be reset. If the handle controller is not in the reset area ( Figure 2 The dark circle area in the figure resets the mapping algorithm by temporarily interrupting it, that is, when the gamepad controller moves, the virtual hand freezes. After the gamepad controller returns to the reset area, the virtual hand is unfrozen.

[0072] Furthermore, the present invention also provides a solution for cross-viewport operations. When a user selects an object, the target location may be outside his viewport. If a variable virtual interaction area is constructed at this time, the object cannot be moved to the target location. A head movement phase is introduced after the object is selected to support cross-viewport operations. The button is not released after the user presses the button to select the object. The variable virtual interaction area will be continuously updated with the user's head movement and will always face forward until the user releases the button when he sees the target location. During this process, the mapping algorithm remains activated, allowing the object to move with the updated variable virtual interaction area. Throughout the process, no additional buttons are added, which conforms to the natural flow of user interaction.

[0073] The hardware configuration used in the embodiment of the present invention is HTC Vive Cosmos, including a head-mounted display device (HMD) with position tracking function and two handheld wireless controllers. In order to capture eye movement data, the aSee VR eye tracker is integrated into the HMD. The average sampling rate of the eye tracker is 250Hz, the accuracy is 0.5°, and the maximum field of view is 110°. The HMD is connected to a desktop computer equipped with an Intel i7 processor, an NVIDIA 3080Ti graphics card, and 16G memory. The computer uses a Windows 10 64-bit system, and the implementation of the present invention is completed on the Unity3D platform.

[0074] The present invention discloses a virtual reality object manipulation method based on a variable virtual interaction area, which introduces and constructs a hand interaction hemispherical area in real space. Compared with the existing method, the user can always maintain a comfortable operation posture during the entire operation process, while the traditional method requires the user to stretch his arms unnaturally or requires the user to frequently move the handle to operate the object, causing the user to quickly feel fatigued during the operation. The present invention introduces and constructs a variable virtual interaction area in a virtual space. Compared with the existing method, the operation accuracy can be automatically changed according to the needs of the user in different operation stages. In the coarse-grained stage, the user can move the virtual object to the vicinity of the target position with a continuous handle movement; in the fine-grained stage, the variable virtual interaction area is reduced, so that the movement amplitude of the virtual object is greatly reduced. The problem that the traditional method cannot take into account both efficiency and accuracy in the operation of long-distance objects is solved, and the total movement distance of the handle during the operation is greatly reduced, reducing the burden on the user. The interactive heat body proposed by the present invention obtains the input of multiple interaction modes of the user in real time, reflects the user's interaction status in real time, and supports the algorithm to analyze the user's operation status, which makes the update of the variable virtual interaction area meet the user's operation intention and makes the operation process natural and simple.

[0075] Embodiment 2

[0076] like Figure 5 As shown, an embodiment of the present invention provides a virtual reality object manipulation system based on a variable virtual interaction area, including the following modules:

[0077] The hand interaction hemispherical area construction module 61 is used to construct the hand interaction hemispherical area H: the hand interaction hemispherical area is a ball table, and the single-handed comfortable interaction area and the single-handed maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated according to the single-handed comfortable interaction area and the single-handed maximum interaction area; the hand interaction hemispherical area is the effective area for the user to control the handle controller T;

[0078] Constructing a variable virtual interaction area module 62, for constructing a variable virtual interaction area V of the virtual hand O: the variable virtual interaction area is a frustum, the size of which changes with the user's operation; firstly, the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area are determined according to the user's field of view volume parameters, and then a bounding box in a virtual scene is created, the bounding box contains possible interaction areas in the virtual scene, and then a ray is emitted from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user performs remote operations on virtual objects through the virtual hand;

[0079] The interactive heat volume construction module 63 is used to construct the interactive heat volume: the interactive heat volume is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxel are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range;

[0080] The variable virtual interaction area updating module 64 is used to determine whether the operation stage needs to be switched according to the interaction heat body, and to update the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained, and enhanced fine-grained. In the coarse-grained stage, it is determined whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, the fine-grained stage is entered, and the interaction heat body is updated by the user's hand interaction. When the interaction is stable, it is switched from fine-grained to enhanced fine-grained, and the anchor point and depth of the variable virtual interaction area are updated again.

[0081] The mapping module 65 is used to obtain the relative position of the handle controller T in H, and then map the relative position to V to calculate the world position of the virtual hand O.

[0082] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A virtual reality object manipulation method based on a variable virtual interaction area, characterized in that: include: Step S1: Construct the hand interaction hemisphere area :The hand interaction hemispherical area is a ball table. The single-handed comfortable interaction area and the single-handed maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated based on the single-handed comfortable interaction area and the single-handed maximum interaction area; the hand interaction hemispherical area is the user's control handle controller The effective area includes: Step S11: Defining the one-handed comfortable interaction area ,in, For a hemisphere, for The center of the body is determined by the position of the user's elbow. for The radius of yes direction, i.e. the direction of the user's body; Step S12: defining the maximum single-handed interaction area ,in, For a hemisphere, for The center of the body is determined by the position of the user's shoulders. for The radius of Step S13: Pass and Determine the hand interaction hemispheric area ,in, is a flat truncated hemisphere, is the anchor point, that is Center ; Flattened truncated hemisphere The depth of Radius ; The representative center is based on , and Directions Calculate and obtain; Represents the radius by constructing a line perpendicular to plane, through Center and with Intersect, move the intersection point to the center The distance as ; Step S2: Constructing the virtual hand Variable virtual interaction area :The variable virtual interaction area is a frustum, the size of which changes with the user's operation; firstly, the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area are determined according to the user's field of view parameters, and then a bounding box in a virtual scene is created, the bounding box contains the possible interaction area in the virtual scene, and then a ray is emitted from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user performs remote operations on virtual objects through the virtual hand, specifically including: Define the variable virtual interaction region as a frustum with 5 parameters ;in, It is the anchor point; is the angle from vertical; is the ratio between the width and height of the frustum; It is the direction; is the depth of the frustum: creating a bounding box for the virtual scene , the bounding box The type uses an AABB box, using the center line and bounding box of the variable virtual interaction area Intersect to get the intersection point ,but From the anchor point arrive distance; Step S3: constructing an interactive heat volume: the interactive heat volume is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxel are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range; Step S4: Determine whether the operation stage needs to be converted according to the interaction heat body, and update the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained and enhanced fine-grained: In the coarse-grained stage, determine whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, enter the fine-grained stage, and the interaction heat body is updated by the user's hand interaction. When the interaction is stable, switch from fine-grained to enhanced fine-grained, and update the anchor point and depth of the variable virtual interaction area again; Step S5: Get the handle controller exist The relative position of The virtual hand is calculated world location.

2. The virtual reality object manipulation method based on a variable virtual interaction area according to claim 1, characterized in that: The step S3: constructing an interactive heat volume: the interactive heat volume is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxels are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range, specifically including: Step S31: define the interactive heat volume as a three-dimensional voxel array, which is used to dynamically record the user's interactive input in the virtual space; wherein the voxel is a cube and can be expressed as ,in, express The side length of is the caloric value, It's the state. It is life span; Step S32: The interactive heat body is updated in the following two user interaction modes according to different application scenarios: a. When the interactive heat volume is applied to gaze, for each frame, the gaze point is first obtained, then the voxel to which the gaze point belongs is calculated, and the voxel is updated; b. When the interactive heat body is applied to the movement of the hand, each frame will get the point of the virtual hand in the variable virtual interactive area, and then the virtual hand The voxel is updated in the same way as gaze mode.

3. The virtual reality object manipulation method based on a variable virtual interaction area according to claim 2, characterized in that: The step S4: determining whether the operation stage needs to be converted according to the interaction heat body, and updating the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained and enhanced fine-grained: in the coarse-grained stage, judging whether the interaction center and interaction dispersion of the interaction heat body meet the conditions, if so, entering the fine-grained stage, the interaction heat body is updated by the user's hand interaction, when the interaction is stable, switching from fine-grained to enhanced fine-grained, and updating the anchor point and depth of the variable virtual interaction area again, specifically including: Step S41: When the operation starts, the user's operation is in the coarse-grained stage, and the interaction heat body is updated according to the user's gaze situation; First, condition 1 is judged: the interaction center and interaction dispersion of the current frame of the interaction heat body are calculated; secondly, the distance between the interaction center and the center of the virtual hand is calculated as the interaction deviation distance; when the interaction deviation distance is less than a threshold, condition 1 is satisfied; Next, condition 2 is determined: an interaction discreteness sequence is constructed based on the interaction discreteness of the most recent 20 frames, and an Augmented Dickey-Fuller test is used to determine whether the user's interaction enters a stable state in the current frame. If so, condition 2 is satisfied. When both the condition 1 and the condition 2 are satisfied, the anchor point and the depth of the variable virtual interaction area are updated, and the operation phase is changed from the coarse-grained phase to the fine-grained phase, that is, the process proceeds to step S42; Step S42: In the fine-grained stage, the interactive heat body is updated by the user's hand interaction; the condition 2 is used to determine whether it has entered a stable state, and if so, it enters the enhanced fine-grained stage, that is, it goes to step S43; Step S43: updating the anchor point and depth of the variable virtual interaction area based on the interaction heat body.

4. The virtual reality object manipulation method based on a variable virtual interaction area according to claim 3 is characterized in that: Step S5: Obtaining the handle controller exist The relative position of The virtual hand is calculated The world location of Step S51: The handle controller exist Normalize the position in the local coordinate system to obtain the normalized vector , and then according to the normalized vector ,Will Anchor point Relative to the virtual hand The offset on the three axes as an offset vector ; Step S52: The offset vector Added to the variable virtual interaction area Anchor point , to obtain the virtual hand The original mapping position of Then, detect Anchor point and Is the line between the two lines consistent with the bounding box? intersect; if so, the intersection As the virtual hand World Location , otherwise As the world position of the virtual hand .

5. A virtual reality object manipulation system based on a variable virtual interaction area, characterized in that: Includes the following modules: Construct the hand interaction hemispherical area module, which is used to construct the hand interaction hemispherical area :The hand interaction hemispherical area is a ball table. The single-handed comfortable interaction area and the single-handed maximum interaction area are determined by the given user's arm data, and then the hand interaction hemispherical area is calculated based on the single-handed comfortable interaction area and the single-handed maximum interaction area; the hand interaction hemispherical area is the user's control handle controller The effective area includes: Step S11: defining the one-handed comfortable interaction area ,in, For a hemisphere, for The center of the body is determined by the user's elbow position. for The radius of yes direction, i.e. the direction of the user's body; Step S12: defining the maximum single-handed interaction area ,in, For a hemisphere, for The center of the body is determined by the position of the user's shoulders. for The radius of Step S13: Pass and Determine the hand interaction hemispheric area ,in, is a flat truncated hemisphere, is the anchor point, that is Center ; Flattened truncated hemisphere The depth of Radius ; The representative center is based on , and Directions Calculate and obtain; Represents the radius by constructing a line perpendicular to plane, through Center and with Intersect, move the intersection point to the center The distance as ; Construct a variable virtual interaction area module for constructing a virtual hand Variable virtual interaction area :The variable virtual interaction area is a frustum, the size of which changes with the user's operation; firstly, the anchor point, near plane distance, field of view range and aspect ratio of the variable virtual interaction area are determined according to the user's field of view parameters, and then a bounding box in a virtual scene is created, the bounding box contains the possible interaction area in the virtual scene, and then a ray is emitted from the center of the variable virtual interaction area to collide with the bounding box, wherein the depth of the variable virtual interaction area is the distance from its center to the collision point; the variable virtual interaction area is the active area of ​​the user's virtual hand, and the user performs remote operations on virtual objects through the virtual hand, specifically including: Define the variable virtual interaction region as a frustum with 5 parameters ;in, It is the anchor point; is the angle from vertical; is the ratio between the width and height of the frustum; It is the direction; is the depth of the frustum: creating a bounding box for the virtual scene , the bounding box The type uses an AABB box, using the center line and bounding box of the variable virtual interaction area Intersect to get the intersection point ,but From the anchor point arrive distance; Constructing an interactive heat volume module, which is used to construct an interactive heat volume: the interactive heat volume is a three-dimensional voxel array, which is used to record the interactive information of the user in the virtual space in real time; firstly, the parameters of the voxel are determined, including the side length, heat value, state and life span, and all parameters except the side length are updated during the operation, and the virtual space is discretized into a closely arranged voxel array with the bounding box as the range; The module for updating the variable virtual interaction area is used to determine whether the operation stage needs to be converted according to the interaction heat body, and to update the parameters of the variable virtual interaction area, wherein the operation stage is divided into three stages: coarse-grained, fine-grained and enhanced fine-grained. In the coarse-grained stage, it is determined whether the interaction center and interaction dispersion of the interaction heat body meet the conditions. If they meet the conditions, the fine-grained stage is entered, and the interaction heat body is updated by the user's hand interaction. When the interaction is stable, it is switched from fine-grained to enhanced fine-grained, and the anchor point and depth of the variable virtual interaction area are updated again. Mapping module, used to obtain the handle controller exist The relative position of The virtual hand is calculated world location.

Images