Virtual reality complex scene interaction method and device based on eye movement tracking and electronic equipment

Abstract

The invention provides a virtual reality complex scene interaction method and device based on eye movement tracking and electronic equipment, and the method comprises the steps: tracking user eyeball movement data, and determining a user focusing region in a virtual reality scene in real time; the method comprises the following steps: projecting all objects in a user focusing area into a circular plane suspended in front of a user body on the basis of keeping a relative position relation under a user visual angle, and intelligently adjusting the positions of the objects projected into the circular plane so that all the objects are separated from each other and are not overlapped with each other; and objects in the circular plane are selected and interacted. The visual line information of the user is utilized, and the user is helped to quickly and accurately select the target object in the complex virtual scene. Compared with a traditional ray interaction mode, the method has the advantages that the selection time can be remarkably shortened, the accuracy of target object selection is improved, and especially under the condition that a large number of overlapped or shielded objects exist in a scene, the defect that a shielded target is difficult to select through a traditional method can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual reality and augmented reality, and in particular to a method, device and electronic device for complex scene interaction in virtual reality based on eye tracking. Background Art

[0002] With the development of the virtual reality (VR) industry, more and more people are beginning to use VR headsets and use them for office work, play, and movie watching in virtual reality scenes. In this process, users often need to interact with and select objects in the virtual scene. The traditional VR interaction method is to allow users to emit rays through the controller and then select objects or icons in the virtual scene.

[0003] Although this ray-based method is widely used in VR, it also has obvious drawbacks. This method requires the user to hold the controller high for a long time to aim at the object or icon to be interacted with. If the object is far away from the user and the object is small, it is difficult for the ray to keep aiming at the object, especially when the human hand has a certain natural jitter problem during natural movement, and when the user presses the controller button with his hand, the controller will also tilt and vibrate to a certain extent, which makes it difficult for the ray-based interaction method to have a good accuracy rate when facing distant objects and small objects.

[0004] In addition, when the target object is blocked by other obstructions in the scene, it is even more difficult for the user to aim the controller ray directly at the target object. The ray will be blocked by the obstruction in front of the target object, and the target object cannot be selected. Although the user can adjust the camera posture direction, the position of the controller and the angle of the emitted ray to make the ray bypass the obstruction and select the target, when the obstructions in the scene are too dense, the user may even find it difficult to find the location of the target object in the first place, and thus cannot select the target by adjusting his posture. Sometimes the target cannot be selected no matter how it is adjusted. Therefore, the ray-based method also has the disadvantages of being difficult to obtain obstructed targets and not being suitable for scenes with dense obstructions. Summary of the invention

[0005] In order to solve the problems faced by traditional VR ray interaction technology in virtual reality scenes, such as insufficient accuracy, long operation time, and difficulty in selecting occluded targets, the present invention proposes and develops a progressive refinement interaction method based on eye tracking, which helps users quickly and accurately select target objects in complex virtual scenes by utilizing the user's line of sight information and combining progressive interaction design. Compared with traditional ray interaction methods, eye tracking technology can significantly shorten the time of target object selection and improve the accuracy of target object selection, especially when there are a large number of overlapping or occluded objects in the scene, which can effectively solve the problem that traditional methods are difficult to select occluded targets.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The present invention provides a virtual reality complex scene interaction method based on eye tracking, comprising:

[0008] Tracking user eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area;

[0009] Project all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0010] Objects in the circular plane are selected and interacted with to achieve complex scene interaction of virtual reality based on eye tracking.

[0011] Preferably, the user eye movement data includes user eye position and user eye direction information; the tracking of the user eye movement data determines the user focus area in the virtual reality scene in real time, and then determines all objects in the user focus area, specifically:

[0012] Track the user's eye movement data through the head camera, and obtain the user's eye position and eye direction information in the local coordinate system of the user's head in real time;

[0013] According to the transformation matrix from the local coordinate system of the user's head to the world coordinate system, the user's eye position and the user's eye direction information in the local coordinate system of the user's head are transformed into the user's eye position and the user's eye direction in the world coordinate system;

[0014] According to the user's eyeball position in the world coordinate system, a transparent cone is generated at the center of the user's eyes, the vertex of the cone is located at the center of the eyes, and the direction of the cone refers to the direction from the vertex to the bottom surface, which is the direction of the user's eyeball; the area where the cone is located is the user's focusing area;

[0015] For all objects in the virtual reality scene, it is determined whether the object is located in the cone. If so, the object is in the user focus area; otherwise, the object is not in the user focus area, and then all objects in the user focus area are determined.

[0016] Preferably, the determination of whether the object is located within the cone is obtained by vector dot product calculation, and the specific formula is:

[0017]

[0018] in:

[0019] dir obj is the vector from the cone vertex to the object;

[0020] dir cone is the cone orientation vector, that is, the orientation vector from the cone vertex to the base;

[0021] dir edge is the cone side vector;

[0022] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing the vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, indicating that the object is inside the cone; otherwise, the function y is 0, indicating that the object is not inside the cone.

[0023] Preferably, it also includes:

[0024] If the object is located within the user focus area, a circular highlight outline ignoring the depth is drawn on the object to highlight the object.

[0025] Preferably, all objects in the user's focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap, specifically:

[0026] Performing coordinate transformation on all objects in the user focus area, transforming the global coordinates of each object in the world coordinate system into the circular plane local coordinate system, and obtaining the local coordinates of each object in the circular plane local coordinate system;

[0027] Scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates;

[0028] Projecting the scaled coordinates to corresponding positions in the circular plane;

[0029] Collision detection is performed on each object projected into the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap.

[0030] Preferably, the following formula is used to scale the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates:

[0031]

[0032] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0033] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0034] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0035] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0036] r sphere Represents the radius of the circular plane.

[0037] Preferably, the collision detection is performed on each object projected into the circular plane so that overlapping objects are mutually repelled and separated until they no longer overlap, specifically as follows:

[0038] For each object projected into the circular plane, traverse in sequence from the center to the periphery; for each traversed object, represented as the currently detected object O, detect whether the object O has an overlapping collision body C. If not, traverse and detect the next object; if so, push the collision body C away from the object O, and determine the position of the updated collision body C by the following formula:

[0039]

[0040] in:

[0041] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0042] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0043] O detectRepresents the position coordinates of object O;

[0044] β is a minimum value, which is a known given value;

[0045] D represents the distance that the collision body C is pushed away;

[0046] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0047] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0048] Preferably, it also includes:

[0049] Configure a device controller; the device controller has an eye tracking start button event, a projection button event, and an interaction button event;

[0050] Monitoring the button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area, until the projection button event is detected, stopping the tracking of the user's eye movement data;

[0051] When the projection button event is detected, all objects in the latest user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap;

[0052] When the interaction button event is detected, it is determined whether there is an interactive object in the interaction area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

[0053] The present invention also provides a device for implementing the virtual reality complex scene interaction method based on eye tracking, comprising:

[0054] A user eye movement data tracking unit, used to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and then determine all objects in the user's focus area;

[0055] A projection unit is used to project all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining the relative position relationship under the user's viewing angle, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0056] The interaction unit is used to select and interact with objects in the circular plane to achieve complex scene interaction of virtual reality based on eye tracking.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for complex virtual reality scene interaction based on eye tracking is implemented.

[0058] The virtual reality complex scene interaction method, device and electronic device based on eye tracking provided by the present invention have the following advantages:

[0059] The virtual reality complex scene interaction method, device and electronic device based on eye tracking provided by the present invention can obtain the center position and orientation of the user's eyes in real time, project occluded objects onto the user's near plane in the virtual reality complex scene and solve the occlusion problem, so that the user can obtain the required objects faster and easier in the virtual reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the present invention and related technical solutions, the following briefly introduces the drawings required in the embodiments or prior art. Obviously, these drawings are only some embodiments of the present invention, and those skilled in the art can deduce other possible drawings based on these drawings without creative work.

[0061] Figure 1 A flowchart of a virtual reality complex scene interaction method based on eye tracking provided by the present invention;

[0062] Figure 2 A scene example diagram of the virtual reality complex scene interaction method based on eye tracking provided by the present invention, which shows the highlighted display of objects in the user's focus area;

[0063] Figure 3 An example image of an object in the focus area provided by the present invention being projected onto the user's near plane;

[0064] Figure 4 An example diagram of detecting whether an object is within a focus area provided by the present invention;

[0065] Figure 5A schematic diagram of the structure of the virtual reality complex scene interaction device based on eye tracking provided by the present invention;

[0066] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0067] In order to more clearly explain the purpose, technical solutions and advantages of the present invention, the technical solutions will be described in detail and completely in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described below are only part of the present invention and do not cover all possible embodiments. Any other embodiments that can be derived by ordinary technicians in the field without creative work on the basis of the embodiments of the present invention belong to the protection scope of the present invention.

[0068] Since it is difficult for existing virtual reality interaction technologies to easily, quickly and accurately acquire occluded targets in virtual reality scenes, there are problems such as difficulty in interaction in dense scenes, and users need to expend a lot of unnecessary physical effort walking in a virtual reality environment to acquire distant target objects. Therefore, the present invention provides a virtual reality complex scene interaction method, device and electronic device based on eye tracking, which are used to solve the above problems existing in the prior art.

[0069] The present invention provides a virtual reality complex scene interaction method based on eye tracking, comprising:

[0070] Step S1, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area;

[0071] Step S2, projecting all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and intelligently adjusting the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0072] Step S3, selecting and interacting with objects in the circular plane to achieve complex scene interaction of virtual reality based on eye tracking.

[0073] The following is a detailed description of steps S1 to S3:

[0074] Step S1, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area;

[0075] In this step, the user's eye movement data includes the user's eye position and the user's eye direction information; this step is specifically as follows:

[0076] Step S11, tracking the user's eye movement data through the head camera, and obtaining the user's eye position and the user's eye direction information in the local coordinate system of the user's head in real time;

[0077] Step S12, converting the user's eye position and the user's eye orientation information in the user's head local coordinate system into the user's eye position and the user's eye orientation in the world coordinate system according to the conversion matrix from the user's head local coordinate system to the world coordinate system;

[0078] Step S13, generating a transparent cone at the center of the user's eyes according to the user's eye position in the world coordinate system, wherein the vertex of the cone is located at the center of the eyes, and the direction of the cone refers to the direction from the vertex to the bottom surface, which is the direction of the user's eyes; the area where the cone is located is the user's focusing area;

[0079] Step S14, for all objects in the virtual reality scene, determine whether the object is located in the cone. If so, the object is in the user focus area; otherwise, the object is not in the user focus area, and then determine all objects in the user focus area.

[0080] As an embodiment, the determination of whether the object is located in the cone is obtained by vector dot product calculation, and the specific formula is:

[0081]

[0082] in:

[0083] dir obj is the vector from the cone vertex to the object;

[0084] dir cone is the cone orientation vector, that is, the orientation vector from the cone vertex to the base;

[0085] dir edge is the cone side vector;

[0086] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing the vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, indicating that the object is inside the cone; otherwise, the function y is 0, indicating that the object is not inside the cone.

[0087] In practical applications, if the object is located in the user focus area, the method further includes: drawing a layer of depth-ignoring circular highlight outline on the object to highlight the object.

[0088] Step S2, projecting all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and intelligently adjusting the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0089] This step specifically includes:

[0090] Step S21, performing coordinate transformation on all objects in the user focus area, transforming the global coordinates of each object in the world coordinate system into the circular plane local coordinate system, and obtaining the local coordinates of each object in the circular plane local coordinate system;

[0091] Step S22, scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates;

[0092] As an embodiment, the following formula is used to scale the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates:

[0093]

[0094] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0095] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0096] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0097] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0098] r sphere Represents the radius of the circular plane.

[0099] Step S23, projecting the scaled coordinates to corresponding positions in the circular plane;

[0100] Step S24, performing collision detection on each object projected into the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap.

[0101] As an embodiment, step S24 specifically includes:

[0102] For each object projected into the circular plane, traverse in sequence from the center to the periphery; for each traversed object, represented as the currently detected object O, detect whether the object O has an overlapping collision body C. If not, traverse and detect the next object; if so, push the collision body C away from the object O, and determine the position of the updated collision body C by the following formula:

[0103]

[0104] in:

[0105] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0106] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0107] O detect Represents the position coordinates of object O;

[0108] β is a minimum value, which is a known given value;

[0109] D represents the distance that the collision body C is pushed away;

[0110] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0111] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0112] Step S3, selecting and interacting with objects in the circular plane to achieve complex scene interaction of virtual reality based on eye tracking.

[0113] In practical applications, when performing steps S1 to S3, the following steps are also included:

[0114] Configure a device controller; the device controller has an eye tracking start button event, a projection button event, and an interaction button event;

[0115] Monitoring the button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area, until the projection button event is detected, stopping the tracking of the user's eye movement data;

[0116] When the projection button event is detected, all objects in the latest user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap;

[0117] When the interaction button event is detected, it is determined whether there is an interactive object in the interaction area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

[0118] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for complex virtual reality scene interaction based on eye tracking is implemented.

[0119] The embodiment of the present invention provides a virtual reality complex scene interaction method based on eye tracking, and one implementation method is:

[0120] Input the user's eye position and eye direction information, and output the virtual reality scene area where the user is currently focused, i.e., the user's focus area; highlight the objects in the user's focus area;

[0121] When the user presses a button on the controller, the highlighted object will be projected onto a circular plane in front of the user while maintaining its relative position from the user's perspective. Objects in the plane will detect whether they overlap with each other. If so, they will spread out until they no longer overlap.

[0122] In this embodiment: the user's eye position and eye direction information are converted into a coordinate system based on the conversion matrix from the head camera to the world coordinate system, and the user's eye position and eye direction information are converted from the local coordinate system relative to the user's head to the world coordinate system relative to the virtual reality scene, and then the user's focus area is determined.

[0123] In this embodiment: on the basis of maintaining the relative position relationship of the object under the user's perspective, the object is projected into a circular plane in front of the user, including: generating a cone at the center of the user's eye position, and the bottom surface of the cone is facing the direction of the user's eyeballs. For all interactive objects in the scene, the dot product of the vector from its position to the cone vertex and the cone orientation vector is calculated, and the result is compared with the dot product of the cone orientation vector and the cone edge vector to determine whether the object is in the user's focusing cone area. If the object is in this area, a layer of circular highlight outline that ignores depth is drawn on the object. When the user presses a button on the virtual reality device controller, the interactive object will be projected into a plane in front of the user.

[0124] In this embodiment, the interactive object is projected onto a plane in front of the user, including: monitoring the user's device controller button event to determine whether the user presses the button. If the button is pressed, the position of the object in the user's focus area is converted from the world coordinate system to the user's camera coordinate system, and through collision detection, the overlapping objects are mutually repelled and separated until they no longer overlap. The user can then use the controller of the other hand to obtain the object in the projection plane.

[0125] In this embodiment: the method for obtaining the object in the plane includes: monitoring the user's virtual reality device controller button, when detecting that the user presses the interaction button of the controller, judging whether there is an interactive object in the interactive area of ​​the user's handle according to collision detection, if there is, then changing the position of the interactive object to the user's hand position, thereby completing a selection. If not, prompting the user to repeat the selection until there is an interactive object in the hand interaction area.

[0126] The virtual reality complex scene interaction method based on eye tracking provided in this embodiment is a progressive refinement interaction technology based on eye tracking, which allows users to accurately, quickly and easily obtain target objects even in scenes with dense obstructions, thereby greatly reducing users' unnecessary interaction time and interaction physical burden, and improving users' interaction experience.

[0127] The embodiment of the present invention provides a virtual reality complex scene interaction method based on eye tracking, and one implementation method is:

[0128] a) Obtaining and processing user eye movement data: processing the user eye movement data captured by a virtual reality helmet with an eye tracking function to obtain the user's eye center position information and the user's eye direction in the world coordinate system under the virtual reality scene;

[0129] b) Calculate the user's focus area and the objects in the user's focus area: dynamically update the user's focus area according to the center position and orientation of the user's eyes, and use vector calculation to determine all the interactive objects in the user's focus area of ​​the virtual reality scene, and highlight the interactive objects;

[0130] c) Projecting objects in the user's focus area onto a plane: Projecting objects in the user's focus area onto a plane in front of the user. On this screen, the projected objects will maintain their original size and relative position from the user's perspective, and detect whether there are objects overlapping each other after projection. If there are overlapping objects, separate the overlapping objects in opposite directions until they no longer overlap;

[0131] d) Obtaining objects in the controller interaction area: When the user presses the "Interact" button on the controller, the collision detection method is used to detect whether there are interactive objects in the user controller interaction area. If so, the position of the interactive object is changed to the position of the user's hand to achieve the effect of the user "grabbing" the object with his hand; if not, the user is prompted to perform the interactive operation again until the object is obtained.

[0132] The virtual reality complex scene interaction method based on eye tracking provided by the present invention obtains the center position and eye orientation of the user's eyes in real time, projects occluded objects onto the user's near plane in the virtual reality complex scene and solves the occlusion problem, thereby allowing the user to obtain the desired object faster and easier in the virtual reality scene.

[0133] Combine the following Figure 1-Figure 3 An embodiment of a virtual reality complex scene interaction method based on eye tracking of the present invention is described. Figure 1 A flowchart of the virtual reality complex scene interaction method based on eye tracking provided by the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0134] Step 1: System activation and eye tracking start. The user first activates the system by pressing the Trigger button on the left-hand controller. When the user presses the Trigger button on the left-hand controller, the eye tracking function starts immediately, and the system begins to monitor the user's eye movement data in real time. The system will align the user's line of sight with the world coordinate system of the virtual scene and convert the eye movement data into position coordinates in the virtual scene.

[0135] Step 2, eye tracking and highlighting. The system uses an algorithm to determine the user's current gaze area, that is, the user's focus area, and applies a highlight effect to all objects in the user's focus area. At this time, the objects in the user's focus area will be prominently marked to help the user more clearly perceive the target objects that may be of interest at the moment. This highlight effect can significantly improve selection efficiency, especially in scenes with dense objects or occlusions. Users do not need to manually adjust the viewing angle or controller position, and can quickly locate the target area by natural eye movement.

[0136] like Figure 2 As shown, the method provided by the present invention obtains the user's eye movement data to update the user's focus area in real time, and at the same time highlights the objects in the user's focus area, so that the user can intuitively view the interactive objects in the user's focus area that is currently being viewed;

[0137] Specifically, Figure 2 The real-time update process of the focus area in the virtual reality complex scene interaction method based on eye tracking includes the following four steps: a, b, c and d:

[0138] Step a, obtaining the user's eye movement data and converting it into the world coordinate system;

[0139] Step b, based on the user's eye information in the world coordinate system, a transparent cone is generated at the center of the user's eyes, the cone vertex is located at the center of the eyes, the cone direction is the combined direction of the user's eyes, and the cone position and direction are updated in real time according to the user's eye movement;

[0140] Step c, based on the results of the cone position and orientation updated in step b, perform dot product of the positions of all objects in the scene with the cone orientation vector, and determine whether the object is within the cone according to the dot product result;

[0141] In this method, the cone area is the user focus area; judging whether the object is in the user focus area is obtained by calculating the vector dot product, such as Figure 4 As shown, the calculation formula can be expressed as:

[0142]

[0143] in:

[0144] dir obj is the vector from the cone vertex to the object;

[0145] dir cone is the cone orientation vector, that is, the orientation vector from the cone vertex to the base;

[0146] dir edge is the cone side vector;

[0147] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing the vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, indicating that the object is inside the cone; otherwise, the function y is 0, indicating that the object is not inside the cone.

[0148] Step d: For the object in the cone, this method will call the engine rendering interface to draw a layer of highlighted outline ignoring the depth on the outline of the object to prompt the user where the user is currently looking.

[0149] Step 3, projecting onto a circular plane. When the user releases the Trigger button on the left-hand controller, the system stops eye tracking and then performs a key step: all highlighted objects in the user's focus area are projected onto a circular plane in front of the user according to their relative positions in the scene. The circular plane is a virtual two-dimensional interface that floats in front of the user to facilitate the user to continue to refine the selection. On this circular plane, objects that may have been difficult to select due to occlusion or overlap will be automatically processed. Specifically, the system will spread out the occluded objects until they are no longer obscured by other objects. This process intelligently adjusts the position of objects so that all objects can be clearly presented in the user's field of view, thereby eliminating the difficulty of selection caused by object occlusion.

[0150] In the present invention, the method for projecting objects within the user focus area onto a circular plane adopts a coordinate system conversion method and a collision detection method to complete the projection task without overlapping each other. The process involves multiple iterations, aiming to separate overlapping objects from each other until they no longer overlap.

[0151] Specifically, Figure 3 As shown:

[0152] Step 31, the projection process first converts the global coordinates of the object in the world coordinate system to the coordinates of the circular plane local coordinate system, and sets the depth value to 0, thereby ensuring that all objects are on the same plane.

[0153] The process formula for converting the global coordinates of the object in the world coordinate system to the coordinates in the circular plane local coordinate system can be expressed as:

[0154] e local =M trans ·e global

[0155] Among them, e localRepresents the position coordinates of the object relative to the local coordinate system of the circular plane; M trans represents the transformation matrix that transforms the object from the world coordinate system to the local coordinate system of the circular plane; e global Represents the global coordinates of the object in the world coordinate system.

[0156] During the projection process, in order to ensure that the object can remain within the circular plane after projection, the position information of the object is also scaled to a certain extent. The scaling formula can be expressed as:

[0157]

[0158] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0159] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0160] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0161] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0162] r sphere Represents the radius of the circular plane.

[0163] Step 32, after the collision detection function, the overlapping objects will be mutually excluded and separated in opposite directions according to their positions, and the process will be iterated continuously until all objects in the plane no longer overlap, thus laying the foundation for the subsequent close-range selection of target objects.

[0164] In each iteration, the overlapping objects repel each other, and the position update formula of each object can be expressed as:

[0165]

[0166] in:

[0167] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0168] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0169] O detect Represents the position coordinates of object O;

[0170] β is a minimum value, which is a known given value and is used to prevent the collision body C from overlapping with the object at the edge after updating its position;

[0171] D represents the distance that the collision body C is pushed away;

[0172] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0173] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0174] Step 4: Select an object with the right controller. At this point, the user can use the Trigger button on the right controller to complete the final selection of the object. When the user presses the Trigger button on the right controller, the system will automatically determine whether there is an interactive object in the interactive area of ​​the current right controller. If there is an object in the interactive area, the system will select the object and complete the selection operation; if the interactive area is empty, the user can release the Trigger button to readjust the selection until the target object is successfully selected.

[0175] Figure 5 The schematic diagram of the structure of the virtual reality complex scene interaction device based on eye tracking provided by the present invention is as follows: Figure 5 As shown, it includes a user eye movement data tracking unit 510, a projection unit 520 and an interaction unit 530;

[0176] A user eye movement data tracking unit 510, used to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and then determine all objects in the user's focus area;

[0177] The projection unit 520 is used to project all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining the relative position relationship under the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0178] The user eye movement data tracking unit 510 and the projection unit 520 can also be used to: obtain user eye movement data and scene object information; construct a corresponding user focus area with the scene object information and eye movement data, and highlight the objects in the user focus area for further selection by the user; wherein the user focus area will be dynamically updated with the user's eye movement, so that the user can view the objects in the user focus area more intuitively.

[0179] The interaction unit 530 is used to select and interact with objects in the circular plane to achieve complex scene interaction of virtual reality based on eye tracking.

[0180] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630, and the virtual reality complex scene interaction algorithm based on eye tracking includes: updating the user eye movement data model, eye movement features and its focus area algorithm, and outputting the object cluster focused by the user; wherein the focus area is updated in real time, so that the user can intuitively view the focused object.

[0181] In addition, the logic instructions in the aforementioned memory 630 can be implemented in the form of a software module and can be sold or used as an independent product. These instructions can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially, or its contribution to the prior art, can also be embodied in the form of a software product. The computer software product can be stored in a certain storage medium, including a number of instructions to enable a computing device (such as a personal computer, a server, a network device, etc.) to perform all or part of the operating steps described in each embodiment of the present invention. The above-mentioned storage medium includes, but is not limited to: a U disk, an external hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk, an optical disk, and other media that can store program codes.

[0182] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the virtual reality complex scene interaction algorithm based on eye tracking provided by the above-mentioned methods, and the method includes: updating the user's eye movement data model, eye movement characteristics and focus area update algorithm, and outputting the object cluster focused by the user; wherein the focus area is updated in real time to allow the user to intuitively view the focused object.

[0183] In order to solve the problems faced by traditional VR ray interaction technology in virtual reality scenes, such as insufficient accuracy, long operation time, and difficulty in selecting occluded targets, the present invention provides a virtual reality complex scene interaction method, device and electronic device based on eye tracking, which is a progressive refinement interaction method based on eye tracking. This method helps users quickly and accurately select target objects in complex virtual scenes by utilizing the user's line of sight information and combining progressive interaction design. Compared with traditional ray interaction methods, eye tracking can significantly shorten the time of target object selection and improve the accuracy of target object selection, especially when there are a large number of overlapping or occluded objects in the scene, which can effectively solve the problem that traditional methods are difficult to select occluded targets.

[0184] The core innovation of the present invention is that it uses eye tracking as the initial input signal, and automatically screens and progressively refines the selection of objects near the user's gaze point, avoiding the tedious operation of the user manually adjusting the ray. The user only needs to guide the system to gradually narrow the target range through natural gaze behavior, and finally achieve rapid selection of any target object. This method not only improves the user's interactive experience, but also reduces the possibility of misoperation, making the selection process in virtual reality more intuitive and efficient.

[0185] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual reality complex scene interaction method based on eye tracking, characterized in that: include: Tracking user eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area; Project all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap; Objects in the circular plane are selected and interacted with to achieve complex scene interaction of virtual reality based on eye tracking.

2. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: The user eye movement data includes the user eye position and the user eye direction information; the tracking of the user eye movement data determines the user focus area in the virtual reality scene in real time, and then determines all objects in the user focus area, specifically: Track the user's eye movement data through the head camera, and obtain the user's eye position and eye direction information in the local coordinate system of the user's head in real time; According to the transformation matrix from the local coordinate system of the user's head to the world coordinate system, the user's eye position and the user's eye direction information in the local coordinate system of the user's head are transformed into the user's eye position and the user's eye direction in the world coordinate system; According to the user's eyeball position in the world coordinate system, a transparent cone is generated at the center of the user's eyes, the vertex of the cone is located at the center of the eyes, and the direction of the cone refers to the direction from the vertex to the bottom surface, which is the direction of the user's eyeball; the area where the cone is located is the user's focusing area; For all objects in the virtual reality scene, determine whether the object is located in the cone, and if so, the object is within the user focus area; Otherwise, the object is not within the user focus area, and then all objects within the user focus area are determined.

3. The method for complex scene interaction in virtual reality based on eye tracking according to claim 2, characterized in that: The determination of whether the object is located within the cone is obtained by calculating the vector dot product, and the specific formula is: in: dir obj is the vector from the cone vertex to the object; dir cone is the cone orientation vector, that is, the orientation vector from the cone vertex to the base; dir edge is the cone side vector; Therefore, if the vector dir from the cone vertex to the object obj With the cone facing the vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, indicating that the object is inside the cone; otherwise, the function y is 0, indicating that the object is not inside the cone.

4. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: Also includes: If the object is located within the user focus area, a circular highlight outline ignoring the depth is drawn on the object to highlight the object.

5. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: The method projects all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining the relative position relationship under the user's perspective, and intelligently adjusts the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap. Specifically, Performing coordinate transformation on all objects in the user focus area, transforming the global coordinates of each object in the world coordinate system into the circular plane local coordinate system, and obtaining the local coordinates of each object in the circular plane local coordinate system; Scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates; Projecting the scaled coordinates to corresponding positions in the circular plane; Collision detection is performed on each object projected into the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap.

6. The method for complex scene interaction in virtual reality based on eye tracking according to claim 5, characterized in that: The following formula is used to scale the local coordinates of each object in the local coordinate system of the circular plane to obtain the scaled coordinates: Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane; P represents the local coordinates of the scaled object in the local coordinate system of the circular plane; P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane; P far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane; r spaere Represents the radius of the circular plane.

7. The method for complex scene interaction in virtual reality based on eye tracking according to claim 5, characterized in that: The collision detection is performed on each object projected into the circular plane so that overlapping objects are mutually repelled and separated until they no longer overlap, specifically: For each object projected into the circular plane, traverse in sequence from the center to the periphery; for each traversed object, represented as the currently detected object O, detect whether the object O has an overlapping collision body C. If not, traverse and detect the next object; if so, push the collision body C away from the object O, and determine the position of the updated collision body C by the following formula: in: R c Represents the radius of the collision body C; R o represents the radius of object O; C overlap Represents the position coordinates of the collision body C before being pushed away; O detect Represents the position coordinates of object O; β is a minimum value, which is a known given value; D represents the distance that the collision body C is pushed away; (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O; C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

8. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: Also includes: Configure device controllers; The device controller has an eye tracking start button event, a projection button event and an interaction button event; Monitoring the button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area, until the projection button event is detected, stopping the tracking of the user's eye movement data; When the projection button event is detected, all objects in the latest user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap; When the interaction button event is detected, it is determined whether there is an interactive object in the interaction area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

9. A device for implementing the virtual reality complex scene interaction method based on eye tracking as described in any one of claims 1 to 8, characterized in that: include: A user eye movement data tracking unit, used to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and then determine all objects in the user's focus area; A projection unit is used to project all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining the relative position relationship under the user's viewing angle, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap; The interaction unit is used to select and interact with objects in the circular plane to achieve complex scene interaction of virtual reality based on eye tracking.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a virtual reality complex scene interaction method based on eye tracking as described in any one of claims 1 to 8 is implemented.

Images