Virtual reality complex scene interaction method, device and electronic device based on eye tracking

Abstract

The present invention provides a method, device and electronic device for interacting with complex virtual reality scenes based on eye tracking, including: tracking the user's eye movement data to determine the user's focus area in the virtual reality scene in real time; projecting all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining the relative position relationship from the user's perspective, and intelligently adjusting the positions of the objects projected into the circular plane so that all objects are separated from each other and do not overlap; and selecting and interacting with objects in the circular plane. The present invention utilizes the user's line of sight information to help the user quickly and accurately select target objects in complex virtual scenes. Compared with traditional ray interaction methods, the present invention can significantly shorten the selection time and improve the accuracy of target object selection, especially when there are a large number of overlapping or occluded objects in the scene, and can effectively solve the problem that traditional methods have difficulty in selecting occluded targets.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual reality and augmented reality, and in particular to a method, device and electronic device for interacting with complex virtual reality scenes based on eye tracking. Background Art

[0002] With the development of the virtual reality (VR) industry, more and more people are using VR headsets and using them for work, entertainment, and movie viewing. During this process, users often need to interact with and select objects in the virtual scene. Traditional VR interaction methods require users to use controllers to shoot rays to select objects or icons in the virtual scene.

[0003] Although this ray-based method is widely used in VR, it also has obvious drawbacks. This method requires the user to hold the controller high for a long time to aim at the object or icon they want to interact with. If the object is far away from the user and the object is small, it is difficult for the ray to remain aimed at the object. In particular, the human hand will have a certain degree of natural jitter during natural movement, and when the user presses the controller button with his hand, the controller will also tilt and vibrate to a certain extent. This makes it difficult for the ray-based interaction method to have a good accuracy rate when facing distant objects and small objects.

[0004] In addition, when the target object is blocked by other obstructions in the scene, it is even more difficult for the user to aim the controller ray directly at the target object. The ray will be blocked by the obstruction in front of the target object, and the target object cannot be selected. Although the user can adjust the camera posture direction, the position of the controller and the angle of the emitted ray to make the ray bypass the obstruction and select the target, when the obstructions in the scene are too dense, it is even difficult for the user to find the location of the target object in the first place, and thus cannot select the target by adjusting their posture. Sometimes the target cannot be selected no matter how it is adjusted. Therefore, the ray-based method has the disadvantages of being difficult to obtain obstructed targets and is not suitable for scenes with dense obstructions. Summary of the Invention

[0005] To address the challenges of traditional VR ray-guided interaction technology in virtual reality scenarios, such as insufficient accuracy, time-consuming operations, and difficulty selecting occluded targets, this paper proposes and develops a progressive refinement interaction method based on eye tracking. This method leverages the user's line of sight and combines it with a progressive interaction design to help users quickly and accurately select target objects in complex virtual scenes. Compared to traditional ray-guided interaction methods, eye tracking technology can significantly shorten target selection time and improve target selection accuracy, especially in scenes with a large number of overlapping or occluded objects. This effectively addresses the difficulty of traditional methods in selecting occluded targets.

[0006] The technical solution adopted in the present invention is as follows:

[0007] The present invention provides a method for interacting with complex virtual reality scenes based on eye tracking, comprising:

[0008] Tracking user eye movement data to determine the user's focus area in the virtual reality scene in real time, and then determining all objects within the user's focus area;

[0009] Project all objects within the user's focus area onto a circular plane suspended in front of the user, maintaining their relative positional relationship from the user's perspective. Furthermore, intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap.

[0010] Objects within the circular plane are selected and interacted with to achieve complex virtual reality scene interaction based on eye tracking.

[0011] Preferably, the user eye movement data includes user eye position and user eye direction information; the tracking of the user eye movement data determines the user's focus area in the virtual reality scene in real time, and further determines all objects within the user's focus area, specifically:

[0012] Track the user's eye movement data through the head camera, and obtain the user's eye position and eye direction information in the local coordinate system of the user's head in real time;

[0013] According to the transformation matrix from the local coordinate system of the user's head to the world coordinate system, the user's eye position and eye direction information in the local coordinate system of the user's head are converted into the user's eye position and eye direction in the world coordinate system;

[0014] Based on the user's eye position in the world coordinate system, a transparent cone is generated at the center of the user's eyes. The apex of the cone is located at the center of the eyes. The orientation of the cone, which is the direction from the apex to the base, is the orientation of the user's eyes. The area where the cone is located is the user's focus area.

[0015] For all objects in the virtual reality scene, it is determined whether the object is located in the cone. If so, the object is within the user focus area; otherwise, the object is not within the user focus area, and then all objects within the user focus area are determined.

[0016] Preferably, the determination of whether the object is located within the cone is performed by vector dot product calculation, and the specific formula is:

[0017]

[0018] in:

[0019] dir obj is the vector from the cone vertex to the object;

[0020] dir cone is the cone facing vector, that is, the facing vector from the cone vertex to the base;

[0021] dir edge is the cone side vector;

[0022] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, which means the object is inside the cone; otherwise, the function y is 0, which means the object is not inside the cone.

[0023] Preferably, it also includes:

[0024] If the object is within the user focus area, a depth-ignoring circular highlight outline is drawn on the object to highlight the object.

[0025] Preferably, all objects in the user's focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap. Specifically,

[0026] Performing coordinate transformation on all objects in the user focus area, transforming the global coordinates of each object in the world coordinate system into the local coordinate system of the circular plane, and obtaining the local coordinates of each object in the local coordinate system of the circular plane;

[0027] Scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates;

[0028] Projecting the scaled coordinates to corresponding positions within the circular plane;

[0029] Collision detection is performed on each object projected into the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap.

[0030] Preferably, the local coordinates of each object in the circular plane local coordinate system are scaled using the following formula to obtain scaled coordinates:

[0031]

[0032] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0033] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0034] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0035] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0036] r sphere Represents the radius of the circular plane.

[0037] Preferably, the collision detection is performed on each object projected into the circular plane so that overlapping objects are mutually repelled and separated until they no longer overlap, specifically as follows:

[0038] For each object projected onto the circular plane, traverse it in sequence from the center to the periphery. For each traversed object, represented as the currently detected object O, check whether the object O has an overlapping collision body C. If not, traverse and detect the next object. If so, push the collision body C away from the object O. The position of the updated collision body C is determined by the following formula:

[0039]

[0040] in:

[0041] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0042] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0043] O detectRepresents the position coordinates of object O;

[0044] β is a minimum value, which is a known given value;

[0045] D represents the distance that the collision body C is pushed away;

[0046] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0047] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0048] Preferably, it also includes:

[0049] Configuring a device controller; the device controller has an eye tracking start button event, a projection button event, and an interaction button event;

[0050] monitoring button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects within the user's focus area, until the projection button event is detected, stopping tracking the user's eye movement data;

[0051] When the projection button event is detected, all objects in the current user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positions from the user's perspective. Furthermore, the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap.

[0052] When the interaction button event is detected, it is determined whether there is an interactive object in the interaction area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

[0053] The present invention also provides a device for implementing the above-mentioned method for interacting with complex virtual reality scenes based on eye tracking, comprising:

[0054] A user eye movement data tracking unit, configured to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and further determine all objects within the user's focus area;

[0055] a projection unit, configured to project all objects within the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationships from the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap;

[0056] The interaction unit is used to select and interact with objects in the circular plane to achieve complex virtual reality scene interaction based on eye tracking.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for interacting with complex virtual reality scenes based on eye tracking is implemented.

[0058] The eye-tracking-based virtual reality complex scene interaction method, device, and electronic device provided by the present invention have the following advantages:

[0059] The eye-tracking-based virtual reality complex scene interaction method, device, and electronic device provided by the present invention obtain the center position and orientation of the user's eyes in real time, project occluded objects onto the user's near plane in complex virtual reality scenes, and solve the occlusion problem, thereby allowing the user to obtain the desired objects faster and easier in virtual reality scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To more clearly illustrate the present invention and related technical solutions, the following briefly describes the drawings required in the embodiments or prior art. It is obvious that these drawings only represent some embodiments of the present invention, and those skilled in the art can deduce other possible drawings based on these drawings without inventive effort.

[0061] Figure 1 A flow chart of the eye-tracking-based virtual reality complex scene interaction method provided by the present invention;

[0062] Figure 2 This is a scene example diagram of the virtual reality complex scene interaction method based on eye tracking provided by the present invention, which shows the highlighting of objects in the user's focus area;

[0063] Figure 3 An example image of an object within the focus area provided by the present invention being projected onto the user's near plane;

[0064] Figure 4 This is an example diagram of detecting whether an object is within the focus area provided by the present invention;

[0065] Figure 5This is a schematic diagram of the structure of the virtual reality complex scene interaction device based on eye tracking provided by the present invention;

[0066] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0067] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the technical solutions will be described in detail and completely in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described below are only part of the present invention and do not cover all possible embodiments. Any other embodiments that can be derived by ordinary technicians in this field without creative work on the basis of the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0068] Because existing virtual reality interaction technologies cannot easily, quickly, and accurately acquire occluded targets in virtual reality scenes, interactions in dense scenes are difficult, and users need to expend a lot of unnecessary physical effort walking in the virtual reality environment to acquire distant target objects. Therefore, the present invention provides a method, device, and electronic device for interacting with complex virtual reality scenes based on eye tracking, which are used to solve the above problems existing in the prior art.

[0069] The present invention provides a method for interacting with complex virtual reality scenes based on eye tracking, comprising:

[0070] Step S1, tracking user eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area;

[0071] Step S2: Projecting all objects within the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationships from the user's perspective, and intelligently adjusting the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap.

[0072] Step S3: Select and interact with objects in the circular plane to achieve complex virtual reality scene interaction based on eye tracking.

[0073] The following describes steps S1 to S3 in detail:

[0074] Step S1, tracking user eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects in the user's focus area;

[0075] In this step, the user's eye movement data includes the user's eye position and the user's eye direction information; this step is specifically as follows:

[0076] Step S11, tracking the user's eye movement data through the head camera, and obtaining the user's eye position and eye direction information in the local coordinate system of the user's head in real time;

[0077] Step S12, converting the user's eye position and eye orientation information in the local coordinate system of the user's head into the user's eye position and eye orientation in the world coordinate system according to the conversion matrix from the local coordinate system of the user's head to the world coordinate system;

[0078] Step S13: Generate a transparent cone at the center of the user's eyes based on the user's eye position in the world coordinate system. The apex of the cone is located at the center of the eyes. The orientation of the cone, which is the direction from the apex to the base, is the orientation of the user's eyes. The area where the cone is located is the user's focus area.

[0079] Step S14: for all objects in the virtual reality scene, determine whether the object is located in the cone. If so, the object is within the user focus area; otherwise, the object is not within the user focus area, and then determine all objects within the user focus area.

[0080] As an embodiment, the determination of whether the object is located within the cone is obtained by vector dot product calculation, and the specific formula is:

[0081]

[0082] in:

[0083] dir obj is the vector from the cone vertex to the object;

[0084] dir cone is the cone facing vector, that is, the facing vector from the cone vertex to the base;

[0085] dir edge is the cone side vector;

[0086] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, which means the object is inside the cone; otherwise, the function y is 0, which means the object is not inside the cone.

[0087] In practical applications, if the object is located within the user focus area, the method further includes: drawing a layer of depth-ignoring circular highlight outline on the object to highlight the object.

[0088] Step S2: Projecting all objects within the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationships from the user's perspective, and intelligently adjusting the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap.

[0089] This step specifically includes:

[0090] Step S21, performing coordinate transformation on all objects within the user focus area, transforming the global coordinates of each object in the world coordinate system into the local coordinate system of the circular plane, and obtaining the local coordinates of each object in the local coordinate system of the circular plane;

[0091] Step S22, scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates;

[0092] As an embodiment, the following formula is used to scale the local coordinates of each object in the circular plane local coordinate system to obtain the scaled coordinates:

[0093]

[0094] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0095] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0096] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0097] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0098] r sphere Represents the radius of the circular plane.

[0099] Step S23, projecting the scaled coordinates to corresponding positions within the circular plane;

[0100] Step S24: performing collision detection on the objects projected into the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap.

[0101] As an embodiment, step S24 specifically includes:

[0102] For each object projected onto the circular plane, traverse it in sequence from the center to the periphery. For each traversed object, represented as the currently detected object O, check whether the object O has an overlapping collision body C. If not, traverse and detect the next object. If so, push the collision body C away from the object O. The position of the updated collision body C is determined by the following formula:

[0103]

[0104] in:

[0105] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0106] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0107] O detect Represents the position coordinates of object O;

[0108] β is a minimum value, which is a known given value;

[0109] D represents the distance that the collision body C is pushed away;

[0110] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0111] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0112] Step S3: Select and interact with objects in the circular plane to achieve complex virtual reality scene interaction based on eye tracking.

[0113] In practical applications, when performing steps S1 to S3, the following steps are also included:

[0114] Configuring a device controller; the device controller has an eye tracking start button event, a projection button event, and an interaction button event;

[0115] monitoring button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects within the user's focus area, until the projection button event is detected, stopping tracking the user's eye movement data;

[0116] When the projection button event is detected, all objects in the current user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positions from the user's perspective. Furthermore, the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap.

[0117] When the interaction button event is detected, it is determined whether there is an interactive object in the interaction area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

[0118] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for interacting with complex virtual reality scenes based on eye tracking is implemented.

[0119] The present invention provides a method for interacting with complex virtual reality scenes based on eye tracking. One implementation method is as follows:

[0120] Input the user's eye position and eye direction information, and output the virtual reality scene area that the user is currently focused on, i.e., the user focus area; objects within the user focus area are highlighted;

[0121] When the user presses a button on the controller, the highlighted object is projected onto a circular plane in front of the user, maintaining its relative position from the user's perspective. Objects within the plane detect if they overlap, and if so, they spread out until they no longer overlap.

[0122] In this embodiment, the user's eye position and eye direction information are transformed into a coordinate system based on the conversion matrix from the head camera to the world coordinate system. The user's eye position and eye direction information are transformed from the local coordinate system relative to the user's head to the world coordinate system relative to the virtual reality scene, and then the user's focus area is determined.

[0123] In this embodiment: on the basis of maintaining the relative position relationship of the object in the user's perspective, the object is projected into a circular plane in front of the user, including: generating a cone at the center of the user's eye position, and the bottom surface of the cone is facing the direction of the user's eyeballs. For all interactive objects in the scene, the dot product of the vector from its position to the cone vertex and the cone orientation vector is calculated, and the result is compared with the dot product of the cone orientation vector and the cone edge vector to determine whether the object is within the user's focusing cone area. If the object is within this area, a layer of circular highlight outline that ignores depth is drawn on the object. When the user presses a button on the virtual reality device controller, the interactive object will be projected into a plane in front of the user.

[0124] In this embodiment, interactive objects are projected onto a plane in front of the user, including monitoring button events on the user's device controller to determine whether the user has pressed a button. If the button has been pressed, the positions of objects within the user's focus area are converted from the world coordinate system to the user's camera coordinate system. Collision detection is then performed to mutually repel and separate overlapping objects until they no longer overlap. The user can then use the controller in their other hand to capture objects within the projected plane.

[0125] In this embodiment, the method for obtaining objects within a plane includes monitoring the user's virtual reality device controller buttons. When the user presses an interaction button on the controller, collision detection is used to determine whether there is an interactive object within the user's controller's interactive area. If so, the position of the interactive object is relocated to the user's hand position, thereby completing a selection. If not, the user is prompted to repeat the selection until an interactive object is found within the hand's interactive area.

[0126] The virtual reality complex scene interaction method based on eye tracking provided in this embodiment is a progressive refinement interaction technology based on eye tracking, which allows users to accurately, quickly and easily obtain target objects even in scenes with dense obstructions, thereby greatly reducing users' unnecessary interaction time and interaction physical burden, and improving users' interaction experience.

[0127] The present invention provides a method for interacting with complex virtual reality scenes based on eye tracking. One implementation method is as follows:

[0128] a) Obtaining and processing user eye movement data: Processing the user's eye movement data captured by a virtual reality headset with an eye tracking function to obtain the user's eye center position information and the user's eye direction in the world coordinate system of the virtual reality scene;

[0129] b) Calculating the user's focus area and the objects within it: Dynamically updating the user's focus area based on the user's eye center position and orientation, and using vector calculation to determine all interactive objects in the virtual reality scene within the user's focus area, and highlighting the interactive objects;

[0130] c) Projecting objects within the user's focus area onto a flat surface: Projecting objects within the user's focus area onto a flat surface in front of the user. The projected objects will maintain their original size and relative position from the user's perspective. The system will also detect if any objects overlap after projection. If any objects overlap, they will be separated in opposite directions until they no longer overlap.

[0131] d) Obtaining objects in the controller interaction area: When the user presses the "Interact" button on the controller, the collision detection method is used to detect whether there are interactive objects in the user controller interaction area. If so, the position of the interactive object is changed to the position of the user's hand to achieve the effect of the user "grabbing" the object with his hand; if not, the user is prompted to perform the interactive operation again until the object is obtained.

[0132] The eye-tracking-based virtual reality complex scene interaction method provided by the present invention obtains the center position and orientation of the user's eyes in real time, projects occluded objects onto the user's near plane in complex virtual reality scenes, and solves the occlusion problem, thereby allowing users to obtain desired objects faster and easier in virtual reality scenes.

[0133] The following combination Figure 1-Figure 3 Describe an embodiment of the present invention's method for interacting with complex virtual reality scenes based on eye tracking. Figure 1 This is a flow chart of the virtual reality complex scene interaction method based on eye tracking provided by the present invention, such as Figure 1 As shown, the method includes:

[0134] Step 1: System Activation and Eye Tracking. The user first activates the system by pressing the Trigger button on the left controller. Once the user presses the Trigger button, eye tracking begins, and the system begins monitoring the user's eye movements in real time. The system aligns the user's gaze direction with the world coordinate system of the virtual scene, converting the eye movement data into positional coordinates within the virtual scene.

[0135] In step 2, eye tracking and highlighting, the system uses an algorithm to determine the user's current gaze area, known as the user's focus area, and applies a highlighting effect to all objects within the user's focus area. At this point, objects in the user's focus area are prominently marked to help the user more clearly perceive the target objects of potential interest. This highlighting effect can significantly improve selection efficiency, especially in scenes with dense objects or occlusions. Users can quickly locate the target area through natural eye movements without having to manually adjust the viewing angle or controller position.

[0136] like Figure 2 As shown, the method provided by the present invention obtains the user's eye movement data to update the user's focus area in real time, and at the same time highlights the objects in the user's focus area, so that the user can intuitively view the interactive objects in the user's focus area that they are currently viewing;

[0137] Specifically, Figure 2 The real-time update process of the focus area in the eye-tracking-based virtual reality complex scene interaction method includes the following four steps: a, b, c, and d:

[0138] Step a: Obtain user eye movement data and convert it into the world coordinate system;

[0139] Step b: Based on the user's eye information in the world coordinate system, a transparent cone is generated at the center of the user's eyes. The cone's apex is located at the center of the eyes, and the cone's orientation is the combined direction of the user's eyes. The cone's position and orientation are updated in real time based on the user's eye movement.

[0140] Step c: Based on the results of the cone position and orientation updated in step b, perform a dot product between the positions of all objects in the scene and the cone orientation vector, and determine whether the object is within the cone based on the dot product result.

[0141] In this method, the conical area is the user focus area; whether the object is in the user focus area is determined by vector dot product calculation, such as Figure 4 As shown, the calculation formula can be expressed as:

[0142]

[0143] in:

[0144] dir obj is the vector from the cone vertex to the object;

[0145] dir cone is the cone facing vector, that is, the facing vector from the cone vertex to the base;

[0146] dir edge is the cone side vector;

[0147] Therefore, if the vector dir from the cone vertex to the object obj With the cone facing vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, which means the object is inside the cone; otherwise, the function y is 0, which means the object is not inside the cone.

[0148] In step d, for objects within the cone, this method calls the engine rendering interface to draw a layer of highlighted outline on the outline of the object, ignoring the depth, to prompt the user where they are currently looking.

[0149] Step 3, projection onto a circular plane. When the user releases the Trigger button on the left-hand controller, the system stops eye tracking and then performs a key step: all highlighted objects in the user's focus area are projected onto a circular plane in front of the user according to their relative positions in the scene. The circular plane is a virtual two-dimensional interface that floats in front of the user, making it easier for the user to continue to refine their selection. On this circular plane, objects that may have been difficult to select due to occlusion or overlap will be automatically processed. Specifically, the system will spread out the occluded objects until they are no longer obscured by other objects. This process intelligently adjusts the position of objects so that all objects can be clearly presented in the user's field of view, thereby eliminating the difficulty of selection caused by object occlusion.

[0150] In the present invention, the method for projecting objects within the user focus area onto a circular plane uses a coordinate system conversion method and a collision detection method to complete the projection task without overlapping. The process involves multiple iterations, aiming to separate overlapping objects from each other until they no longer overlap.

[0151] Specifically, if Figure 3 As shown:

[0152] In step 31 , the projection process first converts the global coordinates of the object in the world coordinate system to the coordinates of the circular plane local coordinate system, and sets the depth value to 0, thereby ensuring that all objects are on the same plane.

[0153] The process formula for converting the global coordinates of the object in the world coordinate system to the coordinates in the local coordinate system of the circular plane can be expressed as:

[0154] e local =M trans ·e global

[0155] Among them, e localRepresents the position coordinates of the object relative to the local coordinate system of the circular plane; M trans Represents the transformation matrix that transforms the object from the world coordinate system to the local coordinate system of the circular plane; e global Represents the global coordinates of the object in the world coordinate system.

[0156] During the projection process, in order to ensure that the object remains within the circular plane after projection, the object's position information is also scaled to a certain extent. The scaling formula can be expressed as:

[0157]

[0158] Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane;

[0159] P represents the local coordinates of the scaled object in the local coordinate system of the circular plane;

[0160] P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane;

[0161] O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane;

[0162] r sphere Represents the radius of the circular plane.

[0163] In step 32, after the collision detection function, overlapping objects will be separated in opposite directions according to their positions. This process will be repeated until all objects in the plane no longer overlap, laying the foundation for the subsequent close-range selection of target objects.

[0164] In each iteration, as overlapping objects repel each other, the position update formula for each object can be expressed as:

[0165]

[0166] in:

[0167] R c Represents the radius of the collision body C; R o represents the radius of object O;

[0168] C overlap Represents the position coordinates of the collision body C before being pushed away;

[0169] O detect Represents the position coordinates of object O;

[0170] β is a minimum value, a known given value, used to prevent the collision body C from overlapping with the object at the edge after updating its position;

[0171] D represents the distance that the collision body C is pushed away;

[0172] (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O;

[0173] C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

[0174] Step 4: Select an object with the right controller. At this point, the user can use the Trigger button on the right controller to finalize the object selection. When the user presses the Trigger button on the right controller, the system automatically determines whether there are any interactive objects within the right controller's interactive area. If an object is within the interactive area, the system selects it and completes the selection. If the interactive area is empty, the user can release the Trigger button and readjust the selection until the target object is successfully selected.

[0175] Figure 5 This is a structural diagram of the virtual reality complex scene interaction device based on eye tracking provided by the present invention, such as Figure 5 As shown, it includes a user eye movement data tracking unit 510, a projection unit 520 and an interaction unit 530;

[0176] A user eye movement data tracking unit 510 is used to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and then determine all objects in the user's focus area;

[0177] The projection unit 520 is configured to project all objects within the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positions from the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap.

[0178] The user eye movement data tracking unit 510 and the projection unit 520 can also be used to: obtain user eye movement data and scene object information; construct a corresponding user focus area based on the scene object information and eye movement data, and highlight the objects in the user focus area for the user to make further selections; wherein, the user focus area will be dynamically updated with the user's eye movement, so that the user can view the objects in the user focus area more intuitively.

[0179] The interaction unit 530 is used to select and interact with objects in the circular plane to achieve complex virtual reality scene interaction based on eye tracking.

[0180] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communications bus 640. The processor 610 may call the logic instructions in the memory 630 to implement an eye-tracking-based virtual reality complex scene interaction algorithm, which includes: updating the user's eye movement data model, eye movement features, and focus area with an algorithm to output a cluster of objects focused on by the user; wherein the focus area is updated in real time to allow the user to intuitively view the focused object.

[0181] In addition, the logical instructions in the aforementioned memory 630 can be implemented in the form of a software module and can be sold or used as an independent product. These instructions can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially, or its contribution to the prior art, can also be embodied in the form of a software product. The computer software product can be stored in a certain storage medium, containing a number of instructions to enable a computing device (such as a personal computer, server, network device, etc.) to perform all or part of the operating steps described in the various embodiments of the present invention. The above-mentioned storage medium includes but is not limited to: U disk, external hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk, CD-ROM and other media that can store program code.

[0182] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the virtual reality complex scene interaction algorithm based on eye tracking provided by the above-mentioned methods. The method includes: updating the user eye movement data model, eye movement characteristics and its focus area algorithm, and outputting the object cluster focused by the user; wherein the real-time update process of the focus area is used to allow the user to intuitively view the focused object.

[0183] To address the problems faced by traditional VR ray-based interaction technology in virtual reality scenes, such as insufficient accuracy, time-consuming operation, and difficulty selecting occluded targets, the present invention provides a method, device, and electronic device for complex virtual reality scene interaction based on eye tracking. This method is a progressively refined interaction method based on eye tracking. By utilizing the user's line of sight information and combining it with a progressive interaction design, the method helps users quickly and accurately select target objects in complex virtual scenes. Compared with traditional ray-based interaction methods, eye tracking can significantly shorten the time it takes to select target objects and improve the accuracy of target object selection. This is especially true when there are a large number of overlapping or occluded objects in the scene, effectively resolving the shortcomings of traditional methods in that they have difficulty selecting occluded targets.

[0184] The core innovation of this invention lies in its use of eye tracking as the initial input signal. By automatically screening and progressively refining the selection of objects near the user's gaze point, the system avoids the tedious manual adjustment of the ray. The user simply guides the system through natural gaze behavior, gradually narrowing the target range, ultimately achieving rapid selection of any target object. This method not only enhances the user's interactive experience but also reduces the possibility of misoperation, making the selection process in virtual reality more intuitive and efficient.

[0185] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present embodiment without inventive effort.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for complex scene interaction in virtual reality based on eye tracking, characterized in that: include: Tracking user eye movement data to determine the user's focus area in the virtual reality scene in real time, and then determining all objects within the user's focus area; Project all objects within the user's focus area onto a circular plane suspended in front of the user, maintaining their relative positional relationship from the user's perspective. Furthermore, intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap. Selecting and interacting with objects within the circular plane to achieve complex virtual reality scene interaction based on eye tracking; The method projects all objects in the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationship from the user's perspective, and intelligently adjusts the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap. Specifically, Performing coordinate transformation on all objects in the user focus area, transforming the global coordinates of each object in the world coordinate system into the local coordinate system of the circular plane, and obtaining the local coordinates of each object in the local coordinate system of the circular plane; Scaling the local coordinates of each object in the circular plane local coordinate system to obtain scaled coordinates; Projecting the scaled coordinates to corresponding positions within the circular plane; Performing collision detection on each object projected onto the circular plane, so that overlapping objects are mutually repelled and separated until they no longer overlap; The collision detection is performed on each object projected into the circular plane so that overlapping objects are mutually repelled and separated until they no longer overlap, specifically: For each object projected onto the circular plane, traverse it in sequence from the center to the periphery. For each traversed object, represented as the currently detected object O, check whether the object O has an overlapping collision body C. If not, traverse and detect the next object. If so, push the collision body C away from the object O. The position of the updated collision body C is determined by the following formula: in: R c Represents the radius of the collision body C; R o represents the radius of object O; C overlap Represents the position coordinates of the collision body C before being pushed away; O detect Represents the position coordinates of object O; β is a minimum value, which is a known given value; D represents the distance that the collision body C is pushed away; (C overlap -O detect ) norm Represents the unit direction vector from the position of the collision body C before being pushed away to the position of the object O; C new Represents the position coordinates of the collision body C after being pushed away, that is, the updated position coordinates of the collision body C.

2. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: The user eye movement data includes user eye position and user eye direction information; the tracking of the user eye movement data determines the user's focus area in the virtual reality scene in real time, and then determines all objects within the user's focus area, specifically: Track the user's eye movement data through the head camera, and obtain the user's eye position and eye direction information in the local coordinate system of the user's head in real time; According to the transformation matrix from the local coordinate system of the user's head to the world coordinate system, the user's eye position and eye direction information in the local coordinate system of the user's head are converted into the user's eye position and eye direction in the world coordinate system; Based on the user's eye position in the world coordinate system, a transparent cone is generated at the center of the user's eyes. The apex of the cone is located at the center of the eyes. The orientation of the cone, which is the direction from the apex to the base, is the orientation of the user's eyes. The area where the cone is located is the user's focus area. For all objects in the virtual reality scene, determine whether the object is located within the cone, and if so, the object is within the user focus area; Otherwise, the object is not within the user focus area, and then all objects within the user focus area are determined.

3. The method for complex scene interaction in virtual reality based on eye tracking according to claim 2, characterized in that: The determination of whether the object is located within the cone is obtained by vector dot product calculation, and the specific formula is: in: dir obj is the vector from the cone vertex to the object; dir cone is the cone facing vector, that is, the facing vector from the cone vertex to the base; dir edge is the cone side vector; Therefore, if the vector dir from the cone vertex to the object obj With the cone facing vector dir cone The dot product result is greater than or equal to the cone side vector dir edge and the cone facing vector dir cone If the function y is the dot product of , then the function y is 1, which means the object is inside the cone; otherwise, the function y is 0, which means the object is not inside the cone.

4. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: Also includes: If the object is within the user focus area, a depth-ignoring circular highlight outline is drawn on the object to highlight the object.

5. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: Use the following formula to scale the local coordinates of each object in the local coordinate system of the circular plane to obtain the scaled coordinates: Where: P" represents the local coordinates of the object before scaling in the local coordinate system of the circular plane; P represents the local coordinates of the scaled object in the local coordinate system of the circular plane; P center Represents the local coordinates of the center of the circular plane in the local coordinate system of the circular plane; O far Represents the local coordinates of the object farthest from the center of the circular plane in the local coordinate system of the circular plane; r sphere Represents the radius of the circular plane.

6. The method for complex scene interaction in virtual reality based on eye tracking according to claim 1, characterized in that: Also includes: Configure device controllers; The device controller has an eye tracking start button event, a projection button event, and an interaction button event; monitoring button events of the device controller, and when the eye tracking start button event is detected, tracking the user's eye movement data, determining the user's focus area in the virtual reality scene in real time, and then determining all objects within the user's focus area, until the projection button event is detected, stopping tracking the user's eye movement data; When the projection button event is detected, all objects in the current user focus area are projected onto a circular plane suspended in front of the user while maintaining their relative positions from the user's perspective. Furthermore, the positions of the objects projected onto the circular plane are intelligently adjusted so that all objects are separated from each other and do not overlap. When the interactive button event is detected, it is determined whether there is an interactive object in the interactive area based on collision detection. If so, the position of the interactive object is changed to the hand interaction area, thereby completing a selection and interaction; if not, the user is prompted to repeat the selection until there is an interactive object in the hand interaction area.

7. A device for implementing the eye tracking-based virtual reality complex scene interaction method according to any one of claims 1 to 6, characterized in that: include: A user eye movement data tracking unit, configured to track the user's eye movement data, determine the user's focus area in the virtual reality scene in real time, and further determine all objects within the user's focus area; a projection unit, configured to project all objects within the user's focus area onto a circular plane suspended in front of the user while maintaining their relative positional relationships from the user's perspective, and intelligently adjust the positions of the objects projected onto the circular plane so that all objects are separated from each other and do not overlap; The interaction unit is used to select and interact with objects in the circular plane to achieve complex virtual reality scene interaction based on eye tracking.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for complex scene interaction in virtual reality based on eye tracking as described in any one of claims 1 to 6 is implemented.

Images