A method, system and device for identifying eye movement fixation points in a three-dimensional environment

Abstract

The present invention belongs to the field of gaze point recognition, and specifically relates to a method, system and device for eye movement gaze point recognition in a three-dimensional environment. It aims to solve the problems of insufficient adaptability of three-dimensional space under the constraints of a two-dimensional plane and the lack of an error correction mechanism. The present invention includes: constructing a three-dimensional gaze cone model of the user and a dynamic scene model with semantic labels, screening candidate targets based on the intersection of geometric bounding boxes, calculating the object score based on the intersection volume ratio and dynamic weight, and using a time series analysis model to process time series data, and finally identifying the target object that the user is looking at. The present invention achieves precise positioning and error correction of gaze points in three-dimensional scenes by integrating spatial geometric relationships, semantic weights and temporal continuity analysis.

Description

Technical Field

[0001] The present invention belongs to the field of gaze point recognition, and in particular relates to a method, system and device for recognizing eye movement gaze points in a three-dimensional environment. Background Art

[0002] During flight and training, pilots need to pay attention to the surroundings, including the instruments directly ahead, the overhead control panel, the central console, and the environment outside the forward side windows. Eye trackers cannot cover all of these areas. Furthermore, assessing the pilot's attention distribution is an essential component of flight training. In recent years, eye tracking technology has been widely used in fields such as human-computer interaction, psychology research, and virtual reality. Current mainstream eye trackers detect gaze points based on infrared light sources and image analysis. They project infrared light onto the eye, using a high-speed camera to capture the reflected image of the sclera, iris, and pupil. Key features such as the pupil center and corneal reflection point are then extracted, and gaze direction is determined by calculating the eye movement vector. To achieve spatial localization, existing technologies typically define a virtual display coordinate system in the upper left corner of the screen (stimulus display area), with the X-axis extending horizontally to the right, the Y-axis extending vertically downward (with smaller Y values ​​at the top of the screen and larger Y values ​​at the bottom), and the Z-axis perpendicular to the screen plane, pointing toward the subject. The Z value of the screen position is always zero. This coordinate system maps the gaze point onto a two-dimensional plane and combines eye position and gaze direction parameters to implement basic interactive functions.

[0003] However, the above technical solutions have significant limitations:

[0004] Insufficient adaptability to three-dimensional space within the constraints of a two-dimensional plane: Existing virtual display coordinate systems can only represent gaze points within a two-dimensional plane and cannot accurately identify the depth information (Z-axis direction) of the gaze target in a real three-dimensional environment (such as a virtual reality scene, a multi-screen interactive system, or a spatial projection interface). This results in gaze point recognition being limited to screen surface projection, making it difficult to meet the requirements of scenarios such as augmented reality and three-dimensional modeling. Therefore, based on the current three-dimensional simulation training environments of simulators with circular screens, column screens, and LED displays, there is a lack of methods to identify pilots' gaze points in training scenarios. This makes it impossible to evaluate pilots' attention allocation during training, making it difficult to identify problems with pilots' attention allocation, thereby reducing the quality of training.

[0005] Lack of error correction mechanism: Limited by hardware accuracy (such as infrared light source scattering, camera resolution, and ambient light interference) and algorithm robustness (such as pupil ellipse fitting deviation and corneal reflection point drift), existing technologies lack a dynamic error compensation mechanism. Especially under long-term use or complex environments (such as slight head movements or users wearing glasses), gaze point positioning errors will continue to accumulate, affecting interaction accuracy and user experience.

[0006] Based on this, the present invention proposes a method, system and device for identifying eye movement gaze points in a three-dimensional environment. Summary of the Invention

[0007] In order to solve the above-mentioned problems in the prior art, namely, insufficient three-dimensional spatial adaptability under two-dimensional plane constraints and the lack of error correction mechanism, the present invention provides a method, system and device for eye movement gaze point recognition in a three-dimensional environment.

[0008] In a first aspect, the present invention provides a method for identifying eye gaze points in a three-dimensional environment, the method comprising:

[0009] Construct a three-dimensional gaze cone model based on the user's eye position and gaze direction vector;

[0010] Constructing a dynamic three-dimensional scene model containing semantic labels and assigning a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being looked at in the scene;

[0011] Obtaining the geometric bounding box of each object in the 3D scene model, and calculating the intersection of the geometric bounding box and the 3D gaze cone model. If at least one vertex in the geometric bounding box of an object is within the 3D gaze cone model, then the object is included in the candidate gaze target set;

[0012] The intersection volume of the candidate target and the three-dimensional gaze cone model is calculated, and a weight coefficient is generated based on the ratio of the intersection volume to the target volume. The weight coefficient is combined with the dynamic weight of the object to obtain the final score. The score sequence within the time window is processed through a timing analysis model to output the target object that the user is looking at.

[0013] Furthermore, the three-dimensional gaze cone model is constructed by:

[0014] Obtaining the user's eye position as the vertex of the three-dimensional gaze cone model;

[0015] Obtaining a sight line direction vector as the central axis of the three-dimensional gaze cone model;

[0016] determining a half-opening angle of the three-dimensional gaze cone model according to the measurement accuracy of the eye tracker and the line of sight diffusion angle;

[0017] Calculating the gaze distance of the three-dimensional gaze cone model in three-dimensional space by binocular disparity calculation or preset distance interception method;

[0018] A three-dimensional gaze cone model is constructed based on the vertex, central axis, half-opening angle and gaze distance.

[0019] Furthermore, the semantic tags include at least one of airport buildings and equipment, environmental elements, and urban elements.

[0020] Furthermore, the intersection of the geometric bounding box and the three-dimensional gaze cone model is calculated by:

[0021] Calculate the direction vectors of the vertices of the target object's geometric bounding box relative to the user's eye position;

[0022] Determining whether the angle between the direction vector and the sight direction vector does not exceed the half-open angle of the three-dimensional gaze cone model;

[0023] If the included angles corresponding to all vertices of the geometric bounding box are greater than the half-open angle, the object is excluded;

[0024] If the included angle corresponding to at least one vertex in the geometric bounding box does not exceed the half-open angle, the object is included in the candidate gaze target set.

[0025] Furthermore, the geometric bounding box includes an axis-aligned bounding box or a polygonal mesh.

[0026] Furthermore, the weight coefficient is combined with the dynamic weight of the object to obtain the final score, and the method is as follows:

[0027] Calculating the ratio of the intersection volume to the sum of the target object volume and the three-dimensional gaze cone model volume to obtain a weight coefficient;

[0028] The weight coefficient is multiplied by the dynamic weight of the object to obtain a final score.

[0029] Furthermore, the half-open angle is calculated as follows:

[0030] Calculate the maximum deviation angle based on the measurement accuracy error of the eye tracker;

[0031] The maximum deviation angle is added to the line of sight diffusion angle to obtain the half-opening angle.

[0032] Furthermore, the score sequence within the time window is processed by the time series analysis model to output the target object that the user is looking at. The method is as follows:

[0033] The time series analysis model is a long short-term memory network. The input of the time series analysis model is the score sequence of candidate objects in the time window, and the output is the probability distribution of each object being watched.

[0034] Another aspect of the present invention provides a system for identifying eye gaze points in a three-dimensional environment. The system is based on a method for identifying eye gaze points in a three-dimensional environment and includes:

[0035] a three-dimensional gaze cone model construction module configured to construct a three-dimensional gaze cone model based on the user's eye position and the sight direction vector;

[0036] a weight assignment module configured to construct a dynamic three-dimensional scene model including semantic labels and assign a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being gazed at in the scene;

[0037] a candidate gaze target set construction module configured to obtain a geometric bounding box of each object in the three-dimensional scene model, and calculate the intersection of the geometric bounding box and the three-dimensional gaze cone model; if at least one vertex in the geometric bounding box of an object is located within the three-dimensional gaze cone model, the object is included in the candidate gaze target set;

[0038] The target object generation module is configured to calculate the intersection volume of the candidate target and the three-dimensional gaze cone model, generate a weight coefficient based on the ratio of the intersection volume to the target volume, combine the weight coefficient with the dynamic weight of the object to obtain a final score, and process the score sequence within the time window through a timing analysis model to output the target object that the user is looking at.

[0039] A third aspect of the present invention provides an electronic device, comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to at least one of the processors; wherein,

[0042] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye movement gaze point recognition method in a three-dimensional environment.

[0043] Beneficial effects of the present invention:

[0044] Accurately identify gaze points in three-dimensional space: By constructing a three-dimensional gaze cone model and combining the user's line of sight direction vector with the eye position, the constraints of the traditional two-dimensional plane coordinate system can be broken through. This can directly represent the gaze direction and potential gaze area in three-dimensional space, effectively supporting the depth information recognition needs in three-dimensional scenes such as virtual reality, multi-screen interaction, and spatial projection, and solving the problem that traditional technology can only map screen surface projections.

[0045] Dynamic scene weighting and semantic fusion: By assigning dynamic weights to objects in 3D scenes and associating semantic labels, combined with a comprehensive scoring mechanism that uses the intersection volume of candidate objects and dynamic weights, the system significantly improves the accuracy of identifying fixated objects in complex scenes. For example, in dynamic interactive scenes, even if the geometric bounding box of a high-weighted object partially enters the fixation cone due to device error, semantic weight compensation can still prioritize recognition, reducing the false positive rate.

[0046] Error self-correction and enhanced robustness: Utilizing a time series analysis model to smooth the sequence of gaze target scores within a time window, this method suppresses transient errors caused by hardware noise, momentary occlusion, or slight head movements, enabling dynamic error correction. Furthermore, the intersection volume calculation method of the geometric bounding box and the gaze cone quantifies the spatial correlation of gaze targets, avoiding the false or missed detections often associated with traditional thresholding methods due to fluctuations in device accuracy.

[0047] Computational efficiency and scene adaptability optimization: A fast intersection detection algorithm based on geometric bounding boxes can efficiently screen candidate gaze targets in three-dimensional scenes, reducing real-time computing load; combined with a hierarchical scoring mechanism with dynamic weights, it can adapt to both static and dynamic scenes, expanding the application scope of the technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 This is a schematic diagram of the process flow of an eye movement fixation point recognition method in a three-dimensional environment according to the present invention;

[0050] Figure 2 It is a schematic diagram of a process of constructing a three-dimensional gaze cone model in a method for identifying eye movement gaze points in a three-dimensional environment according to the present invention;

[0051] Figure 3 It is a schematic diagram of the intersection calculation process of the geometric bounding box and the three-dimensional gaze cone model in the eye movement gaze point recognition method in a three-dimensional environment of the present invention. DETAILED DESCRIPTION

[0052] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0053] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] A first embodiment of the present invention provides a method for identifying eye movement fixation points in a three-dimensional environment, the method comprising:

[0055] Step S10, constructing a three-dimensional gaze cone model based on the user's eye position and sight direction vector;

[0056] Step S20, constructing a dynamic three-dimensional scene model including semantic labels, assigning a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being looked at in the scene;

[0057] Step S30: obtaining a geometric bounding box of each object in the 3D scene model, and calculating the intersection of the geometric bounding box and the 3D gaze cone model. If at least one vertex in the geometric bounding box of an object is within the 3D gaze cone model, the object is included in the candidate gaze target set.

[0058] Step S40, calculate the intersection volume of the candidate target and the three-dimensional gaze cone model, generate a weight coefficient based on the ratio of the intersection volume to the target volume, combine the weight coefficient with the dynamic weight of the object to obtain the final score, and process the score sequence in the time window through the timing analysis model to output the target object that the user is looking at.

[0059] In order to more clearly illustrate the eye movement fixation point recognition method in a three-dimensional environment of the present invention, the following is combined with Figure 1 Each step in the embodiment of the present invention is described in detail, including step S10 to step S40, and each step is described in detail as follows:

[0060] Step S10, constructing a three-dimensional gaze cone model based on the user's eye position and sight direction vector;

[0061] See also Figure 2 In this embodiment, the three-dimensional gaze cone model is constructed by:

[0062] Step S11, obtaining the user's eye position as the vertex of the three-dimensional gaze cone model;

[0063] Step S12, obtaining a sight line direction vector as the central axis of the three-dimensional gaze cone model;

[0064] Step S13, determining the half-opening angle of the three-dimensional gaze cone model according to the measurement accuracy of the eye tracker and the line of sight diffusion angle;

[0065] Step S14, calculating the gaze distance of the three-dimensional gaze cone model in the three-dimensional space by binocular disparity calculation or a preset distance interception method;

[0066] Step S15: constructing a three-dimensional gaze cone model based on the vertex, the central axis, the half-opening angle, and the gaze distance.

[0067] In this embodiment, the half-open angle is calculated as follows:

[0068] Calculate the maximum deviation angle based on the measurement accuracy error of the eye tracker;

[0069] The maximum deviation angle is added to the line of sight diffusion angle to obtain the half-opening angle.

[0070] Specifically, in a dynamic 3D scene, the line of sight provided by the eye tracker is usually composed of two elements: a starting point and a direction vector. The starting point is generally the position of the user's eyes (such as the center of the eyeball or the midpoint where the two eyes converge), indicating the source of the line of sight; the direction vector indicates the direction of the eye's gaze, which is a unit vector starting from the starting point and pointing in the direction the user is looking. With the help of a non-contact eye tracker, this line of sight ray can be obtained at every moment. ,in is the eye position, is the direction vector of the sight line.

[0071] Ideally, a gaze ray would accurately pass through the target object or point the user is looking at. However, in practice, due to factors such as eye tracker accuracy limitations and errors in head motion compensation, the measured gaze direction has a certain degree of uncertainty. This means that the actual gaze direction may deviate from the measured direction by a small angle. Therefore, simply representing the gaze direction with an infinitely extended ray may not adequately represent the true gaze point location.

[0072] To better characterize the uncertainty of gaze direction, we introduce the concept of a gaze cone. A gaze cone is a conical volume surrounding the measured gaze ray. It defines a conical field of view in space, representing the area where the eye may be looking. Constructing this cone requires considering two parameters: gaze distance and gaze uncertainty.

[0073] Gaze distance (or focus estimate) refers to the depth at which the eye tracker infers the intersection of gaze rays in space. If the system can estimate the distance at which the user is gazing (e.g., focus distance calculated from binocular disparity), a preliminary gaze point can be determined at that distance. Otherwise, a default distance can be chosen (e.g., extending the gaze to a fixed depth in the scene or the position of the first object encountered) to intercept the gaze ray.

[0074] The uncertainty of gaze comes from the measurement error of the eye tracker and the residual error of head motion compensation. Usually the eye tracker will give an accuracy index (such as ±1°), which indicates the maximum angle that the gaze direction may deviate from the true direction. This uncertainty angle reflects the tolerance of the gaze direction, which we use represents the maximum deviation angle + the line of sight divergence angle (i.e. the half-opening angle of the cone). For example, if the eye tracker accuracy is approximately , the sight diffusion angle is , then it is advisable As the angular parameter for constructing the gaze cone.

[0075] Combining the above two factors determines the shape of the gaze cone: if the gaze distance is known to be , then at the distance for Uncertainty of sight will correspond to a circular area centered on the expected gaze point, with a radius of approximately That is, the farther the distance, the larger the spatial area covered by the uncertainty angle; conversely, at close distances, the area covered by the uncertainty angle is smaller.

[0076] Geometrically, the foveation cone can be defined by the cone's vertex, axis, and angle:

[0077] Cone Vertex: Take the eye position As the vertex of the cone.

[0078] Axis: The unit vector in the direction of the line of sight d As the central axis of the cone.

[0079] Half-Angle: The maximum deviation of the cone determined by the eye tracker's measurement accuracy constitutes the half-opening angle of the cone. (i.e. the angle between the axis and the side of the cone).

[0080] According to the above definition, the conditions for determining the interior points of a cone can be described in algebraic form. Let any point in space be , whose direction vector relative to the eye is .point is inside the cone if and only if The angle between the axis of sight d is not greater than Using the dot product to express this condition, we can get the judgment formula:

[0081] ;

[0082] in That is the vector The direction of the unit's line of sight d The cosine of the angle. If the value is not less than (Equivalent to an angle no greater than ), then it means In the range of the cone of gaze, otherwise it is outside the cone. For vertices along the sight axis d Open half angle A "viewing light cone" covers all possible real gaze directions.

[0083] Step S20, constructing a dynamic three-dimensional scene model including semantic labels, assigning a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being looked at in the scene;

[0084] In this embodiment, the semantic tags include at least one of airport buildings and equipment, environmental elements, and urban elements.

[0085] The airport buildings and equipment include at least runways, terminals, PAPI lights, airport vehicles, other aircraft and runway edge lights; environmental elements include at least mountains, sea surfaces, lake surfaces, river surfaces, clouds and vegetation; and urban elements include at least buildings, bridges, cars, etc.

[0086] Step S20 of the present invention essentially involves constructing a data twin 3D scene and a dynamic weighting mechanism. A digital twin is a detailed 3D model representing the real-world environment, including all relevant objects. It is created using 3D scanning technology or manual modeling. Computer vision techniques are used to assign semantic labels to each 3D object. For example, object classification (e.g., runway, terminal, PAPI light) is achieved using 3D object detection algorithms or manual annotation. Scene types (e.g., cruising, landing, taxiing, etc.) are identified using a machine learning model based on the characteristics of object arrangement and labels.

[0087] Based on the identified scene, each object is assigned a dynamic weight, reflecting its likelihood of being gazed at within that scene. For example, during a night flight landing, the runway centerline might have a weight of 0.5, the PAPI lights 0.3, and the runway edge lights 0.2. These weights can be predefined (based on statistics or expert knowledge) or learned from user gaze data.

[0088] Step S30: obtaining a geometric bounding box of each object in the 3D scene model, and calculating the intersection of the geometric bounding box and the 3D gaze cone model. If at least one vertex in the geometric bounding box of an object is within the 3D gaze cone model, the object is included in the candidate gaze target set.

[0089] See also Figure 3 In this embodiment, the intersection of the geometric bounding box and the three-dimensional gaze cone model is calculated by:

[0090] Step S31, calculating the direction vectors of the vertices of the geometric bounding box of the target object relative to the user's eye position;

[0091] Step S32, determining whether the angle between the direction vector and the sight direction vector does not exceed the half-open angle of the three-dimensional gaze cone model;

[0092] Step S33: if the included angles corresponding to all vertices of the geometric bounding box are greater than the half-open angle, then the object is excluded;

[0093] Step S34: If the angle corresponding to at least one vertex in the geometric bounding box does not exceed the half-open angle, the object is included in the candidate gaze target set.

[0094] The geometric bounding box includes an axis-aligned bounding box or a polygonal mesh.

[0095] This embodiment uses the gaze cone to efficiently filter out objects of interest in the scene that may be gazed at by the user. The method is to check whether the bounding volume (such as axis-aligned bounding box AABB or its polygonal mesh) of each potential target (object or area) intersects with the gaze cone. A simple judgment method is to calculate the direction vector of the target relative to the eye and determine whether the angle between it and the gaze direction is within 0. If the entire geometry of the target object falls outside the cone (that is, all key points have an angle greater than ), then the object is almost impossible to be the focus of attention and can be excluded; on the contrary, as long as any point of the target is within the cone, it means that the sight cone has covered the object and it should be included in the candidate object of interest.

[0096] Apply the above cosine condition to the 8 vertices of each object's AABB: if any vertex satisfies , then the object is determined to intersect with the gaze cone and needs to be included in the candidate list. Further, we can perform more refined detection on objects that pass this preliminary screening (for example, checking the intersection of the object's fine grid and the cone) to determine which part of the object the gaze may actually fall on.

[0097] In this embodiment, the weight coefficient is combined with the dynamic weight of the object to obtain the final score, and the method is as follows:

[0098] Calculating the ratio of the intersection volume to the sum of the target object volume and the three-dimensional gaze cone model volume to obtain a weight coefficient;

[0099] The weight coefficient is multiplied by the dynamic weight of the object to obtain a final score.

[0100] The intersection volume of the cone and each potential target object is calculated. The area weight coefficient is defined as: intersection volume / (target volume + cone volume). This coefficient is multiplied by the object's dynamic weight to obtain the final score. For example, if the intersection volume is 10, the target volume is 20, and the cone volume is 15, the coefficient is 10 / (20 + 15) = 0.25; if the dynamic weight is 0.5, the final score is 0.125.

[0101] Step S40, calculate the intersection volume of the candidate target and the three-dimensional gaze cone model, generate a weight coefficient based on the ratio of the intersection volume to the target volume, combine the weight coefficient with the dynamic weight of the object to obtain the final score, and process the score sequence in the time window through the timing analysis model to output the target object that the user is looking at.

[0102] In this embodiment, the score sequence within the time window is processed by a time series analysis model to output the target object that the user is looking at. The method is as follows:

[0103] The time series analysis model is a long short-term memory network. The input of the time series analysis model is the score sequence of candidate objects in the time window, and the output is the probability distribution of each object being watched.

[0104] Within a set detection time window (e.g., 200ms), a score sequence is collected for each object. These sequences are processed using an LSTM network, which takes the time series scores as input and outputs the probability distribution of the object being gazed. The LSTM model is trained on labeled gaze data, which includes gaze sequences of known target objects and captures spatiotemporal features (e.g., dynamic changes in gaze position and duration).

[0105] Based on the score sequence, the spatiotemporal features (such as the dynamic changes in gaze position and time) are integrated to finally determine the object the user is looking at.

[0106] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0107] A second embodiment of the present invention provides a system for recognizing eye gaze points in a three-dimensional environment, based on a method for recognizing eye gaze points in a three-dimensional environment. The system includes:

[0108] a three-dimensional gaze cone model construction module configured to construct a three-dimensional gaze cone model based on the user's eye position and the sight direction vector;

[0109] a weight assignment module configured to construct a dynamic three-dimensional scene model including semantic labels and assign a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being gazed at in the scene;

[0110] a candidate gaze target set construction module configured to obtain a geometric bounding box of each object in the three-dimensional scene model, and calculate the intersection of the geometric bounding box and the three-dimensional gaze cone model; if at least one vertex in the geometric bounding box of an object is located within the three-dimensional gaze cone model, the object is included in the candidate gaze target set;

[0111] The target object generation module is configured to calculate the intersection volume of the candidate target and the three-dimensional gaze cone model, generate a weight coefficient based on the ratio of the intersection volume to the target volume, combine the weight coefficient with the dynamic weight of the object to obtain a final score, and process the score sequence within the time window through a timing analysis model to output the target object that the user is looking at.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0113] It should be noted that the above embodiment provides an eye movement gaze point recognition system in a three-dimensional environment, and only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention.

[0114] An electronic device according to a third embodiment of the present invention includes:

[0115] at least one processor; and

[0116] a memory communicatively connected to at least one of the processors; wherein,

[0117] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye movement gaze point recognition method in a three-dimensional environment.

[0118] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned eye movement gaze point recognition method in a three-dimensional environment.

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0121] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0122] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0123] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for identifying eye gaze points in a three-dimensional environment, characterized in that: The method includes: Construct a three-dimensional gaze cone model based on the user's eye position and gaze direction vector; Constructing a dynamic three-dimensional scene model containing semantic labels and assigning a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being looked at in the scene; Obtaining the geometric bounding box of each object in the 3D scene model, and calculating the intersection of the geometric bounding box and the 3D gaze cone model. If at least one vertex in the geometric bounding box of an object is within the 3D gaze cone model, then the object is included in the candidate gaze target set; The intersection volume of the candidate target and the three-dimensional gaze cone model is calculated, and a weight coefficient is generated based on the ratio of the intersection volume to the target volume. The weight coefficient is combined with the dynamic weight of the object to obtain the final score. The score sequence within the time window is processed through a timing analysis model to output the target object that the user is looking at.

2. The method for identifying eye gaze points in a three-dimensional environment according to claim 1, wherein: The three-dimensional gaze cone model is constructed by: Obtaining the user's eye position as the vertex of the three-dimensional gaze cone model; Obtaining a sight line direction vector as the central axis of the three-dimensional gaze cone model; determining a half-opening angle of the three-dimensional gaze cone model according to the measurement accuracy of the eye tracker and the line of sight diffusion angle; Calculating the gaze distance of the three-dimensional gaze cone model in three-dimensional space by binocular disparity calculation or preset distance interception method; A three-dimensional gaze cone model is constructed based on the vertex, central axis, half-opening angle and gaze distance.

3. The method for identifying eye movement gaze points in a three-dimensional environment according to claim 1, wherein: The semantic tags include at least one of airport buildings and equipment, environmental elements, and urban elements.

4. The method for identifying eye movement fixation points in a three-dimensional environment according to claim 2, wherein: The intersection of the geometric bounding box and the three-dimensional gaze cone model is calculated by: Calculate the direction vectors of the vertices of the target object's geometric bounding box relative to the user's eye position; Determining whether the angle between the direction vector and the sight direction vector does not exceed the half-open angle of the three-dimensional gaze cone model; If the included angles corresponding to all vertices of the geometric bounding box are greater than the half-open angle, the object is excluded; If the included angle corresponding to at least one vertex in the geometric bounding box does not exceed the half-open angle, the object is included in the candidate gaze target set.

5. The method for identifying eye gaze points in a three-dimensional environment according to claim 4, wherein: The geometric bounding box includes an axis-aligned bounding box or a polygonal mesh.

6. The method for identifying eye movement gaze points in a three-dimensional environment according to claim 1, wherein: The weight coefficient is combined with the dynamic weight of the object to obtain the final score, which is as follows: Calculating the ratio of the intersection volume to the sum of the target object volume and the three-dimensional gaze cone model volume to obtain a weight coefficient; The weight coefficient is multiplied by the dynamic weight of the object to obtain a final score.

7. The method for identifying eye movement gaze points in a three-dimensional environment according to claim 2, wherein: The half-open angle is calculated as follows: Calculate the maximum deviation angle based on the measurement accuracy error of the eye tracker; The maximum deviation angle is added to the line of sight diffusion angle to obtain the half-opening angle.

8. The method for identifying eye movement gaze points in a three-dimensional environment according to claim 1, wherein: The score sequence within the time window is processed by the time series analysis model to output the target object that the user is looking at. The method is as follows: The time series analysis model is a long short-term memory network. The input of the time series analysis model is the score sequence of candidate objects in the time window, and the output is the probability distribution of each object being watched.

9. A system for recognizing eye gaze points in a three-dimensional environment, based on the method for recognizing eye gaze points in a three-dimensional environment according to any one of claims 1 to 8, characterized in that: The system includes: a three-dimensional gaze cone model construction module configured to construct a three-dimensional gaze cone model based on the user's eye position and the sight direction vector; a weight assignment module configured to construct a dynamic three-dimensional scene model including semantic labels and assign a dynamic weight to each object in the scene, wherein the dynamic weight reflects the probability of the object being gazed at in the scene; a candidate gaze target set construction module configured to obtain a geometric bounding box of each object in the three-dimensional scene model, and calculate the intersection of the geometric bounding box and the three-dimensional gaze cone model; if at least one vertex in the geometric bounding box of an object is located within the three-dimensional gaze cone model, the object is included in the candidate gaze target set; The target object generation module is configured to calculate the intersection volume of the candidate target and the three-dimensional gaze cone model, generate a weight coefficient based on the ratio of the intersection volume to the target volume, combine the weight coefficient with the dynamic weight of the object to obtain a final score, and process the score sequence within the time window through a timing analysis model to output the target object that the user is looking at.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the eye movement gaze point recognition method in a three-dimensional environment as described in any one of claims 1-8.

Images