Spatial gaze tracking method and apparatus based on a human eye model

Abstract

The application provides a spatial gaze tracking method and device based on a human eyeball model, comprising: obtaining a target image to be processed; generating an eye key point detection result and an eye region segmentation result according to the target image to be processed; inputting the eye key point detection result and the eye region segmentation result into a pre-constructed ellipse fitting model to obtain an ellipse fitting result; obtaining a rotation center of an eyeball based on a first preset formula using a first preset rule according to the ellipse fitting result to determine an optical axis parameter; determining a visual axis based on a second preset formula using a second preset rule according to the optical axis parameter, and further determining a spatial position of a visual line. The application obtains the visual axis through four steps of eye key point detection, eye region segmentation, ellipse fitting and optical axis calculation based on a new visual axis model, and further determines the spatial position of the visual line, so that the spatial gaze tracking with higher accuracy, lower error, lower cost and more convenience is realized.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a spatial gaze tracking method and apparatus based on a human eye model. Background Technology

[0002] The Metaverse, as a paradigm of the next-generation internet, is gaining attention for its ubiquity and immersive experiences. Incorporating elements such as 5G, AI, blockchain, and content creation, its core is the virtual experience of Extended Reality (XR). Therefore, Extended Reality (XR), which provides a fusion and interaction between the physical and virtual worlds, holds the promise of becoming a fork in the road connecting the metaverse and the physical world through 3D displays and interactions. While XR is prevalent in various fields such as healthcare, industry, and entertainment, 3D display and interaction remain the last mile problem in practical application.

[0003] The main challenge behind 3D display and interaction is spatial gaze tracking, a key enabling technology that is currently far from being applicable. This is because spatial gaze is defined based on the convergence of the binocular visual axes, such as... Figure 1 As shown in (a)-1(c), existing methods typically employ approximate models, such as approximating the visual axis to other easily perceived axes (e.g., the pupillary axis). However, this compromises accuracy. Figure 2 As shown, different approximations will produce different errors for the same viewing angle, especially for the depth required for 3D display. Some experiments have shown that the depth perception error can be as high as 303.90 cm.

[0004] In summary, existing spatial gaze tracking technologies suffer from low accuracy. Summary of the Invention

[0005] This invention provides a spatial gaze tracking method and apparatus based on a human eye model to address the shortcomings of low accuracy in existing spatial gaze tracking technologies, thereby achieving spatial gaze tracking with higher accuracy and lower error.

[0006] This invention provides a spatial gaze tracking method based on a human eye model, comprising:

[0007] Acquire the target image to be processed;

[0008] Based on the target image to be processed, generate eye key point detection results and eye region segmentation results;

[0009] Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset.

[0010] The eye key point detection results and the eye region segmentation results are input into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset.

[0011] Based on the ellipse fitting result, the rotation center of the eyeball is obtained using the first preset rule and the first preset formula, so as to determine the optical axis parameters;

[0012] Based on the optical axis parameters, the visual axis is determined using a second preset rule and a second preset formula, thereby determining the spatial position of the line of sight.

[0013] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, wherein the eye region segmentation model specifically includes:

[0014] An encoder, which is used to learn recognizable features of an image;

[0015] A decoder is used to map the features of the encoder to a high-resolution pixel space to obtain dense classification.

[0016] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, which trains an eye keypoint detection model using a pre-built neural network and a pre-built dataset, and then further includes:

[0017] The first loss is calculated using the third preset formula;

[0018] The third preset formula includes:

[0019]

[0020] Where, loss lm Denotes the first loss function. x max The x-axis value represents the maximum value of the eye key point detection result, y. max The vertical axis represents the maximum value of the eye key point detection result, x. minThe x-axis represents the minimum value of the eye key point detection result, y min o1 represents the minimum value of the vertical axis of the eye key point detection result, and o1 represents the difference between the predicted value and the actual ground value.

[0021] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, which trains an eye region segmentation model using a pre-built neural network and a pre-built dataset, and then further includes:

[0022] The weights of each value in the output probability distribution vector are calculated using the fourth preset formula.

[0023] The second loss is calculated using the fifth preset formula;

[0024] The fourth preset formula includes:

[0025]

[0026] in, d represents the weight of pixel o2. i Let σ represent the Euclidean distance from the pixel to the i-th near boundary, and σ represent the standard deviation of the normal distribution.

[0027] The fifth preset formula includes:

[0028]

[0029] Where, loss seg Denotes the second loss function, o predicted It is the predicted label for pixel o2, o goundtruth It is the ground real-time label for pixel o2. It is the weight of pixel o2.

[0030] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, wherein the detection results of eye key points and the segmentation results of eye regions are input into a pre-constructed ellipse fitting model to obtain ellipse fitting results, specifically including:

[0031] Using the results of eye keypoint detection and eye region segmentation as constraints, ellipses of the iris and pupil are fitted to obtain the target parameter x. c ,y c ,a c ,b c ,θ c ;

[0032] Using the target parameters, x is generated based on the ellipse fitting model. c ,y c ,a c ,b c ,θc The iris and pupil are oval;

[0033] Where, x c It is the x-coordinate of the center coordinate, y c It is the ordinate of the center coordinate, a c It is the length of the semi-spindle, b c It is the length of the semi-secondary axis, θ c It is the rotation angle.

[0034] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, which obtains an ellipse fitting model by training a pre-built neural network using a pre-built dataset, and then further includes:

[0035] The third loss is calculated using the sixth preset formula;

[0036] Calculate the total loss using the seventh preset formula;

[0037] The sixth preset formula includes:

[0038]

[0039] Where, loss fit This represents the third loss function, and o3 represents the actual ground value. Indicates the predicted value;

[0040] The seventh preset formula includes:

[0041] l loss =α lm loss lm +β seg loss seg +γ fit loss fit

[0042] Where, loss lm Let loss be the first loss function. seg Denotes the second loss function, loss fit Let α represent the third loss function. lm β represents the weight of the first loss function. seg γ represents the weight of the second loss function. fit This represents the weight of the third loss function.

[0043] According to the present invention, a spatial gaze tracking method based on a human eye model is provided. Based on the ellipse fitting result, the rotation center of the eyeball is obtained using a first preset rule and a first preset formula to determine the optical axis parameters. Specifically, the method includes:

[0044] The orientation of the pupil and iris in three-dimensional space is determined by using the target parameters obtained from ellipse fitting and the effective focal length of the camera, so as to obtain the first preset formula;

[0045] The rotation center of the eyeball is established by using the first preset formula, and the optical axis is obtained by connecting the center of the pupil and the rotation center of the eyeball.

[0046] The first preset formula includes:

[0047] lx + my + nz = 0

[0048] Among them l 2 +m 2 +n 2 =1, where x, y, z are the coordinates of the plane containing the pupil and iris.

[0049] According to the present invention, a spatial gaze tracking method based on a human eye model is provided, wherein the second preset formula includes:

[0050] v = Mo

[0051] Where o represents the unit vector along the optical axis, v represents the unit vector along the visual axis, and M represents the transformation vector;

[0052]

[0053] Where c represents the distance between the centers of eye rotation, α d Let β represent the first intermediate variable. d The second intermediate variable is represented by d, which represents the user's depth of vision, and a and b represent the visual axis.

[0054]

[0055] Where D and d together indicate the position of the user's gaze in space, D represents the direction of the user's gaze, d represents the depth of the user's gaze, c represents the distance between the centers of eye rotation, and A represents the rotation angle.

[0056] The present invention also provides a spatial gaze tracking device based on a human eye model, comprising:

[0057] Image acquisition unit, used to acquire the target image to be processed;

[0058] An image processing unit is used to generate eye key point detection results and eye region segmentation results based on the target image to be processed;

[0059] Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset.

[0060] An ellipse fitting unit is used to input the eye key point detection results and the eye region segmentation results into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset.

[0061] The first calculation unit is used to determine the optical axis parameters by obtaining the rotation center of the eyeball based on the ellipse fitting result using a first preset rule and a first preset formula.

[0062] The second calculation unit is used to determine the visual axis based on the optical axis parameters, using a second preset rule and a second preset formula, and then determine the spatial position of the line of sight.

[0063] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the spatial gaze tracking method based on the human eye model as described above.

[0064] This invention provides a spatial gaze tracking method and apparatus based on a human eye model. The method involves acquiring a target image to be processed; generating eye keypoint detection results and eye region segmentation results based on the target image; wherein the eye keypoint detection results are generated by inputting the target image to a pre-built eye keypoint detection model, which is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation results are generated by inputting classification results to a pre-built eye region segmentation model, where the classification results are obtained by classifying the target image according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset; the eye keypoint detection results and the eye region segmentation results are input to a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is trained on a pre-built neural network using a pre-built dataset; based on the ellipse fitting result, the rotation center of the eyeball is obtained using a first preset rule and a first preset formula to determine the optical axis parameters; based on the optical axis parameters, the visual axis is determined using a second preset rule and a second preset formula, thereby determining the spatial position of the gaze. Based on a new visual axis model, this invention obtains the visual axis through four steps: eye key point detection, eye region segmentation, ellipse fitting, and optical axis calculation, thereby determining the spatial position of the gaze. This achieves more accurate, lower error, lower cost, and more convenient spatial gaze tracking. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0066] Figure 1 These are different visual models for gaze tracking in the spatial gaze tracking method based on the human eyeball model provided by this invention, wherein... Figure 1 (a)-1(c) are visual models of gaze tracking in the prior art. Figure 1 (d) is the visual model for gaze tracking provided by this invention;

[0067] Figure 2 This is a schematic diagram of the convergence and divergence differences in the spatial gaze tracking method based on the human eyeball model provided by the present invention;

[0068] Figure 3 This is a schematic diagram of existing 3D display and 3D interaction for adjusting the display focus;

[0069] Figure 4 This is a user study of the parallax phenomenon in the spatial gaze tracking method based on the human eye model provided by this invention;

[0070] Figure 5 This is a flowchart illustrating the spatial gaze tracking method based on a human eye model provided by the present invention.

[0071] Figure 6 This is a schematic diagram of the network structure of the eye key point detection model in an embodiment of the spatial gaze tracking method based on a human eye model provided by the present invention. Figure 6 (a) and schematic diagram of convolutional architecture Figure 6 (b);

[0072] Figure 7 This is a pixel classification illustration of an embodiment of the spatial gaze tracking method based on a human eye model provided by the present invention. Figure 7 (a) Schematic diagram of the network structure of the eye region segmentation model Figure 7 (b) and schematic diagram of upsampling convolution architecture Figure 7 (c);

[0073] Figure 8 This is a schematic diagram of the ellipse fitting model network structure of an embodiment of the spatial gaze tracking method based on the human eyeball model provided by the present invention;

[0074] Figure 9 This is a schematic diagram of the eye rotation center of an embodiment of the spatial gaze tracking method based on a human eye model provided by the present invention;

[0075] Figure 10 This is a schematic diagram of the three-dimensional and two-dimensional pupil relationship of an embodiment of the spatial gaze tracking method based on the human eyeball model provided by the present invention;

[0076] Figure 11 This is a schematic diagram of spatial gaze calculation in an embodiment of the spatial gaze tracking method based on the human eyeball model provided by the present invention;

[0077] Figure 12 This is a schematic diagram of the spatial gaze tracking device based on a human eyeball model provided by the present invention;

[0078] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0079] Figure label:

[0080] 1210: Image acquisition unit; 1220: Image processing unit; 1230: Ellipse fitting unit; 1240: First calculation unit; 1250: Second calculation unit;

[0081] 1310: Processor; 1320: Communication interface; 1330: Memory; 1340: Communication bus. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0083] Spatial gaze tracking can capture a user's visual attention and provide information about visual direction and depth. This information plays an indispensable role in 3D display and interaction. Specifically, true 3D display requires adaptive adjustment of the display focus based on the user's attention, such as... Figure 3 As shown in (a). Without spatial gaze information, XR devices must use fixed-focal-length displays to simulate differences and parallax, causing convergence conflict and leading to user fatigue or dizziness. Similarly, in 3D interaction, when spatial gaze tracking is missing, the user's interaction with virtual objects will be limited to a 2D plane. Furthermore, insufficient spatial gaze tracking reduces the effectiveness of 3D interaction. For example, virtual gaze pointers in XR support directional spatial interaction. If the spatial gaze information is monocular or inaccurate, then using such pointers to interact with virtual objects can easily lead to occlusion problems or misoperations, such as... Figure 3 As shown in (b).

[0084] Spatial gaze tracking is far more difficult than the thoroughly studied planar gaze tracking problem. Planar gaze tracking establishes a connection between eye movements and the coordinate system of the target plane. It requires the existence of a known plane (usually the display plane), which makes it impossible to provide depth information in three-dimensional space. Spatial gaze tracking uses the parallax of the two eyes to determine the gaze in space, without relying on any prior assumptions. Compared to planar gaze tracking, spatial gaze tracking simultaneously obtains both the direction and depth information of the user's visual attention, which is very difficult to obtain in reality.

[0085] The general procedure for spatial gaze tracking involves capturing images of the user's eyes with a camera, analyzing these images, and inferring the visual axis representing the user's gaze. Monocular vision can approximate the user's gaze direction using a single axis. In contrast, binocular vision can utilize the convergence of the binocular axes to precisely determine the direction and depth of the gaze. The visual model is a crucial aspect of the entire procedure.

[0086] Since spatial gaze is defined based on the convergence of the binocular visual axes, existing methods typically employ approximate models. Figure 1 (a) through (c) illustrate three approximate visual models used in most works. All models consist of two important axes of the eye: the optical axis and the visual axis. The optical axis passes through the anterior apex (A) and the center of rotation (C) of the cornea. Meanwhile, existing models use different approximations of the visual axis, which represents visual acuity. Figure 1 The model in (a) uses the line (C) connecting the object (O) and the center of eye rotation as the visual axis, which we call the central axis. Its angle with the optical axis is η. Figure 1 In (b), the visual axis is approximated as a line (E) perpendicular to the cornea (P) and passing through the center of the pupil. This line is the pupillary axis, which forms an angle κ with the actual visual axis. We denote the angle between this angle and the optical axis as γ. Figure 1 In (c), the line of sight connecting the object (O) and the center of the pupil (E) is also considered as an approximation of the visual axis in the figure, which forms an angle ω with the optical axis.

[0087] However, these visual models cannot accurately depict human vision. For example... Figure 2 As shown, different approximations will produce different errors for the same viewing angle, especially at the depth required for 3D display.

[0088] To demonstrate that these visual models cannot accurately depict human vision, the following experiment was conducted, measuring the left eye as an example (the right eye is symmetrical). The experimental setup is as follows: Figure 4 As shown in (a)

[0089] 1. Setup Steps: Place a monitor connected to a laptop behind a table. The frame on the monitor is divided into two parts; the left part is displayed in black, and the right part in red. The dividing line between the two parts is initially positioned in the center of the monitor and can be controlled using the keyboard wirelessly connected to the laptop. Pressing the left or right arrow keys will move the dividing line one pixel to the left or right. Place an obstacle in front of the volunteer's eyes, and place a headrest in front of the obstacle to stabilize the volunteer's head and block the right eye. This way, when the volunteer looks straight ahead, they will see the center of the monitor with their left eye. Additionally, place a target with a red cross in the center on the left side of the table to guide the volunteer's line of sight.

[0090] 2. Experimental Procedure: Six volunteers (aged 23-27) participated. At the start of the experiment, the volunteers sat at a table, their heads fixed to a support frame so that their view was partially obstructed when they looked straight ahead. They focused on the precise center of the screen. Then, the volunteers turned their eyes to look at the target while the support frame remained fixed to their heads. The volunteers then noticed a red frame appearing on the monitor, partially obstructed. The volunteers then manipulated the division between the two parts using the keyboard, completely obscuring the red portion. The distance the division line moved was recorded.

[0091] 3. Experimental results: This experimental phenomenon is caused by parallax. Figure 4 (b) explains why volunteers could perceive the extra portion within the red box while keeping their heads fixed and simply rotating their eyes. Due to the structure of the eyeball, the human eye refracts incident light. Like a camera lens, the human eye has three pairs of cardioid points: the front / back focal point, the front / back principal point, and the front / back nodal point. Here, we focus on the front nodal point, also known as the parallax-free point in photography. This point is characterized by the fact that no parallax occurs in the image when the eye / camera rotates around it. This is primarily due to the optical properties of the front and back nodals: any light entering the front nodal point will exit at the back nodal point at the same angle. The focal point is the area on the retina in the human visual system where vision is most acute. It is also the central area of ​​visual imaging, and it does not coincide with the center of the retina. Therefore, light entering the eye from the fixation point exits at the back nodal point and reaches the eye socket.

[0092] Experimental observations show that the anterior node is located in front of the center of rotation in the eyeball. To further determine its position, the distance NC between the anterior node and the center of rotation is calculated. This can be obtained by solving two similar triangles, such as... Figure 4 As shown in (b):

[0093]

[0094] Where, d NC The distance the dividing line moves is represented by θ, where NC represents the distance between the previous node and the rotation center. NC L1 represents the eye rotation angle, L2 represents the distance from the eye to the monitor, and L1 represents the distance from the eye to the obstruction.

[0095] Then NC can be calculated as:

[0096]

[0097] Where, d NC The distance the dividing line moves is represented by θ, where NC represents the distance between the previous node and the rotation center. NCL1 represents the eye rotation angle, L2 represents the distance from the eye to the monitor, and L1 represents the distance from the eye to the obstruction.

[0098] Figure 4 (c) shows the results of the NC distance measurement, with an average value of 8.45 mm and a range of 6.35 mm to 9.59 mm (a variation of 3.23 mm).

[0099] The parallax phenomenon observed in the experiment only occurs when light rays from an object bypass the obstacle and enter the user's eye; that is, it occurs when the incident light ray intersects the optical axis in front of the center of the eye (e.g., ...). Figure 4 (b) shows). In current tracking models, for example... Figure 1 As shown in (b), the intersection of the pupillary axis (e.g.) and the optical axis is the center of the pupil. Considering the typical size of the eyeball, this does not match the experimental results. That is, the current gaze tracking model is basically unsuitable. Furthermore, since the pupillary axis enters the cornea perpendicularly and passes through the center of the pupil, it is difficult to guarantee that it will eventually fall into the eye socket.

[0100] In reality, the visual axis is not a ray determined by visible points. Instead, it is composed of segmented lines determined by invisible points. How to obtain information about invisible points related to the visual axis? This becomes a new problem when applying the new model for spatial gaze tracking.

[0101] Based on this, the spatial gaze tracking method based on a human eye model provided by this invention proposes a new visual axis model, such as... Figure 1 As shown in (d), the visual axis connects the fixed point (O) on the object to the front node (N), then leaves from the back node and lands on the concave point (F). This not only explains the experimental phenomenon described above but also contributes to accurate spatial gaze tracking. Figure 2 As shown, it is clear that different models exhibit different depth perceptions when the eyes are looking directly forward. Furthermore, both angle and depth perception differ when the eyes are not looking directly forward. Simultaneously, this invention provides a spatial gaze tracking method based on a novel visual axis model.

[0102] The following is combined with Figures 1-11 This invention describes a spatial gaze tracking method based on a human eye model. Figure 5 This is a flowchart illustrating the spatial gaze tracking method based on a human eye model provided by the present invention, as shown below. Figure 5 As shown, this invention provides a multi-distribution sampling method for image-text correlation based on a diffusion model, comprising:

[0103] Step 110: Obtain the target image to be processed. The target image to be processed is obtained using a camera; the image is of the eye, and this image is then used as input for subsequent steps.

[0104] Step 120: Generate eye key point detection results and eye region segmentation results based on the target image to be processed;

[0105] Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is obtained by training a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is obtained by training a pre-built neural network using a pre-built dataset.

[0106] Specifically, the following challenges may be encountered when applying the new view axis model:

[0107] Challenge 1: Key elements in the visual model are essentially virtual and invisible, making them impossible to capture directly. The convergence of the binocular visual axes determines spatial gaze. The nodal point and the center of rotation of the eyeball are key elements defining the visual axis, and they are essentially virtual and invisible. It is impossible to capture these elements directly using visual methods.

[0108] Challenge 2: XR applications place high demands on the accuracy of gaze tracking. In the case of 3D displays, the display's focal length must match the user's depth of field (DoF), thus requiring a high accuracy of 0.541°, which necessitates precise calculation of the visual axis. In 3D interactive scenarios, this requirement is similar, or even higher, depending on the density of objects in the XR display.

[0109] Challenge 3: The design should be lightweight and efficient to accommodate resource-constrained XR devices. Most extended reality devices are mobile, so the design should be as compact as possible to meet computing power and power consumption requirements. Additionally, the design needs to be efficient, i.e., low latency, to meet the real-time requirements of 3D display and interaction.

[0110] To address the aforementioned challenges, this invention proposes a multi-stage method for indirectly calculating the visual axis. First, the pupil and iris, which determine the optical axis, are calculated separately (1, 2, 3). A neural network is used to ensure accurate calculations, while network overhead is well controlled through network structure optimization. Then, the optical axis is derived by locating the pupil center and the eyeball rotation center using the elliptical coordinates of the pupil and iris (4). Finally, in a personal calibration stage, the visual axis is derived using the geometric relationship between the optical axis and the visual axis, and then spatial gaze is calculated (5).

[0111] First, eye landmark detection is performed, which is a preparatory task for ellipse fitting. It extracts eye landmarks for subsequent ellipse fitting and generates eye landmark detection results. The eye landmark detection results are generated by inputting the eye image into a pre-built eye landmark detection model; the eye landmark detection model is based on a pre-built neural network trained using a pre-built dataset.

[0112] In the design of the eye keypoint detection model, considering the significant advantages of Convolutional Neural Networks (CNNs) in image processing, the eye keypoint detection model was designed as a 13-layer CNN, with the network structure as follows: Figure 6 As shown in (a). In practical operation, considering the computational requirements of mobile platforms, the eye keypoint detection model design also referenced MobileNets and adopted depthwise separable convolution (DS Conv Block) instead of traditional convolution (Standard Conv Block) to reduce overhead. Depthwise separable convolution divides traditional convolution into depthwise convolution and pointwise convolution, as shown in (a). Figure 6 As shown in (b), to reduce computational cost. Considering the input W×H×N, output Wou×Hou×M, kernel size K×K, and the computational ratio of depthwise separable convolution to traditional convolution, the following parameters are used.

[0113]

[0114] Eye landmark detection primarily focuses on local features; therefore, the eye landmark detection model is designed with a shallow network, resulting in a relatively small receptive field. Simultaneously, skip connections are used to concatenate feature maps with different receptive fields, deriving local feature descriptions at different scales and thus improving network performance. Furthermore, skip connections prevent the network from failing to converge at the beginning of training. Specifically, three layers are concatenated and then flattened to generate eye landmarks, i.e., the eye landmark detection results, through a fully connected layer.

[0115] After the model is built, the eye keypoint detection model is trained using a pre-built dataset. The pre-built dataset can be the TEyeD dataset, which is a combination and extension of existing datasets, including NVGaze, LPW, GIW, ElSe, ExCuSe, and PNET.

[0116] In one embodiment, after inputting the eye image into a trained eye landmark detection model, a total of 50 eye landmarks are generated, of which 8 represent the pupil, 8 represent the iris, and the remaining landmarks represent the eyelids. The pupil and iris landmarks are provided for ellipse fitting. The eyelid landmarks are used to detect blinks and skip blink boxes (blinks are considered when the landmarks of the upper and lower eyelids are too close together).

[0117] Since eye landmarks are typically represented as a series of coordinate values, the training loss of the eye keypoint detection model is evaluated based on the distance between the model's predicted and actual values; this is known as the first loss. In determining the loss function, this invention adjusts the Smooth L1 loss to a more suitable loss function for evaluating the eye keypoint detection model; this is denoted as the first loss function.

[0118] Smooth L1 loss is a combination of L1-loss and L2-loss. Compared to the standard L1 loss function, it converges faster. Compared to the L2 loss function, it is significantly less sensitive to outliers and foreign factors, and is less prone to distortion because the gradient changes much smaller.

[0119] Specifically, considering the dense spatial distribution of eyemarks, the loss function is made more robust by normalizing the smooth L1 loss. As shown in the third preset formula, the modified first loss function is displayed as follows:

[0120]

[0121] Where, loss lm Denotes the first loss function. x max The x-axis value represents the maximum value of the eye key point detection result, y. max The vertical axis represents the maximum value of the eye key point detection result, x. min The x-axis represents the minimum value of the eye key point detection result, y min o1 represents the minimum value of the vertical axis of the eye key point detection result, and o1 represents the difference between the predicted value and the actual ground value.

[0122] Eye region segmentation is another preparatory step for ellipse fitting, aiming to learn pixel classification features to provide a basis for ellipse fitting. Specifically, the eye image is first classified according to pixels to obtain classification results. In some embodiments, the pixels in the eye image are divided into three categories: pupil, iris, and others (e.g., sclera, eyelids, etc.), such as... Figure 7 As shown in (a). After classification, the classification result is obtained, and the classification result is input into the eye region segmentation result generated by the pre-built eye region segmentation model.

[0123] The eye region segmentation model is based on a pre-built neural network trained on a pre-built dataset. Unlike eye keypoint detection, both local and global information are crucial for eye region segmentation. For local information, the same convolutional layers as in eye keypoint detection are used to maintain a small receptive field. For global information, many existing works propose methods to expand the receptive field by using dilated convolutions, dense connections, or pyramid-shaped blocks to extract more global features at high resolution. However, this method of directly upsampling from low-resolution features may lead to degradation of high-resolution features. Inspired by U-ne, this invention proposes a U-shaped encoder-decoder network, such as... Figure 7 As shown in (b), the eye region segmentation model consists of two main parts: an encoder and a decoder.

[0124] Figure 7 (b) The left-hand portion represents the encoder, used to learn recognizable features of the image. The downsampling block is the same depthwise separable convolutional block used in the eye keypoint detection model, such as... Figure 6 As described in (b). Along this path, the size of the feature map decreases while the number of channels increases.

[0125] Figure 7 The right side of (b) is the decoder, which maps the encoder's features to a high-resolution pixel space to obtain dense classification. Here, the decoder utilizes... Figure 7 Block (c) is shown. Deconvolution is used for upsampling (UpsampleConv Block), while 1×1 convolution is used to adjust the number of channels.

[0126] In eye region segmentation models, skip connections are used to connect deep feature maps with corresponding shallow feature maps of the same size. The deep / shallow feature ratio has its significance: deeper networks have larger perceptual domains and focus on global features (more abstract and essential); shallower networks focus more on local features (such as texture). Merging preserves more multidimensional information and allows lower network layers to flexibly choose between shallow and deep features, which is more beneficial for segmentation.

[0127] After the model is built, the eye region segmentation model is trained using a pre-built dataset. The pre-built dataset can be the TEyeD dataset, which is a combination and extension of existing datasets, including NVGaze, LPW, GIW, ElSe, ExCuSe, and PNET.

[0128] This design utilizes both local and global features to achieve more accurate segmentation. It's important to note that the eye region segmentation model uses a decoder during training to learn the necessary segmentation features. However, in practice, only the encoder is used to generate constraints for ellipse fitting.

[0129] Since segmentation is a pixel-level classification task, the cross-entropy loss function can be used to evaluate the training loss of the eye region segmentation model, i.e., the second loss. The cross-entropy loss independently evaluates the class prediction for each pixel, and then averages the loss across all pixels, ensuring that the learning effect for each pixel in the image is essentially the same. However, the pupil, iris, and others are unevenly distributed, which will lead to the training process being dominated by the class with the most pixels. That is, the model will primarily learn the features of the class samples with the most pixels, and the trained model will be more biased towards predicting pixels of that class. To alleviate the class imbalance problem in the image, each value in the output probability distribution vector needs to be weighted, making the model pay more attention to samples with fewer pixels. Furthermore, this invention considers pixels closer to the edges or boundaries of each class to be more meaningful than other pixels. This makes the weight calculation resemble a normal distribution, as shown in the fourth preset formula:

[0130]

[0131] in, d represents the weight of pixel o2. i Let represent the Euclidean distance from the pixel to the i-th near boundary, and σ represent the standard deviation of the normal distribution. The weights can be pre-computed offline for each real image, thus avoiding introducing additional computation to the network. Based on this, as shown in the fifth preset formula, the second loss function is expressed as:

[0132]

[0133] Where, loss seg Denotes the second loss function, o predicted It is the predicted label for pixel o2, o goundtruth It's the ground live tag for pixel o2, w o It is the weight of pixel o2.

[0134] Step 130: Input the eye key point detection results and the eye region segmentation results into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset.

[0135] Specifically, after obtaining the eye key point detection results and eye region segmentation results, they are input into a pre-built ellipse fitting model as constraints to fit the ellipse of the iris and pupil, and the ellipse fitting results are obtained.

[0136] A general ellipse can be represented as:

[0137] F(a,x)=a·x=A1x 2+B1xy+C1y 2 +D1x+E1y+F1=0

[0138] Where a = [A1B1C1D1E1F1] T , x = [x 2 yy 2 y1]. From this, we can calculate the five parameters of the ellipse: x c ,y c ,a c ,b c ,θ c ;

[0139] Where, x c It is the x-coordinate of the center coordinate, y c It is the ordinate of the center coordinate, a c It is the length of the semi-spindle, b c It is the length of the semi-secondary axis, θ c It is the rotation angle (the angle from the positive horizontal axis to the principal axis of the ellipse). A1, B1, C1, D1, E1, F1 and x c ,y c ,a c ,b c ,θ c They can be converted to each other.

[0140] Based on this, an ellipse fitting model is constructed to generate x. c ,y c ,a c ,b c ,θ c The iris and pupil are oval. Specifically, as... Figure 8 As shown, Figure 8 The network architecture of the ellipse fitting model is described, which takes the high-dimensional features of the encoder contained in the eye region segmentation results as input and the landmarks of the eye contained in the eye keypoint detection results as other inputs to add additional constraints to the ellipse fitting, making it more accurate.

[0141] After the model is built, the ellipse fitting model is trained using a pre-built dataset. The pre-built dataset can be the TEyeD dataset, which is a combination and extension of existing datasets, including NVGaze, LPW, GIW, ElSe, ExCuSe, and PNET.

[0142] After training is complete, as shown in the sixth preset formula, the third loss function is used to evaluate the training loss (L1 loss) of the ellipse fitting model:

[0143]

[0144] Where, lossfit This represents the third loss function, and o3 represents the actual ground value. This represents the predicted value.

[0145] Since the ellipse fitting model is trained together with the eye keypoint detection model and the eye region segmentation model, the total loss is a weighted sum of these losses, as shown in the seventh preset formula:

[0146] l loss =α lm loss lm +β seg loss seg +γ fit loss fit

[0147] Where, loss lm Let loss be the first loss function. seg Denotes the second loss function, loss fit Let α represent the third loss function. lm β represents the weight of the first loss function. seg γ represents the weight of the second loss function. fit This represents the weight of the third loss function.

[0148] Step 140: Based on the ellipse fitting result, the rotation center of the eyeball is obtained using the first preset rule and the first preset formula to determine the optical axis parameters.

[0149] Specifically, after obtaining the ellipse fitting result, the optical axis is calculated. The first pre-defined rule includes: the optical axis connects the center of the pupil and the center of rotation of the eyeball. The center of the pupil is obtained through ellipse fitting. The center of rotation is a virtual position within the eyeball, which cannot be directly inferred from visual data but can be inferred based on the characteristics of this virtual position. When the user's eye rotates, the center of rotation remains stationary. Therefore, if we consider the iris as a plane tangent to the eyeball, the normal vector of this plane will pass through the center of rotation. Calculating the normal vectors of this plane at multiple locations determines the center of rotation. Figure 9 As shown, Figure 9 This explains the above situation.

[0150] First, the position of the iris in three-dimensional space needs to be determined. The iris ellipse in the image coordinate frame reflects the perspective projection of the circular feature from the three-dimensional space of the iris to the two-dimensional space of the image. The three-dimensional orientation of the circular feature relative to the camera frame can be estimated using the five basic parameters of the ellipse obtained from the ellipse fitting model and the effective focal length of the camera.

[0151] In other words, the circular intersection point and its direction with the cone are determined. Given a three-dimensional cone that passes through an image plane forming an ellipse and a vertex that is the center of the camera, the following steps are taken: Figure 10 As shown.

[0152] Vertex O c A three-dimensional cone with an elliptical base (α,β,γ) can be represented as:

[0153] F(x,y)≡a′x 2 +2h′xy+b′y 2 +2g′x+2f′y+d′=0

[0154] This can be written in a homogeneous coordinate system as follows:

[0155]

[0156] F(x,y,t)≡a′x 2 +2h′xy+b′y 2 +2g′xt+2f′yt+d′t 2 =0

[0157] Here, t is an auxiliary variable in a homogeneous coordinate system.

[0158] The general form can be expressed as:

[0159] a1x 2 +b1y 2 +c1z 2 +2f1yz+2g1zx+2h1xy+2u1x+2v1y+2w1z+d1=0

[0160] The parameters a1, b1, c1, d1, f1, g1, h1, u1, v1, and w1 can be derived from the ellipse parameters. All parallel plane portions of the cone are similar cones in position. Therefore, the planes intersecting the cone can be represented by the first pre-defined formula:

[0161] lx + my + nz = 0

[0162] Among them l 2 +m 2 +n 2 =1. Then we can determine the orientation of the pupil and iris in three-dimensional space.

[0163] Nevertheless, ambiguity persists, and the solution is not unique. There are always two symmetrical planes. This problem can be avoided through consistency between adjacent frames. Specifically, when an ellipse on a plane undergoes a slight translation (e.g., in this case, the fitted ellipses of two adjacent frames can approximate each other), the normal vector of its corresponding circle remains unchanged. Considering this approximation assumption, invariance can be guaranteed with a small threshold, filtering out ambiguity values ​​that do not meet the condition. This can be done simultaneously during calibration.

[0164] Then, the center of rotation of the eyeball can be established using the orientation calculated from multiple locations. The optical axis can be determined by the relationship between the center of rotation of the eyeball and the center of the pupil.

[0165] Step 150: Based on the optical axis parameters, determine the visual axis using the second preset rule and the second preset formula, and then determine the spatial position of the line of sight.

[0166] The second pre-defined rule includes: using several iris positions, determining the optical axis through the center of eye rotation, and then calculating the visual axis based on its spatial relationship with the optical axis. Preliminary experiments show that the NC distance varies between individuals. Therefore, this value needs to be determined individually for each user. Although the NC distance can be precisely quantified in practice, the arduous experimental setup and lengthy process limit its use to confirmatory experiments. To reduce complexity, this invention employs a calibration-based method to calculate the NC distance and orientation, rather than direct measurement.

[0167] Specifically, assume the optical axis is considered a unit vector with the center of rotation as its origin. In this case, the viewing axis can be determined by translating and rotating the optical axis, as shown in the second preset formula.

[0168] v = Mo

[0169] Where o represents a unit vector along the optical axis, v represents a unit vector along the visual axis, and M is the transformation vector.

[0170] Based on this, the present invention proposes a calibration method that generates various iris positions through eye rotation, and then derives the positional parameters between the optical axis and the visual axis using a given stimulus. The steps, namely the second preset rule, are as follows: display a stimulus at a given angle and a given depth, and have the user look at it; then, change the angle and depth of the stimulus and perform the same operation. This cycle is repeated multiple times to determine the optical axis and the visual axis.

[0171] After deriving the visual axis from the optical axis, we can perform fixation tracking by observing the convergence of the binocular visual axes, such as... Figure 11 As shown, a and b represent the visual axis, and d represents the user's gaze direction and depth. We can then calculate the length of d using Apollo's theorem.

[0172]

[0173] Where c represents the distance between nodes in front of the eye (on the baseline, it can also represent the interpupillary distance (IPD) or the distance between the centers of eye rotation), α d Let β represent the first intermediate variable. d α represents the second intermediate variable. d and β d It is simply calculated from the rotation angles A and B of the visual axes of the two eyes. Using the geometry of a triangle, after calculating the length d, its direction can be determined.

[0174]

[0175] Where D and d together indicate the position of the user's gaze in space, D represents the direction of the user's gaze, d represents the depth of the user's gaze, c represents the distance between the centers of eye rotation, and A represents the rotation angle.

[0176] This invention is based on a novel parallax transformation model. By capturing and analyzing user eye movements, and utilizing ocular visual features, it employs algorithms such as eye keypoint detection, eye region segmentation, and pupil and iris fitting to solve for the user's eyeball model. Finally, using the user's binocular vision, it derives the invisible and virtual parameters of the visual axis and determines the user's gaze in space, achieving a 3D gaze tracking technology suitable for extended reality 3D display and 3D interaction. This invention is implemented as a multi-stage method, which is both lightweight and accurate.

[0177] The spatial gaze tracking based on a human eye model proposed in this invention can be considered a multi-level VR streaming media system. Utilizing human hierarchical vision, it provides corresponding multi-level streaming video to achieve high-quality 360° VR streaming and a satisfactory Quality of Experience (QoE). Through our multi-level VR streaming media system, we can significantly optimize the bandwidth requirements of VR streaming, thereby achieving smooth playback of VR streaming media on existing WiFi networks without introducing additional infrastructure. This invention solves the problem of missing depth information in extended reality 3D display and 3D interaction at a relatively low cost, and extensive experiments have been conducted to evaluate its performance under different settings. The results show that this invention significantly reduces spatial gaze tracking errors. Several case studies also demonstrate the 3D interaction capabilities of this invention.

[0178] Based on the above embodiments, the eye region segmentation model in this method specifically includes:

[0179] An encoder, which is used to learn recognizable features of an image;

[0180] A decoder is used to map the features of the encoder to a high-resolution pixel space to obtain dense classification.

[0181] Specifically, the eye region segmentation model consists of two main parts: an encoder and a decoder.

[0182] Figure 7 (b) The left-hand portion represents the encoder, used to learn recognizable features of the image. The downsampling block is the same depthwise separable convolutional block used in the eye keypoint detection model, such as... Figure 6 As described in (b). Along this path, the size of the feature map decreases while the number of channels increases.

[0183] Figure 7 The right side of (b) is the decoder, which maps the encoder's features to a high-resolution pixel space to obtain dense classification. Here, the decoder utilizes... Figure 7 Block (c) is shown. Deconvolution is used for upsampling (UpsampleConv Block), while 1×1 convolution is used to adjust the number of channels.

[0184] In eye region segmentation models, skip connections are used to connect deep feature maps with corresponding shallow feature maps of the same size. The deep / shallow feature ratio has its significance: deeper networks have larger perceptual domains and focus on global features (more abstract and essential); shallower networks focus more on local features (such as texture). Merging preserves more multidimensional information and allows lower network layers to flexibly choose between shallow and deep features, which is more beneficial for segmentation.

[0185] Based on the above embodiments, in this method, an eye keypoint detection model is trained using a pre-built dataset based on a pre-built neural network, and then the method further includes:

[0186] The first loss is calculated using the third preset formula;

[0187] The third preset formula includes:

[0188]

[0189] Where, loss lm Denotes the first loss function. x max The x-axis value represents the maximum value of the eye key point detection result, y. max The vertical axis represents the maximum value of the eye key point detection result, x. min The x-axis represents the minimum value of the eye key point detection result, y min o1 represents the minimum value of the vertical axis of the eye key point detection result, and o1 represents the difference between the predicted value and the actual ground value.

[0190] Specifically, considering the dense spatial distribution of eyemarks, the loss function is made more robust by normalizing the smooth L1 loss. As shown in the third preset formula, the modified first loss function is displayed as follows:

[0191]

[0192] Where, loss lm Denotes the first loss function. x max The x-axis value represents the maximum value of the eye key point detection result, y. max The vertical axis represents the maximum value of the eye key point detection result, x. min The x-axis represents the minimum value of the eye key point detection result, y min o1 represents the minimum value of the vertical axis of the eye key point detection result, and o1 represents the difference between the predicted value and the actual ground value.

[0193] Based on the above embodiments, in this method, an eye region segmentation model is trained using a pre-built dataset based on a pre-built neural network, and then the method further includes:

[0194] The weights of each value in the output probability distribution vector are calculated using the fourth preset formula.

[0195] The second loss is calculated using the fifth preset formula;

[0196] The fourth preset formula includes:

[0197]

[0198] in, d represents the weight of pixel o2. i Let σ represent the Euclidean distance from the pixel to the i-th near boundary, and σ represent the standard deviation of the normal distribution.

[0199] The fifth preset formula includes:

[0200]

[0201] Where, loss seg Denotes the second loss function, o predicted It is the predicted label for pixel o2, o goundtruth It's the ground live tag for pixel o2, w o It is the weight of pixel o2.

[0202] Specifically, since segmentation is a pixel-level classification task, the cross-entropy loss function can be used to evaluate the training loss of the eye region segmentation model, i.e., the second loss. The cross-entropy loss independently evaluates the class prediction for each pixel, and then averages the loss across all pixels, ensuring that the learning effect for each pixel in the image is essentially the same. However, the pupil, iris, and others are unevenly distributed, which will lead to the training process being dominated by the class with the most pixels. That is, the model will primarily learn the features of the class samples with the most pixels, and the trained model will be more biased towards predicting pixels of that class. To alleviate the class imbalance problem in the image, each value in the output probability distribution vector needs to be weighted, making the model pay more attention to samples with fewer pixels. Furthermore, this invention considers pixels closer to the edges or boundaries of each class to be more meaningful than other pixels. This makes the weight calculation resemble a normal distribution, as shown in the fourth preset formula:

[0203]

[0204] in, d represents the weight of pixel o2. i Let represent the Euclidean distance from the pixel to the i-th near boundary, and σ represent the standard deviation of the normal distribution. The weights can be pre-computed offline for each real image, thus avoiding introducing additional computation to the network. Based on this, as shown in the fifth preset formula, the second loss function is expressed as:

[0205]

[0206] Where, loss seg Denotes the second loss function, o predicted It is the predicted label for pixel o2, o goundtruth It's the ground live tag for pixel o2, w o It is the weight of pixel o2.

[0207] Based on the above embodiments, in this method, the detection results of the key eye points and the segmentation results of the eye region are input into a pre-constructed ellipse fitting model to obtain the ellipse fitting result, specifically including:

[0208] Using the results of eye keypoint detection and eye region segmentation as constraints, ellipses of the iris and pupil are fitted to obtain the target parameter x. c ,y c ,a c ,b c ,θ c ;

[0209] Using the target parameters, x is generated based on the ellipse fitting model. c ,y c ,a c ,bc ,θ c The iris and pupil are oval;

[0210] Where, x c It is the x-coordinate of the center coordinate, y c It is the ordinate of the center coordinate, a c It is the length of the semi-spindle, b c It is the length of the semi-secondary axis, θ c It is the rotation angle.

[0211] Specifically, after obtaining the eye key point detection results and eye region segmentation results, they are input into a pre-built ellipse fitting model as constraints to fit the ellipse of the iris and pupil, and the ellipse fitting results are obtained.

[0212] A general ellipse can be represented as:

[0213] F(a,x)=a·x=A1x 2 +B1xy+C1y 2 +D1x+E1y+F1=0

[0214] Where a = [A1B1C1D1E1E1] T , x = [x 21 yy 2 y1]. From this, we can calculate the five parameters of the ellipse: x c ,y c ,a c ,b c ,θ c ;

[0215] Where, x c It is the x-coordinate of the center coordinate, y c It is the ordinate of the center coordinate, a c It is the length of the semi-spindle, b c It is the length of the semi-secondary axis, θ c It is the rotation angle (the angle from the positive horizontal axis to the principal axis of the ellipse). A1, B1, C1, D1, E1, F1 and x c ,y c ,a c ,b c ,θ c They can be converted to each other.

[0216] Based on this, an ellipse fitting model is constructed to generate x. c ,y c ,a c ,b c ,θ c The iris and pupil are oval. Specifically, as... Figure 8 As shown, Figure 8The network architecture of the ellipse fitting model is described, which takes the high-dimensional features of the encoder contained in the eye region segmentation results as input and the landmarks of the eye contained in the eye keypoint detection results as other inputs to add additional constraints to the ellipse fitting, making it more accurate.

[0217] Based on the above embodiments, in this method, an ellipse fitting model is trained using a pre-built dataset based on a pre-built neural network, and then the method further includes:

[0218] The third loss is calculated using the sixth preset formula;

[0219] Calculate the total loss using the seventh preset formula;

[0220] The sixth preset formula includes:

[0221]

[0222] Where, loss fit This represents the third loss function, and o3 represents the actual ground value. Indicates the predicted value;

[0223] The seventh preset formula includes:

[0224] l loss =α lm loss lm +β seg loss seg +γ fit loss fit

[0225] Where, loss lm Let loss be the first loss function. seg Denotes the second loss function, loss fit Let α represent the third loss function. lm β represents the weight of the first loss function. seg γ represents the weight of the second loss function. fit This represents the weight of the third loss function.

[0226] Specifically, after training is completed, as shown in the sixth preset formula, the third loss function is used to evaluate the training loss of the ellipse fitting model:

[0227]

[0228] Where, loss fit This represents the third loss function, and o3 represents the actual ground value. This represents the predicted value.

[0229] Since the ellipse fitting model is trained together with the eye keypoint detection model and the eye region segmentation model, the total loss is a weighted sum of these losses, as shown in the seventh preset formula:

[0230] l loss =α lm loss lm +β seg loss seg +γ fit loss fit

[0231] Where, loss lm Let loss be the first loss function. seg Denotes the second loss function, loss fit Let α represent the third loss function. lm β represents the weight of the first loss function. seg γ represents the weight of the second loss function. fit This represents the weight of the third loss function.

[0232] Based on the above embodiments, in this method, according to the ellipse fitting result, the rotation center of the eyeball is obtained using a first preset rule and a first preset formula to determine the optical axis parameters, specifically including:

[0233] The orientation of the pupil and iris in three-dimensional space is determined by using the target parameters obtained from ellipse fitting and the effective focal length of the camera, so as to obtain the first preset formula;

[0234] The rotation center of the eyeball is established by using the first preset formula, and the optical axis is obtained by connecting the center of the pupil and the rotation center of the eyeball.

[0235] The first preset formula includes:

[0236] lx + my + nz = 0

[0237] Among them l 2 +m 2 +n 2 =1, where x, y, z are the coordinates of the plane containing the pupil and iris.

[0238] Specifically, after obtaining the ellipse fitting result, the optical axis is calculated. The first pre-defined rule includes: the optical axis connects the center of the pupil and the center of rotation of the eyeball. The center of the pupil is obtained through ellipse fitting. The center of rotation is a virtual position within the eyeball, which cannot be directly inferred from visual data but can be inferred based on the characteristics of this virtual position. When the user's eye rotates, the center of rotation remains stationary. Therefore, if we consider the iris as a plane tangent to the eyeball, the normal vector of this plane will pass through the center of rotation. Calculating the normal vectors of this plane at multiple locations determines the center of rotation. Figure 9 As shown, Figure 9 This explains the above situation.

[0239] First, the position of the iris in three-dimensional space needs to be determined. The iris ellipse in the image coordinate frame reflects the perspective projection of the circular feature from the three-dimensional space of the iris to the two-dimensional space of the image. The three-dimensional orientation of the circular feature relative to the camera frame can be estimated using the five basic parameters of the ellipse obtained from the ellipse fitting model and the effective focal length of the camera.

[0240] In other words, the circular intersection point and its direction with the cone are determined. Given a three-dimensional cone that passes through an image plane forming an ellipse and a vertex that is the center of the camera, the following steps are taken: Figure 10 As shown.

[0241] Vertex O c A three-dimensional cone with an elliptical base (α,β,γ) can be represented as:

[0242] F(x,y)≡a′x 2 +2h′xy+b′y 2 +2g′x+2f′y+d′=0

[0243] This can be written in a homogeneous coordinate system as follows:

[0244]

[0245] F(x,y,t)≡a′x 2 +2h′xy+b′y 2 +2g′xt+2f′yt+d′t 2 =0

[0246] Here, t is an auxiliary variable in a homogeneous coordinate system.

[0247] The general form can be expressed as:

[0248] a1x 2 +b1y 2 +c1z 2 +2f11yz+2g1zx+2h1xy+2u1x+2v1y+2w1z+d1=0 where the parameters a1, b1, c1, d1, f1, g1, h1, u1, v1, w1 can be derived from the ellipse parameters. All parallel plane portions of the cone are similar cones in similar positions. Therefore, the planes intersecting the cone can be expressed by the first presupposed formula:

[0249] lx + my + nz = 0

[0250] Among them l 2 +m 2 +n2 =1. Then we can determine the orientation of the pupil and iris in three-dimensional space.

[0251] Nevertheless, ambiguity persists, and the solution is not unique. There are always two symmetrical planes. This problem can be avoided through consistency between adjacent frames. Specifically, when an ellipse on a plane undergoes a slight translation (e.g., in this case, the fitted ellipses of two adjacent frames can approximate each other), the normal vector of its corresponding circle remains unchanged. Considering this approximation assumption, invariance can be guaranteed with a small threshold, filtering out ambiguity values ​​that do not meet the condition. This can be done simultaneously during calibration.

[0252] Then, the center of rotation of the eyeball can be established using the orientation calculated from multiple locations. The optical axis can be determined by the relationship between the center of rotation of the eyeball and the center of the pupil.

[0253] Based on the above embodiments, in this method, the second preset formula includes:

[0254] v = Mo

[0255] Where o represents the unit vector along the optical axis, v represents the unit vector along the visual axis, and M represents the transformation vector;

[0256]

[0257] Where c represents the distance between the centers of eye rotation, α d Let β represent the first intermediate variable. d The second intermediate variable is represented by d, which represents the user's depth of vision, and a and b represent the visual axis.

[0258]

[0259] Where D and d together indicate the position of the user's gaze in space, D represents the direction of the user's gaze, d represents the depth of the user's gaze, c represents the distance between the centers of eye rotation, and A represents the rotation angle.

[0260] Specifically, the second pre-defined rule includes: using several iris positions, determining the optical axis through the center of eye rotation, and then calculating the visual axis based on its spatial relationship with the optical axis. Preliminary experiments show that the distance of NC varies among individuals. Therefore, this value needs to be determined individually for each user. Although the NC distance can be precisely quantified in practice, the arduous experimental setup and lengthy process limit its use to confirmatory experiments. To reduce complexity, this invention tends to employ a calibration-based method to calculate the NC distance and orientation, rather than direct measurement.

[0261] Specifically, assume the optical axis is considered a unit vector with the center of rotation as its origin. In this case, the viewing axis can be determined by translating and rotating the optical axis, as shown in the second preset formula.

[0262] v = Mo

[0263] Where o represents a unit vector along the optical axis, v represents a unit vector along the visual axis, and M is the transformation vector.

[0264] Based on this, the present invention proposes a calibration method that generates various iris positions through eye rotation, and then derives the positional parameters between the optical axis and the visual axis using a given stimulus. The steps, namely the second preset rule, are as follows: display a stimulus at a given angle and a given depth, and have the user look at it; then, change the angle and depth of the stimulus and perform the same operation. This cycle is repeated multiple times to determine the optical axis and the visual axis.

[0265] After deriving the visual axis from the optical axis, we can perform fixation tracking by observing the convergence of the binocular visual axes, such as... Figure 11 As shown, a and b represent the visual axis, and d represents the user's gaze direction and depth. We can then calculate the length of d using Apollo's theorem.

[0266]

[0267] Where c represents the distance between nodes in front of the eye (on the baseline, it can also represent the interpupillary distance (IPD) or the distance between the centers of eye rotation), α d Let β represent the first intermediate variable. d α represents the second intermediate variable. d and β d It is simply calculated from the rotation angles A and B of the visual axes of the two eyes. Using the geometry of a triangle, after calculating the length d, its direction can be determined.

[0268]

[0269] Where D and d together indicate the position of the user's gaze in space, D represents the direction of the user's gaze, d represents the depth of the user's gaze, c represents the distance between the centers of eye rotation, and A represents the rotation angle.

[0270] In one specific embodiment, a 3D-printed frame was used, with two webcams attached (each costing $8) as a capture prototype. The equipment cost of this embodiment is very low compared to existing technologies. This invention relates to three neural network models, all implemented using PyTorch and PyTorch-Lightning. The TEyeD dataset was used for model training; it is a combination and extension of existing datasets, including NVGaze, LPW, GIW, ElSe, ExCuSe, and PNET. The TEyeD dataset contains landmark labels, segmentation labels, and ellipse parameters. This dataset was divided into training / validation / testing parts to ensure fair comparisons in experiments. Furthermore, this invention includes data augmentation techniques such as image flipping, rotation, and Gaussian blur to improve model robustness and reduce overfitting. For model training, we used four NVIDIA GeForce 1080Ti GPUs (each with 12 GiB of memory), while a single NVIDIA GeForce 1080 (8 GiB of memory) was used for gaze tracking.

[0271] In the above specific embodiments, the spatial gaze tracking method based on a human eye model provided by the present invention acquires a target image to be processed; generates eye keypoint detection results and eye region segmentation results based on the target image to be processed; wherein, the eye keypoint detection results are generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is obtained by training a pre-built neural network using a pre-built dataset; the eye region segmentation results are generated by inputting classification results into a pre-built eye region segmentation model, wherein the classification results are obtained by classifying the target image to be processed according to pixels. The eye region segmentation model is obtained through classification; it is based on a pre-built neural network trained using a pre-built dataset; the eye key point detection results and the eye region segmentation results are input into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is based on a pre-built neural network trained using a pre-built dataset; based on the ellipse fitting result, the rotation center of the eyeball is obtained using a first preset rule and a first preset formula to determine the optical axis parameters; based on the optical axis parameters, the visual axis is determined using a second preset rule and a second preset formula, thereby determining the spatial position of the gaze. This invention, based on a new visual axis model, obtains the visual axis through four steps: eye key point detection, eye region segmentation, ellipse fitting, and optical axis calculation, thereby determining the spatial position of the gaze, achieving higher accuracy, lower error, lower cost, and more convenient spatial gaze tracking.

[0272] The spatial gaze tracking device based on the human eye model provided by the present invention is described below. The spatial gaze tracking device based on the human eye model described below can be referred to in correspondence with the spatial gaze tracking method based on the human eye model described above.

[0273] Figure 12 This is a schematic diagram of the spatial gaze tracking device based on a human eye model provided in an embodiment of the present invention, as shown below. Figure 12 As shown, this embodiment of the invention provides a spatial gaze tracking device based on a human eye model, comprising: an image acquisition unit 1210; an image processing unit 1220; an ellipse fitting unit 1230; a first calculation unit 1240; and a second calculation unit 1250; wherein,

[0274] Image acquisition unit 1210 is used to acquire the target image to be processed;

[0275] Image processing unit 1220 is used to generate eye key point detection results and eye region segmentation results based on the target image to be processed;

[0276] Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset.

[0277] Ellipse fitting unit 1230 is used to input the eye key point detection results and the eye region segmentation results into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset.

[0278] The first calculation unit 1240 is used to determine the optical axis parameters by obtaining the rotation center of the eyeball based on the ellipse fitting result using a first preset rule and a first preset formula.

[0279] The second calculation unit 1250 is used to determine the visual axis based on the optical axis parameters and the second preset rule and the second preset formula, thereby determining the spatial position of the line of sight.

[0280] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 can call logical instructions in the memory 1330 to execute a spatial gaze tracking method based on a human eye model. This method includes: acquiring a target image to be processed; generating eye keypoint detection results and eye region segmentation results based on the target image to be processed; wherein the eye keypoint detection results are generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is obtained by training a pre-built neural network using a pre-built dataset; the eye region segmentation results are generated by inputting classification results into a pre-built eye region segmentation model, wherein the classification results are obtained by inputting the target image to be processed into a pre-built eye region segmentation model. The target image is obtained by classifying pixels; the eye region segmentation model is obtained by training a pre-built neural network using a pre-built dataset; the eye key point detection results and the eye region segmentation results are input into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset; based on the ellipse fitting result, the rotation center of the eyeball is obtained using a first preset rule and a first preset formula to determine the optical axis parameters; based on the optical axis parameters, the visual axis is determined using a second preset rule and a second preset formula, thereby determining the spatial position of the gaze.

[0281] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0282] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0283] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0284] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatial gaze tracking method based on a human eye model, characterized in that, include: Acquire the target image to be processed; Based on the target image to be processed, generate eye key point detection results and eye region segmentation results; Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset. The eye key point detection results and the eye region segmentation results are input into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset. Based on the ellipse fitting result, the rotation center of the eyeball is obtained using the first preset rule and the first preset formula, so as to determine the optical axis parameters; Based on the optical axis parameters, the visual axis is determined using a second preset rule and a second preset formula, thereby determining the spatial position of the line of sight. The step of determining the optical axis parameters by obtaining the rotation center of the eyeball based on the ellipse fitting result using a first preset rule and a first preset formula specifically includes: The orientation of the pupil and iris in three-dimensional space is determined by using the target parameters obtained from ellipse fitting and the effective focal length of the camera, so as to obtain the first preset formula; The rotation center of the eyeball is established by using the first preset formula, and the optical axis is obtained by connecting the center of the pupil and the rotation center of the eyeball. The first preset formula includes: ； in , The coordinates of the plane containing the pupil and iris; The second preset formula includes: ； in, Represents the unit vector along the optical axis. Represents the unit vector along the view axis. Represents the transformation vector; ； in, Indicates the distance between the centers of eye rotation. Indicates the first intermediate variable. Indicates the second intermediate variable. Indicates the user's line of sight depth. and Indicates the line of sight; ； in, and Together they indicate the user's line of sight in space. Indicates the user's gaze direction. Indicates the rotation angle.

2. The spatial gaze tracking method based on a human eye model according to claim 1, characterized in that, The eye region segmentation model specifically includes: An encoder, which is used to learn recognizable features of an image; A decoder is used to map the features of the encoder to a high-resolution pixel space to obtain dense classification.

3. The spatial gaze tracking method based on a human eye model according to claim 1, characterized in that, An eye keypoint detection model is trained using a pre-built neural network and a pre-built dataset. This process then includes: The first loss is calculated using the third preset formula; The third preset formula includes: ； in, Denotes the first loss function. , This represents the maximum value of the horizontal axis of the eye key point detection results. This represents the maximum value of the vertical axis of the eye key point detection results. This represents the minimum value of the horizontal axis of the eye key point detection results. This represents the minimum value of the vertical axis of the eye key point detection results. This indicates the difference between the predicted value and the actual ground value.

4. The spatial gaze tracking method based on a human eye model according to claim 2, characterized in that, An eye region segmentation model is obtained by training a pre-built neural network using a pre-built dataset, followed by: The weights of each value in the output probability distribution vector are calculated using the fourth preset formula. The second loss is calculated using the fifth preset formula; The fourth preset formula includes: ； in, Represents pixels The weight, Represents from pixel to the 1st The Euclidean distance to the nearest boundary, This represents the standard deviation of a normal distribution. The fifth preset formula includes: ； in, This represents the second loss function. It is a pixel Predicted labels, It is a pixel Ground-based real-time labels.

5. The spatial gaze tracking method based on a human eye model according to claim 1, characterized in that, The eye key point detection results and the eye region segmentation results are input into a pre-constructed ellipse fitting model to obtain the ellipse fitting result, specifically including: Using the results of eye keypoint detection and eye region segmentation as constraints, ellipses of the iris and pupil are fitted to obtain the target parameters. ; Using the target parameters, generate based on the ellipse fitting model The iris and pupil are oval; in, It is the x-coordinate of the center coordinate. It is the ordinate of the center coordinate. It is the length of the semi-spindle. It is the length of the semi-secondary axis. It is the rotation angle.

6. The spatial gaze tracking method based on a human eye model according to claim 1, characterized in that, An ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset, followed by: The third loss is calculated using the sixth preset formula; Calculate the total loss using the seventh preset formula; The sixth preset formula includes: ； in, This represents the third loss function. Represents the actual ground conditions. Indicates the predicted value; The seventh preset formula includes: ； in, Denotes the first loss function. This represents the second loss function. Indicates the weights of the first loss function. Indicates the weights of the second loss function. This represents the weight of the third loss function.

7. A spatial gaze tracking device based on a human eyeball model, characterized in that, include: Image acquisition unit, used to acquire the target image to be processed; An image processing unit is used to generate eye key point detection results and eye region segmentation results based on the target image to be processed; Specifically, the eye keypoint detection result is generated by inputting the target image to be processed into a pre-built eye keypoint detection model; the eye keypoint detection model is trained on a pre-built neural network using a pre-built dataset; the eye region segmentation result is generated by inputting the classification result into a pre-built eye region segmentation model, wherein the classification result is obtained by classifying the target image to be processed according to pixels; the eye region segmentation model is trained on a pre-built neural network using a pre-built dataset. An ellipse fitting unit is used to input the eye key point detection results and the eye region segmentation results into a pre-built ellipse fitting model to obtain an ellipse fitting result; the ellipse fitting model is obtained by training a pre-built neural network using a pre-built dataset. The first calculation unit is used to determine the optical axis parameters by obtaining the rotation center of the eyeball based on the ellipse fitting result using a first preset rule and a first preset formula. The second calculation unit is used to determine the visual axis based on the optical axis parameters, using a second preset rule and a second preset formula, and then determine the spatial position of the line of sight. The first computing unit specifically includes: The orientation of the pupil and iris in three-dimensional space is determined by using the target parameters obtained from ellipse fitting and the effective focal length of the camera, so as to obtain the first preset formula; The rotation center of the eyeball is established by using the first preset formula, and the optical axis is obtained by connecting the center of the pupil and the rotation center of the eyeball. The first preset formula includes: ； in , The coordinates of the plane containing the pupil and iris; The second preset formula includes: ； in, Represents the unit vector along the optical axis. Represents the unit vector along the view axis. Represents the transformation vector; ； in, Indicates the distance between the centers of eye rotation. Indicates the first intermediate variable. Indicates the second intermediate variable. Indicates the user's line of sight depth. and Indicates the line of sight; ； in, and Together they indicate the user's line of sight in space. Indicates the user's gaze direction. Indicates the rotation angle.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the spatial gaze tracking method based on a human eye model as described in any one of claims 1 to 6.

Images