Head-mounted eye movement data calibration method and system applied to GRAIL system
By obtaining the user's head posture and eye gaze direction in the head-mounted eye movement system, combined with the neural network fitting model, the problem of cumbersome calibration and low accuracy of the head-mounted eye movement tracking system in the prior art is solved, and the problem of "small accuracy" is realized. High-precision gaze calibration and computing resource optimization.
Patent Information
- Application Number
- CN202411308241.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-08
AI Technical Summary
The calibration process of the existing head-mounted eye tracking system is cumbersome and takes a long time. Users are prone to fatigue and it is difficult to accurately estimate the head position when the user's head sways slightly or participates in exercise, which affects the wide range of system applications.
Using a non-immersive virtual environment calibration method from a head-mounted eye tracker to a GRAIL system, the actual coordinates of the user's eye gaze target position are calculated by obtaining the user's head position, eye gaze direction and screen correction point position, and Estimate the three-dimensional line of sight vector and determine the correct focus on the display device. The method includes using a motion acquisition device to obtain the position of the marking point, Hoff transform to detect the circular positioning of the pupil, Purchinban recognizes the detection of the line of sight direction, and combining with the neural network to fit the mapping model, to achieve high-precision calibration of the gaze point.
It reduces the number of calibration points, improves gaze accuracy and computing resource utilization, supports large-scale movement of the user's head, reduces computing resource consumption, and avoids the need for additional equipment and complex algorithms.
Smart Images

Figure CN118902378A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision camera calibration, and in particular to a calibration method for head-mounted eye movement data applied to a GRAIL system. Background Art
[0002] With the improvement of computing device performance and the reduction of cost, eye tracking technology has become increasingly mature. For example, there are stand-mounted eye tracking systems and head-mounted eye tracking systems. Head-mounted eye trackers have gradually become an ideal choice for pervasive human-computer interaction applications due to their non-invasiveness, flexibility, and mobility.
[0003] However, existing eye tracking systems require multi-point calibration before each use to determine the positional relationship between the eye tracker and the display, so as to obtain the correct gaze point coordinates on the display. This calibration process takes a lot of time to record calibration data, and users need to frequently look at multiple different calibration points, which can easily cause eye fatigue or even incorrect calibration. These problems may hinder the widespread application of eye tracking technology.
[0004] In addition, the existing calibration methods for head-mounted eye tracking systems have difficulty estimating the user's head position. When the user's head shakes slightly, or even when the user's head needs to participate in movement, it is impossible to calibrate the exact coordinates of the user's gaze point based solely on the user's eye movement tracking data. How to obtain or estimate the accurate position of the user's head limits the widespread application of head-mounted eye tracking systems.
[0005] In this case, a solution to reduce the number of calibration points and simplify the calibration model is proposed. Sufficient calibration data can be obtained through a small number of calibration points, or the number of calibration points can be reduced to two points, a single point, or even no calibration points by using regression-based, appearance-based methods, and eyeball models. However, there are many problems with the current solution. For example, after reducing the number of calibration points, the gaze accuracy is poor, and more hardware equipment is required; when using hidden calibration points, due to the constraints of the user's trajectory tracking ability, the gaze point lags behind the calibration point when the trajectory moves at high speed.
[0006] In addition, some solutions for estimating the user's head posture have also been proposed, such as adding inertial sensors or fixed cameras to the head-mounted eye tracking system, and estimating the user's head posture through the imu odometer or visual odometer calculation method. However, due to the computing performance of the eye tracking system, the head posture estimation method based on the odometer method has low accuracy. And due to the cumulative influence of the odometer error, additional loop detection and global optimization algorithms are required, which increases the consumption of computing resources.
[0007] Therefore, in view of the limitations of the current head-mounted eye tracking system calibration scheme, a head-mounted eye tracking system calibration method applied to the GRAIL system is studied, which has positive significance for improving the performance of the head-mounted eye tracking system of the GRAIL system and expanding its application scope. Summary of the invention
[0008] The purpose of the present invention is to provide a method for calibrating head-mounted eye movement data applied to the GRAIL system in response to the problems and improvement needs in the above-mentioned existing solutions.
[0009] To achieve the above object, the present invention adopts the following technical solutions:
[0010] A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system comprises the following steps:
[0011] Step 1: Obtain the user's head posture, the user's eye gaze direction, and the position of the screen calibration point;
[0012] Step 2: based on the user's head posture, the user's eye gaze direction, and the screen correction point position obtained in step 1, calibrate the actual gaze point coordinates when the user's eyes are looking at the target position;
[0013] Step 3: Based on the user's head posture obtained in step 1 and the actual gaze point of the user's eyes when looking at the target position calibrated in step 2, estimate the user's three-dimensional sight line vector to determine the correct gaze point coordinates on the display device.
[0014] Furthermore, the method for obtaining the user's head posture in step 1 comprises the following steps:
[0015] a1. Use a motion acquisition device to obtain the three-dimensional position coordinates of the marker points, which are distributed on the eye tracker; including at least two groups, each group includes two marker points, and the two marker points are symmetrically arranged on the two side frame ear planes; the first group of marker points includes P M1 and P M3 , The second set of marking points includes P M2 and P M4 ;
[0016] a2. Determine the three-dimensional position coordinates of the head T according to the following calculation formula based on the marked points h and the head direction vector D h :
[0017]
[0018] a3. According to the head direction vector D h , calculate the rotation matrix R of the user's gaze direction h, that is, the direction in which the user's eyes are looking at the screen; the rotation matrix R can be obtained from the knowledge of spatial geometry h The calculation formula is:
[0019]
[0020] Where I is the identity matrix, s=||V||, is the z-axis of the world coordinate system;
[0021] a4. According to the rotation matrix R h and the head 3D position coordinates T h The user's head posture can be determined.
[0022] Furthermore, the method for obtaining the user's eye gaze direction in step 1 comprises the following steps:
[0023] b1. Use an eye tracker to collect eye movement data, which includes a user's eye area map;
[0024] b2. Based on the eye area map obtained in b1, the user's pupil is located using the Hough transform detection circle;
[0025] b3. Using Purkinje spots to detect the direction of the user's eye line of sight. Purkinje spots are bright spots formed by the reflection of the infrared light source of the head-mounted eye tracking data acquisition device on the cornea;
[0026] b4. Based on the user pupil location obtained in step b3 and the user eye sight direction obtained in step b4, obtain the user's eye gaze direction.
[0027] Furthermore, the method for locating the user's pupil by using the Hough transform detection circle based on the eye area map obtained in b1 in step b2 comprises the following steps:
[0028] Using an edge detection algorithm in step b1 to obtain edge points of the user's pupil;
[0029] Given a pupil radius range (maximum size and minimum size), set multiple levels of search radius with different values within the given radius range;
[0030] Draw a series of circles with the edge point of the user's pupil as the center and the search radius r of each level as the radius; each circle is assigned a weight value λ, and the weight value λ is superimposed on the pixels covered by the circle;
[0031] Assuming that the eye area image is an m×n rectangle, the weight value at the pixel (x, y) is:
[0032]
[0033] in, Search function for circle;
[0034] According to the above method, a weight value is assigned to each pixel to form a weight map, and the obtained weight map is filtered and sorted by setting a threshold. The pixel point with the highest weight is used as the center of the fitting circle, and the search is performed within the radius of the user's pupil. The center of the circle with the highest weight and its corresponding radius are found to obtain the user's pupil center and radius, thereby locating the user's pupil.
[0035] Furthermore, the method of using Purkinje spot recognition to detect the direction of the user's eye line of sight in step b3 includes the following steps:
[0036] Use the Gaussian Laplacian (LOG) operator to perform Gaussian filtering and Laplacian operator processing on the user's eye area image to obtain bright spots with high gray values, and connect the areas with high gray values;
[0037] The center of the connected area is calculated, and the final bright spot features and the corresponding radius are estimated based on the center to obtain the position of each bright spot, thereby identifying each bright spot.
[0038] Furthermore, the method of using Purkinje spots to detect the direction of the user's eye line of sight in step b3 further includes filtering misidentified bright spots according to the distance between the bright spots and the pupil.
[0039] Furthermore, the step b4 is to calculate the relative position (x q ,y q ) represents the user's gaze direction, that is, the state vector T related to the user's eye gaze direction is obtained eye =[x q ,y q ,r].
[0040] Furthermore, the method for obtaining the screen correction point position in step 1 comprises the following steps:
[0041] Use the following formula:
[0042]
[0043] Realize the mapping from the correction point P(u,v) in the computer image coordinate system to the actual three-dimensional coordinate point P′(x′,y′,z′) of the curved screen in space.
[0044] Furthermore, the method for calibrating the actual gaze point of the user's eyes when gazing at the target position based on the user's head posture, the user's eye gaze direction, and the screen correction point position in step 2 comprises the steps of:
[0045] Setting a calibration point on the curved screen that the user is looking at, and the calibration point can move along a preset trajectory;
[0046] The user's eyes are made to follow the trajectory of the calibration points on the curved screen, and the head posture and calibration point data are synchronously processed through the timestamp t to form the mapping point data D = (x q ,y q ,r,T h ,R h ,u M ,v M ,t);
[0047] Calculate the mean u and standard deviation α of the mapping point pairs to exclude head poses outside the range of u±3α;
[0048] The RANSAC algorithm is used to filter out abnormal mapping point pairs and obtain data d = (x q ,y q ,r,T h ,R h );
[0049] Use neural network to realize data d=(x q ,y q ,r,T h ,R h ) to (u M ,v M ) is fitted to the mapping model; the neural network has 8 inputs, 108 intermediate neurons, and two outputs; the loss function of the neural network is defined as:
[0050]
[0051] The function used to update the neural network parameters is:
[0052]
[0053] Based on the forward prediction of the neural network model, the pixel coordinates of the actual gaze point when the user's eyes are looking at the target are obtained (u p ,v p ).
[0054] Furthermore, the method in step 3 based on the user's head posture and the actual gaze point when the user's eyes gaze at the target position comprises the steps of:
[0055] According to the actual pixel coordinates of the user's gaze point when the user's eyes are looking at the target (u p ,v p ), using the same calculation method as the actual three-dimensional coordinate point of the curved screen in space, the actual three-dimensional coordinates of the user's gaze point when the user's eyes are looking at the target (x s ,ys ,z s );
[0056] According to I=(x s ,y s ,z s )-T h The user's three-dimensional sight direction I is calculated to determine the correct gaze point coordinates on the display device.
[0057] A calibration system for head-mounted eye movement data applied to a GRAIL system, comprising a head-mounted device and a GRAIL system;
[0058] The head mounted device includes an eye tracker and marking points arranged on the eye tracker, and the eye tracker collects eye movement data of the user;
[0059] The GRAIL system includes a motion acquisition device, a computing platform and a display device; the motion acquisition device acquires the three-dimensional position coordinates of the marker points; the computing platform receives the user's eye movement data provided by the eye tracker and the three-dimensional position coordinates of the marker points provided by the motion acquisition device, and estimates the user's three-dimensional line of sight vector according to the eye movement data and the three-dimensional position coordinates of the marker points according to the above-mentioned calibration method from the head-mounted eye tracker to the non-immersive virtual environment of the GRAIL system, so as to determine the correct gaze point coordinates on the display device.
[0060] The present invention provides a method and system for correcting eye movement data. The method collects eye movement data and marker data provided on an eye tracker to obtain the user's head posture, the user's eye gaze direction, and the screen correction point position. Based on the user's head posture, the user's eye gaze direction, and the screen correction point position, the actual gaze point when the user's eyes are gazing at a target position is calibrated; based on the user's head posture and the actual gaze point, the user's three-dimensional sight line vector is estimated to determine the correct gaze point coordinates on a display device.
[0061] The present invention allows a wide range of movement of the user's head during data collection, and uses a trajectory tracking method to form mapping point pair data when calculating the actual gaze point when the user's eyes are looking at the target position. Without adding additional equipment, the calibration result has higher accuracy. Compared with the traditional method of estimating the user's head posture through the imu odometer or visual odometer calculation method, the present invention does not require additional loop detection and global optimization algorithms, and consumes less computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0063] Figure 1 A structural block diagram of a method for calibrating head-mounted eye movement data applied to a GRAIL system is provided for an embodiment;
[0064] Figure 2 A model diagram of a calibration method for head-mounted eye movement data applied to a GRAIL system in an embodiment;
[0065] Figure 3 A schematic diagram of an eye tracker for implementing a method for calibrating head-mounted eye movement data applied to a GRAIL system;
[0066] Figure 4 This is a flow chart of a method for calibrating head-mounted eye movement data applied to a GRAIL system according to an embodiment of the present invention;
[0067] Figure 5 is a schematic diagram of marking points of an eye tracker in an embodiment;
[0068] Figure 6 A schematic diagram of mapping from two-dimensional coordinates on a pixel plane to three-dimensional coordinates on a curved screen provided in an embodiment. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings and embodiments in the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0070] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0071] like Figure 1 As shown, a head-mounted eye movement data calibration system applied to a GRAIL system in this embodiment includes a head-mounted device and a GRAIL system.
[0072] The head-mounted device includes a head-mounted eye-tracking data acquisition device and a head-mounted marking point arranged on the head-mounted eye-tracking data acquisition device. The head-mounted eye-tracking data acquisition device is an eye tracker, and its structure is as follows: Figure 3 As shown, the head-mounted markers are referred to as markers in the following description. The eye tracker is used to collect user eye movement data.
[0073] The GRAIL system includes a motion acquisition device, a computing platform and a display device. The motion acquisition device is used to acquire the three-dimensional position coordinates of the marker points; the computing platform receives the user's eye movement data provided by the eye tracker and the three-dimensional position coordinates of the marker points provided by the motion acquisition device, and estimates the user's three-dimensional sight line vector based on the eye movement data and the three-dimensional position coordinates of the marker points to determine the correct gaze point coordinates on the display device.
[0074] The display device of this embodiment is a curved screen. Based on the calibration system from the head-mounted eye tracker to the non-immersive virtual environment of the GRAIL system, this embodiment provides a model diagram of the calibration system from the head-mounted eye tracker to the non-immersive virtual environment of the GRAIL system, such as Figure 2 As shown, it includes a curved screen, one or more computing platforms and multiple motion acquisition devices.
[0075] The curved projection screen displays the images output by the computing platform 113 through the four projectors of the GRAIL system.
[0076] The computing platform stores the calibration method of the embodiment from the head-mounted eye tracker to the non-immersive virtual environment of the GRAIL system in the form of software. The computing platform receives data collected by the eye tracker and the motion acquisition device, runs the calibration method from the head-mounted eye tracker to the non-immersive virtual environment of the GRAIL system, calibrates the actual gaze point when the user's eyes are looking at the target position, and estimates the user's three-dimensional sight line vector.
[0077] The motion acquisition device includes a set of vicon motion capture cameras, which use a set of motion capture infrared cameras with 10 different angles to collect the 3D positions of reflective markers within the camera's experimental range in real time.
[0078] The eye tracker has a built-in infrared light source and an infrared camera. It collects the user's eye area based on the infrared light source and the infrared camera, calculates the pupil coordinates, and calculates the eye gaze direction relative to the user's eye position based on the position of the Purkinje spot on the user's eyeball.
[0079] Figure 5The marking points are shown. In this embodiment, a small reflective ball is used as a marking point. The marking point is fixedly attached to the eye tracker. It includes at least two groups, each group includes two marking points, and the two marking points are symmetrically arranged on the two side frame ear planes. The first group of marking points includes P M1 and P M3 , The second set of marking points includes P M2 and P M4 ; Used to cooperate with the motion capture infrared camera of the motion acquisition device to collect the user's head posture in real time.
[0080] The eye tracker and the GRAIL system are connected via a communication signal line or wireless communication to achieve data transmission or interaction.
[0081] like Figure 4 As shown, this embodiment provides a method for calibrating head-mounted eye movement data applied to a GRAIL system, comprising the following steps:
[0082] Step S200: Obtaining the user's head posture;
[0083] Step S210: obtaining the user's eye gaze direction;
[0084] Step S220: obtaining the position of the screen calibration point;
[0085] There is no sequence requirement for the steps S200 to S220 during implementation.
[0086] Step S230: based on the user's head posture, the user's eye gaze direction and the screen correction point position, calibrate and obtain the actual gaze point when the user's eyes are gazing at the target.
[0087] Step S240: Estimate the user's three-dimensional sight line vector based on the user's head posture, the user's eye gaze direction and the actual gaze point to determine the correct gaze point coordinates on the display device.
[0088] like Figure 5 As shown, the method of obtaining the user's head posture in step 1 includes:
[0089] Use the motion acquisition device to obtain the marker point P M1 , P M2 , P M3 With P M4 The three-dimensional position coordinates of the head-mounted eye tracking data acquisition device, the head-mounted markers and the user's head are so small that the mutual movement is negligible, and the head-mounted markers and the user's head can be approximately fixedly connected. Directly obtain the three-dimensional position coordinates of the midpoint of both eyes and the head direction vector:
[0090] The three-dimensional position coordinates of the midpoint of both eyes are taken as the head T h The three-dimensional position of Get the user's rotation matrix, where I is the unit matrix, s=V, is the z-axis of the world coordinate system, that is, the direction facing the screen. Finally, according to the rotation matrix R h and the head 3D position coordinates T h The user's head posture can be determined.
[0091] It should be noted that, in order to further simplify the loss of computing resources, the present implementation can also adopt the method of directly setting the marker point P at the midpoint of the line connecting the centers of the pupils of the two eyes of the eye tracker. M5 , use the motion acquisition device to directly collect the marker point P M5 Location coordinates.
[0092] The method of obtaining the user's eye gaze direction in step S210 includes:
[0093] b1. Use an eye tracker to collect eye movement data, which includes a user's eye area map;
[0094] b2. Based on the eye area map obtained in b1, the user's pupil is located using the Hough transform detection circle. Specifically:
[0095] Using an edge detection algorithm in step b1 to obtain edge points of the user's pupil;
[0096] Given a pupil radius range (maximum size and minimum size), set multiple levels of search radius with different values within the given radius range;
[0097] With the edge point of the user's pupil as the center and the search radius r of each level as the radius, a series of circles are drawn. Each circle is accompanied by a weight value λ, and the pixels covered by the circle are superimposed with the weight value. Assuming that the eye area image is an m×n rectangle, the weight value at the pixel (x, y) is:
[0098]
[0099] in, Search function for circle;
[0100] The obtained weight map is filtered and sorted by setting a threshold, and the pixel with the highest weight is used as the center of the fitting circle. The search is performed within the radius of the user's pupil. The center of the circle with the highest weight and its corresponding radius are found to obtain the user's pupil center and radius, thereby locating the user's pupil.
[0101] b3. Use Purkinje spots to detect the direction of the user's eye line of sight. The Purkinje spots are bright spots formed by the reflection of the infrared light source of the head-mounted eye tracking data acquisition device on the cornea; specifically:
[0102] Use the Gaussian Laplacian (LOG) operator to perform Gaussian filtering and Laplacian operator processing on the user's eye area image to obtain bright spots with high gray values, and connect the areas with high gray values;
[0103] The center of the connected area is obtained by calculating the centroid of the bright spot, and the final bright spot features and the corresponding radius are estimated based on the center to obtain the position of each bright spot, thereby identifying each bright spot.
[0104] In actual use, reflective spots similar to Purkinje spots may appear, but these reflective spots are generally far away from the pupil area. For this reason, this embodiment uses Purkinje spots to detect the direction of the user's eye line of sight for each identified bright spot, and also includes filtering out misidentified bright spots based on the distance between the bright spot and the pupil.
[0105] b4. Based on the user pupil location obtained in step b3 and the user eye sight direction obtained in step b4, obtain the user eye gaze direction:
[0106] The eyeball rotation is simplified to the rotation of a sphere with a curvature radius of about 7.8 mm around the center of the sphere. The position of the eye tracker relative to the user's eye remains basically fixed, so the user's eyeball rotation direction and the position of the Purkinje plaque captured by the eye tracker are fixed. The relative position of the pupil center and the Purkinje plaque center on the x-axis and y-axis (x q ,y q ) can represent the user's gaze direction. The error caused by the simplification of the eye movement model and the slight displacement change between the eye tracker and the user's eye can be compensated to a certain extent in the subsequent mapping fitting process. Thus, the state vector T related to the user's eye gaze direction is obtained. eye =[x q ,y q ,r].
[0107] The method of obtaining the screen correction point position in step S220 is as follows: Figure 6 As shown, the steps include:
[0108] Use the following formula:
[0109]
[0110] Realize the mapping from the point P(u,v) in the computer image coordinate system to the actual three-dimensional coordinate point P′(x′,y′,z′) of the curved screen in space. s (xso ,y so ,z so ) The coordinate position is known.
[0111] The method of step S230 for calibrating the actual gaze point of the user's eyes when they are gazing at the target position based on the user's head posture, the user's eye gaze direction, and the screen correction point position includes:
[0112] Set the user to look at the calibration point in the curved screen, and the calibration point can move along a preset trajectory; make the user's eyes follow the trajectory of the calibration point in the curved screen; synchronize the head posture and the calibration point data through the timestamp t to form a mapping point pair data D = (x q ,y q ,r,T h ,R h ,u M ,v M ,t). The trajectory motion process of this embodiment lasts about 10s. The frame rate of the head-mounted eye tracking device is 250-300Hz, and it can sample 2000-3500 sampling points during the calibration period. The user's eye-related data collected by the head-mounted eye tracking device and the calibration point position data collected by the motion acquisition device in the GRAIL system are synchronized through timestamps.
[0113] Due to the constraints of human eye trajectory tracking ability, eye jitter, pupil positioning error and other factors, the collected mapping point pairs may contain noise data caused by blinking and closing eyes. If these noise data are used to calculate the coordinates of the gaze point, an incorrect mapping relationship will be caused. In addition, the mapping point pairs themselves will also produce errors, which will affect the calculation accuracy of the gaze point. Therefore, it is necessary to eliminate error data and filter noise from the collected data. Therefore, this embodiment first calculates the average value u and standard deviation α of the mapping point pairs, and excludes sampling points outside the range of u±3α. Then use the RANSAC algorithm to filter out abnormal mapping point pair data. Among them, the process of filtering out abnormal mapping point pair data based on the RANSAC algorithm is as follows:
[0114] 1) The mapping point pair data D is randomly divided into 4 groups, and 1 group is used as the evaluation sample;
[0115] 2) Select one group from the remaining data and calculate the value of d=(x q ,y q ,r,T h ,R h ) to (u M ,v M )’s mapping model;
[0116] 3) Use the above mapping model to test the remaining two groups of mapping point pairs, and the points whose errors are within the threshold are considered as in-game points;
[0117] 4) If the number of in-office points is greater than 50% of the test data, the mapping model is calculated again using all in-office points, otherwise go to step 2) to continue execution;
[0118] 5) Put the evaluation samples into the new model for testing, and evaluate the model with standard deviation, save the optimal point, otherwise go to step 2) to continue.
[0119] A neural network with 8 inputs, 108 intermediate neurons and two outputs is used to solve d = (x q ,y q ,r,T h ,R h ) to (u M ,v M ) is fitted. The loss function is defined as Where f(s) represents the neural network model's response to the input d=(x q ,y q ,r,T h ,R h ) generated prediction results. M =(u M ,v M ) represents the true coordinates of the calibration points. The training set is divided in the filtered data D according to Update the neural network parameters to obtain the optimal mapping model.
[0120] Based on the optimal mapping model, the actual pixel coordinates of the user's gaze point when looking at the target are predicted (u p ,v p ).
[0121] The method of estimating the user's three-dimensional sight line vector based on the user's head posture and the actual gaze point when the user's eyes gaze at the target position in step S240 includes:
[0122] According to the actual pixel coordinates of the user's gaze point when the user's eyes are looking at the target (u p ,v p ), according to the following formula:
[0123]
[0124] Calculate the actual three-dimensional coordinates of the user's gaze point when the user's eyes are looking at the target (x s ,y s ,z s ). Then according to I=(x s ,y s ,z s )-T h The user's three-dimensional sight direction I is calculated.
[0125] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system, characterized in that: The following steps are involved: Step 1: Obtain the user's head posture, the user's eye gaze direction, and the position of the screen calibration point; Step 2: based on the user's head posture, the user's eye gaze direction, and the screen correction point position obtained in step 1, calibrate the actual gaze point coordinates when the user's eyes are looking at the target position; Step 3: Based on the user's head posture obtained in step 1 and the actual gaze point of the user's eyes when looking at the target position calibrated in step 2, estimate the user's three-dimensional sight line vector to determine the correct gaze point coordinates on the display device.
2. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 1, characterized in that: The method of obtaining the user's head posture in step 1 comprises the following steps: a1. Use a motion acquisition device to obtain the three-dimensional position coordinates of the marker points, which are distributed on the eye tracker; including at least two groups, each group includes two marker points, and the two marker points are symmetrically arranged on the two side frame ear planes; the first group of marker points includes P M1 and P M3 , the second set of marking points includes P M2 and P M4 ; a2. Determine the three-dimensional position coordinates of the head T according to the following calculation formula based on the marked points h and the head direction vector D h : a3. According to the head direction vector D h , calculate the rotation matrix R of the user's gaze direction h , that is, the direction in which the user's eyes are looking at the screen; the rotation matrix R can be obtained from the knowledge of spatial geometry h The calculation formula is: Where I is the identity matrix, s=||V‖, is the z-axis of the world coordinate system; a4. According to the rotation matrix R h and the head 3D position coordinates T h The user's head posture can be determined.
3. The method of calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 1, characterized in that: The method of obtaining the user's eye gaze direction in step 1 comprises the following steps: b1. Use an eye tracker to collect eye movement data, which includes a user's eye area map; b2. Based on the eye area map obtained in b1, the user's pupil is located using the Hough transform detection circle; b3. Using Purkinje spots to detect the direction of the user's eye line of sight. Purkinje spots are bright spots formed by the reflection of the infrared light source of the head-mounted eye tracking data acquisition device on the cornea; b4. Based on the user pupil location obtained in step b3 and the user eye sight direction obtained in step b4, obtain the user's eye gaze direction.
4. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 3, characterized in that: The method for realizing user pupil positioning by using a Hough transform detection circle based on the eye region map obtained in step b1 in step b2 comprises the following steps: Using an edge detection algorithm in step b1 to obtain edge points of the user's pupil; Given a pupil radius range (maximum size and minimum size), set multiple levels of search radius with different values within the given radius range; Draw a series of circles with the edge point of the user's pupil as the center and the search radius r of each level as the radius; each circle is assigned a weight value λ, and the weight value λ is superimposed on the pixels covered by the circle; Assuming that the eye area image is an m×n rectangle, the weight value at the pixel (x, y) is: in, Search function for circle; According to the above method, a weight value is assigned to each pixel to form a weight map, and the obtained weight map is filtered and sorted by setting a threshold. The pixel point with the highest weight is used as the center of the fitting circle, and the search is performed within the radius of the user's pupil. The center of the circle with the highest weight and its corresponding radius are found to obtain the user's pupil center and radius, thereby locating the user's pupil.
5. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 4, characterized in that: The method of using Purkinje spot recognition to detect the direction of the user's eye line of sight in step b3 comprises the following steps: Use the Gaussian Laplacian (LOG) operator to perform Gaussian filtering and Laplacian operator processing on the user's eye area image to obtain bright spots with high gray values, and connect the areas with high gray values; The center of the connected area is calculated, and the final bright spot features and the corresponding radius are estimated based on the center to obtain the position of each bright spot, thereby identifying each bright spot.
6. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 5, characterized in that: The method of using Purkinje spots to detect the direction of the user's eye line of sight in step b3 also includes filtering out misidentified bright spots according to the distance between the bright spots and the pupil; the step b4 is to calculate the relative position (x) between the center of the user's pupil and the center of the Purkinje spots on the x-axis and y-axis. q ,y q ) represents the user's gaze direction, that is, the state vector T related to the user's eye gaze direction is obtained eye =[x q ,y q ,r].
7. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 1, characterized in that: The method for obtaining the screen correction point position in step 1 comprises the following steps: Use the following formula: Realize the mapping from the correction point P(u,v) in the computer image coordinate system to the actual three-dimensional coordinate point P′(x′,y′,z′) of the curved screen in space.
8. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 1, characterized in that: The method for calibrating the actual gaze point of the user's eyes when gazing at the target position based on the user's head posture, the user's eye gaze direction, and the screen correction point position in step 2 comprises the following steps: Setting a calibration point on the curved screen that the user is looking at, and the calibration point can move along a preset trajectory; The user's eyes are made to follow the trajectory of the calibration points on the curved screen, and the head posture and calibration point data are synchronously processed through the timestamp t to form the mapping point data D = (x q ,y q ,r,T h ,R h ,u M ,v M ,t); Calculate the mean u and standard deviation α of the mapping point pairs to exclude head poses outside the range of u±3α; The RANSAC algorithm is used to filter out abnormal mapping point pairs and obtain data d = (x q ,y q ,r,T h ,R h ); Use neural network to realize data d=(x q ,y q ,r,T h ,R h ) to (u M ,v M ) is fitted to the mapping model; the neural network has 8 inputs, 108 intermediate neurons, and two outputs; the loss function of the neural network is defined as: The function used to update the neural network parameters is: Based on the forward prediction of the neural network model, the pixel coordinates of the actual gaze point when the user's eyes are looking at the target are obtained (u p ,v p ).
9. A method for calibrating a head-mounted eye tracker to a non-immersive virtual environment of a GRAIL system according to claim 1, characterized in that: The method in step 3 based on the user's head posture and the actual gaze point when the user's eyes gaze at the target position comprises the steps of: According to the actual pixel coordinates of the user's gaze point when the user's eyes are looking at the target (u p ,v p ), using the same calculation method as the actual three-dimensional coordinate point of the curved screen in space, the actual three-dimensional coordinates of the user's gaze point when the user's eyes are looking at the target (x s ,y s ,z s ); According to I=(x s ,y s ,z s )-T h The user's three-dimensional sight direction I is calculated to determine the correct gaze point coordinates on the display device.
10. A calibration system for head-mounted eye movement data applied to a GRAIL system, comprising a head-mounted device and a GRAIL system; The head mounted device includes an eye tracker and marking points arranged on the eye tracker, and the eye tracker collects eye movement data of the user; The GRAIL system includes a motion acquisition device, a computing platform and a display device; The motion acquisition device acquires the three-dimensional position coordinates of the marking point; The computing platform receives the user's eye movement data provided by the eye tracker and the three-dimensional position coordinates of the marker points provided by the motion acquisition device, and estimates the user's three-dimensional line of sight vector according to the eye movement data and the three-dimensional position coordinates of the marker points, so as to determine the correct gaze point coordinates on the display device according to the calibration method for head-mounted eye movement data applied to the GRAIL system as described in any one of claims 1 to 9.
Citation Information
Cited By
Human factor intelligent user gaze analysis method, device and system and edge computing equipment
CN119148860A
Human factor intelligent user gaze analysis method, device and system, edge computing device
CN119148860B
