Eye movement tracking method, eye movement tracking device, eye movement tracking system and storage medium

Abstract

The invention discloses an eye movement tracking method, eye movement tracking equipment, an eye movement tracking system and a computer readable storage medium. The eye movement tracking method comprises the steps that static frame image information of an eyeball area and a preset range area around eyeballs is collected through an infrared camera, and a first eyeball model is obtained according to the static frame image information; collecting dynamic event flow information generated by eyeball movement through an event camera, and generating a second eyeball model according to the dynamic event flow information; and establishing a high-frame-rate eye movement model based on the first eyeball model and the second eyeball model, and determining eye movement information and predicting eye movement information according to the high-frame-rate eye movement model. According to the embodiment of the invention, the event camera is introduced to collect the dynamic event flow information generated by eyeball movement, the recognition of the moving target is enhanced, the dynamic event flow information and the static frame image information are complementary and participate in eye movement tracking together, and the high-resolution eye movement tracking is realized while the user experience is improved. The frame rate of the eye movement tracking device is effectively improved, and the power consumption of the eye movement tracking device is reduced.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and particularly relates to an eye tracking method, an eye tracking device, an eye tracking system, and a computer-readable storage medium. Background Art

[0002] Eye tracking is a technology that obtains information by tracking and recording the movement of the human eye and is a key technology in the field of human-computer interaction. With the development of electronic technology and the progress of computer vision and image processing technology, eye tracking technology has been greatly improved and applied. When current eye tracking devices perform eye tracking, since each pixel of the image captured by the camera sensor requires energy and bandwidth for acquisition, communication, and processing, it is difficult to balance the sensor between resolution, frame rate, and power. Summary of the Invention

[0003] Embodiments of the present application provide an eye tracking method, an eye tracking device, an eye tracking system, and a computer-readable storage medium to solve at least one of the above technical problems.

[0004] The eye tracking method of the embodiments of the present application includes:

[0005] Collect static frame image information of the eyeball region and a predetermined range region around the eyeball through an infrared camera, generate a low-frame-rate initial eyeball model according to the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low-frame-rate initial eyeball model according to the key point information to obtain a first eyeball model;

[0006] Collect dynamic event stream information generated by the movement of the eyeball through an event camera, and generate a second eyeball model according to the dynamic event stream information;

[0007] Establish a high-frame-rate eye movement model based on the first eyeball model and the second eyeball model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

[0008] In some embodiments, the performing key point detection on the static frame image information through a key point detection model to obtain key point information includes:

[0009] Perform image preprocessing on the static frame image information to obtain a grayscale image;

[0010] Perform edge detection on the grayscale image to obtain an edge image;

[0011] Calculate a structure tensor matrix according to the edge image, and perform threshold processing on the edge image to obtain a binary image;

[0012] Perform contour extraction on the binary image to obtain pupil contour information and corneal contour information;

[0013] Calculate the corner response function based on the eigenvalues of the structure tensor matrix;

[0014] Perform non-maximum suppression based on the corner response function to obtain eyelid information.

[0015] In some embodiments, the generating the second eyeball model based on the dynamic event stream information includes:

[0016] Generate a high-frame-rate initial eyeball model;

[0017] Perform motion estimation on the dynamic event stream information through a motion detection model to obtain motion information;

[0018] Update the high-frame-rate initial eyeball model according to the motion information to obtain the second eyeball model.

[0019] In some embodiments, the performing motion estimation on the dynamic event stream information through a motion detection model to obtain motion information includes:

[0020] Decode the dynamic event stream information to obtain event timestamps, pixel position information, and pixel polarity information;

[0021] Perform filtering on the pixel polarity information according to the event timestamps and the pixel position information;

[0022] Perform event clustering on the pixel polarity information according to the event timestamps and the pixel position information;

[0023] Perform motion estimation according to the pixel polarity information after event clustering to obtain the motion information.

[0024] In some embodiments, the eye movement tracking method further includes:

[0025] Perform time alignment and spatial alignment on the infrared camera and the event camera so that the parameters of the first eyeball model and the second eyeball model are aligned.

[0026] In some embodiments, the determining the eye movement information and the predicted eye movement information according to the high-frame-rate eye movement model includes:

[0027] Determine the eyeball position information and the pupil position information according to the high-frame-rate eye movement model;

[0028] Determine the predicted eyeball position information and the predicted pupil position information according to the high-frame-rate eye movement model;

[0029] Calculate the eye movement information based on the eye position information and the pupil position information;

[0030] Calculate the predicted eye movement information based on the predicted eye position information and the predicted pupil position information.

[0031] In some embodiments, the number of the infrared cameras is one or more; and / or the number of the event cameras is one or more.

[0032] The eye movement tracking device according to the embodiment of the present application includes:

[0033] The infrared camera is configured to collect static frame image information of the eye region and a predetermined range region around the eye. The eye movement tracking module is configured to generate a low frame rate initial eye model according to the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low frame rate initial eye model according to the key point information to obtain a first eye model;

[0034] The event camera is configured to collect dynamic event stream information generated by eye movement. The eye movement tracking module is further configured to generate a second eye model according to the dynamic event stream information;

[0035] The eye movement tracking module is further configured to establish a high frame rate eye movement model based on the first eye model and the second eye model, and determine the eye movement information and the predicted eye movement information according to the high frame rate eye movement model.

[0036] The eye movement tracking system according to the embodiment of the present application includes one or more processors and a memory. The memory stores a computer program. When the computer program is executed by the processor, the eye movement tracking method according to any one of the above embodiments is implemented.

[0037] The computer-readable storage medium according to the embodiment of the present application stores a computer program thereon. When the program is executed by a processor, the eye movement tracking method according to any one of the above embodiments is implemented.

[0038] The eye movement tracking method, eye movement tracking device, eye movement tracking system and computer-readable storage medium according to the embodiments of the present application introduce an event camera to collect dynamic event stream information generated by eye movement, strengthen the recognition of moving targets. The dynamic event stream information is complementary to the static frame image information and jointly participates in the eye movement tracking process, which can effectively improve the frame rate of the eye movement tracking device and reduce the power consumption of the eye movement tracking device while achieving high-resolution eye movement tracking.

[0039] The additional aspects and advantages of the embodiments of the present application will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0041] Figure 1 is a schematic flowchart of an eye movement tracking method according to some embodiments of the present application;

[0042] Figure 2 is a schematic diagram of the working process of an eye movement tracking method according to some embodiments of the present application;

[0043] Figure 3 is a schematic diagram of the working process of time alignment and spatial alignment of an infrared camera and an event camera according to some embodiments of the present application;

[0044] Figure 4 is a schematic flowchart of an eye movement tracking method according to some embodiments of the present application;

[0045] Figure 5 is a schematic diagram of the working process of a key point detection model according to some embodiments of the present application;

[0046] Figure 6 is a schematic diagram of key point information according to some embodiments of the present application;

[0047] Figure 7 is a schematic flowchart of an eye movement tracking method according to some embodiments of the present application;

[0048] Figure 8 is a schematic flowchart of an eye movement tracking method according to some embodiments of the present application;

[0049] Figure 9 is a schematic diagram of the working process of a motion detection model according to some embodiments of the present application;

[0050] Figure 10 is a schematic flowchart of an eye movement tracking method according to some embodiments of the present application;

[0051] Figure 11 is a schematic diagram of the position of the pupil relative to the eyeball according to some embodiments of the present application;

[0052] Figure 12 is a schematic diagram of the modules of an eye movement tracking device according to some embodiments of the present application;

[0053] Figure 13 is a schematic diagram of the modules of an eye movement tracking system according to some embodiments of the present application;

[0054] Figure 14 is a schematic diagram of the connection state between a computer-readable storage medium and a processor according to some embodiments of the present application. Specific Embodiments

[0055] The following further describes the embodiments of the present application with reference to the accompanying drawings. The same or similar reference numerals in the drawings represent the same or similar elements or elements having the same or similar functions throughout. Additionally, the embodiments of the present application described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application and should not be construed as a limitation of the present application.

[0056] Please refer to Figure 1 and Figure 2 The embodiments of the present application provide an eye movement tracking method, including:

[0057] 010: Collect static frame image information of the eye region and a predetermined range region around the eye through the infrared camera 10, generate a low-frame-rate initial eye model based on the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low-frame-rate initial eye model according to the key point information to obtain a first eye model;

[0058] 020: Collect dynamic event stream information generated by eye movement through the event camera 20, and generate a second eye model based on the dynamic event stream information;

[0059] 030: Establish a high-frame-rate eye movement model based on the first eye model and the second eye model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

[0060] The eye movement tracking method of the embodiments of the present application introduces the event camera 20 to collect dynamic event stream information generated by eye movement, strengthens the recognition of moving targets, the dynamic event stream information is complementary to the static frame image information, and they jointly participate in the eye movement tracking process, which can achieve high-resolution eye movement tracking while effectively improving the frame rate of the eye movement tracking device 100 and reducing the power consumption of the eye movement tracking device 100.

[0061] Specifically, the eye region may include the area covered by the pupil and the cornea, and the predetermined range region around the eye may include the area covered by the upper and lower eyelids and the upper and lower eyelashes. The static frame image information is the data of a continuous image sequence captured by the infrared camera 10, and each frame includes image data, a timestamp, a resolution, and frame rate information, providing a continuous image representation of the static scene of the eye region and the predetermined range region around the eye. A low-frame-rate initial eye model can be generated based on the static frame image information.

[0062] The low-frame-rate initial eyeball model can be specifically established according to the spherical geometric characteristics of the eyeball using methods such as statistics and machine learning. The low-frame-rate initial eyeball model can include parameters such as the center of the eyeball, radius, corneal curvature, refractive index, pupil, and visual axis. Then, by using the key-point detection model to detect the key points in the static frame image information, the key-point information of the eyeball region and a predetermined range region around the eyeball can be obtained, such as pupil contour information, corneal contour information, eyelid information, etc. Writing the obtained key-point information into the low-frame-rate initial eyeball model and updating the above parameters in the low-frame-rate initial eyeball model can obtain the first eyeball model.

[0063] The working principle of the event camera 20 can be as follows: The event camera 20 detects whether the eyeball moves by detecting the change in the logarithm of the light intensity of each pixel point in the field of view. When the change amount of the logarithm of the light intensity of the pixel point reaches a certain threshold, an event containing time, position, and polarity information is output. All pixels in the event camera 20 simultaneously and asynchronously detect the change in the logarithm of the light intensity and output events, forming an asynchronous event stream, that is, dynamic event stream information. One pixel corresponds to one event in the dynamic event stream information, and the dynamic event stream information can include multiple events. The second eyeball model can be generated according to the dynamic event stream information.

[0064] Both the first eyeball model and the second eyeball model can be two-dimensional models. The first eyeball model generated according to the static frame image information collected by the infrared camera 10 can provide key-point information. The second eyeball model generated according to the dynamic event stream information collected by the event camera 20 can provide motion information. Combining the key-point information provided by the first eyeball model, the motion information provided by the second eyeball model, and relevant physiological knowledge, and using machine learning algorithms to further process and analyze these data, a high-frame-rate eye movement model can be generated for high-frame-rate eye movement tracking.

[0065] The machine learning algorithm takes the key-point information obtained from the first eyeball model and the motion information obtained from the second eyeball model as inputs; by learning the complex patterns and correlations behind these key-point information and motion information, the machine learning algorithm can generate detailed information about eye movement and output it as a high-frame-rate eye movement model; the high-frame-rate eye movement model includes not only the type and speed of eye movement but also details such as the direction, duration, and frequency of eye movement.

[0066] When performing eye movement tracking, the infrared camera 10 captures images of the eye region and a predetermined range region around the eye, obtaining static frame image information of the eye region and the predetermined range region around the eye. When the eye moves, the event camera 20 outputs event information according to the change in the light intensity of the eye region, that is, the dynamic event stream information generated by eye movement. It can be understood that when the change amount of the logarithm of the light intensity is less than a certain threshold, the event camera 20 is in a standby state and does not output dynamic event stream information. For example, when the eye is in a static state, the event camera 20 is in a standby state, which can significantly reduce the system power consumption. In addition, since the event camera 20 has no frame rate limit, a high-frame-rate eye movement model can be established based on the static frame image information and the dynamic event stream information, so as to determine the eye movement information and predict the eye movement information according to the high-frame-rate eye movement model, so as to achieve high-resolution and high-frame-rate eye movement tracking. Among them, the eye movement information can include various types, such as the position coordinates of the eye fixation point, the line-of-sight direction, the eye orientation, the pupil position, the pupil size, the feature points of the eye region, or one or more of other eye-related information, etc.

[0067] It can be understood that in the related art, usually only the pupil information changes in the continuous eye movement data obtained by the camera sensor, and the information recorded by most pixels is redundant and the effective information is less. When processing images, a lot of repetitive work is carried out. In addition, in near-eye gaze tracking, when the eye remains stationary, the camera sensor is still in an uninterrupted working state. The repetitive image processing process and the uninterrupted working bring high power consumption to the eye movement tracking system.

[0068] In the embodiment of the present application, the event camera 20 is used to collect the dynamic event stream information generated by eye movement. The characteristics of the event camera 20 enable it to be triggered to work when the eye moves and output the dynamic event stream information, while being in a standby state when the eye is stationary. On the one hand, it can reduce the output of redundant information and avoid repetitive work. On the other hand, when the eye remains stationary, the event camera 20 does not need to work uninterruptedly, thereby reducing the power consumption of the eye movement tracking device 100.

[0069] In the related art, the accuracy of eye movement tracking is often affected by the environment. When the ambient light of the user is insufficient or too strong, the images captured by the camera sensor have underexposure and overexposure phenomena, which will cause the loss of information of the eye and its surrounding parts recorded by the camera sensor, affecting the subsequent eye movement tracking process.

[0070] In the embodiments of the present application, the event camera 20 is used to collect the dynamic event stream information generated by the eye movement. The event camera 20 outputs the event stream information according to the change in the light intensity of the eye area. Insufficient or excessive ambient light will not have too much impact on the working process of the event camera 20 (the event camera 20 outputs the event stream information only related to the change in light intensity). Therefore, in the application scenarios with insufficient or excessive ambient light, the event camera 20 can still have a good recognition effect on the eye movement, thus ensuring the eye movement tracking effect.

[0071] In addition, in order to avoid the situation of underexposure or overexposure of the static frame image information collected by the infrared camera 10 due to insufficient or excessive ambient light, resulting in the loss of information in the eye area and the predetermined range area around the eye, affecting the subsequent eye movement tracking process. In the eye movement tracking solution of the embodiments of the present application, multiple small light-emitting diodes can also be used for illumination. The multiple light-emitting diodes can be evenly arranged on the annular lamp board and installed on the eye movement tracking device 100 to achieve an ideal illumination environment for eye movement tracking. For example, 10 small light-emitting diodes can be evenly arranged on the annular lamp board for illumination. In the ideal illumination environment, the infrared camera 10 then performs corresponding information collection. In this way, the static frame image information of the eye area and the predetermined range area around the eye collected by the infrared camera 10 will not be affected by insufficient or excessive ambient light, so as to further ensure the eye movement tracking effect and achieve a relatively stable and accurate eye movement tracking.

[0072] In some embodiments, the number of infrared cameras 10 is one or more; and / or the number of event cameras 20 is one or more.

[0073] Specifically, the number of both the infrared camera 10 and the event camera 20 can be one or more. For example, the number of both the infrared camera 10 and the event camera 20 is one; or, the number of the infrared camera 10 is one and the number of the event camera 20 is two; or, the number of the infrared camera 10 is two and the number of the event camera 20 is one; or, the number of both the infrared camera 10 and the event camera 20 is two, etc. Examples are not given one by one here, and the specific number can be set accordingly according to different actual application scenarios. For example, for some application scenarios with high requirements for eye movement tracking, the number of the infrared camera 10 and the event camera 20 can be set to multiple to have a better eye movement tracking effect.

[0074] It should be noted that compared with the event camera 20, the frame rate of the static frame image information collected by the infrared camera 10 is lower, and the frame rate of the dynamic event stream information collected by the event camera 20 is higher. Therefore, the first eye model corresponds to a low-frame-rate eye model, and the second eye model corresponds to a high-frame-rate eye model.

[0075] In some embodiments, the eye movement tracking method further includes: performing time alignment and spatial alignment on the infrared camera 10 and the event camera 20 so that the parameters of the first eyeball model and the second eyeball model are aligned.

[0076] Please refer to Figure 3 , and the parameter alignment includes time alignment and spatial alignment of the first eyeball model and the second eyeball model.

[0077] For time alignment, before collecting information using the infrared camera 10 and the event camera 20, an external clock can be used to align the timestamps of the infrared camera 10 and the event camera 20 with each other.

[0078] For spatial alignment, before collecting information using the infrared camera 10 and the event camera 20, different calibration methods can be used for internal parameter calibration according to different cameras. For example, the infrared camera 10 can use the corner point calibration method for internal parameter calibration to obtain the internal parameters of the infrared camera 10, so that the static frame image information collected by the infrared camera 10 can be mapped from the pixel coordinate system to the infrared camera 10 coordinate system. The event camera 20 uses the multi-view calibration method for internal parameter calibration to obtain the internal parameters of the event camera 20, so that the dynamic event stream information collected by the event camera 20 can be mapped from the pixel coordinate system to the event camera 20 coordinate system. After the infrared camera 10 collects the static frame image information and the event camera 20 collects the dynamic event stream information, the Zhang Zhengyou calibration method is used to perform external parameter calibration on the infrared camera 10 and the event camera 20. The external parameter calibration can obtain the external parameters of the infrared camera 10 and the event camera 20, so that the static frame image information collected by the infrared camera 10 can be mapped from the infrared camera 10 coordinate system to the world coordinate system, and the dynamic event stream information collected by the event camera 20 can be mapped from the event camera 20 coordinate system to the world coordinate system. After performing the external parameter calibration, the information collected by the infrared camera 10 and the event camera 20 is aligned to the same coordinate system, so that the first eyeball model and the second eyeball model generated according to the above information are aligned to the same coordinate system to generate a high-frame-rate eye movement model. It can be understood that the information collected by the infrared camera 10 and the event camera 20 can be aligned to the world coordinate system, or can be aligned to the infrared camera 10 coordinate system or the event camera 20 coordinate system, which is not limited here.

[0079] Please refer to Figure 4 and Figure 5 , in some embodiments, key point detection is performed on the static frame image information through a key point detection model to obtain key point information (i.e., 010), including:

[0080] 011: Performing image preprocessing on the static frame image information to obtain a grayscale image;

[0081] 012: Performing edge detection on the grayscale image to obtain an edge image;

[0082] 013: Calculate the structure tensor matrix based on the edge image, and perform threshold processing on the edge image to obtain a binary image;

[0083] 014: Extract the contours from the binary image to obtain pupil contour information and corneal contour information;

[0084] 015: Calculate the corner response function according to the eigenvalues of the structure tensor matrix;

[0085] 016: Perform non-maximum suppression according to the corner response function to obtain eyelid information.

[0086] Specifically, when performing key point detection, first perform image preprocessing on the static frame image information. The image preprocessing may include grayscale processing. Performing grayscale processing on the static frame image information can obtain the corresponding grayscale image. Then, perform edge detection on the grayscale image, specifically by calculating the gradient value of each pixel point of the grayscale image, and determining whether the pixel point is an edge pixel point according to the gradient value. The image composed of all the edge pixel points is the edge image. Among them, different operators can be used to calculate the gradient value of the pixel points according to the actual situation. For example, the Sobel operator can be used to calculate the gradient value of the pixel points of the grayscale image to obtain the edge image.

[0087] When calculating the structure tensor matrix based on the edge image, for each pixel point in the edge image, a 2×2 structure tensor matrix T can be calculated:

[0088]

[0089] where Rx is the gradient value of the pixel point in the horizontal direction, and Ry is the gradient value of the pixel point in the vertical direction.

[0090] When calculating the structure tensor matrix, for the pixel points at the four edges of the edge image, they can be ignored, that is, the pixel points at the four edges are not considered, and only the structure tensor matrices of other pixel points are calculated; or a circle of pixel points can be added at the four edges of the edge image, and then the structure tensor matrix is calculated.

[0091] In addition, threshold processing also needs to be performed on the edge image. The process of threshold processing can be: determine a grayscale value threshold, set the grayscale value of the pixel points greater than the grayscale value threshold to the grayscale maximum value (255), and the corresponding pixel points are determined as background pixel points; set the grayscale value of the pixel points less than the grayscale value threshold to the grayscale minimum value (0), and the corresponding pixel points are determined as foreground pixel points, so as to realize the binarization of the edge image and obtain a binary image. Among them, an adaptive threshold algorithm can be used to perform threshold processing on the edge image.

[0092] After obtaining the binary image, perform contour extraction on the binary image. Specifically, extract the pupil contour and the corneal contour from the binary image. At this time, there may be interfering contours in the extracted contours. It is necessary to further screen the extracted pupil contour and corneal contour according to the characteristics such as the shape and size of the actual pupil and cornea to obtain the pupil contour information and the corneal contour information (as Figure 6 shown). The pupil contour information and the corneal contour information include the position information of the pixel points of the pupil contour and the corneal contour.

[0093] After that, the corner response function of the pixel point can be obtained according to the eigenvalues of the structure tensor matrix of the pixel point. For example, the Harris corner response function or other corner response functions can be obtained according to the eigenvalues, which is not limited here. Perform non-maximum suppression on the corner response function of each pixel point. If the corner response function of the pixel point is greater than the corner response functions of other pixel points in its neighborhood, then take this corner response function as the local maximum response value, and the corresponding pixel point as the corner. All the finally obtained corners are the eyelid information.

[0094] Please refer to Figure 2 and Figure 7 , in some embodiments, generating the second eyeball model (i.e., 020) according to the dynamic event stream information includes:

[0095] 021: Generate a high-frame-rate initial eyeball model;

[0096] 022: Perform motion estimation on the dynamic event stream information through a motion detection model to obtain motion information;

[0097] 023: Update the high-frame-rate initial eyeball model according to the motion information to obtain the second eyeball model.

[0098] Specifically, the high-frame-rate initial eyeball model can be established according to the spherical geometric characteristics of the eyeball using methods such as statistics and machine learning. The high-frame-rate initial eyeball model can include parameters such as eyeball motion parameters, the center of the eyeball, radius, corneal curvature, refractive index, pupil, and visual axis. The event camera 20 collects the dynamic event stream information generated by the eyeball movement. By performing motion estimation on the dynamic event stream information through a motion detection model, the motion information of the eyeball can be obtained. Write the obtained motion information into the high-frame-rate initial eyeball model and update the above parameters in the high-frame-rate initial eyeball model to obtain the second eyeball model.

[0099] Please refer to Figure 8 and Figure 9 , in some embodiments, performing motion estimation on the dynamic event stream information through a motion detection model to obtain motion information (i.e., 022) includes:

[0100] 0221: Decode the dynamic event stream information to obtain the event timestamp, pixel position information, and pixel polarity information;

[0101] 0222: Filter the pixel polarity information based on the event timestamp and pixel position information;

[0102] 0223: Cluster the pixel polarity information based on the event timestamp and pixel position information;

[0103] 0224: Perform motion estimation based on the pixel polarity information after event clustering to obtain motion information.

[0104] Specifically, the motion detection model can read the dynamic event stream information, decode each event in the dynamic event stream information to obtain the event timestamp, pixel position information, and pixel polarity information of each event. The event polarity information is determined according to the change in the logarithm of the light intensity of the pixel. When the light intensity of the pixel increases, the event polarity information is +1; when the light intensity of the pixel decreases, the event polarity information is -1.

[0105] When filtering the pixel polarity information based on the event timestamp and pixel position information, different filtering methods can be selected according to the application scenario. For example, a time surface-based filtering method can be adopted. A time surface is constructed based on the event timestamp, and on the time surface, the method of using a convolutional smoothing window is used to smooth the time surface. Perform weighted averaging on the smoothed time surface, and different weights are assigned according to different timestamps. Perform threshold processing on the weighted-averaged time surface. By setting an appropriate threshold, the noise and redundancy in multiple events can be reduced. In this way, the dynamic event stream information can be made more stable and accurate.

[0106] When clustering the pixel polarity information based on the event timestamp and pixel position information, the similarity between multiple events can be calculated according to the event timestamp and pixel position information, that is, the similarity is calculated according to the time interval and pixel distance between multiple events. According to the similarity between multiple events, it is judged whether multiple events belong to the same trajectory, so that events with adjacent times and adjacent pixel positions can be clustered. After event clustering, based on the event timestamp and pixel position information, motion estimation is performed on the pixel polarity information, and the motion information of the eyeball can be obtained. For example, the direction of eyeball movement can be obtained.

[0107] The specific process of motion estimation is as follows: For the set of events within a fixed time window (Δt) and a small spatial window (L x L), there is a functional relationship between the pixel position information and the timestamps numerically. An equation of the function of event pixel position information - timestamp can be calculated according to a certain method. Based on this equation, the rate of change of event pixel position information relative to the timestamp can be calculated, that is, the motion information of local pixels (the motion information includes the motion direction and the magnitude of the motion speed). Generally, Δt and L are selected as 1 ms and 3 pixels, or can be adjusted according to actual needs. Combining with the theoretical range of the magnitude of eye movement speed, the abnormal values of the motion magnitude of local pixels are removed, and the motion direction is corrected according to the pixel polarity information. Then, a weighted operation is performed on the corrected local pixel motion information to obtain the overall eye motion information within the current time window.

[0108] Please refer to Figure 2 and Figure 10 , in some embodiments, determining the current eye movement information and predicted eye movement information (i.e., 030) according to the high - frame - rate eye movement model includes:

[0109] 031: Determine the eye position information and pupil position information (hereinafter abbreviated as eye information and pupil information) at the current moment and the predicted eye information and pupil information according to the high - frame - rate eye movement model;

[0110] 032: Calculate the eye movement information according to the eye information and pupil information; calculate the predicted eye movement information according to the predicted eye information and predicted pupil information.

[0111] Specifically, the high - frame - rate eye movement model is generated by combining the first eye model and the second eye model, including the eye position information and the pupil position information. The position information can specifically be the two - dimensional position coordinates of the eye and the two - dimensional position coordinates of the pupil. According to the eye information and the pupil information, the position information of the pupil relative to the eye can be obtained, that is, a two - dimensional position coordinate of the pupil relative to the eye is obtained. As Figure 11 shown, the pupil is located at the lower right of the eye. According to the eye information and the pupil information, a two - dimensional position coordinate of the pupil relative to the eye can be obtained. Based on the method of polynomial fitting, the position coordinates of the eye fixation point can be calculated, that is, the eye movement information. The process of polynomial fitting can be: fitting a polynomial according to the known data of multiple position information of the pupil relative to the eye and the position coordinates of the eye fixation point. This polynomial can represent the relationship between the position information of the pupil relative to the eye and the position coordinates of the eye fixation point. When performing eye movement tracking, the position information of the pupil relative to the eye determined according to the high - frame - rate eye movement model is input into the above - mentioned polynomial, and the position coordinates of the eye fixation point can be output. In this way, high - frame - rate eye movement tracking can be achieved.

[0112] In addition, since the high-frame-rate eye movement model includes the motion information provided by the second eyeball model, the eyeball information and pupil information at the next moment can be predicted based on the current eyeball information and pupil information, combined with the motion information, and the predicted eye movement information can be obtained according to the method mentioned above.

[0113] Please refer to Figure 12 , an eye movement tracking device 100 is provided in an embodiment of the present application, including an infrared camera 10, an event camera 20, and an eye movement tracking module 30. The infrared camera 10 is used to collect static frame image information of the eyeball region and a predetermined range region around the eyeball. At this time, the eye movement tracking module 30 is used to generate a low-frame-rate initial eyeball model based on the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low-frame-rate initial eyeball model according to the key point information to obtain a first eyeball model. The event camera 20 is used to collect dynamic event stream information generated by the eyeball movement. At this time, the eye movement tracking module 30 is also used to generate a second eyeball model based on the dynamic event stream information. The eye movement tracking module 30 is further used to establish a high-frame-rate eye movement model based on the first eyeball model and the second eyeball model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

[0114] In some embodiments, the eye movement tracking module 30 is specifically configured to perform image preprocessing on the static frame image information to obtain a grayscale image; perform edge detection on the grayscale image to obtain an edge image; calculate a structure tensor matrix according to the edge image, and perform threshold processing on the edge image to obtain a binary image; perform contour extraction on the binary image to obtain pupil contour information and corneal contour information; calculate a corner response function according to the eigenvalues of the structure tensor matrix; perform non-maximum suppression according to the corner response function to obtain eyelid information.

[0115] In some embodiments, the eye movement tracking module 30 is specifically configured to generate a high-frame-rate initial eyeball model; perform motion estimation on the dynamic event stream information through a motion detection model to obtain motion information; update the high-frame-rate initial eyeball model according to the motion information to obtain a high-frame-rate initial eyeball model.

[0116] In some embodiments, the eye movement tracking module 30 is specifically configured to decode the dynamic event stream information to obtain an event timestamp, pixel position information, and pixel polarity information; perform filtering processing on the pixel polarity information according to the event timestamp and pixel position information; perform event clustering on the pixel polarity information according to the event timestamp and pixel position information; perform motion estimation according to the pixel polarity information after event clustering to obtain motion information.

[0117] In some embodiments, the eye movement tracking module 30 is further configured to perform time alignment and spatial alignment on the infrared camera 10 and the event camera 20, so as to align the parameters of the first eyeball model and the second eyeball model.

[0118] In some embodiments, the eye movement tracking module 30 is specifically configured to determine the eyeball position information and the pupil position information according to the high-frame-rate eye movement model; determine the predicted eyeball position information and the predicted pupil position information according to the high-frame-rate eye movement model; calculate the eye movement information according to the eyeball position information and the pupil position information; calculate the predicted eye movement information according to the predicted eyeball position information and the predicted pupil position information.

[0119] In some embodiments, the number of infrared cameras 10 is one or more; and / or the number of event cameras 20 is one or more.

[0120] The eye movement tracking device 100 according to the embodiments of the present application may be a wearable eye tracker, which is used to capture the gaze direction of the wearer in real time and provide robust and accurate eye movement tracking data. The wearable eye tracker may be composed of a head-mounted unit, a recording unit, and a controller installed on an external device. The wearable eye tracker may assemble the infrared camera 10 and the event camera 20 beside the spectacle lenses, which are respectively used to collect the static frame image information of the eyeball area and the predetermined range area around the eyeball, and collect the dynamic event stream information generated by the eyeball movement. In addition, the wearable eye tracker may also be equipped with other types of sensors, such as an accelerometer, a gyroscope, and a magnetometer, etc., which are used to detect the movement of the head direction. These sensors work together to provide the head posture and the line-of-sight direction of the wearer when interacting with the real world. The wearable eye tracker stores the data on an internal memory card and uses a wireless method to transfer the memory card data to the host computer where the corresponding software runs, so that researchers can analyze the data and visualize it. Finally, by analyzing the position and movement of the reflection, the wearable eye tracker can confirm the gaze direction and the eyeball movement of the wearer.

[0121] It should be noted that the explanations of the eye movement tracking method in the foregoing embodiments are equally applicable to the eye movement tracking device 100 according to the embodiments of the present application, and will not be elaborated here.

[0122] Please refer to Figure 13 , the embodiments of the present application further provide an eye movement tracking system 200, which includes one or more processors 210 and a memory 220. When the computer program stored in the memory 220 is executed by the processor 210, the eye movement tracking method according to any one of the foregoing embodiments is implemented.

[0123] For example, when the computer program is executed by the processor 210, the following eye movement tracking method is implemented:

[0124] 010: Collect static frame image information of the eye region and a predetermined range around the eye by an infrared camera, generate a low-frame-rate initial eye model based on the static frame image information, perform key-point detection on the static frame image information by a key-point detection model to obtain key-point information, and update the low-frame-rate initial eye model according to the key-point information to obtain a first eye model;

[0125] 020: Collect dynamic event stream information generated by eye movement by an event camera 20, and generate a second eye model based on the dynamic event stream information;

[0126] 030: Establish a high-frame-rate eye movement model based on the first eye model and the second eye model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

[0127] For another example, when a computer program is executed by a processor 210, an eye movement tracking method as follows is implemented:

[0128] 011: Perform image preprocessing on the static frame image information to obtain a grayscale image;

[0129] 012: Perform edge detection on the grayscale image to obtain an edge image;

[0130] 013: Calculate a structure tensor matrix according to the edge image, and perform threshold processing on the edge image to obtain a binary image;

[0131] 014: Perform contour extraction on the binary image to obtain pupil contour information and corneal contour information;

[0132] 015: Calculate a corner response function according to the eigenvalues of the structure tensor matrix;

[0133] 016: Perform non-maximum suppression according to the corner response function to obtain eyelid information.

[0134] It should be noted that the explanations of the eye movement tracking method and the eye movement tracking device 100 in the foregoing embodiments are equally applicable to the eye movement tracking system 200 of the embodiments of the present application, and will not be elaborated herein.

[0135] Please refer to Figure 14 , the embodiments of the present application further provide a computer-readable storage medium 300, on which a computer program 310 is stored. When the program is executed by a processor 320, the eye movement tracking method of any of the foregoing embodiments is implemented.

[0136] For example, when the program is executed by a processor 320, an eye movement tracking method as follows is implemented:

[0137] 010: Collect static frame image information of the eye region and a predetermined range around the eye by the infrared camera 10, generate a low-frame-rate initial eye model based on the static frame image information, perform key-point detection on the static frame image information by the key-point detection model to obtain key-point information, and update the low-frame-rate initial eye model according to the key-point information to obtain the first eye model;

[0138] 020: Collect dynamic event stream information generated by eye movement by the event camera 20, and generate a second eye model based on the dynamic event stream information;

[0139] 030: Establish a high-frame-rate eye movement model based on the first eye model and the second eye model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

[0140] For another example, when the program is executed by the processor 320, the following eye movement tracking method is implemented:

[0141] 011: Perform image preprocessing on the static frame image information to obtain a grayscale image;

[0142] 012: Perform edge detection on the grayscale image to obtain an edge image;

[0143] 013: Calculate the structure tensor matrix according to the edge image, and perform threshold processing on the edge image to obtain a binary image;

[0144] 014: Perform contour extraction on the binary image to obtain pupil contour information and corneal contour information;

[0145] 015: Calculate the corner response function according to the eigenvalues of the structure tensor matrix;

[0146] 016: Perform non-maximum suppression according to the corner response function to obtain eyelid information.

[0147] It should be noted that the explanations of the eye movement tracking method and the eye movement tracking device 100 in the foregoing embodiments are equally applicable to the computer-readable storage medium 300 of the embodiments of the present application, and will not be elaborated herein.

[0148] The eye movement tracking method, the eye movement tracking device 100, the eye movement tracking system 200, and the computer-readable storage medium 300 of the embodiments of the present application introduce the event camera 20 to collect dynamic event stream information generated by eye movement, strengthen the recognition of moving targets, the dynamic event stream information is complementary to the static frame image information, and jointly participate in the eye movement tracking process, which can effectively improve the frame rate of the eye movement tracking device 100 and reduce the power consumption of the eye movement tracking device 100 while achieving high-resolution eye movement tracking.

[0149] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0150] Any process or method description represented in the flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a manner not shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of this application belong.

[0151] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable storage medium 300 for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, the computer-readable storage medium 300 can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of the computer-readable storage medium 300 include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium 300 can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0152] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0153] Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium 300. When the program is executed, it includes one or a combination of the steps of the method embodiments. In addition, in each embodiment of the present application, each functional unit can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage media mentioned above can be read-only memories, magnetic disks, optical disks, etc.

[0154] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. An eye movement tracking method, characterized in that, Including: Collect static frame image information of the eye region and a predetermined range region around the eye through an infrared camera, generate a low-frame-rate initial eye model according to the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low-frame-rate initial eye model according to the key point information to obtain a first eye model; Collect dynamic event stream information generated by eye movement through an event camera, and generate a second eye model according to the dynamic event stream information; Establish a high-frame-rate eye movement model based on the first eye model and the second eye model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

2. The eye movement tracking method according to claim 1, wherein The performing key point detection on the static frame image information through the key point detection model to obtain key point information includes: Perform image preprocessing on the static frame image information to obtain a grayscale image; Perform edge detection on the grayscale image to obtain an edge image; Calculate a structure tensor matrix according to the edge image, and perform threshold processing on the edge image to obtain a binary image; Perform contour extraction on the binary image to obtain pupil contour information and corneal contour information; Calculate a corner response function according to the eigenvalues of the structure tensor matrix; Perform non-maximum suppression according to the corner response function to obtain eyelid information.

3. The eye movement tracking method according to claim 1, characterized in that, The generating the second eye model according to the dynamic event stream information includes: Generate a high-frame-rate initial eye model; Perform motion estimation on the dynamic event stream information through a motion detection model to obtain motion information; Update the high-frame-rate initial eye model according to the motion information to obtain the second eye model.

4. The eye tracking method according to claim 3, wherein: The performing motion estimation on the dynamic event stream information through the motion detection model to obtain motion information includes: Decode the dynamic event stream information to obtain event timestamps, pixel position information, and pixel polarity information; Perform filtering processing on the pixel polarity information according to the event timestamps and the pixel position information; Perform event clustering on the pixel polarity information according to the event timestamps and the pixel position information; Perform motion estimation according to the pixel polarity information after event clustering to obtain the motion information.

5. The eye movement tracking method according to claim 1, wherein The eye movement tracking method further includes: Perform time alignment and spatial alignment on the infrared camera and the event camera so that the parameters of the first eye model and the second eye model are aligned.

6. The eye movement tracking method according to claim 1, wherein The determining eye movement information and predicted eye movement information according to the high-frame-rate eye movement model includes: Determine eye position information and pupil position information according to the high-frame-rate eye movement model; Determine predicted eye position information and predicted pupil position information according to the high-frame-rate eye movement model; Calculate the eye movement information according to the eye position information and the pupil position information; Calculate the predicted eye movement information according to the predicted eye position information and the predicted pupil position information.

7. The eye movement tracking method according to claim 1, characterized in that, The number of the infrared cameras is one or more; and / or the number of the event cameras is one or more.

8. An eye movement tracking device, characterized in that, Including an infrared camera, an event camera, and an eye movement tracking module; The infrared camera is used to collect static frame image information of the eye region and a predetermined range region around the eye. The eye movement tracking module is used to generate a low-frame-rate initial eye model according to the static frame image information, perform key point detection on the static frame image information through a key point detection model to obtain key point information, and update the low-frame-rate initial eye model according to the key point information to obtain a first eye model; The event camera is used to collect dynamic event stream information generated by eye movement. The eye movement tracking module is further used to generate a second eye model according to the dynamic event stream information; The eye movement tracking module is further used to establish a high-frame-rate eye movement model based on the first eye model and the second eye model, and determine eye movement information and predicted eye movement information according to the high-frame-rate eye movement model.

9. An eye movement tracking system, characterized in that, It includes one or more processors and a memory. The memory stores a computer program. When the computer program is executed by the processor, the eye movement tracking method according to any one of claims 1-7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the eye movement tracking method according to any one of claims 1-7 is implemented.

Images