Human eye movement tracking system and method

Abstract

The present invention discloses a human eye movement tracking system and method. The system includes: a first camera for capturing a first facial image of a user; a second camera for capturing a screen image of a display screen, which is configured to display a customized stimulus image; an infrared illuminator for emitting infrared light toward the user's eyeballs; a signal processing unit for sending an instruction to the display screen to display the stimulus image, and determining the user's initial line of sight based on the first facial image, the stimulus image, and the position of the infrared illuminator; setting a camera located in the direction of the first line of sight as an auxiliary camera, and using the auxiliary camera to capture a second facial image of the user; and splicing a target eye image of the user based on the first and second facial images. The target line of sight direction is then determined based on the target eye image and the screen image. Application of the embodiments of the present invention can improve the accuracy of the determined user's line of sight.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image processing, and more particularly to a human eye movement tracking system and method. Background Art

[0002] Infrared eye tracking uses infrared light to track a user's visual focus. The basic principle is this: a beam of near-infrared light is directed at the user's face. The user's eyeball reflects the light, forming a light spot at the reflection point. This light spot is captured by the camera when the eye image is captured. Image processing is then used to determine the pupil center. The corneal reflection point is then used as the reference point for the relative position of the eye camera and the eyeball. Based on the pupil center position obtained through image processing, the offset of the pupil center relative to the corneal reflection point is used to derive the line of sight coordinates, thereby determining the gaze point.

[0003] Due to individual differences in human eye shape, size, structure, facial posture, etc., there is a nonlinear relationship between the projection point position of a point on the eye's spherical surface in the camera reference system and the eye rotation angle, and there is a model error between the estimated gaze direction and the actual gaze direction. Therefore, using a single camera to capture user eye movement data to obtain user gaze data is not technically accurate enough. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a human eye movement tracking system and method to improve the accuracy of the determined user sight line data.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] The present invention provides a human eye movement tracking system, which includes: a first camera, a second camera, an auxiliary camera, an infrared illuminator, a signal processing unit and a display screen, wherein:

[0007] The first camera is used to capture a first facial image of the user;

[0008] The second camera user captures the screen image of the display screen, and the display screen is used to display the customized stimulus picture;

[0009] The infrared illuminator is used to emit infrared light towards the user's eyeballs;

[0010] The signal processing unit is used to send an instruction to display a stimulation picture to the display screen, determine the user's initial line of sight direction based on the first facial image, the stimulation picture, and the position of the infrared lighting; set the camera located in the first line of sight direction as an auxiliary camera, and use the auxiliary camera to capture the user's second facial image; and splice the user's target eyeball image based on the user's first facial image and the second facial image, and obtain the target line of sight direction based on the target eyeball image and the screen image.

[0011] Optionally, one or a combination of the first camera, the second camera, and the infrared lighting lamp is installed on the wearable device.

[0012] Optionally, the signal processing unit is further used to obtain eye movement parameters of the user, wherein the eye movement parameters include: number of blinks, pupil diameter, eye center coordinates, and eye center movement trajectory.

[0013] The present invention further provides a method for tracking human eye movement, which is applied to the signal processing unit in the above system, and the method comprises:

[0014] Sending a command to display a stimulus image to the display screen, obtaining a first facial image of the user captured by the first camera; using a pre-trained eye detection model to identify the pupil center position and the area of ​​the reflected light spot from the first facial image;

[0015] The user's initial gaze direction is calculated using the gaze mapping model based on the offset of the pupil center relative to the light spot center.

[0016] Setting a camera located in the first sight direction as an assisting camera, and using the assisting camera to capture a second facial image of the user;

[0017] According to the respective positions of the first camera and the auxiliary camera, the first facial image and the second facial image are spliced ​​into a target eye image, and the target sight direction is obtained according to the target eye image and the screen image.

[0018] Optionally, identifying the pupil center position from the first facial image using a pre-trained eye detection model includes:

[0019] Identifying an eye image region from the first facial image, and first cropping the eye image from the user's facial image based on a contour line of the eye image region;

[0020] The eye image is processed in sequence by grayscale, Gaussian filtering and binarization to obtain a preprocessed image;

[0021] Use the pre-trained eye detection model to identify the pupil area from the pre-processed image;

[0022] Screening out a target area from the pupil area according to a preset binarization threshold;

[0023] The pupil edge is detected from the target area using an edge detection algorithm, and the pupil edge is fitted into an ellipse to obtain the fitted target area;

[0024] Eliminate abnormal pupils in the target area according to preset pupil constraint conditions;

[0025] The center point of the target area after excluding the abnormal pupil is taken as the pupil center position.

[0026] Optionally, setting a camera located in the first sight direction as an assisting camera includes:

[0027] A virtual ray is generated with the user's eyes as the starting point and the direction corresponding to the first line of sight as the direction, and a camera located on the virtual ray is used as a candidate camera;

[0028] For each alternative camera, an alternative camera that can capture the user's eyeballs is used as an auxiliary camera.

[0029] Optionally, setting a camera located in the first sight direction as an assisting camera includes:

[0030] The user's eyes are used as the starting point, the direction corresponding to the first line of sight is used as the central axis, and the angle is set as the cone angle to generate a first virtual cone. The cameras within the range of the first virtual cone are used as candidate cameras.

[0031] For each alternative camera, an alternative camera that can capture the user's eyeballs is used as an auxiliary camera.

[0032] Optionally, stitching the first facial image and the second facial image into a target eye image according to respective positions of the first camera and the auxiliary camera includes:

[0033] According to the distribution positions of the first camera and the auxiliary cameras, the eye area corresponding to each camera is evenly divided;

[0034] Calculating the size of the clipping cone angle according to the corresponding eyeball area, the distance between the first camera and the eyeball, and the distance between the auxiliary camera and the eyeball;

[0035] For each of the first camera and the auxiliary camera, a second virtual cone is constructed by cropping the cone angle with the optical axis of the camera as the central axis, and an area enclosed by an intersection line of the second virtual cone and the captured eye image is used as a cropping area. A corresponding sub-image is cropped from the captured eye image according to the cropping area;

[0036] According to the coordinates of the corresponding camera, the pixel points of each sub-image are converted into the same coordinate system. According to the eyeball area corresponding to each sub-image, the sub-images are spliced ​​into a complete eyeball image, and the complete eyeball image is used as the target eyeball image.

[0037] Optionally, the process of acquiring the sight line mapping model includes:

[0038] using the screen image captured by the second camera, identifying the display position of the stimulus picture from the screen image;

[0039] The user's line of sight direction is obtained according to the pupil center position and the display position of the stimulus image; the line of sight mapping model is corrected using the position of the infrared lighting lamp, the center position of the reflected light spot, and the offset of the pupil center position relative to the reflected light spot.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] The present invention determines the user's initial line of sight direction through the first camera, and then determines the auxiliary camera based on the initial line of sight direction. Since the auxiliary camera is determined based on the initial line of sight direction, the user's eye image taken by the camera is more accurate. At the same time, the first camera is closer, so the shooting results of the first camera and the auxiliary camera are spliced ​​into a target eye image. Therefore, the overall distortion of the target eye image is smaller than the total distortion of the single camera shooting, and thus the target line of sight determined based on the target eye image is more accurate.

[0042] Moreover, multiple cameras each capture sub-images within a very small range in the direction of their own optical axis, and then use the sub-images to stitch together the target eye image. Since the distortion of the image captured by the camera along the optical axis is minimal, the accuracy of eye tracking is further improved.

[0043] In addition, the cooperation between multiple cameras can eliminate errors and external interference when shooting with a single camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic structural diagram of a human eye movement tracking system provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the relative positions of the reflected light spot and the pupil in the human eye movement tracking method provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the principle of pupil center position identification in a human eye movement tracking method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0048] Example 1

[0049] Embodiment 1 of the present invention provides a human eye movement tracking system, Figure 1 A schematic diagram of the structure of a human eye movement tracking system provided by an embodiment of the present invention is shown in FIG. Figure 1 The system includes: a first camera 3, a second camera 6, an auxiliary camera, an infrared illuminator 2, a signal processing unit 5, and a display screen 0. The first camera is set on the base 1 and is used to collect the wearer's eye image. The second camera is set on the base and is used to collect the visual field image of the stimulation picture directly in front of the wearer. The signal processing unit is connected to the first camera, the second camera, the infrared illuminator, and the display screen 0 respectively. The display screen 0 is directly in front of the visual field, and customized information can be projected on the screen, wherein,

[0050] The first camera is used to capture a first facial image of the user; the first facial image includes pixel content corresponding to the user's eyeball 4.

[0051] The second camera user captures the screen image of display screen 0, and display screen 0 is used to display the customized stimulus picture;

[0052] The infrared illuminator is used to emit infrared light towards the user's eyeballs;

[0053] The signal processing unit 5 is configured to send a command to the display screen 0 to display a stimulus image, determine the user's initial gaze direction based on the first facial image, the stimulus image, and the position of the infrared illuminator 2, set a camera located in the first gaze direction as an auxiliary camera, and use the auxiliary camera to capture the user's second facial image; and splice the user's target eye image based on the first and second facial images, and obtain the target gaze direction based on the target eye image and the screen image. The signal processing unit includes image processing and communication modules. Image processing is used to extract eye movement information and track the visual axis. The communication module matches and parses the eye tracking results with custom information. This unit can upload real-time processed data to a register, server, host computer, or memory.

[0054] like Figure 1As shown, for example, the base 1 can be a common pair of glasses or a bracket of a similar frame. The first camera 3 can be an iris camera, whose function is to image part of the human eye. The first camera 3 is on the frame-type base, and is installed in a position slightly below and in front of the human eye. The shooting angle is based on capturing the entire human eye and face. In an embodiment not shown, it can also be fixed on the left, right or top of the frame. The first camera 3 is communicatively connected to the signal processing unit 5, and the first facial image is transmitted to the signal processing unit 5. The specific connection method can be through a data cable connection or a wireless connection.

[0055] Infrared illuminator 2 is used to provide fill light for the camera and provide a visual axis reference. This infrared illuminator must be eye-safe. It can be placed anywhere within the base frame, as long as the light is incident on the eye. The relative spatial position between infrared illuminator 2 and first camera 3 can be determined based on actual needs. In practical applications, two or more infrared illuminators may be used.

[0056] Display screen 0 is placed directly in front of the eye movement monitoring device and is also connected to signal processing unit 5. The screen can display information sent by the information unit, such as a cross frame in the center of the screen. The connection between display screen 0 and signal processing unit 5 can be wired or wireless, and the screen can display a cross, a dot, or other items anywhere on the screen. To adapt to the shape of the user's eyes and improve viewing comfort, the display screen can be spherical.

[0057] The signal processing unit 5 determines the user's initial line of sight direction using the pupil cornea reflection method based on the first facial image, the stimulus picture and the position of the infrared lighting lamp 2; the specific determination process is existing technology and will not be repeated here in the embodiment of the present invention.

[0058] A camera located in the extension direction of the first line of sight is then set as an auxiliary camera, and a second facial image of the user is captured using the auxiliary camera. A first eye region is identified from the first facial image, and a second eye region is identified from the second facial image. A corresponding sub-image is then cut from the first eye region, and another corresponding sub-image is cut from the second eye region. Iris feature point matching is then used to stitch the sub-images together to form a target eye image. The target line of sight direction is then determined based on the target eye image and the screen image.

[0059] The present invention can detect and track eye movement information including the visual axis through the first camera and the customized information on the display screen 0, and the required hardware equipment is simple and low-cost.

[0060] Furthermore, in a specific implementation of Example 1 of the present invention, the signal processing unit is also used to obtain the user's eye movement parameters, wherein the eye movement parameters include: number of blinks, pupil diameter, eye center coordinates, and eye center movement trajectory.

[0061] Example 2

[0062] Embodiment 2 of the present invention is described using gaze tracking as an example. In practical applications, the embodiment of the present invention can also be used to count data such as the number of blinks.

[0063] Based on Example 1 of the present invention, Example 2 of the present invention provides a method for tracking human eye movement, which is applied to a signal processing unit in the system described in Example 1. The method includes:

[0064] S101: Sending an instruction to display a stimulus image to a display screen, obtaining a first facial image of a user captured by a first camera; and using a pre-trained eye detection model to identify the pupil center position and the area of ​​the reflected light spot.

[0065] Specifically, Figure 2 Schematic diagram of the relative positions of the reflected light spot and the pupil in the human eye movement tracking method provided by an embodiment of the present invention, as shown in FIG. Figure 2 As shown, the signal processing unit sends a cross-frame instruction to the display screen. Upon receiving the instruction, the display screen displays the cross-frame. When the user views the cross-frame, they rotate their eyeballs 201. The infrared illuminator emits infrared light, which is reflected by the eyeballs to form a reflected light spot. The first camera captures a first facial image of the user. The first facial image includes the user's eyeball area, that is, the user's pupil 203 and the reflected light spot 205.

[0066] Figure 3 A schematic diagram of the principle of pupil center position recognition in a method for tracking human eye movement provided by an embodiment of the present invention is shown in FIG. Figure 3 The signal processing unit may utilize a preset recognition algorithm, such as a neural network algorithm, to identify the eye image region from the first facial image. However, since the eye image region identified at this time is only an approximate region and is not sufficiently precise, an edge detection algorithm is required to further identify the corresponding contour line from the eye image region. The eye image is first cropped from the user's facial image based on the contour line of the eye image region.

[0067] Then, the eye image is gray-scaled, Gaussian filtered, and binarized using a preset algorithm to obtain a pre-processed image;

[0068] Use the pre-trained eye detection model to identify the pupil area from the pre-processed image;

[0069] Screening out a target area from the pupil area according to a preset binarization threshold;

[0070] The edge detection algorithm is used to detect the pupil edge from the target area. Since the first camera is not set at the crosshair position of the screen, there is a certain angle between the shooting direction of the first camera and the user's visual axis direction. Therefore, when the first camera captures the pupil, the pupil is elliptical in shape. Therefore, the pupil edge can be fitted into an ellipse to obtain the fitted target area.

[0071] According to the preset pupil constraint conditions, if the pupil diameter exceeds the preset range, it means that the model cannot effectively identify the pupil, or if the ratio of the major axis to the minor axis of the ellipse fitted by the pupil shape exceeds the set range, it means that the fitting is not accurate enough. In this case, the first facial image taken cannot be used, thereby achieving the purpose of excluding abnormal pupils in the target area;

[0072] The center point of the target area after excluding the abnormal pupil is taken as the pupil center position.

[0073] Similarly, a neural network recognition algorithm can be used to identify the area of ​​the reflected light spot.

[0074] S102: Calculate the user's initial sight direction using a sight mapping model according to the offset of the pupil center position relative to the light spot center position.

[0075] Exemplarily, a pupil-corneal reflection method, that is, a two-dimensional line of sight estimation method, may be used to estimate the initial line of sight direction.

[0076] In practical applications, the screen image captured by the second camera can be used to identify the display position of the stimulus picture from the screen image; the user's line of sight direction is obtained based on the pupil center position and the display position of the stimulus picture; and the line of sight mapping model is corrected using the position of the infrared lighting, the center position of the reflected light spot, and the offset of the pupil center position relative to the reflected light spot.

[0077] When correcting the gaze mapping model, the two-dimensional eye movement features extracted from the eye image are input as the independent variable of the mapping function, and the dependent variable of the function is the desired gaze direction or gaze point.

[0078] In order to obtain the sight mapping function, online calibration is required for each user. The sight mapping function model is expressed as follows:

[0079] Px=f(Vx,Vy)

[0080] Py=g(Vx,Vy)

[0081] Among them, (Px, Py) is the coordinate of the sight point, which is the coordinate of the stimulus image obtained by the second camera, and (Vx, Vy) is the pupil reflection spot vector. Figure 3 The signal processing unit uses the first camera to capture the first facial image and calculates the vector of the light spot reflected on the pupil. The eyeball's focus point on the display screen can be determined by line of sight estimation, and the relationship between the eyeball and the information on the display screen can be determined by the focus point. For example, using a 9-point 6-parameter model, the line of sight mapping model can be expressed as

[0082] Px=a0+a1Vx+a2Vy+a3VxVy+a4Vx2+a5Vy2

[0083] Py=b0+b1Vx+b2Vy+b3VxVy+b4Vx2+b5Vy2

[0084] Among them, Vx is the horizontal component of the relative offset vector of the pupil center relative to the center of the reflected light spot; Vy is the vertical component of the relative offset vector of the pupil center relative to the center of the reflected light spot; a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, and b5 are correction coefficients, which are the correction targets of the model and are determined during the correction process.

[0085] Then, using the nine first facial images, 18 equations are generated to perform polynomial regression on the coefficients a0, a1, a2, a3, a4, and a5, and b0, b1, b2, b3, b4, and b5, solving the coefficients to determine the mapping model. The mapping model modification process is known from the prior art and will not be further described in detail in this embodiment of the present invention.

[0086] S103: Setting a camera located in the first sight line direction as an assisting camera, and using the assisting camera to capture a second facial image of the user.

[0087] In practical applications, a two-dimensional line of sight estimation method can be used to estimate the user's focus point on the display screen. A virtual ray is constructed with the user's eyeball as the starting point and the focus point as the point on the ray, and then the initial line of sight direction can be obtained.

[0088] Since some alternative cameras may not be able to capture the user's eyes, in order to avoid this situation, for each alternative camera, an alternative camera that can capture the user's eyes will be used as an auxiliary camera.

[0089] In another specific implementation of this step, to increase the number of assisting cameras, a first virtual cone can be generated, starting with the user's eye and the direction corresponding to the first line of sight as the central axis. An angle is set as the cone angle. The first virtual cone is symmetrically distributed along the central axis and extends along the central axis. Cameras within the first virtual cone are selected as candidate cameras. For each candidate camera, the candidate camera that can capture the user's eye is selected as an assisting camera.

[0090] Furthermore, the signal processing unit transmits a capture assistance request to the alternative camera, wherein the capture assistance request includes the first coordinate of the first camera. The alternative camera that receives the capture assistance request calculates its optimal capture range based on its second coordinate and the optimal capture distance pre-calibrated by the manufacturer. When the first coordinate falls within the optimal capture range, the alternative camera responds to the capture assistance request and is used as the camera to capture the user's facial image. By applying this embodiment of the present invention, assisting cameras with better capture effects can be selected, thereby improving the accuracy of gaze tracking.

[0091] S104: splicing the first facial image and the second facial image into a target eye image according to the respective positions of the first camera and the auxiliary camera, and acquiring a target sight direction according to the target eye image and the screen image.

[0092] Specifically, the eye area corresponding to each camera is evenly divided based on the distribution of the first camera and the auxiliary cameras. For example, if the first camera is on the left, and the auxiliary cameras are in the middle and right, then each camera corresponds to one-third of the eye area. The left third of the eye area is measured from the first facial image captured by the first camera, the middle third of the eye area is measured from the auxiliary camera in the middle, and the right third of the eye area is measured from the right camera.

[0093] In actual applications, the pattern formed by the connecting lines of the projection points of the distribution positions of each camera on the vertical plane can be a triangle, a quadrilateral, a pentagon, a hexagon, etc. Therefore, the eyeball area corresponding to each camera should also be in the corresponding position in the eyeball.

[0094] In order to briefly explain the technical process for achieving the above-mentioned effects, the embodiment of the present invention uses a first camera and an assisting camera as an example to explain the principle. The first camera is located on the left side relative to the assisting camera. Therefore, the eyeball area corresponding to the first camera is one-half on the left side, and the eyeball area corresponding to the assisting camera is one-half on the right side, that is, half of the eyeball image is obtained from the first facial image taken by the first camera and the second facial image taken by the assisting camera. Therefore, half of the eyeball image is obtained from the first facial image taken by the first camera and the second facial image taken by the assisting camera. Therefore, the cone angle corresponding to the first camera is calculated with half of the diameter of the eyeball image in the first facial image as the arc length and the distance from the first camera to the eyeball as the radius, and then the second virtual cone is obtained with the line connecting the first camera to the eyeball, that is, the optical axis direction of the first camera, as the central axis. Similarly, the method for obtaining the second virtual cone of the assisting camera is similar to the above method.

[0095] using an area enclosed by an intersection line of the second virtual cone and the captured eye image as a cropping area, and cropping a corresponding sub-image from the captured eye image according to the cropping area;

[0096] According to the coordinates of the corresponding cameras, the pixel points of each sub-image are converted to the same coordinate system, and the sub-image corresponding to the first camera and the sub-image corresponding to the assisting camera are scaled to the same size. The sub-image corresponding to the first camera is placed on the left, and the sub-image corresponding to the assisting camera is placed on the right. The sub-images are spliced ​​into a complete eyeball image using the method of feature point overlap, and the complete eyeball image is used as the target eyeball image.

[0097] Furthermore, during stitching, because sub-images corresponding to multiple cameras are used to stitch the target eye image, even if the grayscale value of the pupil in a sub-image is lower than a set value, the remaining sub-images can be used to fit the complete pupil image, thereby obtaining the offset vector of the pupil center relative to the reflected light spot. If a single camera is used to obtain the pupil center point, and if the pupil has a low reflectivity to infrared light at that angle, gaze tracking in this situation may result in significant errors. Therefore, embodiments of the present invention can achieve a high error tolerance.

[0098] Then, the offset vector of the pupil center point relative to the reflected light spot obtained from the target eyeball image is input into the modified line of sight mapping model, and the user's target focus point can be obtained, and the line between the user's eyeball and the target focus point is used as the user's target line of sight.

[0099] In practical applications, obtaining the target sight direction based on the target eyeball image and the screen image may be an existing technology.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A human eye movement tracking system, characterized in that: The system includes: a first camera, a second camera, an auxiliary camera, an infrared illuminator, a signal processing unit and a display screen, wherein: The first camera is used to capture a first facial image of the user; The second camera user captures the screen image of the display screen, and the display screen is used to display the customized stimulus picture; The infrared illuminator is used to emit infrared light towards the user's eyeballs; The signal processing unit is configured to send an instruction to display a stimulation image to a display screen, determine the user's initial sight direction based on the first facial image, the stimulation image, and the positions of the infrared lighting; set a camera located in the first sight direction as an auxiliary camera, and use the auxiliary camera to capture a second facial image of the user; and splice a target eye image of the user based on the first facial image and the second facial image of the user, and obtain a target sight direction based on the target eye image and the screen image; splice the first facial image and the second facial image into the target eye image based on the respective positions of the first camera and the auxiliary camera, including: According to the distribution positions of the first camera and the auxiliary cameras, the eye area corresponding to each camera is evenly divided; Calculating the size of the clipping cone angle according to the corresponding eyeball area, the distance between the first camera and the eyeball, and the distance between the auxiliary camera and the eyeball; For each of the first camera and the auxiliary camera, a second virtual cone is constructed by cropping the cone angle with the optical axis of the camera as the central axis, and an area enclosed by an intersection line of the second virtual cone and the captured eye image is used as a cropping area. A corresponding sub-image is cropped from the captured eye image according to the cropping area; According to the coordinates of the corresponding camera, the pixel points of each sub-image are converted into the same coordinate system. According to the eyeball area corresponding to each sub-image, the sub-images are spliced ​​into a complete eyeball image, and the complete eyeball image is used as the target eyeball image.

2. The human eye movement tracking system according to claim 1, characterized in that: One or a combination of the first camera, the second camera, and the infrared lighting lamp is installed on the wearable device.

3. The human eye movement tracking system according to claim 1, characterized in that: The signal processing unit is further configured to obtain eye movement parameters of the user, wherein the eye movement parameters include: number of blinks, pupil diameter, eyeball center coordinates, and eyeball center movement trajectory.

4. A method for tracking human eye movement, characterized in that: Applied to the signal processing unit in the system according to any one of claims 1 to 3, the method comprising: Sending a command to display a stimulus image to the display screen, obtaining a first facial image of the user captured by the first camera; using a pre-trained eye detection model to identify the pupil center position and the area of ​​the reflected light spot from the first facial image; The user's initial gaze direction is calculated using the gaze mapping model based on the offset of the pupil center relative to the light spot center. Setting a camera located in the first sight direction as an assisting camera, and using the assisting camera to capture a second facial image of the user; splicing the first facial image and the second facial image into a target eye image according to the respective positions of the first camera and the auxiliary camera, and obtaining a target sight direction according to the target eye image and the screen image; The step of stitching the first facial image and the second facial image into a target eye image according to respective positions of the first camera and the auxiliary camera includes: According to the distribution positions of the first camera and the auxiliary cameras, the eye area corresponding to each camera is evenly divided; Calculating the size of the clipping cone angle according to the corresponding eyeball area, the distance between the first camera and the eyeball, and the distance between the auxiliary camera and the eyeball; For each of the first camera and the auxiliary camera, a second virtual cone is constructed by cropping the cone angle with the optical axis of the camera as the central axis, and an area enclosed by an intersection line of the second virtual cone and the captured eye image is used as a cropping area. A corresponding sub-image is cropped from the captured eye image according to the cropping area; According to the coordinates of the corresponding camera, the pixel points of each sub-image are converted into the same coordinate system. According to the eyeball area corresponding to each sub-image, the sub-images are spliced ​​into a complete eyeball image, and the complete eyeball image is used as the target eyeball image.

5. The method for tracking eye movement according to claim 4, wherein: The method of identifying the pupil center position from the first facial image using a pre-trained eye detection model includes: Identifying an eye image region from the first facial image, and first cropping the eye image from the user's facial image based on a contour line of the eye image region; The eye image is processed in sequence by grayscale, Gaussian filtering and binarization to obtain a preprocessed image; Use the pre-trained eye detection model to identify the pupil area from the pre-processed image; Screening out a target area from the pupil area according to a preset binarization threshold; The pupil edge is detected from the target area using an edge detection algorithm, and the pupil edge is fitted into an ellipse to obtain the fitted target area; Eliminate abnormal pupils in the target area according to preset pupil constraint conditions; The center point of the target area after excluding the abnormal pupil is taken as the pupil center position.

6. The method for tracking eye movement according to claim 4, wherein: The step of setting the camera located in the first sight direction as an assisting camera includes: A virtual ray is generated with the user's eyes as the starting point and the direction corresponding to the first line of sight as the direction, and a camera located on the virtual ray is used as a candidate camera; For each alternative camera, an alternative camera that can capture the user's eyeballs is used as an auxiliary camera.

7. The method for tracking eye movement according to claim 4, wherein: The step of setting the camera located in the first sight direction as an assisting camera includes: The user's eyes are used as the starting point, the direction corresponding to the first line of sight is used as the central axis, and the angle is set as the cone angle to generate a first virtual cone. The cameras within the range of the first virtual cone are used as candidate cameras. For each alternative camera, an alternative camera that can capture the user's eyeballs is used as an auxiliary camera.

8. The method for tracking eye movement according to claim 4, wherein: The acquisition process of the sight line mapping model includes: using the screen image captured by the second camera, identifying the display position of the stimulus picture from the screen image; The user's line of sight direction is obtained according to the pupil center position and the display position of the stimulus image; the line of sight mapping model is corrected using the position of the infrared lighting lamp, the center position of the reflected light spot, and the offset of the pupil center position relative to the reflected light spot.

Images