An eye tracking method and apparatus for detecting head motion, removing head motion

Abstract

Embodiments of the present application provide an eye movement tracking method and device for detecting head movement and removing head movement, which can be related to the field of eye movement tracking. The method comprises: setting a first marker point at a first position around the eye of a test subject, collecting a first image of the eye region of the test subject, and determining a first pixel coordinate of the pupil in the first image and a second pixel coordinate of the first marker point; determining a position offset of the pupil relative to the head of the test subject according to the first pixel coordinate and the second pixel coordinate; and determining a gaze point position of the test subject according to the determined position offset. The method sets a marker point around the eye, as the marker point moves synchronously with the head, the position change of the marker point is used to calculate the change of the head movement, and the gaze point is positioned based on the position offset of the pupil relative to the head of the test subject, thereby eliminating the influence of the head movement of the test subject and improving the accuracy of the eye movement tracking result.

Description

Technical Field

[0001] This application belongs to the field of computer technology and may relate to the field of eye tracking. Specifically, this application relates to an eye tracking method and apparatus for detecting head movements and removing head movements. Background Technology

[0002] Eye-tracking technology can monitor and analyze human eye movements in real time, such as saccades, tracking, and fixation. With the continuous development of technology, eye-tracking technology is being applied in more and more fields, such as human-computer interaction and healthcare.

[0003] In eye-tracking, it is often necessary to quickly locate and track the pupil position in the subject's eye images to calculate the fixation point and form an eye trajectory. However, when only pupil position is used for analysis, even slight head movements by the subject can alter the spatial relationship between the pupil and the eye-tracking device, leading to inaccurate tracking results. Therefore, eliminating the influence of subject head movements is crucial for improving the accuracy of eye-tracking results. Summary of the Invention

[0004] This application provides an eye-tracking method and apparatus for detecting and removing head movements, which can eliminate errors caused by subject head movements and improve the accuracy of eye-tracking results. To achieve this objective, the technical solution provided by this application is as follows:

[0005] On one hand, embodiments of this application provide an eye-tracking method for detecting head movements and removing head movements, wherein a first marker point is set at a first position around the eyes of the subject, and the method is executed by an eye-tracking device, the eye-tracking device including a camera;

[0006] The method includes:

[0007] A first image of the subject's eye region is captured by the camera, and the first image includes the first marker point;

[0008] Determine the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point in the first image;

[0009] The gaze vector of the subject is determined based on the first pixel coordinate and the second pixel coordinate; wherein the gaze vector is the positional offset of the subject's pupil relative to the subject's head.

[0010] The gaze point position of the subject is determined based on the gaze vector.

[0011] Optionally, the eye-tracking device may also include a display;

[0012] Determining the gaze point position of the subject based on the gaze vector includes:

[0013] Based on the gaze vector and a pre-determined mapping relationship, the gaze point position of the subject on the display is determined;

[0014] The mapping relationship reflects the correspondence between the subject's gaze vector and the coordinate position on the display.

[0015] Optionally, determining the subject's gaze vector based on the first pixel coordinates and the second pixel coordinates includes:

[0016] The deviation between the first pixel coordinates and the second pixel coordinates is determined, and the deviation is used as the gaze vector of the subject.

[0017] Optionally, determining the subject's gaze vector based on the first pixel coordinates and the second pixel coordinates includes:

[0018] Obtain the coordinate system transformation parameters between the camera coordinate system and the world coordinate system;

[0019] The world coordinates of the first marker point are determined based on the coordinate system transformation parameters, the camera intrinsic parameters, and the second pixel coordinates.

[0020] The world coordinates of the pupil are determined based on the coordinate system transformation parameters, the camera intrinsic parameters, and the first pixel coordinates of the pupil.

[0021] The deviation between the world coordinates of the pupil and the world coordinates of the first marker point is determined as the gaze vector of the subject.

[0022] Optionally, the coordinate system transformation parameters are obtained in the following way:

[0023] The camera acquires a second image of the subject's eye area, the second image including the first marker point and the at least two reference points; wherein, the at least two reference points are at least two second marker points set at a second position around the subject's eyes, or, the at least two reference points are left corneal reflective spots and right corneal reflective spots formed when the infrared light source of the eye-tracking device illuminates the subject's eyes during the acquisition of the second image;

[0024] Obtain the world coordinates of at least two reference points and the first marker point in the second image;

[0025] Determine the pixel coordinates of at least two reference points and the first marker point in the second image;

[0026] Based on the pixel coordinates and world coordinates of at least two reference points in the second image, the pixel coordinates and world coordinates of the first marker point, and the camera intrinsic parameters, determine the coordinate system transformation parameters between the camera coordinate system and the world coordinate system.

[0027] Optionally, the mapping relationship is a first mapping relationship;

[0028] The first mapping relationship is obtained in the following way:

[0029] Acquire eye images of the subjects as they fixate on each calibration point;

[0030] For each calibration point, the pixel coordinates of the pupil and the first marker point in the eye image corresponding to the calibration point are determined; based on the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marker point, the gaze vector when the subject gazes at the calibration point is determined.

[0031] The first mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

[0032] Optionally, the eye-tracking device may also include a display;

[0033] Determining the gaze point position of the subject based on the gaze vector includes:

[0034] The intersection of the gaze vector and the display is taken as the gaze point position of the subject on the display.

[0035] Optionally, the eye-tracking device further includes a display; the subject's gaze point position is determined based on the gaze vector and a pre-determined mapping relationship, wherein the mapping relationship is a second mapping relationship;

[0036] The second mapping relationship is obtained in the following way:

[0037] Obtain the coordinate system transformation parameters and the eye images of the subjects when they gaze at each calibration point;

[0038] For each calibration point, the pixel coordinates of the pupil and the first marker point in the eye image corresponding to the calibration point are determined; the world coordinates of the pupil are determined based on the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters; the world coordinates of the first marker point are determined based on the pixel coordinates of the first marker point in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters; and the gaze vector when the subject gazes at the calibration point is determined based on the deviation between the world coordinates of the pupil and the world coordinates of the first marker point in the eye image corresponding to the calibration point.

[0039] The second mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

[0040] Optionally, the subject sets the marker points by wearing a frame-like device, with the first marker point positioned at a first position on the frame-like device and the at least two second marker points positioned at a second position on the frame-like device.

[0041] On the other hand, embodiments of this application provide an eye-tracking device for detecting and removing head movements. A first marker point is set at a first position around the eyes of the subject. The device is deployed in an eye-tracking device, which includes a camera.

[0042] The device includes:

[0043] An eye image acquisition module is used to acquire a first image of the eye region of the subject through the camera, the first image including the first marker point;

[0044] A pupil and marker point localization module is used to determine the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point in the first image.

[0045] A gaze vector determination module is used to determine the gaze vector of the subject based on the first pixel coordinates and the second pixel coordinates; wherein, the gaze vector is the position offset of the subject's pupil relative to the subject's head;

[0046] The gaze point localization module is used to determine the gaze point position of the subject based on the position offset.

[0047] On the other hand, embodiments of this application also provide an eye-tracking system, which includes a camera, a display, a memory, and a processor. The camera is used to acquire eye images of a subject, the memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0048] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0049] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.

[0050] The beneficial effects of the technical solution provided in this application are as follows:

[0051] The eye-tracking method provided in this application involves setting a first marker point at a first position around the subject's eyes, acquiring a first image of the subject's eye region, and determining the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point in the first image. Based on the first and second pixel coordinates, the positional offset of the pupil relative to the subject's head is determined, and the subject's fixation point position is determined based on the determined positional offset. This method sets marker points around the eyes, and since the marker points move synchronously with the head, the positional changes of the marker points are used to calculate changes in head movement. Fixation point localization is based on the positional offset of the pupil relative to the subject's head, eliminating the influence of the subject's head movements and improving the accuracy of eye-tracking results. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0053] Figure 1 This is a schematic diagram of the structure of an eye-tracking system applicable to the embodiments of this application;

[0054] Figure 2 This is a schematic flowchart illustrating an eye-tracking method for detecting and removing head movements, provided in an embodiment of this application.

[0055] Figure 3 This is a schematic diagram of setting the first marker point provided in an embodiment of this application;

[0056] Figures 4a-4d A schematic diagram of a human eye with multiple reflective light spots provided in an embodiment of this application;

[0057] Figures 5a-5b This is a schematic diagram illustrating the setting of three marker points in an embodiment of this application;

[0058] Figure 6This is a schematic diagram of the structure of an eye-tracking device for detecting and removing head movements, provided in an embodiment of this application.

[0059] Figure 7 This is a schematic diagram of the structure of an eye-tracking system provided in an embodiment of this application. Detailed Implementation

[0060] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0061] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0062] When conducting eye-tracking tasks, if tracking is based solely on the subject's pupil position, even slight head movements can alter the spatial relationship between the pupil and the eye-tracking device. If head movements are not eliminated, this can lead to significant positioning errors. For example, even if the subject's gaze point remains unchanged during eye-tracking, a leftward head movement can cause the eye-tracking device to misinterpret it as eye movement, resulting in inaccurate eye-tracking data.

[0063] To address the aforementioned problems, this application provides an eye-tracking method and apparatus for detecting and removing head movements. The method sets a first marker point at a first position around the subject's eyes. Based on a first image of the subject's eye region, it determines the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point. Based on the first and second pixel coordinates, it determines the positional offset of the pupil relative to the subject's head. Based on the determined positional offset, it determines the subject's fixation point position. This method sets marker points around the eyes. Since the marker points move synchronously with the head, the positional changes of the marker points are used to calculate changes in head movement. Fixation point localization is based on the positional offset of the pupil relative to the subject's head, eliminating the influence of the subject's head movements and improving the accuracy of eye-tracking results.

[0064] The eye-tracking method provided in this application can be applied to eye-tracking processes in any scenario, such as virtual reality (VR) technology, screen interaction control, eye pathology examination, and mental illness diagnosis assistance, etc. This application does not impose any limitations on it.

[0065] Figure 1 A schematic diagram of an eye-tracking system applicable to an embodiment of this application is shown. The eye-tracking system includes a first terminal 10 and an eye-tracking device 20. The eye-tracking device 20 includes at least an infrared light source 201, a camera 202, and a display 203. The infrared light source 201 is used to provide supplemental lighting to the eyes to acquire clear eye images. Alternatively, the infrared light source 201 may be a near-infrared light source.

[0066] When conducting eye-tracking tests on subjects, a first marker point can be set in the area around the subject's eyes.

[0067] While the subject observes the content displayed on the monitor 203, a first image of the subject's eye area is captured by the camera 202. The pupil and the first marker point in the first image are detected, and the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point are located. Based on the first pixel coordinates and the second pixel coordinates, the subject's gaze vector is determined. Based on the subject's gaze vector and the mapping relationship determined during the calibration process, the position of the subject's gaze point on the monitor 203 is determined.

[0068] In another embodiment of this application, 3D gaze tracking can also be used to locate the gaze point. Specifically, coordinate system transformation parameters (including transformation matrix R and translation vector T) between the camera coordinate system and the world coordinate system are obtained. Based on the coordinate system transformation parameters, the camera intrinsic parameters of camera 202, and the second pixel coordinates of the first marker point in the first image, the world coordinates of the first marker point are determined. Based on the coordinate system transformation parameters, the camera intrinsic parameters of the camera, and the first pixel coordinates of the pupil in the first image, the world coordinates of the pupil are determined. Based on the deviation between the world coordinates of the pupil and the world coordinates of the first marker point, the gaze vector of the subject is determined. Based on the gaze vector and the mapping relationship determined during the calibration process, the gaze point position of the subject in the display 203 is determined.

[0069] The technical solutions of this application and their effects are described below through several embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0070] Figure 2 This illustration shows a flowchart of an eye-tracking method for detecting and removing head movements, according to an embodiment of this application. This method can be executed by any electronic device, such as a terminal device. The terminal device is a device with eye-tracking / gaze-tracking functionality, such as a head-mounted eye tracker, a desktop eye tracker, or a smartphone. For ease of description, the following explanation will use an eye-tracking device as the execution subject.

[0071] See Figure 2 The eye-tracking method provided in this application embodiment may include the following steps S110 to S140.

[0072] S110: Acquire the first image of the subject's eye area using a camera.

[0073] The eye-tracking device includes a camera (such as an infrared camera, depth camera, or high-speed camera) and a display. When performing eye-tracking tests on subjects, a first marker point can be set at a first location around the subject's eyes. This first location can be any position around the eyes. Because the first marker point is set in the area around the eyes and changes synchronously with the subject's head movement, it can serve as a static reference point for eye movement, thus eliminating the influence of head movements.

[0074] Optionally, this application embodiment does not limit the method of setting the first marker point, as long as a fixed point can be formed around the eye. For example, it can be done by placing a marker, such as pasting the marker material on the first position of the eye area; or by applying the marker material to the first position of the eye area; or it can be done by wearing a frame-like device (such as eyeglasses, an eye mask that exposes the eye area, etc.) and setting the first marker point at the first position of the frame-like device to achieve the setting of the marker point around the eye. This application embodiment also does not limit the material used for the first marker point. The marker material used for the first marker point can be a reflective material, such as a material with a silver reflective surface, or a non-reflective material, such as a completely black material; the shape of the first marker point can also be set to any shape.

[0075] Furthermore, the embodiments of this application do not limit the position of the first marker point. For example, it can be set at the center of the eyebrows, below the eyes, etc., as long as it can be displayed in the image of the eye area collected by eye tracking.

[0076] After the first marker point is set, when the subject observes the target image displayed on the monitor of the eye-tracking device, a first image of the subject's eye area is captured by a camera. The captured first image includes the first marker point.

[0077] For example, such as Figure 3 As shown, the first marker is a marker pasted on the center of the eyebrows. This marker is a hemispherical silver reflective material that can produce a clear reflective spot on the image regardless of the angle from which the eye image is captured.

[0078] S120: Determine the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point in the first image.

[0079] For the first image acquired, the pupil center and the first marker point can be located from the first image, and the pixel coordinates of the pupil center in the first image can be determined, which are the first pixel coordinates of the pupil, and the second pixel coordinates of the first marker point in the first image.

[0080] In this application embodiment, the method for locating the pupil center and the first marker point is not limited. For example, when the first marker point is made of reflective material, the first marker point is brighter and the pupil is darker in the acquired first image. The grayscale threshold histogram method can be used to locate the pupil center and the first marker point based on the grayscale values ​​of each pixel in the first image.

[0081] S130: Determine the subject's gaze vector based on the first pixel coordinates and the second pixel coordinates.

[0082] S140: Determine the subject's gaze point location based on the gaze vector.

[0083] In this embodiment, the deviation between the first pixel coordinates and the second pixel coordinates can be determined, and the determined deviation is used as the subject's gaze vector. The gaze vector is the positional offset of the subject's pupil relative to the subject's head. Since the pupil position changes with the location of the gaze point, and the position of the first marker point changes with the subject's head movement, the gaze vector represents the change in eye movement and is not affected by head movement.

[0084] For example, suppose the pixel coordinates of the left eye pupil in the first captured image are ( , The pixel coordinates of the right pupil are ( , The pixel coordinates of the first marker point are ( , Then the corresponding positional offset of the left eye is: ; The gaze vector of the left eye is ( , The positional offset of the right eye (gaze vector) is: ; The gaze vector of the right eye is ( , ).

[0085] Optionally, the subject's gaze point position on the display is determined based on the gaze vector and a pre-determined mapping relationship. The mapping relationship can be determined during eye-tracking calibration and reflects the correspondence between the subject's gaze vector and the coordinate position on the display.

[0086] Optionally, when performing eye tracking based on a single marker point (i.e., the first marker point) and the pupil position, calibration can be performed in the following manner to obtain the first mapping relationship:

[0087] Acquire eye images of the subjects as they fixate on each calibration point;

[0088] For each calibration point, determine the pixel coordinates of the pupil and the first marker point in the eye image corresponding to that calibration point; based on the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marker point, determine the gaze vector when the subject gazes at that calibration point.

[0089] The first mapping relationship is determined based on the gaze vector corresponding to each calibration point and the position of each calibration point on the display.

[0090] The calibration point is a known location of the calibration point. In this embodiment, the number of calibration points is not limited. A 3-point calibration method, a 5-point calibration method, a 9-point calibration method, a 13-point calibration method, etc. can be used. When the subject looks at the calibration point at each location, the subject's eye image is collected.

[0091] After determining the gaze vector corresponding to each calibration point (i.e. the gaze vector when the subject gazes at each calibration point), multiple first mapping parameters are solved by parameter fitting based on the gaze vector corresponding to each calibration point and the position of each calibration point. Based on the multiple first mapping parameters, the first mapping relationship is obtained.

[0092] Related technologies determine eye movement changes in subjects based on pupil position changes, thereby locating the fixation point. However, head movements in subjects lead to significant positioning errors. In contrast, technologies based on… Figure 2 The eye-tracking method shown introduces a first marker point around the eye. The first marker point changes synchronously with the subject's head. The eye movement change is determined based on the positional offset between the pupil and the first marker point. This can eliminate the influence of the subject's head movement, reduce the interference of eye movement test caused by head movement, reduce the time required for eye movement calibration, and improve the accuracy of eye tracking.

[0093] In related technologies, there is also an eye-tracking method based on pupil-corneal reflex, which determines the fixation point position based on the relative position between the pupil and the corneal reflective spot in the human eye. However, the corneal reflective spot is uncertain and unstable, easily affected by various factors, such as disappearance due to obstruction by the upper and lower eyelids, disappearance of the reflective spot due to imperfect corneal surface, the formation of multiple reflective spots on the cornea, or even multiple reflective spots merging into a large area, etc., which interfere with the localization of the corneal reflective spot and cause significant deviations in its localization. For example, such as... Figures 4a-4d As shown, where, Figure 4a The reflected light spot in the right eye of the middle eye is elongated into a larger spot. Figure 4b The reflected light spot in the left eye is divided into two reflected light spots. Figure 4c The reflected light spot in the right eye is elongated into a larger spot, and a smaller spot appears below it. Figure 4d Three reflected light spots appeared in the left eye, and four reflected light spots appeared in the right eye. However, the eye-tracking method provided in this application has better stability and less noise interference because the first marker point is placed externally in the peripheral area of ​​the eye, and is not affected by factors such as eye structure and tearing.

[0094] In this embodiment of the application, since the first marker point is set in the area around the subject's eyes and moves synchronously with the subject's head, the subject's head movement can be detected by monitoring the positional change of the first marker point.

[0095] In one embodiment of this application, the eye-tracking device can be set to multiple detection modes. When a marker point is detected from the acquired first image, the eye-tracking method provided in this application is used to locate the gaze point based on the first marker point and the pupil. When no marker point is detected, the traditional pupil-corneal reflection method is used to locate the gaze point based on the pupil and the corneal reflected light spot.

[0096] In another embodiment of this application, the relative positional relationship (coordinate system transformation parameter) between the camera coordinate system and the world coordinate system can be calculated based on the first marker point and other reference points, so as to determine the changes in the subject's line of sight in 3D space based on the calculated coordinate system transformation parameter.

[0097] Typically, eye-tracking calibration is required before performing eye-tracking tasks. However, in this embodiment of the application, the coordinate system transformation parameters between the camera coordinate system and the world coordinate system need to be determined before eye-tracking calibration.

[0098] Specifically, firstly, at least two second markers are set at a second location around the subject's eyes. The positions of the different second markers vary. These at least two second markers serve as at least two reference points. Alternatively, the corneal reflection spots formed by the infrared light source in the eye-tracking device shining into the subject's eyes and the corneal reflection spots in the right eye are used as two reference points, respectively. Furthermore, a first marker is set at a first location around the subject's eyes.

[0099] For example, suppose three markers, 1, 2, and 3, are set on the eye area of ​​the subject, where marker 1 is the first marker, and markers 2 and 3 are reference points. Then, markers can be pasted on the subject's eye area, such as... Figure 5a As shown, place marker 1 at the center of the eyebrows, marker 2 on the outer side of the lower eyelid of the left eye, and marker 3 on the outer side of the lower eyelid of the right eye; alternatively, this can be achieved by wearing a framed device, such as... Figure 5b As shown, the markers can be pre-attached to the corresponding positions on the eye mask. Marker 1 is attached to the center of the forehead on the eye mask, marker 2 is attached to the lower left eye position on the eye mask, and marker 3 is attached to the lower right eye position on the eye mask.

[0100] Afterwards, the subject can fixate on any coordinate position on the display of the eye-tracking device (such as the center of the display). The camera then captures a second image of the subject's eye area at this time, and determines the pixel coordinates of the first marker point and at least two reference points in the second image. The world coordinates of the first marker point and at least two reference points are then obtained. Based on the pixel coordinates and world coordinates of the at least two reference points in the second image, the pixel coordinates and world coordinates of the first marker point, and the camera intrinsic parameters, the coordinate system transformation parameters between the camera coordinate system and the world coordinate system are determined.

[0101] The world coordinates of the first marker point and at least two reference points can be calculated in a world coordinate system constructed with any fixed point as the origin. For example, the world coordinates of the first marker point and at least two reference points can be calculated in this world coordinate system, with the subject's nose tip as the origin, the direction parallel to both eyes (e.g., pointing towards the right eye) as the X-axis, the direction perpendicular to the gaze of both eyes as the Y-axis, and the direction perpendicular to the line connecting the eyes upward as the Z-axis.

[0102] Optionally, the same positioning method as the first marker point described above can be used to locate the reference point in the second image, such as using the grayscale threshold histogram method to segment the area with higher grayscale value (highlight area) from the second image, and then locate the pixel coordinates of the reference point.

[0103] The transformation relationship between the world coordinate system and the camera coordinate system is as follows:

[0104]

[0105] This indicates the coordinates of the marker point in the camera coordinate system. This indicates the coordinates of the marker point in the world coordinate system. , This represents the rotation matrix and translation vector from the world coordinate system to the camera coordinate system.

[0106] The transformation relationship between the pixel coordinate system and the camera coordinate system is as follows:

[0107]

[0108] p is the coordinate of the marker point in the pixel coordinate system. To mark the depth of the point in the camera coordinate system, This represents the coordinates of the marker point in the camera coordinate system, and K is the camera intrinsic parameter matrix.

[0109] Substituting the pixel coordinates and world coordinates of at least two reference points in the second image, the pixel coordinates and world coordinates of the first marker point, and the camera's intrinsic parameters into the above coordinate system transformation relationship, we obtain the coordinate system transformation parameters between the camera coordinate system and the world coordinate system. and .

[0110] In this embodiment, the PNP (Perspective-n-Point) algorithm can be used to solve for the coordinate system transformation parameters based on the world coordinates of at least three pairs of points in three-dimensional space and the pixel coordinates of the corresponding points projected onto the camera normalized plane. Of course, other methods can also be used, such as Direct Linear Transformation (DLT), EPnP, Bundle Adjustment (BA), etc.

[0111] After determining the coordinate system transformation parameters between the camera coordinate system and the world coordinate system, the next step is to perform eye-tracking calibration:

[0112] Acquire eye images of the subjects as they gaze at each calibration point; during the calibration process, the subjects gaze sequentially at each calibration point displayed on the monitor, and an eye image of the subjects gazing at each calibration point is captured by a camera;

[0113] For each calibration point, determine the pixel coordinates of the pupil and the first marker point in the eye image corresponding to that calibration point; determine the world coordinates of the pupil in the eye image corresponding to that calibration point based on the pixel coordinates of the pupil, camera intrinsic parameters, and coordinate system transformation parameters; determine the world coordinates of the first marker point in the eye image corresponding to that calibration point based on the pixel coordinates of the first marker point, camera intrinsic parameters, and coordinate system transformation parameters; determine the gaze vector when the subject gazes at that calibration point based on the deviation between the world coordinates of the pupil and the world coordinates of the first marker point in the eye image corresponding to that calibration point.

[0114] The second mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

[0115] After eye-tracking calibration is completed, an eye-tracking test task can be performed. A first image of the subject's eye area is acquired using a camera, and the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point are determined in the first image. The world coordinates of the first marker point are determined based on the coordinate system transformation parameters between the camera coordinate system and the world coordinate system, camera intrinsic parameters, and the second pixel coordinates of the first marker point. The world coordinates of the pupil are determined based on the coordinate system transformation parameters, camera intrinsic parameters, and the first pixel coordinates of the pupil. The deviation between the world coordinates of the pupil and the world coordinates of the first marker point is determined as the subject's gaze vector when the first image was acquired.

[0116] Finally, based on the subject's gaze vector and the second mapping relationship determined during the calibration process, the subject's gaze point position on the display is determined.

[0117] As an optional implementation, the calibration process may not be performed in this embodiment. After the subject's gaze vector (viewing direction) is determined, the intersection of the line where the subject's gaze vector is located and the plane where the display is located can be taken as the position of the subject's gaze point.

[0118] Optionally, the first image acquired during the eye-tracking test may also contain multiple markers. The subject's gaze vector can be determined based on a single marker or multiple markers.

[0119] For example, taking the brow marker as a single marker point, the world coordinates of the left eye pupil can be determined based on the coordinate system transformation parameters, camera intrinsics, and the first pixel coordinates of the left eye pupil; the world coordinates of the right eye pupil can be determined based on the coordinate system transformation parameters, camera intrinsics, and the first pixel coordinates of the right eye pupil; the difference between the world coordinates of the left eye pupil and the world coordinates of the brow marker point is used as the gaze vector of the left eye, and the difference between the world coordinates of the right eye pupil and the world coordinates of the brow marker point is used as the gaze vector of the right eye.

[0120] For example, taking multiple marker points including the marker point below the left eye and the marker point below the right eye as an example, the world coordinates of the marker point below the left eye and the world coordinates of the marker point below the right eye can be determined according to the coordinate system transformation parameters between the camera coordinate system and the world coordinate system, the camera intrinsic parameters, and the pixel coordinates of the marker point below the left eye and the pixel coordinates of the marker point below the right eye. The difference between the world coordinates of the left pupil and the world coordinates of the marker point below the left eye is used as the gaze vector of the left eye, and the difference between the world coordinates of the right pupil and the world coordinates of the marker point below the right eye is used as the gaze vector of the right eye.

[0121] The eye-tracking method provided in this application sets one or more marker points in the eye area and uses the marker points as reference points. Since the positional change of the marker points reflects the change of head movement, the eye-tracking data is analyzed based on the positional offset between the pupil and the marker points. This can solve the problem of head movement to a certain extent and eliminate the influence of the subject's head movement.

[0122] By eliminating head movements, the time spent on repeated testing is reduced, further shortening the testing time for participants, alleviating eye fatigue, improving the participant experience, and increasing the efficiency of eye tracking. On the other hand, it significantly improves the accuracy of eye tracking results, resolving the issue of significant data interference caused by eye characteristics or head movements in some individuals.

[0123] Furthermore, the markers in this embodiment can be set by pasting, wearing, or other methods, which is low-cost, highly flexible, and easier to promote. This embodiment can also record eye-tracking data in real time during eye-tracking tests for verification and analysis based on the recorded video.

[0124] Based on the same principle as the eye-tracking method provided in the embodiments of this application, the embodiments of this application provide an eye-tracking device, such as... Figure 6 As shown, the eye-tracking device 200 may include an eye image acquisition module 210, a pupil and marker point positioning module 220, a gaze vector determination module 230, and a gaze point positioning module 240.

[0125] The eye image acquisition module 210 is used to acquire a first image of the eye region of the subject through the camera, the first image including the first marker point;

[0126] Pupil and marker point localization module 220 is used to determine the first pixel coordinates of the pupil and the second pixel coordinates of the first marker point in the first image;

[0127] The gaze vector determination module 230 is used to determine the gaze vector of the subject based on the first pixel coordinates and the second pixel coordinates; wherein, the gaze vector is the position offset of the subject's pupil relative to the subject's head;

[0128] The gaze point localization module 240 is used to determine the gaze point position of the subject based on the position offset.

[0129] Optionally, the eye-tracking device 200 may also include a display;

[0130] The gaze point positioning module 240 can be used for:

[0131] Based on the gaze vector and a pre-determined mapping relationship, the gaze point position of the subject on the display is determined;

[0132] The mapping relationship reflects the correspondence between the subject's gaze vector and the coordinate position on the display.

[0133] Optionally, the gaze vector determination module 230 can be used to:

[0134] The deviation between the first pixel coordinates and the second pixel coordinates is determined, and the deviation is used as the gaze vector of the subject.

[0135] Optionally, the gaze vector determination module 230 can be used to:

[0136] Obtain the coordinate system transformation parameters between the camera coordinate system and the world coordinate system;

[0137] The world coordinates of the first marker point are determined based on the coordinate system transformation parameters, the camera intrinsic parameters, and the second pixel coordinates.

[0138] The world coordinates of the pupil are determined based on the coordinate system transformation parameters, the camera intrinsic parameters, and the first pixel coordinates of the pupil.

[0139] The deviation between the world coordinates of the pupil and the world coordinates of the first marker point is determined as the gaze vector of the subject.

[0140] Optionally, the coordinate system transformation parameters are obtained in the following way:

[0141] The camera acquires a second image of the subject's eye area, the second image including the first marker point and the at least two reference points; wherein, the at least two reference points are at least two second marker points set at a second position around the subject's eyes, or, the at least two reference points are left corneal reflective spots and right corneal reflective spots formed when the infrared light source of the eye-tracking device illuminates the subject's eyes during the acquisition of the second image;

[0142] Obtain the world coordinates of at least two reference points and the first marker point in the second image;

[0143] Determine the pixel coordinates of at least two reference points and the first marker point in the second image;

[0144] Based on the pixel coordinates and world coordinates of at least two reference points in the second image, the pixel coordinates and world coordinates of the first marker point, and the camera intrinsic parameters, determine the coordinate system transformation parameters between the camera coordinate system and the world coordinate system.

[0145] Optionally, the mapping relationship is a first mapping relationship;

[0146] The first mapping relationship is obtained in the following way:

[0147] Acquire eye images of the subjects as they fixate on each calibration point;

[0148] For each calibration point, the pixel coordinates of the pupil and the first marker point in the eye image corresponding to the calibration point are determined; based on the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marker point, the gaze vector when the subject gazes at the calibration point is determined.

[0149] The first mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

[0150] Optionally, the eye-tracking device may also include a display;

[0151] The gaze point positioning module 240 can be used for:

[0152] The intersection of the gaze vector and the display is taken as the gaze point position of the subject on the display.

[0153] Optionally, the eye-tracking device further includes a display; the subject's gaze point position is determined based on the gaze vector and a pre-determined mapping relationship, wherein the mapping relationship is a second mapping relationship;

[0154] The second mapping relationship is obtained in the following way:

[0155] Obtain the coordinate system transformation parameters and the eye images of the subjects when they gaze at each calibration point;

[0156] For each calibration point, the pixel coordinates of the pupil and the first marker point in the eye image corresponding to the calibration point are determined; the world coordinates of the pupil are determined based on the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters; the world coordinates of the first marker point are determined based on the pixel coordinates of the first marker point in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters; and the gaze vector when the subject gazes at the calibration point is determined based on the deviation between the world coordinates of the pupil and the world coordinates of the first marker point in the eye image corresponding to the calibration point.

[0157] The second mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

[0158] Optionally, the subject sets the marker points by wearing a frame-like device, with the first marker point positioned at a first position on the frame-like device and the at least two second marker points positioned at a second position on the frame-like device.

[0159] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0160] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0161] This application provides an eye-tracking system, including an image acquisition unit, a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application.

[0162] Figure 7 A schematic diagram of the structure of an eye-tracking system applicable to an embodiment of the present invention is shown, as follows: Figure 7 As shown, this system can be used to implement the methods provided in any embodiment of the present invention.

[0163] like Figure 7 As shown, the eye-tracking system 2000 may primarily include at least one processor 2001. Figure 7 The diagram shows components such as a memory 2002, a communication module 2003, an input / output interface 2004, and a camera 2006. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 7 The structure of the eye-tracking system 2000 shown is merely illustrative and does not constitute a limitation on the eye-tracking system to which the methods provided in the embodiments of this application are applicable.

[0164] The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0165] Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0166] The eye-tracking system 2000 can connect to a network via a communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. The communication module 2003 may include a wired network interface and / or a wireless network interface, meaning the communication module may include at least one of a wired communication module or a wireless communication module.

[0167] The eye-tracking system 2000 can connect to necessary input / output devices, such as a keyboard or display device (monitor), via the input / output interface 2004. The eye-tracking system 2000 itself can have a display device, and other external display devices can also be connected via interface 2004. Optionally, a storage device, such as a hard drive, can also be connected via this interface 2004 to store data from the eye-tracking system 2000, retrieve data from the storage device, or store data from the storage device in the memory 2002. It is understood that the input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 2004 can be an integral part of the eye-tracking system 2000 or an external device connected to the eye-tracking system 2000 when needed.

[0168] The bus 2005 used to connect the components may include a pathway for transmitting information between the components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc.

[0169] The camera 2006 can be an infrared camera, a depth camera, etc., used to acquire images of the subject's eyes.

[0170] Optionally, for the solution provided in the embodiments of the present invention, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention.

[0171] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0172] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0173] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0174] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0175] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. An eye-tracking method for detecting and removing head movements, characterized in that, At least one eye marker is set around the eyes of the subject by markers, the at least one eye marker including a first marker set at the center of the eyebrows, the method is performed by an eye-tracking device, the eye-tracking device including a desktop eye tracker, the desktop eye tracker including a camera; The method includes: A first image of the subject's eye region is acquired by the camera, and the first image includes at least one eye marker. The first pixel coordinates of the pupil and the second pixel coordinates of the at least one eye marker in the first image are determined; based on the coordinate system transformation parameters between the camera coordinate system and the world coordinate system, the camera intrinsic parameters, the first pixel coordinates of the pupil, and the second pixel coordinates of the at least one eye marker, the world coordinates of the pupil and the at least one eye marker are determined respectively; the gaze vector of the subject is determined according to the deviation between the world coordinates of the pupil and the world coordinates of the at least one eye marker; wherein, the gaze vector is the positional offset of the subject's pupil relative to the subject's head; Based on the gaze vector, determine the gaze point position of the subject; The coordinate system transformation parameters are obtained in the following way: The camera acquires a second image of the subject's eye area, the second image including the first marker point and at least two reference points; the at least two reference points are at least two second marker points set at a second position around the subject's eyes, or the at least two reference points are left corneal reflective spots and right corneal reflective spots formed when the infrared light source of the eye-tracking device illuminates the subject's eyes during the acquisition of the second image; Based on the pixel coordinates and world coordinates of the at least two reference points, the pixel coordinates and world coordinates of the first marker point, and the camera intrinsic parameters, the coordinate system transformation parameters between the camera coordinate system and the world coordinate system are determined by the perspective N-point projection (PNP) algorithm.

2. The method according to claim 1, characterized in that, The at least one eye marker also includes the at least two second markers, the at least two second markers including a marker below the left eye and a marker below the right eye; the pupil includes a left pupil and a right pupil; Determining the subject's gaze vector based on the deviation between the world coordinates of the pupil and the world coordinates of at least one eye marker includes any of the following: The deviation between the world coordinates of the left pupil and the world coordinates of the first marker point is used as the gaze vector of the left eye, and the deviation between the world coordinates of the right pupil and the world coordinates of the first marker point is used as the gaze vector of the right eye. The deviation between the world coordinates of the left pupil and the world coordinates of the marker point below the left eye is used as the gaze vector of the left eye, and the deviation between the world coordinates of the right pupil and the world coordinates of the marker point below the right eye is used as the gaze vector of the right eye. Wherein, determining the gaze point position of the subject based on the gaze vector includes: The fixation point position of the subject is determined based on the fixation vector of the subject's left eye and the fixation vector of the subject's right eye.

3. The method according to claim 1, characterized in that, The eye-tracking device also includes a display; Determining the gaze point position of the subject based on the gaze vector includes: Based on the gaze vector and a pre-determined mapping relationship, the gaze point position of the subject on the display is determined; The mapping relationship reflects the correspondence between the subject's gaze vector and the coordinate position on the display.

4. The method according to claim 3, characterized in that, The mapping relationship is the first mapping relationship; The first mapping relationship is obtained in the following way: Acquire eye images of the subjects as they fixate on each calibration point; For each calibration point, determine the pixel coordinates of the pupil and the first marker point in the eye image corresponding to that calibration point; The gaze vector of the subject when gazing at the calibration point is determined based on the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marker point. The first mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

5. The method according to claim 1, characterized in that, The eye-tracking device also includes a display; Determining the gaze point position of the subject based on the gaze vector includes: The intersection of the gaze vector and the display is taken as the gaze point position of the subject on the display.

6. The method according to claim 3, characterized in that, The mapping relationship is the second mapping relationship; The second mapping relationship is obtained in the following way: Obtain the coordinate system transformation parameters and the eye images of the subjects when they gaze at each calibration point; For each calibration point, determine the pixel coordinates of the pupil and the first marker point in the eye image corresponding to that calibration point; The world coordinates of the pupil are determined based on the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters. Based on the pixel coordinates of the first marker point in the eye image corresponding to the calibration point, the camera intrinsic parameters, and the coordinate system transformation parameters, the world coordinates of the first marker point are determined; based on the deviation between the world coordinates of the pupil in the eye image corresponding to the calibration point and the world coordinates of the first marker point, the gaze vector when the subject gazes at the calibration point is determined. The second mapping relationship is determined based on the gaze vector corresponding to each calibration point and the coordinate position of each calibration point on the display.

7. The method according to claim 1, characterized in that, The subjects set eye markers by wearing a frame-like device. The first marker was positioned at the center of the forehead corresponding to the frame-like device, and the at least two second markers were positioned below the left and right eyes respectively corresponding to the frame-like device.

8. An eye-tracking device for detecting and removing head movements, characterized in that, At least one eye marker is set around the eyes of the subject by markers. The device is deployed in an eye-tracking device, which includes a desktop eye tracker and a camera. The device includes: An eye image acquisition module is used to acquire a first image of the eye area of ​​the subject through the camera. The first image includes at least one eye marker, and the at least one eye marker includes a first marker set at a first position. A pupil and marker point localization module is used to determine the first pixel coordinates of the pupil and the second pixel coordinates of the at least one eye marker point in the first image; A gaze vector determination module is used to determine the world coordinates of the pupil and the world coordinates of the at least one eye marker based on coordinate system transformation parameters between the camera coordinate system and the world coordinate system, camera intrinsic parameters of the camera, the first pixel coordinates of the pupil, and the second pixel coordinates of the at least one eye marker. Based on the deviation between the world coordinates of the pupil and the world coordinates of the at least one eye marker, the gaze vector of the subject is determined. The gaze vector is the positional offset of the subject's pupil relative to the subject's head. The coordinate system transformation parameters are obtained by acquiring the subject's eye position data through the camera. A second image of the region, the second image including the first marker point and at least two reference points; the at least two reference points are at least two second marker points set at a second position around the subject's eye, or the at least two reference points are left corneal reflective spots and right corneal reflective spots formed when the infrared light source of the eye-tracking device illuminates the subject's eye during the acquisition of the second image; based on the pixel coordinates and world coordinates of the at least two reference points, the pixel coordinates and world coordinates of the first marker point, and the camera intrinsic parameters, the coordinate system transformation parameters between the camera coordinate system and the world coordinate system are determined by the perspective N-point projection (PNP) algorithm; The gaze point localization module is used to determine the gaze point position of the subject based on the gaze vector.

9. An eye-tracking system, characterized in that, The eye-tracking system includes a camera, a display, a memory, and a processor. The camera is used to acquire images of the subject's eyes. The memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images