Eye movement tracking method and device for detecting head movement and removing head movement

Abstract

The embodiment of the invention provides an eye movement tracking method and device for detecting head movement and removing head movement, and relates to the field of eye movement tracking, and the method comprises the steps: setting a first mark point at a first position at the periphery of the eye of a tested person, collecting a first image of the eye region of the tested person, and determining a first pixel coordinate of the pupil in the first image and a second pixel coordinate of the first mark point, determining a position offset of the pupil relative to the head of the testee according to the first pixel coordinate and the second pixel coordinate, and determining a fixation point position of the testee according to the determined position offset. According to the method, the mark points are arranged on the peripheries of the eyes, the mark points and the head move synchronously, the head movement change is calculated through the position change of the mark points, fixation point positioning is carried out based on the position offset of the pupils relative to the head of the tested person, the influence caused by head movement of the tested person is eliminated, and the accuracy of the eye movement tracking result is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology and may relate to the field of eye tracking. Specifically, the present application relates to an eye tracking method and device for detecting and removing head movements. Background Art

[0002] Eye tracking technology can monitor and analyze human eye movements in real time, such as scanning, tracking, fixation and other eye movements. With the continuous development of science and technology, eye tracking technology has been applied in more and more fields, such as human-computer interaction, medical health and so on.

[0003] In the process of eye tracking, it is often necessary to quickly locate and track the position of the pupil in the subject's eye image, so as to calculate the gaze point based on the position of the pupil and form an eye movement trajectory. However, when only the pupil position is used for analysis, any slight head movement of the subject will cause the spatial relationship between the pupil and the eye movement detection device to change, resulting in inaccurate tracking results. Therefore, how to eliminate the influence of the subject's head movement is very important to improve the accuracy of eye tracking results. Summary of the invention

[0004] The embodiment of the present application provides an eye tracking method and device for detecting and removing head movement, which can eliminate the error caused by the head movement of the subject and improve the accuracy of the eye tracking result. To achieve this purpose, the technical solution provided by the embodiment of the present application is as follows: On the one hand, an embodiment of the present application provides an eye tracking method for detecting and removing head movement, wherein a first marker point is set at a first position around the eye of a subject, and the method is performed by an eye tracking device, wherein the eye tracking device includes a camera; The method comprises: Acquire a first image of the eye area of ​​the subject by using the camera, wherein the first image includes the first marking point; Determine a first pixel coordinate of a pupil in the first image and a second pixel coordinate of the first marking point; Determine a gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates; wherein the gaze vector is a position offset of the pupil of the subject relative to the head of the subject; The gaze point position of the subject is determined according to the gaze vector.

[0005] Optionally, the eye tracking device further comprises a display; Determining the gaze point position of the subject according to the gaze vector includes: Determining the gaze point position of the subject on the display according to the gaze vector and a predetermined mapping relationship; The mapping relationship reflects the corresponding relationship between the gaze vector of the subject and the coordinate position on the display.

[0006] Optionally, determining the gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates includes: A deviation between the first pixel coordinate and the second pixel coordinate is determined, and the deviation is used as a gaze vector of the subject.

[0007] Optionally, determining the gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates includes: Get the coordinate system conversion parameters between the camera coordinate system and the world coordinate system; Determine the world coordinates of the first marking point according to the coordinate system conversion parameter, the camera intrinsic parameter of the camera and the second pixel coordinate; Determining the world coordinates of the pupil according to the coordinate system conversion parameter, the camera intrinsic parameter and the first pixel coordinate of the pupil; The deviation between the world coordinates of the pupil and the world coordinates of the first marking point is determined as the gaze vector of the subject.

[0008] Optionally, the coordinate system conversion parameters are obtained in the following manner: A second image of the subject's eye area is captured by the camera, wherein the second image includes the first marking point and the at least two reference points; wherein the at least two reference points are at least two second marking points set at second positions around the subject's eyes, or the at least two reference points are left-eye corneal reflection spots and right-eye corneal reflection spots formed by the infrared light source of the eye tracking device irradiating the eyes of the subject during the process of capturing the second image; Acquire the world coordinates of at least two reference points and the first marking point in the second image; determining pixel coordinates of at least two reference points and the first marking point in the second image; Coordinate system conversion parameters between a camera coordinate system and a world coordinate system are determined according to pixel coordinates and world coordinates of at least two reference points in the second image, pixel coordinates and world coordinates of the first marking point, and the camera intrinsic parameters.

[0009] Optionally, the mapping relationship is a first mapping relationship; The first mapping relationship is obtained in the following manner: Obtaining eye images of the subject when he or she is looking at each calibration point; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the gaze vector of the subject when he or she is looking at the calibration point according to the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marking point; A first mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

[0010] Optionally, the eye tracking device further comprises a display; Determining the gaze point position of the subject according to the gaze vector includes: The intersection of the gaze vector and the display is used as the gaze point position of the subject on the display.

[0011] Optionally, the eye tracking device further includes a display; the gaze point position of the subject is determined based on the gaze vector and a predetermined mapping relationship, and the mapping relationship is a second mapping relationship; The second mapping relationship is obtained in the following manner: Acquiring the coordinate system conversion parameters and the eye images of the subject when he or she is looking at each calibration point; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the world coordinates of the pupil according to the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the world coordinates of the first marking point according to the pixel coordinates of the first marking point in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the gaze vector of the subject when he looks at the calibration point according to the deviation between the world coordinates of the pupil in the eye image corresponding to the calibration point and the world coordinates of the first marking point; A second mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

[0012] Optionally, the subject sets the marking point by wearing a frame-like device, the first marking point is configured at a first position of the frame-like device, and the at least two second marking points are respectively configured at second positions of the frame-like device.

[0013] On the other hand, an embodiment of the present application provides an eye tracking device for detecting and removing head movement, wherein a first mark point is set at a first position around the eye of a subject, and the device is deployed in an eye tracking device, and the eye tracking device includes a camera; The device comprises: An eye image acquisition module, used for acquiring a first image of the eye area of ​​the subject through the camera, wherein the first image includes the first marking point; a pupil and marker point positioning module, configured to determine a first pixel coordinate of the pupil and a second pixel coordinate of the first marker point in the first image; A gaze vector determination module, used to determine the gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates; wherein the gaze vector is a position offset of the pupil of the subject relative to the head of the subject; A gaze point positioning module is used to determine the gaze point position of the subject according to the position offset.

[0014] On the other hand, an embodiment of the present application also provides an eye tracking system, which includes a camera, a display, a memory and a processor, wherein the camera is used to capture eye images of the subject, the memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of the present application.

[0015] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in any optional embodiment of the present application is implemented.

[0016] On the other hand, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method provided in any optional embodiment of the present application.

[0017] The beneficial effects of the technical solution provided by the embodiment of the present application are as follows: The eye tracking method provided in the embodiment of the present application sets a first marking point at a first position around the eye of the subject, collects a first image of the eye area of ​​the subject, and determines the first pixel coordinates of the pupil in the first image and the second pixel coordinates of the first marking point, determines the position offset of the pupil relative to the head of the subject based on the first pixel coordinates and the second pixel coordinates, and determines the position of the subject's gaze point based on the determined position offset. The method sets marking points around the eyes, and since the marking points move synchronously with the head, the position change of the marking points is used to calculate the head movement change, and the gaze point is located based on the position offset of the pupil relative to the subject's head, eliminating the influence caused by the subject's head movement and improving the accuracy of the eye tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0019] Figure 1 A schematic diagram of the structure of an eye tracking system applicable to an embodiment of the present application; Figure 2 A flowchart of an eye tracking method for detecting head movement and removing head movement provided in an embodiment of the present application; Figure 3 A schematic diagram of setting a first marking point provided in an embodiment of the present application; Figure 4a-4d A schematic diagram of a human eye with multiple reflection spots provided in an embodiment of the present application; Figure 5a-Figure 5b A schematic diagram of setting three marking points provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an eye tracking device for detecting and removing head movements provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an eye tracking system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0021] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, A3.

[0022] When performing eye tracking task detection, if the tracking detection is performed only based on the subject's pupil position, then the subject's slight head movement will cause the spatial position relationship between the human pupil and the eye movement detection device to change. If the head movement is not eliminated, it will cause a large deviation in positioning. For example, during the eye tracking detection process, the subject's gaze point did not change, but because the subject's head shifted to the left, the eye tracking device misjudged that an eye movement had occurred at this time, resulting in inaccurate eye movement data collected.

[0023] Based on the above-mentioned problems, an embodiment of the present application provides an eye tracking method and device for detecting and removing head movements. The method sets a first marking point at a first position around the eye of the subject, determines the first pixel coordinates of the pupil and the second pixel coordinates of the first marking point based on the first image of the eye area of ​​the subject collected, determines the position offset of the pupil relative to the head of the subject based on the first pixel coordinates and the second pixel coordinates, and determines the position of the subject's gaze point based on the determined position offset. The method sets marking points around the eyes. Since the marking points move synchronously with the head, the position change of the marking points is used to calculate the change of head movement, and the gaze point is located based on the position offset of the pupil relative to the head of the subject, thereby eliminating the influence of the subject's head movement and improving the accuracy of the eye tracking results.

[0024] The eye tracking method provided in the embodiments of the present application can be applied to the eye tracking process in any scenario, such as virtual reality technology (VR), screen interactive control, eye pathology examination, mental illness diagnosis assistance, etc., and the present application does not impose any restrictions on this.

[0025] Figure 1 A schematic diagram of an eye tracking system applicable to an embodiment of the present application is shown, the eye tracking system includes a first terminal 10 and an eye tracking device 20, wherein the eye tracking device 20 includes at least an infrared light source 201, a camera 202 and a display 203, wherein the infrared light source 201 is used to fill light for the eyes to capture clear eye images, and as an optional method, the infrared light source 201 can adopt a near-infrared light source.

[0026] When performing eye tracking detection on a subject, a first marking point may be set in the area surrounding the subject's eyes.

[0027] When the subject observes the content presented in the display 203, the first image of the subject's eye area is collected by the camera 202, the pupil and the first marking point in the first image are detected, and the first pixel coordinates of the pupil and the second pixel coordinates of the first marking point are located, and the subject's gaze vector is determined according to the first pixel coordinates and the second pixel coordinates, and the position of the subject's gaze point in the display 203 is determined according to the subject's gaze vector and the mapping relationship determined during the calibration process.

[0028] In another embodiment of the present application, 3D line of sight tracking can also be used to locate the gaze point. Specifically, the coordinate system conversion parameters (including the conversion matrix R and the translation vector T) between the camera coordinate system and the world coordinate system are obtained, and the world coordinates of the first marker point are determined according to the coordinate system conversion parameters, the camera intrinsic parameters of the camera 202, and the second pixel coordinates of the first marker point in the first image; the world coordinates of the pupil are determined according to the coordinate system conversion parameters, the camera intrinsic parameters of the camera, and the first pixel coordinates of the pupil in the first image; the gaze vector of the subject is determined according to the deviation between the world coordinates of the pupil and the world coordinates of the first marker point, so as to determine the gaze point position of the subject in the display 203 based on the gaze vector and the mapping relationship determined during the calibration process.

[0029] The following describes several embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0030] Figure 2 The flowchart of an eye tracking method for detecting head movement and removing head movement provided by an embodiment of the present application is shown. The method can be executed by any electronic device, such as a terminal device. The terminal device is a device with eye tracking / gaze tracking function, such as a head-mounted eye tracker, a desktop eye tracker, a smart phone, etc. For the convenience of description, the eye tracking device is used as an example for the execution subject.

[0031] See also Figure 2 The eye tracking method provided in the embodiment of the present application may include the following steps S110 to S140.

[0032] S110: Capturing a first image of the eye area of ​​the subject through a camera.

[0033] The eye tracking device is equipped with a camera (such as an infrared camera, a depth camera, a high-speed camera, etc.) and a display. When performing eye tracking detection on a subject, a first mark point can be set at a first position around the subject's eye, and the first position is any position around the eye. Since the first mark point is set in the area around the eye and changes synchronously with the subject's head, the first mark point can be used as a static reference point for human eye movement to eliminate the influence of head movement based on the first mark point.

[0034] Optionally, the embodiment of the present application does not limit the setting method of the first marking point, as long as a fixed point can be formed around the eye. For example, it can be by placing a marker, such as sticking the marking material at the first position of the eye area; or by applying the marking material to the first position of the eye area; it can also be achieved by wearing a frame-like device (such as a glasses frame, an eye mask that can expose the eye area, etc.), and setting the first marking point at the first position of the frame-like device to achieve the setting of the marking point around the eye. The embodiment of the present application does not limit the material used for the first marking point. The marking material used for the first marking point can be a material that is easy to reflect light, such as a material with a silver reflective surface, or a non-reflective material, such as a completely black material; the shape of the first marking point can also be set to any shape.

[0035] In addition, the embodiment of the present application does not limit the position of the first marking point. For example, it can be set at the center of the eyebrows, below the eyes, etc., as long as it can be presented in the image of the eye area collected by eye tracking.

[0036] After the first marking point is set, when the subject observes the target image displayed on the display of the eye tracking device, the camera collects a first image of the subject's eye area, wherein the collected first image includes the first marking point.

[0037] For example, Figure 3 As shown, the first marking point is a marker pasted on the center of the eyebrows. The marker is a hemispherical silver reflective material. No matter from which angle the eye image is collected, a clear reflective bright spot can be obtained on the image.

[0038] S120: Determine a first pixel coordinate of the pupil and a second pixel coordinate of the first marking point in the first image.

[0039] For the collected first image, the pupil center and the first marking point can be located from the first image, and the pixel coordinates of the pupil center in the first image, i.e. the first pixel coordinates of the pupil, and the second pixel coordinates of the first marking point in the first image can be determined.

[0040] Among them, the embodiments of the present application do not limit the method for locating the pupil center and the first marking point. For example, when the first marking point is made of reflective material, the color of the first marking point is brighter and the pupil color is darker in the collected first image. The grayscale threshold histogram method can be used to locate the pupil center and the first marking point based on the grayscale value of each pixel in the first image.

[0041] S130: Determine the gaze vector of the subject according to the first pixel coordinate and the second pixel coordinate.

[0042] S140: Determine the gaze point position of the subject according to the gaze vector.

[0043] In the embodiment of the present application, the deviation between the first pixel coordinate and the second pixel coordinate can be determined, and the determined deviation is used as the gaze vector of the subject. The gaze vector is the position offset of the pupil of the subject relative to the head of the subject. Since the pupil position changes with the change of the position of the sight point, the position of the first mark point changes with the head movement of the subject, so the gaze vector represents the change of the human eye movement and is not affected by the head movement.

[0044] For example, it is assumed that in the first image captured, the pixel coordinates of the left pupil are ( , ), the pixel coordinates of the right pupil are ( , ), the pixel coordinates of the first marker point are ( , ); then the position offset corresponding to the left eye is: ; , the gaze vector of the left eye is ( , ); the position offset of the right eye (gaze vector) is: ; , the gaze vector of the right eye is ( , ).

[0045] Optionally, the gaze point position of the subject on the display is determined based on the gaze vector and a predetermined mapping relationship, wherein the mapping relationship may be determined during the eye tracking calibration process, and reflects the correspondence between the gaze vector of the subject and the coordinate position on the display.

[0046] Optionally, when eye tracking is performed based on a single marker point (ie, the first marker point) and the pupil position, calibration may be performed in the following manner to obtain a first mapping relationship: Obtain eye images of the subject when he or she is looking at each calibration point; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the gaze vector when the subject is looking at the calibration point according to the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marking point; A first mapping relationship is determined according to the gaze vector corresponding to each calibration point and the position of each calibration point in the display.

[0047] Among them, the calibration point is a calibration point at a known position. The embodiment of the present application does not limit the number of calibration points. A 3-point calibration method, a 5-point calibration method, a 9-point calibration method, a 13-point calibration method, etc. can be used. When the subject looks at the calibration point at each position, the eye image of the subject is collected respectively.

[0048] After determining the gaze vector corresponding to each calibration point (i.e., the gaze vector when the subject is looking at each calibration point), multiple first mapping parameters are solved by parameter fitting based on the gaze vector corresponding to each calibration point and the position of each calibration point, and a first mapping relationship is obtained based on the multiple first mapping parameters.

[0049] In the related art, the eye movement of the subject is determined based on the change of pupil position, and then the gaze point is located. However, when the subject moves his head, the positioning deviation is large. Figure 2 The eye tracking method shown introduces a first marker point around the eye. The first marker point changes synchronously with the subject's head. The eye movement changes determined based on the position offset between the pupil and the first marker point can eliminate the influence of the subject's head movement, reduce the interference of eye movement test caused by head movement, reduce the time consumption of eye movement calibration, and improve the accuracy of eye tracking.

[0050] In the related art, there is also an eye tracking method based on pupil corneal reflection, which determines the position of the gaze point according to the relative position between the pupil and the corneal reflection light spot in the human eye. However, the corneal emission light spot is uncertain and unstable, and is easily affected by many factors. For example, it may disappear due to being blocked by the upper and lower eyelids, the corneal surface is not completely smooth, resulting in the disappearance of the reflection light spot, and it is also possible to form multiple reflection light spots on the cornea, or even multiple reflection light spots connected into a large area..., which may cause certain interference to the positioning of the corneal reflection light spot, resulting in a large deviation in the positioning of the corneal reflection light spot. For example, Figures 4a-4d As shown, Figure 4a The reflected light spot in the middle right eye is stretched to become a larger light spot. Figure 4b The reflected light spot in the middle left eye is divided into two reflected light spots. Figure 4c The reflected light spot in the middle right eye is elongated into a larger light spot, and a smaller light spot appears below. Figure 4d In the figure, 3 reflection spots appear in the left eye and 4 reflection spots appear in the right eye. In the eye tracking method provided in the embodiment of the present application, since the first marking point is externally placed in the peripheral area of ​​the eye, it is not affected by factors such as eye structure and tearing, has better stability, and introduces less noise interference.

[0051] In the embodiment of the present application, since the first marking point is set in the area around the subject's eyes and moves synchronously with the subject's head, it is possible to detect whether the subject's head moves by monitoring the position change of the first marking point.

[0052] In one embodiment of the present application, the eye tracking device can be set with multiple detection modes. When a marker point is detected from the captured first image, the eye tracking method provided in the embodiment of the present application is used to locate the gaze point based on the first marker point and the pupil. When no marker point is detected, the traditional pupil-corneal reflection method is used to locate the gaze point based on the pupil and corneal reflection spot.

[0053] In another embodiment of the present application, the relative position relationship (coordinate system conversion parameters) between the camera coordinate system and the world coordinate system can be calculated based on the first marking point and other reference points, so as to determine the line of sight changes of the subject in the 3D space based on the calculated coordinate system conversion parameters.

[0054] Usually, before performing an eye tracking task, eye calibration needs to be performed first. However, in the embodiment of the present application, before performing eye calibration, the coordinate system conversion parameters between the camera coordinate system and the world coordinate system need to be determined first.

[0055] Specifically, first, at least two second marking points are set at a second position around the subject's eyes, and different second marking points have different positions. The at least two second marking points are used as at least two reference points, or the left corneal reflection light spot and the right corneal reflection light spot formed by the infrared light source of the eye tracking device irradiating the subject's eyes are used as two reference points respectively. In addition, a first marking point is set at a first position around the subject's eyes.

[0056] For example, assuming that three marking points 1, 2, and 3 are respectively set in the eye area of ​​the subject, wherein marking point 1 is the first marking point, and marking points 2 and 3 are reference points, then the marking points can be pasted, such as Figure 5a As shown, a marker 1 is placed between the eyebrows of both eyes, a marker 2 is placed outside the lower eyelid of the left eye, and a marker 3 is placed outside the lower eyelid of the right eye; or a frame-like device can be worn, such as Figure 5b As shown, the markers can be pre-pasted at the corresponding positions of the eye mask, marker 1 is pasted at the corresponding position between the eyebrows of the eye mask, marker 2 is pasted at the corresponding position below the left eye of the eye mask, and marker 3 is pasted at the corresponding position below the right eye of the eye mask.

[0057] Afterwards, the subject can gaze at any coordinate position in the display of the eye tracking device (such as the center position of the display), and capture a second image of the subject's eye area at this time through the camera, and determine the pixel coordinates of the first marking point in the second image, as well as the pixel coordinates of at least two reference points; obtain the world coordinates of the first marking point and the world coordinates of at least two reference points at this time, and determine the coordinate system conversion parameters between the camera coordinate system and the world coordinate system according to the pixel coordinates and world coordinates of at least two reference points in the second image, the pixel coordinates and world coordinates of the first marking point, and the camera intrinsic parameters.

[0058] The world coordinates of the first marking point and at least two reference points may be measured in a world coordinate system constructed with an arbitrary fixed point as the origin. Exemplarily, the nose tip of the subject is used as the origin of the world coordinate system, the direction parallel to the eyes (such as pointing to the right eye) is used as the direction of the X-axis, the direction of the vertical gaze of the eyes is used as the direction of the Y-axis, and the direction perpendicular to the line connecting the eyes is used as the direction of the Z-axis of the world coordinate system. The world coordinates of the first marking point and at least two reference points are measured in the world coordinate system.

[0059] Optionally, the same positioning method as the first marking point mentioned above can be used to locate the reference point from the second image, such as using a grayscale threshold histogram method to segment areas with higher grayscale values ​​(highlight areas) from the second image, and then locate the pixel coordinates of the reference point.

[0060] Among them, the conversion relationship between the world coordinate system and the camera coordinate system is as follows:

[0061] Indicates the coordinates of the marker point in the camera coordinate system. Indicates the coordinates of the marker point in the world coordinate system. , Represents the rotation matrix and translation vector from the world coordinate system to the camera coordinate system.

[0062] The conversion relationship between the pixel coordinate system and the camera coordinate system is as follows:

[0063] p is the coordinate of the marker point in the pixel coordinate system, is the depth of the marker point in the camera coordinate system, Represents the coordinates of the marker point in the camera coordinate system, and K is the camera intrinsic parameter matrix.

[0064] Substitute the pixel coordinates and world coordinates of at least two reference points in the second image, the pixel coordinates and world coordinates of the first marking point, and the camera intrinsic parameters into the above coordinate system conversion relationship to obtain the coordinate system conversion parameters between the camera coordinate system and the world coordinate system. and .

[0065] In the embodiment of the present application, the PNP (Perspective-n-Point) algorithm can be used to solve the coordinate system conversion parameters based on the world coordinates of at least three pairs of points in three-dimensional space and the pixel coordinates of the corresponding points projected on the camera normalized plane. Of course, other methods can also be used to solve, such as direct linear transformation (DLT), EPnP, bundle adjustment (BA, Bundle Adjustment), etc.

[0066] After determining the coordinate transformation parameters between the camera coordinate system and the world coordinate system, the next step is to perform eye tracking calibration: Obtaining eye images of the subject when he or she is gazing at each calibration point; wherein, during the calibration process, the subject sequentially gazes at each calibration point displayed on the display, and the eye images of the subject gazing at each calibration point are captured by a camera; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the world coordinates of the pupil in the eye image corresponding to the calibration point according to the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the world coordinates of the first marking point in the eye image corresponding to the calibration point according to the pixel coordinates of the first marking point in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the gaze vector when the subject is looking at the calibration point according to the deviation between the world coordinates of the pupil in the eye image corresponding to the calibration point and the world coordinates of the first marking point; A second mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

[0067] After the eye movement calibration is completed, the eye tracking test task can be performed. The first image of the subject's eye area is collected by the camera, and the first pixel coordinates of the pupil and the second pixel coordinates of the first marking point in the first image are determined; the world coordinates of the first marking point are determined according to the coordinate system conversion parameters between the camera coordinate system and the world coordinate system, the camera intrinsic parameters and the second pixel coordinates of the first marking point; the world coordinates of the pupil are determined according to the coordinate system conversion parameters, the camera intrinsic parameters and the first pixel coordinates of the pupil; the deviation between the world coordinates of the pupil and the world coordinates of the first marking point is determined as the gaze vector of the subject when the first image is collected.

[0068] Finally, the position of the subject's gaze point on the display is determined according to the subject's gaze vector and the second mapping relationship determined during the calibration process.

[0069] As an optional implementation, the calibration process may not be performed in the embodiment of the present application. After the gaze vector (line of sight) of the subject is determined, the intersection of the straight line where the subject's gaze vector is located and the plane where the display is located can be used as the position of the subject's gaze point.

[0070] Optionally, the first image captured during the eye tracking test may also include multiple marking points, and the gaze vector of the subject may be determined based on a single marking point or based on multiple marking points.

[0071] Exemplarily, taking the case where the single marker point is the eyebrow marker point, the world coordinates of the left eye pupil can be determined according to the coordinate system conversion parameters, the camera intrinsic parameters and the first pixel coordinates of the left eye pupil; the world coordinates of the right eye pupil can be determined according to the coordinate system conversion parameters, the camera intrinsic parameters and the first pixel coordinates of the right eye pupil; the difference between the world coordinates of the left eye pupil and the world coordinates of the eyebrow marker point is used as the gaze vector of the left eye, and the difference between the world coordinates of the right eye pupil and the world coordinates of the eyebrow marker point is used as the gaze vector of the right eye.

[0072] Exemplarily, taking multiple marker points including a marker point under the left eye and a marker point under the right eye as an example, the world coordinates of the marker point under the left eye and the world coordinates of the marker point under the right eye can be determined respectively according to the coordinate system conversion parameters between the camera coordinate system and the world coordinate system, the camera intrinsic parameters, and the pixel coordinates of the marker point under the left eye and the pixel coordinates of the marker point under the right eye; the difference between the world coordinates of the left eye pupil and the world coordinates of the marker point under the left eye is used as the gaze vector of the left eye, and the difference between the world coordinates of the right eye pupil and the world coordinates of the marker point under the right eye is used as the gaze vector of the right eye.

[0073] The eye tracking method provided in the embodiment of the present application sets one or more marking points in the eye area and uses the marking points as reference points. Since the position change of the marking points reflects the change of head movement, the eye movement data is analyzed based on the position offset between the pupil and the marking points. This can solve the problem of head movement to a certain extent and eliminate the influence of the subject's head movement.

[0074] By eliminating head movement, on the one hand, it reduces the time spent on repeated tests, further reduces the test time of the subjects, relieves the subjects' eye fatigue, improves the experience of the subjects, and improves the efficiency of eye tracking. On the other hand, it greatly improves the accuracy of eye tracking results and solves the problem of large data interference caused by eye characteristics or head movement in some people.

[0075] Moreover, the marking points in the embodiment of the present application can be set by pasting, wearing, etc., which is low in cost, more flexible, and easier to promote. In the embodiment of the present application, the eye movement data of the subject during the eye movement test can also be recorded in real time to verify and analyze based on the recorded video.

[0076] Based on the same principle as the eye tracking method provided in the embodiment of the present application, the embodiment of the present application provides an eye tracking device, such as Figure 6 As shown, the eye tracking device 200 may include an eye image acquisition module 210 , a pupil and marker point positioning module 220 , a gaze vector determination module 230 , and a gaze point positioning module 240 .

[0077] An eye image acquisition module 210, configured to acquire a first image of the eye region of the subject through the camera, wherein the first image includes the first marking point; A pupil and marker location module 220, configured to determine a first pixel coordinate of the pupil and a second pixel coordinate of the first marker in the first image; A gaze vector determination module 230 is used to determine the gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates; wherein the gaze vector is a position offset of the pupil of the subject relative to the head of the subject; The gaze point positioning module 240 is used to determine the gaze point position of the subject according to the position offset.

[0078] Optionally, the eye tracking device 200 further includes a display; The gaze point location module 240 can be used to: Determining the gaze point position of the subject on the display according to the gaze vector and a predetermined mapping relationship; The mapping relationship reflects the corresponding relationship between the gaze vector of the subject and the coordinate position on the display.

[0079] Optionally, the gaze vector determination module 230 may be used to: A deviation between the first pixel coordinate and the second pixel coordinate is determined, and the deviation is used as a gaze vector of the subject.

[0080] Optionally, the gaze vector determination module 230 may be used to: Get the coordinate system conversion parameters between the camera coordinate system and the world coordinate system; Determine the world coordinates of the first marking point according to the coordinate system conversion parameter, the camera intrinsic parameter of the camera and the second pixel coordinate; Determining the world coordinates of the pupil according to the coordinate system conversion parameter, the camera intrinsic parameter and the first pixel coordinate of the pupil; The deviation between the world coordinates of the pupil and the world coordinates of the first marking point is determined as the gaze vector of the subject.

[0081] Optionally, the coordinate system conversion parameters are obtained in the following manner: A second image of the subject's eye area is captured by the camera, wherein the second image includes the first marking point and the at least two reference points; wherein the at least two reference points are at least two second marking points set at second positions around the subject's eyes, or the at least two reference points are left-eye corneal reflection spots and right-eye corneal reflection spots formed by the infrared light source of the eye tracking device irradiating the eyes of the subject during the process of capturing the second image; Acquire the world coordinates of at least two reference points and the first marking point in the second image; determining pixel coordinates of at least two reference points and the first marking point in the second image; Coordinate system conversion parameters between a camera coordinate system and a world coordinate system are determined according to pixel coordinates and world coordinates of at least two reference points in the second image, pixel coordinates and world coordinates of the first marking point, and the camera intrinsic parameters.

[0082] Optionally, the mapping relationship is a first mapping relationship; The first mapping relationship is obtained in the following manner: Obtaining eye images of the subject when he or she is looking at each calibration point; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the gaze vector of the subject when he or she is looking at the calibration point according to the deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marking point; A first mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

[0083] Optionally, the eye tracking device further comprises a display; The gaze point location module 240 can be used to: The intersection of the gaze vector and the display is used as the gaze point position of the subject on the display.

[0084] Optionally, the eye tracking device further includes a display; the gaze point position of the subject is determined based on the gaze vector and a predetermined mapping relationship, and the mapping relationship is a second mapping relationship; The second mapping relationship is obtained in the following manner: Acquiring the coordinate system conversion parameters and the eye images of the subject when he or she is looking at each calibration point; For each calibration point, determine the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; determine the world coordinates of the pupil according to the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the world coordinates of the first marking point according to the pixel coordinates of the first marking point in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the gaze vector of the subject when he looks at the calibration point according to the deviation between the world coordinates of the pupil in the eye image corresponding to the calibration point and the world coordinates of the first marking point; A second mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

[0085] Optionally, the subject sets the marking point by wearing a frame-like device, the first marking point is configured at a first position of the frame-like device, and the at least two second marking points are respectively configured at second positions of the frame-like device.

[0086] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0087] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0088] An embodiment of the present application provides an eye tracking system, including an image collector, a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any optional embodiment of the present application can be implemented.

[0089] Figure 7 FIG. 4 shows a schematic diagram of the structure of an eye tracking system applicable to an embodiment of the present invention. Figure 7 As shown, the system can be used to implement the method provided in any embodiment of the present invention.

[0090] like Figure 7As shown in FIG. , the eye tracking system 2000 may mainly include at least one processor 2001 ( Figure 7 ), memory 2002, communication module 2003, input / output interface 2004, camera 2006 and other components. Optionally, the components can be connected and communicated through bus 2005. It should be noted that Figure 7 The structure of the eye tracking system 2000 shown in the figure is merely illustrative and does not constitute a limitation on the eye tracking system to which the method provided in the embodiment of the present application is applicable.

[0091] The memory 2002 may be used to store an operating system and an application program, etc. The application program may include a computer program that implements the method shown in the embodiment of the present invention when called by the processor 2001, and may also include a program for implementing other functions or services. The memory 2002 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0092] The processor 2001 is connected to the memory 2002 via the bus 2005, and implements corresponding functions by calling the application program stored in the memory 2002. The processor 2001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0093] The eye tracking system 2000 can be connected to a network through a communication module 2003 (which may include but is not limited to components such as a network interface) to communicate with other devices (such as a user terminal or a server, etc.) through the network to achieve data interaction, such as sending data to other devices or receiving data from other devices. Among them, the communication module 2003 may include a wired network interface and / or a wireless network interface, etc., that is, the communication module may include at least one of a wired communication module or a wireless communication module.

[0094] The eye tracking system 2000 can be connected to the required input / output devices, such as a keyboard, a display device (display), etc., through the input / output interface 2004. The eye tracking system 2000 itself can have a display device, and can also be connected to other display devices through the interface 2004. Optionally, a storage device, such as a hard disk, can also be connected through the interface 2004, so that the data in the eye tracking system 2000 can be stored in the storage device, or the data in the storage device can be read, and the data in the storage device can also be stored in the memory 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 2004 can be a component of the eye tracking system 2000, or it can be an external device connected to the eye tracking system 2000 when needed.

[0095] The bus 2005 for connecting the components may include a path for transmitting information between the above components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 2005 may be divided into an address bus, a data bus, a control bus, etc.

[0096] Camera 2006 may be an infrared camera, a depth camera, etc., for collecting eye images of the subject.

[0097] Optionally, for the solution provided by the embodiment of the present invention, the memory 2002 can be used to store a computer program for executing the solution of the present invention, and run by the processor 2001. When the processor 2001 runs the computer program, the action of the method or device provided by the embodiment of the present invention is implemented.

[0098] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0099] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0100] It should be noted that the terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown in the figure or described in the text.

[0101] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0102] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. An eye tracking method for detecting and removing head movements, characterized in that: A first marking point is set at a first position around the eye of the subject, and the method is performed by an eye tracking device, wherein the eye tracking device includes a camera; The method comprises: Acquire a first image of the eye area of ​​the subject by using the camera, wherein the first image includes the first marking point; Determine a first pixel coordinate of a pupil in the first image and a second pixel coordinate of the first marking point; Determine a gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates; wherein the gaze vector is a position offset of the pupil of the subject relative to the head of the subject; The gaze point position of the subject is determined according to the gaze vector.

2. The method according to claim 1, characterized in that: The eye tracking device also includes a display; Determining the gaze point position of the subject according to the gaze vector includes: Determining the gaze point position of the subject on the display according to the gaze vector and a predetermined mapping relationship; The mapping relationship reflects the corresponding relationship between the gaze vector of the subject and the coordinate position on the display.

3. The method according to claim 1, characterized in that The determining the gaze vector of the subject according to the first pixel coordinate and the second pixel coordinate includes: A deviation between the first pixel coordinate and the second pixel coordinate is determined, and the deviation is used as a gaze vector of the subject.

4. The method according to claim 1, characterized in that The determining the gaze vector of the subject according to the first pixel coordinate and the second pixel coordinate includes: Get the coordinate system conversion parameters between the camera coordinate system and the world coordinate system; Determine the world coordinates of the first marking point according to the coordinate system conversion parameter, the camera intrinsic parameter of the camera and the second pixel coordinate; Determining the world coordinates of the pupil according to the coordinate system conversion parameter, the camera intrinsic parameter and the first pixel coordinate of the pupil; The deviation between the world coordinates of the pupil and the world coordinates of the first marking point is determined as the gaze vector of the subject.

5. The method according to claim 4, characterized in that The coordinate system transformation parameters are obtained in the following way: A second image of the subject's eye area is captured by the camera, wherein the second image includes the first marking point and the at least two reference points; wherein the at least two reference points are at least two second marking points set at second positions around the subject's eyes, or the at least two reference points are left-eye corneal reflection spots and right-eye corneal reflection spots formed by the infrared light source of the eye tracking device irradiating the eyes of the subject during the process of capturing the second image; Acquire the world coordinates of at least two reference points and the first marking point in the second image; determining pixel coordinates of at least two reference points and the first marking point in the second image; Coordinate system conversion parameters between a camera coordinate system and a world coordinate system are determined according to pixel coordinates and world coordinates of at least two reference points in the second image, pixel coordinates and world coordinates of the first marking point, and the camera intrinsic parameters.

6. The method according to claim 2, characterized in that The mapping relationship is a first mapping relationship; The first mapping relationship is obtained in the following manner: Obtaining eye images of the subject when he or she is looking at each calibration point; For each calibration point, determining the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; Determining a gaze vector of the subject when gazing at the calibration point according to a deviation between the pixel coordinates of the pupil and the pixel coordinates of the first marking point; A first mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

7. The method according to claim 4, characterized in that The eye tracking device also includes a display; Determining the gaze point position of the subject according to the gaze vector includes: The intersection of the gaze vector and the display is used as the gaze point position of the subject on the display.

8. The method according to claim 5, characterized in that The eye tracking device further includes a display; the gaze point position of the subject is determined based on the gaze vector and a predetermined mapping relationship, and the mapping relationship is a second mapping relationship; The second mapping relationship is obtained in the following manner: Acquiring the coordinate system conversion parameters and the eye images of the subject when he or she is looking at each calibration point; For each calibration point, determining the pixel coordinates of the pupil in the eye image corresponding to the calibration point and the pixel coordinates of the first marking point; Determine the world coordinates of the pupil according to the pixel coordinates of the pupil in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; Determine the world coordinates of the first marking point according to the pixel coordinates of the first marking point in the eye image corresponding to the calibration point, the camera intrinsic parameters and the coordinate system conversion parameters; determine the gaze vector of the subject when looking at the calibration point according to the deviation between the world coordinates of the pupil in the eye image corresponding to the calibration point and the world coordinates of the first marking point; A second mapping relationship is determined according to the gaze vector corresponding to each calibration point and the coordinate position of each calibration point in the display.

9. The method according to claim 5, characterized in that The subject sets the marking point by wearing a frame-like device, the first marking point is configured at a first position of the frame-like device, and the at least two second marking points are respectively configured at second positions of the frame-like device.

10. An eye tracking device for detecting and removing head movements, characterized in that: A first marking point is set at a first position around the eye of the subject, and the device is deployed in an eye tracking device, and the eye tracking device includes a camera; The device comprises: An eye image acquisition module, used for acquiring a first image of the eye area of ​​the subject through the camera, wherein the first image includes the first marking point; a pupil and marker point positioning module, configured to determine a first pixel coordinate of the pupil and a second pixel coordinate of the first marker point in the first image; A gaze vector determination module, used to determine the gaze vector of the subject according to the first pixel coordinates and the second pixel coordinates; wherein the gaze vector is a position offset of the pupil of the subject relative to the head of the subject; A gaze point positioning module is used to determine the gaze point position of the subject according to the position offset.

11. An eye tracking system, characterized in that: The eye tracking system includes a camera, a display, a memory and a processor, the camera is used to collect eye images of the subject, the memory stores a computer program, and the processor executes the computer program to implement the method described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images