Machine learning based eye tracking device and method

Abstract

According to one embodiment of the present invention, a machine learning based eye tracking device can easily recognize serial numbers on various banknotes where the distinction between the background and the characters is unclear.

Description

[Technical field]

[0001] The present invention relates to eye-tracking technology, and more particularly to an apparatus and method for eye-tracking based on machine learning. [Background technology]

[0002] Eye tracking technology measures eye movements to determine where a user is looking. It is primarily used in pointing devices that serve as an interface between users and computers, replacing keyboards and mice. The results output by eye tracking devices are used to analyze the user's concentration level or as data to analyze signs of illness and development patterns.

[0003] Typically, eye-tracking devices use additional devices such as infrared cameras, high-magnification zoom lenses, and webcams to capture the pupil center point in an image of the user's eye, and then apply a transformation function based on that to track the eye position.

[0004] However, since a separate measuring device must be worn on the face when tracking the gaze position, it can be inconvenient to use the device, and since expensive devices such as infrared illuminators are used, gaze tracking is expensive.

[0005] Additionally, the complexity of the eye-tracking calculations reduces processing speed and eye-tracking performance.

[0006] The background art of the present invention is disclosed in Korean Patent No. 10-1094766. Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention provides a machine learning based eye tracking device that tracks the gaze using a camera module without a separate lighting device.

[0008] The present invention provides a machine learning-based eye tracking device that estimates eye gaze coordinates based on a regression model and tracks the position where a user's eye gaze intersects with a display.

[0009] The technical problems that the present invention aims to solve are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0010] According to one aspect of the present invention, a machine learning based eye tracking device is provided.

[0011] A machine learning based gaze tracking device according to one embodiment of the present invention includes an input unit that inputs an image including a face, a feature point detection unit that detects a first feature point from the image including the face, a face direction detection unit that detects the direction of the face based on the detected first feature point, an eye direction detection unit that detects the eye direction, which is a feature of the eye, at the detected first feature point, and an gaze tracking unit that performs gaze tracking using the detected feature points, face direction and eye direction feature values ​​as inputs to a learned gaze tracking regression model.

[0012] According to another aspect of the present invention, a machine learning based eye tracking method is provided.

[0013] A machine learning based gaze tracking method according to one embodiment of the present invention may include the steps of: inputting an image including a face; detecting first feature points in the image including the face; detecting a face direction based on the detected first feature points; detecting an eye direction, which is an eye feature, at the detected first feature points; and performing gaze tracking using the detected feature points and the feature values ​​of the face direction and eye direction as inputs to a trained gaze tracking regression model. Effect of the Invention

[0014] According to an embodiment of the present invention, a machine learning based eye tracking device can track the eye gaze using a camera module without a separate lighting device.

[0015] According to an embodiment of the present invention, a machine learning based eye tracking device can estimate eye gaze coordinates based on a regression model to track the intersection of a user's gaze and a display.

[0016] It should be understood that the effects of the present invention are not limited to the effects described above, but include all effects that can be inferred from the configuration of the invention described in the description of the present invention or the claims. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 9] FIG. 9 is a diagram illustrating a machine learning based eye tracking device according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating a machine learning based gaze tracking method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention will be described below with reference to the accompanying drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts that are not related to the description are omitted in order to clearly explain the present invention, and similar parts are given similar reference numerals throughout the specification.

[0019] Throughout this specification, when a part is said to be "connected (connected, in contact, coupled)" to another part, this includes not only "directly connected" but also "indirectly connected" with another member in between. Furthermore, when a part is said to "include" a certain component, this means that it can further include other components, rather than excluding other components, unless otherwise specified.

[0020] The terms used in this specification are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless otherwise clearly indicated in the context. In this specification, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in this specification, and are not intended to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0021] 1 to 9 are diagrams illustrating a machine learning-based eye tracking device according to an embodiment of the present invention.

[0022] Referring to FIG. 1 , the machine learning based eye tracking device 100 includes an input unit 110 , a feature point detection unit 120 , a face direction detection unit 130 , an eye direction detection unit 140 , a model learning unit 150 and an eye tracking unit 160 .

[0023] An image including a face is input to the input unit 110. For example, the input unit 110 inputs an image including a face captured via a web camera or an RGB camera.

[0024] The feature point detection unit 120 detects a first feature point in an image including a face. Here, the first feature point can be detected in coordinates in a three-dimensional space using the 3D FFA model.

[0025] The face direction detection unit 130 detects the face direction based on the detected feature points. Specifically, the face direction detection unit 130 extracts, from the first feature points detected by the feature point detection unit 120, second feature points whose positions do not change due to changes in facial expression, movement, or the like.

[0026] For example, as shown in FIG. 2, the face direction detection unit 130 can extract the corners of both eyes and the start point of the nose groove as second feature points from the first feature points detected in the face image.

[0027] The face direction detection unit 130 detects a plane relative to the position and direction of the face based on the extracted second feature point. Here, the face direction detection unit 130 can calculate a normal vector perpendicular to the plane relative to the detected position and direction of the face.

[0028] The face direction detection unit 130 detects a three-dimensional vector for the face direction based on the calculated normal vector.

[0029] The eyeball direction detection unit 140 detects the direction of the eyeball by extracting a third feature point, which is a feature point of the eyeball, from the first feature point detected by the feature point detection unit 120. The eyeball direction detection unit 140 includes a rotation center calculation unit 141, an iris center calculation unit 143, and a center connection unit 145.

[0030] 3, the rotation center calculation unit 141 extracts a third feature point, which is a feature point of the eyeball, from the first feature point detected by the feature point detection unit 120, and calculates the rotation center of the eyeball using the third feature point. For example, the rotation center calculation unit 141 calculates the rotation center of the eyeball using four third feature points (Inner, Mid, Top, Outer) extracted as shown in FIG.

[0031] The iris center calculation unit 143 calculates the iris center based on the inclination in the eye region of the face image. For example, the iris center calculation unit 143 detects the iris center point and the iris center coordinates based on the inclination in the eye region of the face image. Specifically, the iris center calculation unit 143 can use template matching to detect the iris center point through the pixel gradient at the iris boundary in the eye region. Template matching is a method of constructing an average template of the region to be searched and finding a matching position using the correlation between the template and the input image. In the present invention, the pixel position having the highest value in the eye region can be detected as the iris center point through the correlation between the template filter and the pixel gradient in the eye region, and can be expressed in coordinates.

[0032] 5, the iris center calculation unit 143 can extract the pixel position having the highest value in the eye region through correlation between the template filter and the pixel gradient in the eye region (a). Here, the iris center calculation unit 143 can perform filtering using an image with inverted pixel values ​​to filter the area under the iris having high values ​​due to the curve of the eyelid (b). The iris center calculation unit can detect the position having the highest pixel value as the iris center point by reflecting the filtering result (c, d).

[0033] Referring to FIG. 6, the iris center calculation unit 143 can detect the iris center point from the RGB face image (610), and after extracting the eye region from the face image, can perform gradient-based iris center point detection to detect the center point (620).

[0034] Referring again to FIG. 3, the center connector 145 connects the rotation center coordinates of the eyeball and the iris center coordinates to detect a three-dimensional vector for the direction of the eyeball.

[0035] 7, the model learning unit 150 learns an eye tracking model using the previously detected eye gaze feature value as an input. The model learning unit 150 learns an eye tracking model using the nose groove start point and face direction vector detected by the face direction detection unit 130, the iris center point detected by the iris center calculation unit 143, and the eyeball direction vector detected by the center connection unit 145 as input (e). Specifically, the model learning unit 150 learns an eye tracking model using the eyeball direction vector (x1, y1, z1), face direction vector (x2, y2, z2), nose groove start point (x3, y3, z3), and iris center point (x4, y4, z4) as input (f).

[0036] The model learning unit 150 can learn Support Vector Regression (SVR) with an eye tracking model as shown in Fig. 8. Support Vector Regression (SVR), which is an eye tracking regression model, is a machine learning based model extended with Support Vector Machine (SVM) so that it can be used for regression analysis, and performs eye tracking using a regression function. The model learning unit 150 can learn a first support vector regression (SVR#1) and a second support vector regression (SVR#2).

[0037] Here, the model learning unit 150 learns support vector regression using gaze feature values ​​detected through a calibration process as inputs. That is, the model learning unit 150 can learn x and y coordinates of the gaze position that appears on the display using the nose groove start point, face direction vector, iris center point, and eye direction vector detected when the gaze is directed to nine calibration points as shown in FIG 9.

[0038] The model learning unit 150 can learn the x and y coordinates of the gaze position using the following Equation 1.

[0039]

number

[0040] Here, x represents a matrix of 12 feature values, totaling x, y, and z coordinates of the input nose groove start point, face direction vector, iris center point, and eyeball direction vector, and y represents the coordinate on the display according to the input. T means the regression model parameters that convert the input value into the gaze coordinate, and the following equation 2 is used to find the regression model parameters that minimize the loss.

[0041]

number

[0042] The gaze tracking unit 160 performs gaze tracking using the model learned by the model learning unit 150. The gaze tracking unit 160 can perform semi-permanent gaze tracking by resolving a degree of freedom problem that may occur in a projection transformation process that calculates gaze coordinates on a monitor plane in a two-dimensional space using the movements of the user's face and eyes in a three-dimensional space through the learned regression model.

[0043] FIG. 10 is a diagram illustrating a machine learning based gaze tracking method according to an embodiment of the present invention.

[0044] 10, in step S1001, the machine learning-based eye tracking device 100 inputs an image including a face. For example, the machine learning-based eye tracking device 100 inputs an image including a face acquired via a webcam or an RGB camera.

[0045] In step S1003, the machine learning-based eye tracking device 100 detects a first feature point in an image including a face, where the first feature point can be detected in coordinates in a three-dimensional space using the 3DFFA model.

[0046] In step S1005, the machine learning-based gaze tracking device 100 extracts second feature points whose positions do not change due to changes in facial expression or movement from the first feature points detected by the feature point detection unit 120. For example, the machine learning-based gaze tracking device 100 can extract the corners of both eyes and the start point of the nose groove as the second feature points from the first feature points detected in the face image.

[0047] In step S1007, the machine learning-based eye tracking device 100 detects a plane relative to the position and orientation of the face based on the extracted second feature point, and can calculate a normal vector perpendicular to the plane relative to the detected position and orientation of the face.

[0048] In step S1009, the machine learning based eye tracking device 100 detects a three-dimensional vector for the face direction based on the calculated normal vector.

[0049] In step S1011, the machine learning-based gaze tracking device 100 extracts a third feature point, which is a feature point of the eyeball, from the first feature point detected by the feature point detection unit 120, and calculates the rotation center of the eyeball using the third feature point. For example, the machine learning-based gaze tracking device 100 calculates the rotation center of the eyeball using the four third feature points (Inner, Mid, Top, Outer) extracted as shown in FIG. 4 above.

[0050] In step S1013, the machine learning-based eye tracking device 100 calculates the iris center based on the gradient of the eye region of the face image. Specifically, the machine learning-based eye tracking device 100 can extract the pixel position having the highest value in the eye region through correlation between the template filter and the pixel gradient in the eye region. Here, the machine learning-based eye tracking device 100 can perform filtering using an image with inverted pixel values ​​to filter the area under the iris having high values ​​due to the curve of the eyelid. The machine learning-based eye tracking device 100 can detect the position having the highest pixel value as the iris center point by reflecting the filtering result.

[0051] In step S1015, the machine learning based eye tracking device 100 detects a 3D vector for the eye direction by combining the eye rotation center coordinate and the iris center coordinate.

[0052] In step S1017, the machine learning-based gaze tracking device 100 learns an gaze tracking model using the gaze feature values ​​detected earlier as inputs. Specifically, the model learning unit 150 learns the gaze tracking model using the eyeball direction vector (x1, y1, z1), face direction vector (x2, y2, z2), nasal groove start point (x3, y3, z3), and iris center point (x4, y4, z4) as inputs (f).

[0053] The machine learning-based eye tracking device 100 can learn Support Vector Regression (SVR) as an eye tracking model. The eye tracking regression model, Support Vector Regression (SVR), is a machine learning-based model extended with Support Vector Machine (SVM) so that it can be used for regression analysis, and performs eye tracking using a regression function. The machine learning-based eye tracking device 100 can learn a first support vector regression (SVR#1) and a second support vector regression (SVR#2).

[0054] Here, the machine learning based eye tracking device 100 learns support vector regression using eye gaze feature values ​​detected through a calibration process as inputs. That is, the machine learning based eye tracking device 100 can learn x and y coordinates of the eye gaze position that appears on the display using the nose groove start point, face direction vector, iris center point, and eyeball direction vector detected when the eye gaze is directed to nine calibration points as shown in FIG.

[0055] In step S1019, the machine learning-based eye tracking device 100 performs eye tracking using the model learned by the model learning unit 150. The machine learning-based eye tracking device 100 can perform semi-permanent eye tracking by solving a degree of freedom problem that may occur in a projective transformation process that calculates eye gaze coordinates on a monitor plane in a two-dimensional space using the movements of the user's face and eyes in a three-dimensional space through the learned regression model.

[0056] The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described in a single form can be implemented in a distributed manner, and similarly, components described as being distributed can be implemented in a combined form.

[0057] The scope of the present invention is defined by the following claims, and all modifications and variations that fall within the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention.

Claims

1. In machine learning-based eye tracking devices, An input unit for inputting an image including a face; a feature point detection unit that detects a first feature point in the image including the face; a face direction detection unit that detects a face direction based on the detected first feature point; an eyeball direction detection unit that detects an eyeball direction, which is a feature of the eyeball, at the detected first feature point; a model learning unit that learns an eye gaze tracking model using the detected feature points and directions as inputs; An eye tracking unit that performs eye tracking using the learned eye tracking model; Including, The face direction detection unit A machine learning-based gaze tracking device that extracts the corners of both eyes and the start of the nasal groove as second feature points, whose positions do not change due to changes in facial expression or movement, from the first feature points, and detects the direction of the face based on the extracted second feature points.

2. The eye direction detection unit is a rotation center calculation unit that extracts a third feature point that is a feature point of an eyeball from the first feature points detected by the feature point detection unit, and calculates a rotation center of the eyeball using the third feature point; an iris center calculation unit that calculates an iris center in an eye region of a face image; a center connection unit that connects the rotation center of the eyeball and the center of the iris to detect a three-dimensional vector relative to the direction of the eyeball; The machine learning based eye tracking device of claim 1 , comprising:

3. The iris center calculation unit detecting an iris center point and an iris center coordinate based on the inclination in the eye region; The machine learning based eye tracking device of claim 2 .

4. In machine learning-based eye tracking methods, inputting an image including a face; detecting a first feature point in an image including the face; detecting a face direction based on the detected first feature point; detecting an eye direction, which is a characteristic of an eye, at the detected first feature point; performing gaze tracking using the detected feature points, face direction, and eye direction feature values ​​as inputs to a trained gaze tracking regression model; Including, The step of detecting a face direction based on the detected first feature point includes: extracting, as second feature points, the outer corners of both eyes and the start point of the nose groove, whose positions do not change due to a change in facial expression or movement, from the first feature points; and detecting a face orientation based on the extracted second feature points.

5. The step of detecting an eye direction, which is a feature of the eye, at the detected first feature point includes: extracting a third feature point, which is a feature point of an eyeball, from the detected first feature point, and calculating a rotation center of the eyeball using the third feature point; calculating an iris center in an eye region of a face image; detecting a three-dimensional vector for the eye's orientation by connecting the eye's center of rotation and the iris center; The machine learning based eye tracking method of claim 4 , comprising:

6. The step of calculating an iris center in an eye region of the face image includes: detecting an iris center point and an iris center coordinate based on the inclination in the eye region; The machine learning based eye tracking method of claim 5.

7. A computer program that performs the machine learning based gaze tracking method described in any one of claims 4 to 6, and is recorded on a computer-readable recording medium.

Images