A method for line of sight estimation based on 2D data

Abstract

This invention provides a method for gaze estimation based on 2D data, belonging to the field of CV algorithms. On a single eye's 2D image data, 50 key points and the pupil center key point are labeled. Then, based on the top, bottom, left, and rightmost points from these 50 key points, the gaze point (gaze point) of the pupil center under normal eye-viewing conditions is estimated. The offset of the labeled gaze point from the current pupil center is calculated. This solves the difficulties in acquiring gaze estimation data and the tedious work of converting 3D data to 2D data. Thus, using 2D data achieves the effect of a model trained on 3D data, with low cost; the dataset can be created independently using only ordinary 2D image data.

Description

Technical Field

[0001] This invention relates to the field of CV algorithms, and in particular to a method for gaze estimation based on 2D data. Background Technology

[0002] In recent years, with the rise of the artificial intelligence wave, AI algorithms have developed rapidly, especially in the field of driver assistance. Various AI technologies for driver assistance are becoming more and more mature. The use of AI to detect the driver's gaze and estimate the driver's attention has also made some progress. Estimating the driver's gaze based on gaze is even more common. Research on gaze has always been a niche area, perhaps because data acquisition is more difficult than other types of data, or perhaps because the research on algorithms has not yielded satisfactory results, which has brought great obstacles to research in the field of gaze.

[0003] Current gaze estimation methods based on gaze are primarily geometric. These methods first acquire 3D data using various devices. In this 3D data, the direction of a person's gaze is represented by two angles: pitch (vertical) and yaw (horizontal). Acquiring 3D data involves head pose information because, in the camera coordinate system, the direction of gaze depends not only on the eye position but also on the head posture. For example, when looking down at an object, although the eyes are at an angle relative to the head, in the camera coordinate system, the person is looking straight ahead. Calculating the coordinates of the world coordinates and the human eye's position relative to those world coordinates requires significant effort in data acquisition. This difficulty in data acquisition directly contributes to the arduous path of gaze estimation research. Processing 3D data is also a huge challenge. After acquiring the 3D data, the 3D coordinate information needs to be mapped to a 2D coordinate system before being input into a neural network for training. Another type of method is appearance-based. Appearance-based methods directly learn a model that maps appearance to gaze.

[0004] Existing gaze estimation methods for gaze have the following drawbacks:

[0005] 1) The difficulty of acquiring 3D data. In algorithm research, not only the conception of neural network algorithms affects the research progress, but data is also extremely important. Currently, there is very little data related to line-of-sight estimation in publicly available datasets. Either the datasets are not publicly available, or they were released for a period of time but were discontinued for some reason, or are difficult to download locally. Creating one's own dataset is also very difficult, requiring a great deal of effort and resources, as well as a strong mathematical foundation, for calculations involving camera coordinate systems and world coordinate systems.

[0006] 2) The difficulty of processing 3D data. 3D data contains most of the information as representations of human posture. To use such information, it is necessary to map the 3D information onto ordinary 2D information, or to design specialized neural network algorithms. Acquiring data is not simple in itself, and processing it requires even more steps to consider how to utilize the hard-won data. Obviously, such an approach requires complex procedures and effort.

[0007] 3) When using 2D data, how can we achieve a similar effect to a model trained on 3D data, even if we replace it with our own 2D data?

[0008] 4) The high cost-effectiveness of creating datasets. Some companies even spend a lot of money to purchase specialized equipment for creating datasets in order to obtain the data they need for line-of-sight estimation. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides a method for gaze estimation based on 2D data. By annotating key points on the 2D image data of a single eye, the gaze of key targets is obtained, thus solving the difficulty in acquiring gaze estimation data.

[0010] The technical solution adopted in this invention is as follows:

[0011] A method for gaze estimation based on 2D data includes the following steps:

[0012] 1) Set grid auxiliary lines and concentric circle auxiliary lines: With the upper left corner of the 2D image with a single eye as the origin, draw auxiliary lines with equal intervals of 10×10mm. The crisscrossing auxiliary lines divide the image into a grid shape. Then, based on the center of the pupil in the 2D image as the center, draw concentric circles with equal centers and equal intervals.

[0013] 2) Standard annotation of 51 key points: When the 2D graphic is displayed with the eyes facing forward, set 20 key points on the upper eyelid at equal intervals from left to right; set 20 key points on the lower eyelid at equal intervals from right to left; and set 10 key points on the pupil within the visible range, with the center of the pupil being the 51st key point, denoted as TK(TK_x, TK_y).

[0014] 3) Mark the gaze key points: Obtain the top, bottom, left and right four points out of the 40 key points of the upper and lower eyelids. Using these four points, combined with grid auxiliary lines and concentric circles, estimate the position of the pupil center under normal viewing conditions as the gaze key point TN (TN_x, TN_y).

[0015] 4) Calculate the offset of the marked gaze from the current pupil center: Using the gaze key point TN(TN_x, TN_y) as the origin, draw the X and Y axes, and calculate the offset gaze(gaze_x, gaze_y): gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y;

[0016] 5) Normalize 52 key points: Based on the position of the pupil center in the 2D image data obtained in step 4) and the offset of the horizontal and vertical coordinates of the pupil center when looking forward under normal conditions, and the coordinate information of the key points of the eyes of 51 people, normalize to obtain stable and standard gaze key point information.

[0017] Preferably, in step 2), when marking the upper and lower eyelids, if the eye is to the left or right, the key points on the side that is to the left are gradually closer to the next point and are more densely packed; take the point corresponding to the outer edge of the pupil on the left side of the 2D image as the first point, and mark them in a clockwise direction with equal intervals, marking 10 key points around the pupil.

[0018] Preferably, in step 3), the TN_y of the gaze keypoint TN is calculated by first calculating the Y-axis distance between the highest and lowest points, then finding the Y-axis coordinate of the midpoint between the highest and lowest points, dividing the midpoint's Y-axis coordinate by the grid size to obtain the grid position of the midpoint, and then estimating an offset of 2-3 grids to obtain the estimated TN_y; the TN_x of the gaze keypoint TN is calculated by first calculating the X-axis distance between the leftmost and rightmost points, then finding the X-axis coordinate of the midpoint between the leftmost and rightmost points, dividing the midpoint's X-axis coordinate by the grid size to obtain the grid position of the midpoint, and then estimating an offset of 1-2 grids to obtain the estimated TN_x.

[0019] Preferably, the offset gaze in step 4) is divided into:

[0020] 4.1) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is in the first quadrant, we have:

[0021] gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y;

[0022] 4.2) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is in the second quadrant, we have:

[0023] gaze_x = TK_x - TN_x, gaze_y = -(TK_y- TN_y);

[0024] 4.3) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is in the third quadrant, we have:

[0025] gaze_x = -(TN_x - TK_x), gaze_y = -(TK_y- TN_y);

[0026] 4.4) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is in the fourth quadrant, we have:

[0027] gaze_x = -(TN_x - TK_x), gaze_y = TN_y- TK_y;

[0028] 4.5) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is located on the positive half of the x-axis, we have:

[0029] gaze_x = TN_x - TK_x, gaze_y = 0;

[0030] 4.6) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is located on the negative half of the x-axis, we have:

[0031] gaze_x = -(TN_x - TK_x), gaze_y = 0;

[0032] 4.7) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is located on the positive half of the y-axis, we have:

[0033] gaze_x = 0, gaze_y = TN_y - TK_y;

[0034] 4.8) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is located on the negative half of the y-axis, we have:

[0035] gaze_x = 0, gaze_y = -(TK_y - TN_y);

[0036] 4.9) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics via the x-axis and y-axis with TN as the origin. When TK is located at the origin, we have:

[0037] gaze_x = 0; gaze_y = 0.

[0038] Preferably, in step 5), the larger the x-axis of gaze, the more the current human eye is shifted to the left or right. A positive value indicates looking to the right, while a negative value indicates looking to the left. The larger the y-axis of gaze, the more the current human eye is shifted to the up or down. A positive value indicates looking upwards, while a negative value indicates looking downwards.

[0039] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for gaze estimation based on 2D data. On a single eye's 2D image data, 50 key points are marked. Then, based on the top, bottom, left, and rightmost points among these 50 key points, the gaze point (gaze point) representing the center of the pupil when the human eye is looking straight ahead under normal conditions is estimated. The offset of the marked gaze point from the current pupil center is calculated. This solves the difficulties in acquiring gaze estimation data and the tedious work of converting 3D data to 2D data. Thus, using 2D data achieves the effect of a model trained with 3D data, at a low cost, requiring only ordinary 2D image data to create the dataset. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the steps of a method for estimating line of sight based on 2D data provided by the present invention.

[0041] Figure 2 This is a schematic diagram illustrating a line-of-sight estimation method based on 2D data provided by the present invention. Detailed Implementation

[0042] The preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0043] Figures 1 to 2 This is a preferred embodiment of the gaze estimation method based on 2D data provided by the present invention. For example... Figures 1 to 2 As shown, the gaze estimation method based on 2D data includes the following steps: 1) Setting grid auxiliary lines and concentric circle auxiliary lines: Taking the upper left corner of the 2D image with a single eye as the origin, draw auxiliary lines with equal intervals of 10×10mm. The crisscrossing auxiliary lines divide the image into a grid shape. Then, based on the center of the pupil in the 2D image as the center, draw concentric circles with equal centers and equal intervals.

[0044] 2) Standard annotation of 51 key points: When the 2D graphic is displayed with the eyes facing forward, set 20 key points on the upper eyelid at equal intervals from left to right; set 20 key points on the lower eyelid at equal intervals from right to left; and set 10 key points on the pupil within the visible range, with the center of the pupil being the 51st key point, denoted as TK(TK_x, TK_y).

[0045] 3) Mark the gaze key points: Obtain the top, bottom, left and right four points out of the 40 key points of the upper and lower eyelids. Using these four points, combined with grid auxiliary lines and concentric circles, estimate the position of the pupil center under normal viewing conditions as the gaze key point TN (TN_x, TN_y).

[0046] 4) Calculate the offset of the marked gaze from the current pupil center: Using the gaze key point TN(TN_x, TN_y) as the origin, draw the X and Y axes, and calculate the offset gaze(gaze_x, gaze_y): gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y;

[0047] 5) Normalize 52 key points: Based on the position of the pupil center in the 2D image data obtained in step 4) and the offset of the horizontal and vertical coordinates of the pupil center when looking forward under normal conditions, and the coordinate information of the key points of the eyes of 51 people, normalize to obtain stable and standard gaze key point information.

[0048] In step 2), when marking the upper eyelid, if the eye is displayed from the front, the 20 key points of the upper eyelid are marked at equal intervals from left to right. If the eye is to the left or right, the key points on the side that is to the left are marked more densely and gradually move closer to the next point.

[0049] In step 2), when marking the lower eyelid, with the eye facing forward, the 20 key points of the lower eyelid are marked at equal intervals from right to left, and connected with the points marked on the upper eyelid to form a slightly elliptical pattern that runs along both the upper and lower eyelids; similarly, if the eye is to the left or right, the key points on the side that is to the left will gradually move closer to the next point and become more dense.

[0050] In step 2), when annotating the pupil, take the point corresponding to the outer edge of the pupil on the left side of the 2D image as the first point, and annotate in a clockwise direction at equal intervals, marking 10 key points around the pupil; the 51st point is marked as the center of the pupil. If the eyelid obscures part of the pupil, the visible area of ​​the pupil is taken as the pupil itself.

[0051] In step 3), the TN_y of the Gaze keypoint TN is calculated by first determining the Y-axis distance between the highest and lowest points, then finding the Y-axis coordinate of the midpoint between them. The midpoint's Y-axis coordinate is then divided by the grid size to obtain its grid position. Finally, an estimated offset of 2-3 grids is made to obtain the estimated TN_y. Similarly, the TN_x of the Gaze keypoint TN is calculated by first determining the X-axis distance between the leftmost and rightmost points, then finding the X-axis coordinate of the midpoint between them. The midpoint's X-axis coordinate is then divided by the grid size to obtain its grid position. Finally, an estimated offset of 1-2 grids is made to obtain the estimated TN_x.

[0052] Step 3) yields the coordinates of the gaze keypoint TN on the original 2D data. Step 4) uses the offset of the current pupil center relative to normal human vision. The offset gaze varies depending on the position of the pupil center TK, and can be categorized into several cases:

[0053] 4.1) Taking TN as the origin, the entire image data is divided into four quadrants (one to four) through the x-axis and y-axis, with TN as the origin. When TK is in the first quadrant, we have: gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y, as shown below. Figure 2 As shown;

[0054] 4.2) Taking TN as the origin, the entire image data is divided into one to four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is in the second quadrant, we have: gaze_x = TK_x - TN_x, gaze_y = -(TK_y - TN_y);

[0055] 4.3) Taking TN as the origin, the entire image data is divided into one to four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is in the third quadrant, we have: gaze_x = -(TN_x - TK_x), gaze_y = -(TK_y - TN_y);

[0056] 4.4) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is in the fourth quadrant, we have: gaze_x = -(TN_x - TK_x), gaze_y = TN_y - TK_y;

[0057] 4.5) Taking TN as the origin, the entire image data is divided into one to four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is located on the positive half of the x-axis, we have: gaze_x = TN_x - TK_x, gaze_y = 0;

[0058] 4.6) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is located on the negative half of the x-axis, we have: gaze_x = -(TN_x - TK_x), gaze_y = 0;

[0059] 4.7) Taking TN as the origin, the entire image data is divided into one to four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is located on the positive half of the y-axis, we have: gaze_x = 0, gaze_y = TN_y - TK_y;

[0060] 4.8) Taking TN as the origin, the entire image data is divided into four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is located on the negative half of the y-axis, we have: gaze_x = 0, gaze_y = -(TK_y - TN_y);

[0061] 4.9) Taking TN as the origin, the entire image data is divided into one to four quadrants in mathematics through the x-axis and y-axis with TN as the origin. When TK is located at the origin, we have: gaze_x = 0; gaze_y = 0.

[0062] In step 5), the larger the x-axis of gaze, the more the current person's eye is shifted to the left or right. A positive value indicates looking to the right, while a negative value indicates looking to the left. The larger the y-axis of gaze, the more the current person's eye is shifted to the up or down. A positive value indicates looking upwards, while a negative value indicates looking downwards.

[0063] Based on steps 1) to 5), the 2D gaze estimation and 2D data label information processing were completed. In application, the data obtained in steps 1) to 5) are fed into the algorithm's neural network for learning. The resulting model is applied to the RK3566 development board. The parsed results are consistent with the label information fed into the network for training. The first 51 key points are conventional eye key points, and the last point represents the human eye's gaze direction. During visualization, the current pupil center, pointing from the gaze key point, is the current pupil gaze direction.

[0064] The model produced in this invention is mainly used for driver attention detection as an assisted driving function. The detection and recognition method is as follows: eye detection and recognition are only performed when a face is detected. The face detection model detects the face; if no face is detected, no processing is performed. After a face is detected, the face area is filtered, and the largest face bounding box is taken as the target face. Because the camera is pointing at the driver, the driver's face is usually larger than other people's faces in the image. Then, the eye area is masked out and sent to the gaze estimation and recognition algorithm model for recognition. The driver's eye activities are calculated and statistically analyzed over a continuous period of time. To avoid overly sensitive recognition, left-right scanning thresholds and up-down scanning thresholds (the distance the pupil moves away from the center of the pupil when looking straight ahead) are set to avoid statistical calculations when the pupil is slightly shaking. When the number of left-right or up-down scanning reaches a certain threshold, the driver is reminded to concentrate on driving.

[0065] In summary, the technical solution of this invention can fully and effectively achieve the above-mentioned objectives. Furthermore, the structure and functional principles of this invention have been fully verified in the embodiments, achieving the expected effects and objectives. Without departing from the principles and essence of this invention, various changes or modifications can be made to the embodiments. Therefore, this invention includes all substitutions within the scope mentioned in the patent application claims, and any equivalent changes made within the scope of this patent application are within the scope of the patent application.

Claims

1. A method for line of sight estimation based on 2D data, characterized in that, Comprise the following steps: 1) Set grid auxiliary line and concentric circle auxiliary: take the upper left corner of the 2D image with a single eye as the coordinate origin, draw 10x10mm equally spaced auxiliary lines, the vertical and horizontal auxiliary lines divide the image into a grid shape, and then draw concentric circles with equal and equally spaced circle centers based on the pupil center in the 2D image as the circle center; 2) 51 key points regular annotation: in the case of front view of the 2D image of the eye, 20 key points are set on the upper eyelid in equal intervals from left to right; 20 key points are set on the lower eyelid in equal intervals from right to left; and 10 key points can be set on the pupil within the visible range, with the center of the pupil as the 51st key point, denoted as TK (TK_x, TK_y); 3) Annotation of gaze key points: obtain the four points of the uppermost, lowermost, leftmost and rightmost among the 40 key points on the upper and lower eyelids, estimate the position of the pupil center under normal orthovision as the gaze key point TN (TN_x, TN_y) through the four points, combined with the grid auxiliary line and the concentric circle; 4) Calculate the offset of the labeled gaze from the current pupil center: take the gaze key point TN (TN_x, TN_y) as the origin to do X and Y axes, and the offset gaze (gaze_x, gaze_y): gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y; 5) Normalize the 52 key points: according to the offset of the position of the pupil center on the 2D image data obtained in step 4) and the normal orthovision pupil center horizontal and vertical coordinates and the coordinate information of the 51 human eye key points, the stable standard gaze key point information is obtained after normalization; TN_y of gaze key point TN in step 3) is to calculate the Y-axis distance between the highest point and the lowest point, find the Y-axis coordinate of the middle point between the highest point and the lowest point, divide the size of the grid according to the Y-axis coordinate of the middle point, get the grid position of the middle point, and then estimate the offset of 2~3 grids to get TN_y; TN_x of gaze key point TN is to calculate the x-axis distance between the leftmost point and the rightmost point, find the x-axis coordinate of the middle point between the leftmost and rightmost points, divide the size of the grid according to the x-axis coordinate of the middle point, get the grid position of the middle point, and then estimate the offset of 1~2 grids to get TN_x; According to steps 1) to 5), the production of 2D data gaze estimation and the processing of 2D data label information are completed; after the data obtained in steps 1) to 5) is fed to the algorithm neural network for learning, the model output is applied to the rk3566 development board, and the results parsed are consistent with the label information fed to the network for training, the first 51 key points are regular eye key points, and the last point is the gaze direction of the human eye. When visualized, the current pupil gaze direction is pointed to the current pupil center from the gaze key point.

2. The method for line-of-sight estimation based on 2D data according to claim 1, characterized in that: In step 2, when the eye is left or right, the key points on the left or right side are closer to the next point. The first point is the left point of the pupil, and the other points are marked in order.

3. The method for line-of-sight estimation based on 2D data according to claim 1, characterized in that: In step 4, the offset gaze is divided into: 4.1) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is in the first quadrant: gaze_x = TK_x - TN_x, gaze_y = TN_y - TK_y; 4.2) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is in the second quadrant: gaze_x = TK_x - TN_x, gaze_y = -(TK_y - TN_y); 4.3) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is in the third quadrant: gaze_x = -(TN_x - TK_x), gaze_y = -(TK_y - TN_y); 4.4) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is in the fourth quadrant: gaze_x = -(TN_x - TK_x), gaze_y = TN_y - TK_y; 4.5) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is on the positive half of the x-axis: gaze_x = TN_x - TK_x, gaze_y = 0; 4.6) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is on the negative half of the x-axis: gaze_x = -(TN_x - TK_x), gaze_y = 0; 4.7) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is on the positive half of the y-axis: gaze_x = 0, gaze_y = TN_y - TK_y; 4.8) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is on the negative half of the y-axis: gaze_x = 0, gaze_y = -(TK_y - TN_y); 4.9) Take TN as the origin, and divide the entire image data into one to four quadrants on the x-axis and y-axis with TN as the origin. When TK is at the origin: gaze_x = 0, gaze_y = 0.

4. The method for line-of-sight estimation based on 2D data according to claim 1, characterized in that: In step 5), the greater the x-axis of the gaze indicates that the current eye is more offset to the left or to the right, and a positive value indicates a right gaze, and a negative value indicates a left gaze. The greater the y-axis of the gaze indicates that the current eye is more offset upward or downward, and a positive value indicates an upward gaze, and a negative value indicates a downward gaze.

Images