A non-contact sight tracking error correction method using flash point completion

Abstract

A method for non-contact gaze tracking error correction using flash point completion comprises the following steps: pupil and flash point detection is performed on an input grayscale image; the detected flash points are then matched; pupil and flash point coordinates are corrected, and an intersection ratio is calculated to perform gaze tracking, and head and error correction is performed. Flash point completion is used to resolve flash point loss caused by head movement or other reasons. The present invention can correct head movement, reduce the impact of head movement on gaze estimation, model gaze tracking errors, obtain more accurate gaze tracking, and achieve a larger head movement range.

Description

Technical Field

[0001] The present invention relates to the technical field of eye tracking, and mainly relates to a method for correcting non-contact sight tracking errors by using flash point filling. Technical Background

[0002] Gaze tracking technology has been widely used in fields such as psychology, cognitive science, and human-computer interaction. Traditionally, gaze tracking is performed manually or by recording eye movements using equipment such as microscopes controlled by robotic arms. This method has many limitations, including the time and labor required and the inability to automate or perform in real time.

[0003] Non-contact gaze tracking is significantly limited by environmental and movement constraints. Under conditions of significant variability in external lighting, effective image processing and tracking are difficult. Multiple external light sources may cause multiple light sources to appear near the eye, with the extra light sources partially or completely obscuring the light generated by the other sources, leading to lost light. The intensity of external light sources also varies more widely than indoors. Under varying light intensities, the pupil segmentation threshold must be adjusted accordingly. This threshold determines the quality of pupil and light segmentation, which significantly impacts gaze tracking accuracy. In the case of significant head movement, the device's light source may be obscured by reflected light due to ocular fluid, or lost due to excessive deflection, where the eye and camera light sources are not aligned in the same plane. Errors caused by head movement can significantly reduce detection accuracy, thereby reducing gaze tracking accuracy. Directly processing camera images can cause distortion in the projected image, which can offset light points and ultimately reduce gaze tracking accuracy.

[0004] Patent application CN103761519A discloses a non-contact gaze tracking method based on adaptive calibration. This method combines a BFS algorithm with a spot feature extraction method that uses image geometry and grayscale features to accurately match the spot with the corresponding light source. A one-dimensional edge detection operator and a least-squares ellipse fitting method are used to perform a cyclic fitting to remove noise until the ellipse center is fixed, ultimately obtaining an accurate pupil center. This method effectively improves the accuracy of existing spatial mapping models. However, effective technical solutions are still lacking to address the loss of the flash point, gaze tracking errors caused by large head motion ranges, and coordinate distortion caused by camera imaging. Furthermore, this method is applicable to a small range of head motion, which has certain limitations. Summary of the Invention

[0005] In order to overcome the technical defects existing in the above-mentioned prior art, the purpose of the present invention is to provide a method for correcting non-contact gaze tracking errors by using flash point completion. In view of the problems of small head movement range, flash point loss, and great influence of the environment in non-contact, a dynamic threshold is used and the redundant flash points caused by the light source in the environment are filtered out, so as to solve the problem that the flash points and pupils are difficult to detect and match due to environmental changes; the flash point loss caused by head movement or other reasons is solved by flash point completion, the head movement is corrected, the influence of head movement on gaze estimation is reduced, a larger head movement range is achieved, the gaze tracking error is modeled, and more accurate gaze tracking is obtained.

[0006] To achieve the above objectives, the present invention provides the following technical solutions:

[0007] A method for correcting non-contact sight tracking errors by using flash point filling specifically comprises the following steps:

[0008] 1) Input RGB image and process it into grayscale image, perform pupil and flash point detection on the input grayscale image, and obtain elliptical pupil set and flash point set;

[0009] 2) Find the four flash points generated by the point-concentrated light source detected in step 1), detect and match the four flash points, and if the match is successful, correct the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective based on the triangle similarity principle; use the cross ratio algorithm to track the gaze; use Gaussian process regression to model the gaze tracking error, use the gaze tracking coordinates to obtain the correction value, and directly add the correction value to the gaze tracking to obtain the pupil and flash point gaze tracking correction result; regard the displacement of the eye center, that is, the center between the two pupils, as head displacement, and use this displacement as head correction, which is directly added to the gaze tracking to obtain the head displacement correction result;

[0010] 3) If the matching of the four flash points fails, the detection and matching of the three flash points are performed: the flash point is matched with the corresponding light source that generates the flash point, and the light source of the missing flash point is found; assuming that there is an affine transformation between the flash point coordinates and the corresponding light source coordinates, the affine transformation matrix is ​​solved, and the coordinates of the light source corresponding to the missing flash point are mapped into the flash point plane as the lost flash point coordinates using the affine transformation matrix, and the three flash points are filled into four flash points; after filling the flash points, based on the similarity principle of triangles, the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective is corrected; the line of sight tracking is performed using the cross ratio algorithm; assuming that there is a certain functional relationship between the uncorrected line of sight tracking coordinates and their corrected values, this relationship is modeled and the function is fitted using Gaussian process regression; the correction value is obtained using the line of sight tracking coordinates, and the correction value is directly added to the line of sight tracking to obtain the line of sight tracking correction result; the displacement of the eye center is regarded as the head displacement, and this displacement is used as the head correction, which is directly added to the line of sight tracking to obtain the head displacement correction result; if the matching is not successful, the matching of the two flash points is performed;

[0011] 4) Match the two flash points and find the flash point generated by the light source and not lost; match the flash point with the corresponding light source that generated the flash point; use a multi-layer perceptron to learn the geometric relationship between the flash point coordinates and the major and minor axes of the pupil after distortion, so that the coordinates of the other flash point can be obtained from the two flash points that have not been lost and the major and minor axes of the pupil, and the two flash points are filled into four flash points; after filling the flash points, based on the similarity principle of triangles, correct the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective; use the cross ratio algorithm to track the line of sight, use Gaussian process regression to model the line of sight tracking error, use the line of sight tracking coordinates to obtain the correction value, and directly add the correction value to the line of sight tracking to obtain the line of sight tracking correction result; regard the displacement of the eye center as head displacement, and use this displacement as head correction, and directly add it to the line of sight tracking to obtain the head displacement correction result;

[0012] 5) If none of the above matches are successful, discard the frame and use the next frame for gaze tracking.

[0013] The specific method of step 1) is:

[0014] 1.1) All flash points are searched to form a point set, and all flash points in the point set are sorted. First, all flash points are divided into N groups in ascending order of the vertical coordinate by length T, and then the flash points in each of the N groups are sorted in ascending order of the horizontal coordinate; the relative positions of all flash points are obtained, which are used to match the flash points and their corresponding light sources;

[0015] 1.2) Obtain pupil ellipse set: Input grayscale image and use the maximum inter-class variance method (OTSU method) to obtain the threshold T otsu , and at the threshold [T otsu-60,T otsu -5] range, and preset pupil radius D according to the device resolution. p , pupil radius D p The radius is the mean of multiple experiments; the pupil radius is used to find the range that meets the requirements [D p -15, D p +15] ellipse, multiple pupil ellipses are searched to obtain a pupil ellipse set; according to the obtained pupil radius, the pupil is cut with a side length of radius × 10 to obtain a square eye image, and the image is Gaussian blurred to reduce the image noise after cutting;

[0016] 1.3) Obtaining the optimal pupil ellipse: After obtaining the pupil ellipse set from step 1.2), filter the pupil ellipse using the major-minor axis ratio and length information. Then, evaluate the pupil ellipse using the ratio α of the major axis b to the minor axis a, the degree of contour dispersion β, and the difference in grayscale values ​​inside and outside the contour γ. The pupil ellipse with the highest evaluation score is the optimal pupil ellipse.

[0017] Among them, the ratio of the major axis b to the minor axis a of the contour-fitted pupil ellipse is α, and the range of α is [0,1]. The closer the ellipse is to a circle, the higher the score;

[0018]

[0019] Taking the center of the pupil ellipse as the origin, use the rectangular coordinate system to divide the ellipse into 4 regions, calculate the number of contour points in each region, and obtain a 1×4 vector N p , N p Normalize and subtract 0.25, then the mean of the vector is 0. If the contour points can be evenly distributed in each quadrant, that is, the number of contour points in each quadrant is the same, then the dispersion of the contour points of this contour is called perfect, and the vector N of this contour is p Each element in should be 0, and the sum of the absolute values ​​of each value in N is S. If the contour points are evenly distributed in the four regions, the value of S is 0. If the contour points are only distributed in one of the four regions, the value of S is 1.5. The score of the dispersion of the contour points is β, and the range of β is [0,1]:

[0020]

[0021] Uniformly sample num points from the contour point set of each ellipse, and the range of num is [5, N max ],N max is the number of points in the contour point set, the vector from the ellipse center to the contour point is I, and each contour point sampled is I×0.5 as the inner point of the contour (x i ,y i ) and the point outside the contour at I×1.5 (x o ,yo ), according to the coordinates of the inner and outer points of the contour and the grayscale image, the grayscale values ​​of the grayscale image at the inner and outer points of the contour can be obtained, and the grayscale value difference can be obtained by subtracting them; the absolute value F of the mean of the grayscale value difference is taken, and the grayscale value is normalized by F / 255 to obtain the difference score γ of the inner and outer grayscale values. The range of γ is between [0,1]. The difference γ of the grayscale values ​​inside and outside the contour is;

[0022]

[0023] Then the sum of α, β, and γ is taken as the final score, and the image with the highest score is the best pupil ellipse image;

[0024] 1.4) After finding the best ellipse, search for the flash point around the ellipse: for the best pupil ellipse image obtained in step 1.3), otsu ,T otsu +40] range, preset the flash point radius D according to the resolution of the device and the light source equipment g , search around the pupil ellipse in the range [D g -5,D g +5], search for multiple flash point ellipses, and then get a set of flash point ellipses; calculate the α, β, and γ values ​​of the flash point ellipse, and add the three to get the final score of the flash point ellipse; set a custom threshold for the score, and the score above the threshold is considered to be a flash point ellipse, and the score below the threshold is considered not to be a flash point ellipse.

[0025] After the point set is obtained in step 1), it is assumed that the number of points in the point set is N. gnum , n points need to be matched each time, and the number of points to be matched S is expressed as:

[0026]

[0027] The flash points obtained in step 1.1) are sorted and matched one by one based on the number of points taken S.

[0028] The specific steps of step 2) are as follows:

[0029] 2.1) Matching 4 flash points: Each time, 4 points are selected from the flash point set, and the points to be matched are distributed in a rectangular pattern. The point closest to the pupil is used as the reference point. The reference point and two adjacent vertices form two 1×2 vectors. The diagonal vector v' is added together, and the vector v is obtained from the reference point and its diagonal point. The sum of the Euclidean distance and the cosine similarity of the vectors is used as the similarity value. The higher the similarity value, the higher the degree of similarity. The highest similarity is the best flash point. For two vectors, the smaller the angle between the two vectors, the higher the cosine similarity. The greater the difference in the length of the two vectors, the larger the Euclidean distance. The Euclidean distance is normalized so that the growth trend of the Euclidean distance is the same as that of the cosine similarity. The normalized Euclidean distance is negated and then added to the cosine similarity to obtain the final evaluation value:

[0030]

[0031] A threshold is set for the similarity value. When the similarity value is lower than the threshold, the currently selected 4 flash points are considered unmatched. When the similarity value is greater than the threshold, the match is considered successful. After S matches, the flash point combination with the highest evaluation value is selected from the successful matches as the final selected flash point combination. If no match is successful each time, the flash point set will be judged as not having 4 flash points.

[0032] 2.2) Correction of flash points and pupils

[0033] The pupil center P and the flash point g are separated from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of

[0034] 2.3) Pupil and Flash Point Gaze Tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right.

[0035] In step 3), if the flash point set in step 2) is determined to not contain 4 flash points, 3 flash points are matched, and the matched 3 flash points form a right triangle or an atypical right triangle. The specific steps of flash point matching are as follows:

[0036] 3.1) Each time, three flash points are selected from the flash point set. The number of selections required can be obtained by formula (1). The point closest to the pupil is used as the reference point. Assuming that the three points selected each time are flash points generated by the light source, the corresponding light source of each flash point can be found by the relative relationship between the coordinates. The reference point and the other two flash points (g1, g2, g3) respectively form a 1×2 vector, and the two vectors form a 2×2 matrix corresponding to the light source (l1, l2, l3). The matrix is ​​normalized as a whole to obtain the feature matrix of the flash point to be matched; using the same method, the feature matrix of the light source corresponding to these three flash points is obtained; the cosine similarity of these two feature matrices is used as the similarity value. The higher the similarity value, the better the matching point; a threshold is set for the similarity value. When the similarity value is lower than the threshold, the three flash points currently selected are considered unmatched. When the similarity value is greater than the threshold, the match is considered successful, and the combination with the highest evaluation value is selected as the final selected combination among the successfully matched combinations; the similarity value is calculated as follows:

[0037]

[0038] If the points selected each time are not matched successfully, the set of flash points will be judged as not having 3 flash points;

[0039] 3.2) For the three matched points (g1, g2, g3) and the corresponding light sources (l1, l2, l3), find the affine transformation matrix Matric.

[0040]

[0041] After obtaining the matrix, the coordinates of the missing light source (l4) are mapped into the eyeball plane as the missing flash point coordinates g4, and the missing point coordinates are obtained to fill in the flash point;

[0042]

[0043] 3.3) Correction of the flash point and pupil: Change the pupil center P and the flash point g from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of

[0044] 3.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right.

[0045] 3.5) Head correction: Take the average human pupil radius of 1.5mm to obtain the conversion parameter S from image units to real units scalar , taking the calibrated left and right eye center coordinates P as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H:

[0046] H=S scalar (PP′)

[0047] After the head error correction, the final error correction is required. The error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output. Gaussian process regression is used to model the error.

[0048] The matching method for the two flash points in step 4) is as follows:

[0049] 4.1) Each time, two flashing points are selected from the point set obtained in step 1). The number of selections required can be calculated using formula 4. The point closest to the pupil is used as the reference point. If the reference point is below the center of the pupil, it is considered that the reference point is generated by the lower left or lower right light source of the screen. If it is above the center of the pupil, the flashing point is considered to be generated by the upper left or upper right light source. The vector v is obtained from the reference point and another point. The cosine values ​​of v and the horizontal and vertical directions are calculated.

[0050]

[0051] The largest one is taken as the evaluation value for this time, and the one with the highest evaluation value is the best flash point. For the case of two flash points, if the angle corresponding to the cosine value is greater than 30 degrees, it is considered an unsuccessful match. If it is less than 30 degrees, it is considered a successful match. The combination with the highest evaluation value is selected as the final combination among the successful matches. If all combinations fail to match successfully, this flash point set will be judged as a case without two flash points. In this case, the frame will be discarded and the next frame will be used for eye tracking.

[0052] 4.2) For known flash points, the corresponding light source position must be determined. For two points, a multilayer perceptron is used to predict all four points (g1, g2, g3, g4). The multilayer perceptron input corresponds to the two points and the pupil major and minor axes (a, b). The following formula is used to obtain four points to complete the flash point filling.

[0053] (g1,g2,g3,g4)=MLPs(g1,g2,a,b)

[0054] 4.3) Correction of the flash point and pupil: Change the pupil center P and the flash point g from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of Rp coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of

[0055] 4.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, the vertical downward direction as the y-axis, and the inverse ratio of the horizontal coordinates of the left and right eyes as the weight, assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, then the weight of the left eye is (L / (L+R)), and the weight of the right eye is (R / (L+R)). The left and right eye gaze tracking are multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking; the left eye gaze tracking is more accurate when looking to the left, and the right eye gaze tracking is more accurate when looking to the right; assuming the screen resolution is W×H, the gaze tracking of both eyes is L(Xl,Y l ), R(Xr,Yr), the average eye tracking of both eyes is A(X a ,Y a ), and get the fused gaze tracking F:

[0056]

[0057] 4.5) Head correction: Take the average of the actual human pupil radius to obtain the conversion parameter S from image units to real units scalar , taking the left and right eye center coordinates P in the calibration phase as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H:

[0058] H=S scalar (PP′)

[0059] After head correction, the gaze tracking error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output; the kernel function used in the modeling is as follows:

[0060] cov(x p ,x q )=k1(σ 2 +x p x q )+k2δ

[0061] Among them, k1, k2 are parameters that need to be manually adjusted, δ is white noise,

[0062]

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The present invention matches the flash points emitted by the light source based on the geometric characteristics of the light source in the case of multiple light sources or the light source being lost, and finds the flash points lost due to the environment or user actions and the light sources corresponding to the lost flash points; whereas in the prior art, the accuracy of non-contact eye movement equipment is greatly affected by the environment and has strict restrictions on user actions, making it more difficult to use in practice; the present invention uses light source set feature matching, which can ensure a certain accuracy in different environments with multiple light sources and has fewer restrictions on user actions.

[0065] (2) The present invention uses affine transformation to fill in the fourth point when one flash point is lost and three points remain, so as to perform subsequent cross-ratio calculations. In the prior art, at least four points are required for cross-ratio calculations of sight tracking, and three points cannot be used for cross-ratio calculations of sight tracking. The present invention uses affine transformation to fill in the fourth point, so that sight tracking cross-ratio calculations can be performed even when only three points are matched.

[0066] (3) The present invention uses a multi-layer perceptron to fill in the remaining two flash points when two flash points are lost, thereby performing a subsequent cross-ratio operation. The multi-layer perceptron is used to learn the relationship between the flash points and the major and minor axes of the pupil ellipse. When only two flash points and the major and minor axes of the pupil are known, the other two flash points can be solved based on this information, so that the cross-ratio operation of line of sight tracking can also be performed when only two flash points are present.

[0067] (4) The present invention uses the similarity principle of triangles to correct the errors of the flash point and the pupil center caused by the camera projection principle. In actual situations, due to the camera projection principle, the flash point and the pupil center will be distorted when the three-dimensional eyeball is projected into a plane, thereby causing errors. The present invention projects the flash point and the pupil center onto the same plane and uses the similarity principle of triangles to correct the errors caused by the camera.

[0068] (5) The present invention uses dynamic weights to fuse left and right eye gaze tracking. In practice, the error between the left and right eyes can vary greatly when looking at different areas. The existing method of calculating the average of the left and right eyes reduces the influence of the larger error, but also reduces the influence of the smaller error. The present invention uses dynamic weights to increase the influence of the smaller error and decrease the influence of the larger error.

[0069] (6) The present invention uses Gaussian process regression to model the error based on the distribution of the error, so that the error correction parameters can be dynamically updated according to the coordinates of the predicted point; the existing technology requires different error correction methods to be designed for different environments, and the correction parameters are relatively fixed. The present invention uses Gaussian process regression to model the error, so that correction can be performed in different environments, and the correction parameters are dynamically changing, making the correction more accurate.

[0070] (7) In practice, non-contact eye tracking devices have significant limitations on the head, and errors can increase significantly when the head moves. The present invention corrects the errors caused by head movement, thereby reducing the errors caused by head movement while expanding the range of head movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 This is a diagram showing the relationship between the LED light sources around the screen and the flash points produced by the light sources on the eyeballs.

[0073] Figure 2 This is a schematic diagram of the grouping of flash points on the eyeball plane.

[0074] Figure 3 Schematic diagram of pupil center error caused by the camera.

[0075] Figure 4 This is a diagram illustrating the flash point error caused by the camera.

[0076] Figure 5 This is the structure diagram of the multi-layer perceptron used to fill in the flash points in the case of 2 points. The activation function is ReLU(), and LayerNorm is connected to each layer except the last layer for normalization.

[0077] Figure 6 This is a schematic diagram of correcting the pupil center using the triangle similarity principle, where L is the LED light source and C is the camera.

[0078] Figure 7 This is a schematic diagram of using the triangle similarity principle to correct flash points, where L is the LED light source and C is the camera.

[0079] Figure 8 It is a flow chart of the present invention. Specific implementation plan

[0080] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0081] A method for correcting non-contact sight tracking errors by using flash point filling specifically comprises the following steps:

[0082] 1) Input the RGB image obtained by the camera and process it into a grayscale image. Perform pupil and flash point detection on the input grayscale image to obtain an elliptical pupil set and a flash point set;

[0083] 1.1) All flash points are searched to form a point set, and all flash points in the point set are sorted. First, all flash points are divided into N groups in ascending order of ordinates by length T, and then the flash points in each of the N groups are sorted in ascending order of abscissas; the relative positions of all flash points are obtained to match the flash points and their corresponding light sources, such as Figure 2 As shown;

[0084] 1.2) Obtain pupil ellipse set: Input grayscale image and use the maximum inter-class variance method (OTSU method) to obtain the threshold T otsu , and at the threshold [T otsu -60,T otsu -5] range, and preset pupil radius D according to the device resolution. p , pupil radius D p The pupil radius is 40px, and an ellipse that meets the range [25, 55] is searched with a preset pupil radius of 40px. Multiple pupil ellipses are searched to obtain a pupil ellipse set. According to the obtained pupil radius, the pupil is cut with a side length of radius × 10 to obtain a square eye image. The image is Gaussian blurred to reduce the image noise after cutting and facilitate subsequent processing.

[0085] 1.3) Obtaining the optimal pupil ellipse: After obtaining the pupil ellipse set from step 1.2), filter the pupil ellipse using the major-minor axis ratio and length information. Then, evaluate the pupil ellipse using the ratio α of the major axis b to the minor axis a, the degree of contour dispersion β, and the difference in grayscale values ​​inside and outside the contour γ. The pupil ellipse with the highest evaluation score is the optimal pupil ellipse.

[0086] Among them, the ratio of the major axis b to the minor axis a of the contour-fitted pupil ellipse is α, and the range of α is [0,1]. The closer the ellipse is to a circle, the higher the score;

[0087]

[0088] Taking the center of the pupil ellipse as the origin, use the rectangular coordinate system to divide the ellipse into 4 regions, calculate the number of contour points in each region, and obtain a 1×4 vector N p , N pNormalize and subtract 0.25, then the mean of the vector is 0. If the contour points can be evenly distributed in each quadrant, that is, the number of contour points in each quadrant is the same, then the dispersion of the contour points of this contour is called perfect, and the vector N of this contour is p Each element in should be 0, and the sum of the absolute values ​​of each value in N is S. If the contour points are evenly distributed in the four regions, the value of S is 0. If the contour points are only distributed in one of the four regions, the value of S is 1.5. The score of the dispersion of the contour points is β, and the range of β is [0,1]:

[0089]

[0090] Sample num points uniformly from the contour point set of each ellipse. The vector from the ellipse center to the contour point is I. For each contour point sampled, I×0.5 is taken as the inner point of the contour (x i ,y i ) and the point outside the contour at I×1.5 (x o ,y o ), according to the coordinates of the inner and outer points of the contour and the grayscale image, the grayscale values ​​of the grayscale image at the inner and outer points of the contour can be obtained, and the grayscale value difference can be obtained by subtracting them; the absolute value F of the mean of the grayscale value difference is taken, and the grayscale value is normalized by F / 255 to obtain the difference score γ of the inner and outer grayscale values. The range of γ is between [0,1]. The difference γ of the grayscale values ​​inside and outside the contour is;

[0091]

[0092] Then the sum of α, β, and γ is taken as the final score, and the image with the highest score is the best pupil ellipse image;

[0093] 1.4) After finding the best ellipse, search for the flash point around the ellipse: for the best pupil ellipse image obtained in step 1.3), otsu ,T otsu +40] with a step size of 5. Based on the resolution and light source of the device, the flash point radius is preset to 10px. Ellipses in the range [5,15] are searched around the pupil ellipse. Multiple flash point ellipses are obtained from the search, and then a flash point ellipse set is obtained. The α, β, and γ values ​​of the flash point ellipse are calculated and the three are added together to obtain the final score of the flash point ellipse. A custom threshold is set for the score. Ellipses with scores above the threshold are considered to be flash point ellipses, while those with scores below the threshold are considered not to be flash point ellipses.

[0094] Ten points are uniformly sampled from the contour point set of each ellipse. The vector from the ellipse center to the contour point is I. For each contour point sampled, the points at I×0.5 and I×1.5 are taken as the inner points of the contour (x i ,yi ) and the point outside the contour (x o ,y o ), the grayscale values ​​of the two corresponding points of the image are calculated according to the coordinates of the inner point and the outer point of the contour and the grayscale map, and the grayscale values ​​of the grayscale map at the inner point and the outer point of the contour are obtained. The grayscale value difference is obtained by subtracting the grayscale value difference; the absolute value F of the mean of the grayscale value difference is taken, and the grayscale value is normalized by F / 255 to obtain the difference score γ of the inner and outer grayscale values. The range of γ is between [0,1]. The grayscale value difference can be obtained by formula (3);

[0095] The sum of α, β, and γ is used as the final score, and the one with the highest score is the best pupil ellipse. After finding the best ellipse, search for the flash point around the ellipse. Using formulas (1) to (3), calculate α, β, and γ for the flash point ellipse, and add the three to get the final score of the flash point ellipse. Set a threshold for the score, and the threshold changes according to the specific conditions of the environment and line of sight. Scores above the threshold are considered to be flash point ellipses, and scores below the threshold are considered not to be flash point ellipses. When determining the experimental environment and camera resolution, the threshold should be at least 90% of the maximum score under the system.

[0096] After the point set is obtained in step 1), it is assumed that the number of points in the point set is N. gnum , n points need to be matched each time, and the number of points to be matched S is expressed as:

[0097]

[0098] The flash points obtained in step 1.1) are sorted and matched one by one based on the number of points taken S.

[0099] 2) Find the four flash points generated by the point-concentrated light source detected in step 1), detect and match the four flash points, and if the match is successful, correct the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective based on the triangle similarity principle; use the cross ratio algorithm to track the gaze; use Gaussian process regression to model the gaze tracking error, use the gaze tracking coordinates to obtain the correction value, and directly add the correction value to the gaze tracking to obtain the pupil and flash point gaze tracking correction result; regard the displacement of the eye center, that is, the center between the two pupils, as head displacement, and use this displacement as head correction, which is directly added to the gaze tracking to obtain the head displacement correction result;

[0100] 2.1) Matching 4 flash points: Each time, 4 points are selected from the flash point set, and the points to be matched are distributed in a rectangular pattern. The point closest to the pupil is used as the reference point. The reference point and two adjacent vertices form two 1×2 vectors. The diagonal vector v' is added together, and the vector v is obtained from the reference point and its diagonal point. The sum of the Euclidean distance and the cosine similarity of the vectors is used as the similarity value. The higher the similarity value, the higher the degree of similarity. The highest similarity is the best flash point. For two vectors, the smaller the angle between the two vectors, the higher the cosine similarity. The greater the difference in the length of the two vectors, the larger the Euclidean distance. The Euclidean distance is normalized so that the growth trend of the Euclidean distance is the same as that of the cosine similarity. The normalized Euclidean distance is negated and then added to the cosine similarity to obtain the final evaluation value:

[0101]

[0102] A threshold is set for the similarity value. When the similarity value is lower than the threshold, the currently selected 4 flash points are considered unmatched. When the similarity value is greater than the threshold, the match is considered successful. After S matches, the flash point combination with the highest evaluation value is selected from the successful matches as the final selected flash point combination. If no match is successful each time, the flash point set will be judged as not having 4 flash points.

[0103] 2.2) Correction of flash points and pupils

[0104] The pupil center P and the flash point g are separated from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of

[0105] Make the plane to be projected tangent to the eyeball and parallel to the screen and camera planes; the distance D between the eyeball and the screen is 60 cm, and the pupil radius R is the average value of 2.4 cm for humans; the pixel distance between the eyeball center and the pupil mapping point Vp can be obtained according to the coordinates of the image, and this distance can be used to replace the distance P between the pupil center P and the eyeball center e From the image, we can get the pupil radius in pixels. Taking the actual pupil diameter as the average value of 1.5cm for humans, we can get the conversion parameter S from image units to real units. scalar , through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, the distance H from the eyeball center to the pupil center can be obtained by the Pythagorean theorem. p Collinear, assuming PR can be obtained p The distance is given by △PR p V p ∽△PLC, can be based on PR p The ratio of the distance from the pupil center to L is used to obtain R p V p The distance is used to obtain the corrected R p ; In actual situations, use D instead of the distance from the center of the eye to L, and use d instead of PR p When correcting the flash point, the distance D between the eyeball and the screen is 60cm, the eyeball diameter R is the average of 2.4cm for humans, and the actual pupil diameter is the average of 1.5cm for humans. When correcting the pupil, S is obtained. scalar , through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, we can get the distance H from the eyeball center to the pupil center by the Pythagorean theorem; replace the distance from the eyeball center to L with D, and replace gR with d g , because g, R g , L is collinear so △gR g V g ∽△gLC, R can be obtained according to the ratio of d and D g V g The distance is used to correct R g ;

[0106] 2.3) Pupil and Flash Point Gaze Tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right.

[0107] 3) If the matching of the four flash points fails, the detection and matching of the three flash points are performed: the flash point is matched with the corresponding light source that generates the flash point, and the light source of the missing flash point is found; assuming that there is an affine transformation between the flash point coordinates and the corresponding light source coordinates, the affine transformation matrix is ​​solved, and the coordinates of the light source corresponding to the missing flash point are mapped into the flash point plane as the lost flash point coordinates using the affine transformation matrix, and the three flash points are filled into four flash points; after filling the flash points, based on the similarity principle of triangles, the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective is corrected; the line of sight tracking is performed using the cross ratio algorithm; assuming that there is a certain functional relationship between the uncorrected line of sight tracking coordinates and their corrected values, this relationship is modeled and the function is fitted using Gaussian process regression; the correction value is obtained using the line of sight tracking coordinates, and the correction value is directly added to the line of sight tracking to obtain the line of sight tracking correction result; the displacement of the eye center is regarded as the head displacement, and this displacement is used as the head correction, which is directly added to the line of sight tracking to obtain the head displacement correction result; if the matching is not successful, the matching of the two flash points is performed;

[0108] 3.1) Each time, three flash points are selected from the flash point set. The number of selections required can be obtained by formula (1). The point closest to the pupil is used as the reference point. Assuming that the three points selected each time are flash points generated by the light source, the corresponding light source of each flash point can be found by the relative relationship between the coordinates. The reference point and the other two flash points (g1, g2, g3) respectively form a 1×2 vector, and the two vectors form a 2×2 matrix corresponding to the light source (l1, l2, l3). The matrix is ​​normalized as a whole to obtain the feature matrix of the flash point to be matched; using the same method, the feature matrix of the light source corresponding to these three flash points is obtained; the cosine similarity of these two feature matrices is used as the similarity value. The higher the similarity value, the better the matching point; a threshold is set for the similarity value. When the similarity value is lower than the threshold, the three flash points currently selected are considered unmatched. When the similarity value is greater than the threshold, the match is considered successful, and the combination with the highest evaluation value is selected as the final selected combination among the successfully matched combinations; the similarity value is calculated as follows:

[0109]

[0110] If the points selected each time are not matched successfully, the set of flash points will be judged as not having 3 flash points;

[0111] 3.2) For the three matched points (g1, g2, g3) and the corresponding light sources (l1, l2, l3), find the affine transformation matrix Matric.

[0112]

[0113] After obtaining the matrix, the coordinates of the missing light source (l4) are mapped into the eyeball plane as the missing flash point coordinates g4, and the missing point coordinates are obtained to fill in the flash point;

[0114]

[0115] 3.3) When correcting the flash point, click gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g The coordinates of the eyeball are tangent to the projection plane and parallel to the screen and camera planes. The distance D between the eyeball and the screen is 60 cm, and the pupil radius R is the average value of 2.4 cm for humans. From the image, the pixel distance between the eyeball center and the pupil mapping point Vp can be obtained according to the image coordinates, and this distance can be used to replace the distance P between the pupil center P and the eyeball center. e From the image, we can get the pupil radius in pixels. Taking the actual pupil diameter as the average value of 1.5cm for humans, we can get the conversion parameter S from image units to real units. scalar , through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, the distance H from the eyeball center to the pupil center can be obtained by the Pythagorean theorem. p Collinear, assuming PR can be obtained p The distance is given by △PR p V p ∽△PLC, can be based on PR p The ratio of the distance from the pupil center to L is used to obtain R p V p The distance is used to obtain the corrected R p ; In actual situations, use D instead of the distance from the center of the eye to L, and use d instead of PR pWhen correcting the flash point, the distance D between the eyeball and the screen is 60cm, the eyeball diameter R is the average of 2.4cm for humans, and the actual pupil diameter is the average of 1.5cm for humans. When correcting the pupil, S is obtained. scalar , through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, we can get the distance H from the eyeball center to the pupil center by the Pythagorean theorem; replace the distance from the eyeball center to L with D, and replace gR with d g , because g, R g , L is collinear so △gR g V g ∽△gLC, R can be obtained according to the ratio of d and D g V g The distance is used to correct R g ;

[0116] 3.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right.

[0117] 3.5) Head correction: Take the average human pupil radius of 1.5mm to obtain the conversion parameter S from image units to real units scalar , taking the calibrated left and right eye center coordinates P as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H:

[0118] H=S scalar (PP′) (9)

[0119] After the head error correction, the final error correction is required. The error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output. Gaussian process regression is used to model the error.

[0120] 4) Match the two flash points and find the flash point generated by the light source and not lost; match the flash point with the corresponding light source that generated the flash point; use a multi-layer perceptron to learn the geometric relationship between the flash point coordinates and the major and minor axes of the pupil after distortion, so that the coordinates of the other flash point can be obtained from the two flash points that have not been lost and the major and minor axes of the pupil, and the two flash points are filled into four flash points; after filling the flash points, based on the similarity principle of triangles, correct the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective; use the cross ratio algorithm to track the line of sight, use Gaussian process regression to model the line of sight tracking error, use the line of sight tracking coordinates to obtain the correction value, and directly add the correction value to the line of sight tracking to obtain the line of sight tracking correction result; regard the displacement of the eye center as head displacement, and use this displacement as head correction, and directly add it to the line of sight tracking to obtain the head displacement correction result;

[0121] 4.1) Each time, two flashing points are selected from the point set obtained in step 1). The number of selections required can be calculated using formula 4. The point closest to the pupil is used as the reference point. If the reference point is below the center of the pupil, it is considered that the reference point is generated by the lower left or lower right light source of the screen. If it is above the center of the pupil, the flashing point is considered to be generated by the upper left or upper right light source. The vector v is obtained from the reference point and another point. The cosine values ​​of v and the horizontal and vertical directions are calculated.

[0122]

[0123] The largest one is taken as the evaluation value for this time, and the one with the highest evaluation value is the best flash point. For the case of two flash points, if the angle corresponding to the cosine value is greater than 30 degrees, it is considered an unsuccessful match. If it is less than 30 degrees, it is considered a successful match. The combination with the highest evaluation value is selected as the final combination among the successful matches. If all combinations fail to match successfully, this flash point set will be judged as a case without two flash points. In this case, the frame will be discarded and the next frame will be used for eye tracking.

[0124] 4.2) For known flash points, the corresponding light source position must be determined. For two points, a multilayer perceptron is used to predict all four points (g1, g2, g3, g4). The multilayer perceptron input corresponds to the two points and the pupil major and minor axes (a, b). The following formula is used to obtain four points to complete the flash point filling.

[0125] (g1,g2,g3,g4)=MLPs(g1,g2,a,b) (11)

[0126] The structure diagram of the multi-layer perceptron is as follows Figure 5As shown, LayerNorm is used after each linear layer except the last one for normalization. ReLU is used as the activation function, with a learning rate of 0.05. Training is performed for 150 epochs, with a cosine annealing scheduler dynamically adjusting the learning rate. A 1×6 vector consisting of the two matching points and the pupil coordinates is used as the input to the multilayer perceptron. To facilitate training, a skip-connection structure is added, where the input is simply concatenated with a 1×58 feature map to form a 1×64 feature map.

[0127] 4.3) When correcting the flash point, click gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g The coordinates of the eyeball are tangent to the projection plane and parallel to the screen and camera planes. The distance D between the eyeball and the screen is 60 cm, and the pupil radius R is the average value of 2.4 cm for humans. From the image, the pixel distance between the eyeball center and the pupil mapping point Vp can be obtained according to the image coordinates, and this distance can be used to replace the distance P between the pupil center P and the eyeball center. e From the image, we can get the pupil radius in pixels. Taking the actual pupil diameter as the average value of 1.5cm for humans, we can get the conversion parameter S from image units to real units. scalar , through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, the distance H from the eyeball center to the pupil center can be obtained by the Pythagorean theorem. p Collinear, assuming PR can be obtained p The distance is given by △PR p V p ∽△PLC, can be based on PR p The ratio of the distance from the pupil center to L is used to obtain R p V p The distance is used to obtain the corrected R p ; In actual situations, use D instead of the distance from the center of the eye to L, and use d instead of PR p When correcting the flash point, the distance D between the eyeball and the screen is 60cm, the eyeball diameter R is the average of 2.4cm for humans, and the actual pupil diameter is the average of 1.5cm for humans. When correcting the pupil, S is obtained. scalar, through this parameter, we can get the actual distance h from the pupil center to the eyeball center. According to this distance and the eyeball radius, we can get the distance H from the eyeball center to the pupil center by the Pythagorean theorem; replace the distance from the eyeball center to L with D, and replace gR with d g , because g, R g , L is collinear so △gR g V g ∽△gLC, R can be obtained according to the ratio of d and D g V g The distance is used to correct R g ;

[0128] 4.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, the vertical downward direction as the y-axis, and the inverse ratio of the horizontal coordinates of the left and right eyes as the weight, assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, then the weight of the left eye is (L / (L+R)), and the weight of the right eye is (R / (L+R)). The left and right eye gaze tracking are multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking; the left eye gaze tracking is more accurate when looking to the left, and the right eye gaze tracking is more accurate when looking to the right; assuming the screen resolution is W×H, the gaze tracking of both eyes is L(Xl,Y l ), R(Xr,Yr), the average eye tracking of both eyes is A(X a ,Y a ), and get the fused gaze tracking F:

[0129]

[0130] 4.5) Head correction: Take the average of the actual human pupil radius to obtain the conversion parameter S from image units to real units scalar , taking the left and right eye center coordinates P in the calibration phase as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H:

[0131] H=S scalar (PP′)

[0132] After head correction, the gaze tracking error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output; the kernel function used in the modeling is as follows:

[0133] cov(x p ,x q )=k1(σ 2 +x p x q )+k2δ (12)

[0134] k1, k2 are parameters that need to be manually adjusted, and δ is white noise.

[0135]

[0136] 5) If none of the above matches are successful, discard the frame and use the next frame for gaze tracking.

[0137] In the steps 2), 3) and 4), the flash point and the pupil are corrected. In reality, the pupil center and the flash point are not in the same plane, and even the flash points are not in the same plane. The picture obtained by the camera is a two-dimensional picture. Points that are not in the same plane are projected to the same plane due to the imaging of the camera. What the camera obtains is the projection of the reflected light on the plane, which adds errors to a certain extent. Figure 3 、 Figure 4 As shown; the present invention achieves correction by projecting the incident light on the same plane, as shown Figure 6 Correction of the pupil is shown.

[0138] The left and right eye gaze tracking in step 2), step 3) and step 4) are fused based on the tracking results. In actual situations, the left eye's gaze tracking is more accurate when looking to the left, and the right eye's gaze tracking is more accurate when looking to the right. Taking the upper left corner of the screen as the origin coordinate (0,0), the horizontal direction to the right as the x-axis, and the vertical downward direction as the y-axis, the left eye's gaze tracking error is smaller when looking to the left, and the right eye's gaze tracking error is smaller when looking to the right. Therefore, the result of our fusion should be that the left eye's gaze tracking weight is higher than the right eye's when looking to the left, and the right eye's gaze tracking weight is higher than the left when looking to the right; the present invention uses the inverse ratio of the horizontal coordinates of the left and right eyes as the weight.

[0139] As can be seen from the accompanying drawings and embodiments, compared with the prior art, the present invention matches the flash points emitted by the light source based on the geometric characteristics of the light source, in the case of multiple light sources or the loss of the light source, and finds the flash points lost due to the environment or user actions and the light sources corresponding to the lost flash points; while in the prior art, the accuracy of non-contact eye movement equipment is greatly affected by the environment, and there are strict restrictions on user actions, which is more difficult in practical applications; the present invention uses light source set feature matching, which can ensure a certain accuracy in different environments with multiple light sources, and has fewer restrictions on user actions.

[0140] Using affine transformation, when one flash point is lost and there are 3 points left, the fourth point is filled in to perform subsequent cross-ratio calculations; in the prior art, at least 4 points are required for cross-ratio calculations of line of sight tracking, and 3 points cannot perform cross-ratio calculations of line of sight tracking. The present invention uses affine transformation to fill in the fourth point, so that line of sight tracking cross-ratio calculations can be performed even when only 3 points are matched.

[0141] Using a multi-layer perceptron, if only two flashing points remain after two are lost, the remaining two points are supplemented for subsequent cross-ratio calculations. Using a multi-layer perceptron to learn the relationship between the flashing points and the major and minor axes of the pupil ellipse, the remaining two flashing points can be determined using this information, allowing for cross-ratio calculations for gaze tracking even with only two flashing points.

[0142] The similarity principle of triangles is used to correct the errors in the flash point and pupil center caused by the camera projection principle. In actual situations, due to the camera projection principle, the flash point and pupil center will be distorted when the three-dimensional eyeball is projected into a plane, thereby causing errors. The present invention projects the flash point and pupil center onto the same plane and uses the similarity principle of triangles to correct the errors caused by the camera.

[0143] Dynamic weighting is used to fuse left and right eye gaze tracking. In practice, the error between the left and right eyes can vary significantly when looking at different areas. The existing method of calculating the average of the left and right eyes reduces the influence of the larger error, but also reduces the influence of the smaller error. The present invention uses dynamic weighting to increase the influence of the smaller error and minimize the influence of the larger error.

[0144] According to the distribution of errors, Gaussian process regression is used to model the errors, so that the error correction parameters can be dynamically updated according to the coordinates of the predicted points; the existing technology requires the design of different error correction methods for different environments, and the correction parameters are relatively fixed. The present invention uses Gaussian process regression to model the errors, so that corrections can be made in different environments, and the correction parameters are dynamically changing, making the correction more accurate.

Claims

1. A method for correcting non-contact sight tracking errors using flash point filling, characterized in that: The specific steps include: 1) Input RGB image and process it into grayscale image, perform pupil and flash point detection on the input grayscale image, and obtain elliptical pupil set and flash point set; 2) Find the four flash points generated by the concentrated light source detected in step 1), detect and match the four flash points, and if a match is successful, correct the distortion of the pupil coordinates and the flash point coordinates caused by the camera perspective based on the triangle similarity principle; and use the intersection ratio algorithm to track the line of sight; Gaussian process regression is used to model gaze tracking errors, and correction values ​​are obtained using gaze tracking coordinates. The correction values ​​are directly added to the gaze tracking to obtain the pupil and flash point gaze tracking correction results. The displacement of the eye center, i.e., the center between the two pupils, is considered as the head displacement, and this displacement is used as the head correction. It is directly added to the gaze tracking to obtain the head displacement correction result. 3) If the matching of the four flash points fails, the detection and matching of the three flash points is performed: the flash point is matched with the corresponding light source that generates the flash point to find the light source of the missing flash point; assuming that there is an affine transformation between the flash point coordinates and the corresponding light source coordinates, the affine transformation matrix is ​​solved, and the coordinates of the light source corresponding to the missing flash point are mapped into the flash point plane as the missing flash point coordinates using the affine transformation matrix, thus completing the three flash points into four flash points; After completing the flash points, the distortion of pupil coordinates and flash point coordinates caused by the camera perspective is corrected based on the triangle similarity principle. Gaze tracking is performed using the cross-ratio algorithm. Assuming a functional relationship between the uncorrected gaze tracking coordinates and their corrected values, this relationship is modeled and fitted using Gaussian process regression. The corrected values ​​are calculated using the gaze tracking coordinates and directly added to the gaze tracking data to obtain the gaze tracking correction result. The displacement of the eye center is regarded as the head displacement, and this displacement is used as the head correction, which is directly added to the gaze tracking to obtain the head displacement correction result; If the match is not successful, then match the two flash points; 4) Match the two flash points and find the flash point that is generated by the light source and is not lost; Match the flash point with the corresponding light source that produces it; use a multi-layer perceptron to learn the geometric relationship between the flash point coordinates and the major and minor axes of the pupil after distortion, so that the coordinates of the other flash points can be obtained from the two flash points that have not been lost and the major and minor axes of the pupil, thus completing the two flash points into four flash points; After completing the flash points, the distortion of pupil coordinates and flash point coordinates caused by the camera perspective is corrected based on the triangle similarity principle. Gaze tracking is performed using the cross-ratio algorithm, and the gaze tracking error is modeled using Gaussian process regression. The correction value is obtained using the gaze tracking coordinates, and the correction value is directly added to the gaze tracking to obtain the gaze tracking correction result. The displacement of the eye center is regarded as the head displacement, and this displacement is used as the head correction, which is directly added to the gaze tracking to obtain the head displacement correction result; 5) If none of the above matches are successful, discard the frame and use the next frame for gaze tracking.

2. The method for correcting non-contact sight tracking errors using flash point filling according to claim 1, characterized in that: The specific method of step 1) is: 1.1) All flash points are searched to form a point set, and all flash points in the point set are sorted. First, all flash points are divided into N groups in ascending order of the vertical coordinate by length T. Then, the flash points in each of the N groups are sorted in ascending order of the horizontal coordinate to obtain the relative positions of all flash points, which are used to match the flash points and their corresponding light sources; 1.2) Obtain pupil ellipse set: Input grayscale image and use the maximum inter-class variance method (OTSU method) to obtain the threshold T otsu , and at the threshold [T otsu -60,T otsu -5] range, and preset pupil radius D according to the device resolution. p , pupil radius D p The radius is the mean of multiple experiments; Find the matching range with the preset pupil radius [D p -15, D p +15] ellipse, search for multiple pupil ellipses to obtain a pupil ellipse set; According to the obtained pupil radius, the pupil is cut with a side length of radius × 10 to obtain a square eye image, and the image is Gaussian blurred to reduce the noise of the cut image; 1.3) Obtaining the optimal pupil ellipse: After obtaining the pupil ellipse set from step 1.2), filter the pupil ellipse using the major-minor axis ratio and length information. Then, evaluate the pupil ellipse using the ratio α of the major axis b to the minor axis a, the degree of contour dispersion β, and the difference in grayscale values ​​inside and outside the contour γ. The pupil ellipse with the highest evaluation score is the optimal pupil ellipse. Among them, the ratio of the major axis b to the minor axis a of the contour-fitted pupil ellipse is α, and the range of α is [0,1]. The closer the ellipse is to a circle, the higher the score; Taking the center of the pupil ellipse as the origin, use the rectangular coordinate system to divide the ellipse into 4 regions, calculate the number of contour points in each region, and obtain a 1×4 vector N p , N p Normalize and subtract 0.25, then the mean of the vector is 0. If the contour points can be evenly distributed in each quadrant, that is, the number of contour points in each quadrant is the same, then the dispersion of the contour points of this contour is called perfect, and the vector N of this contour is p Each element in should be 0, and the sum of the absolute values ​​of each value in N is S. If the contour points are evenly distributed in the four regions, the value of S is 0. If the contour points are only distributed in one of the four regions, the value of S is 1.

5. The score of the dispersion of the contour points is β, and the range of β is [0,1]: Uniformly sample num points from the contour point set of each ellipse, and the range of num is [5, N max ],N max is the number of points in the contour point set, the vector from the ellipse center to the contour point is I, and each contour point sampled is I×0.5 as the inner point of the contour (x i ,y i ) and the point outside the contour at I×1.5 (x o ,y o ), according to the coordinates of the inner and outer points of the contour and the grayscale image, the grayscale values ​​of the grayscale image at the inner and outer points of the contour can be obtained, and the grayscale value difference can be obtained by subtracting them; the absolute value F of the mean of the grayscale value difference is taken, and the grayscale value is normalized by F / 255 to obtain the difference score γ of the inner and outer grayscale values. The range of γ is between [0,1]. The difference γ of the grayscale values ​​inside and outside the contour is; Then the sum of α, β, and γ is taken as the final score, and the image with the highest score is the best pupil ellipse image; 1.4) After finding the best ellipse, search for the flash point around the ellipse: for the best pupil ellipse image obtained in step 1.3), otsu ,T otsu +40] range, preset the flash point radius D according to the resolution of the device and the light source equipment g , search around the pupil ellipse in the range [D g -5,D g +5], search for multiple flash point ellipses, and then get a set of flash point ellipses; calculate the α, β, and γ values ​​of the flash point ellipse, and add the three to get the final score of the flash point ellipse; set a custom threshold for the score, and the score above the threshold is considered to be a flash point ellipse, and the score below the threshold is considered not to be a flash point ellipse.

3. The method for correcting non-contact sight tracking errors using flash point filling according to claim 1, characterized in that: After the point set is obtained in step 1), it is assumed that the number of points in the point set is N. gnum , n points need to be matched each time, and the number of points to be matched S is expressed as: The flash points obtained in step 1.1) are sorted and matched one by one based on the number of points taken S.

4. The method for correcting non-contact sight tracking errors using flash point filling according to claim 1, characterized in that: The specific method of step 2) is: 2.1) Matching 4 flash points: Each time, 4 points are selected from the flash point set, and the points to be matched are distributed in a rectangular pattern. The point closest to the pupil is used as the reference point. The reference point and two adjacent vertices form two 1×2 vectors. The diagonal vector v' is added together, and the vector v is obtained from the reference point and its diagonal point. The sum of the Euclidean distance and the cosine similarity of the vectors is used as the similarity value. The higher the similarity value, the higher the degree of similarity. The highest similarity is the best flash point. For two vectors, the smaller the angle between the two vectors, the higher the cosine similarity. The greater the difference in the length of the two vectors, the larger the Euclidean distance. The Euclidean distance is normalized so that the growth trend of the Euclidean distance is the same as that of the cosine similarity. The normalized Euclidean distance is negated and then added to the cosine similarity to obtain the final evaluation value: A threshold is set for the similarity value. When the similarity value is lower than the threshold, the four currently selected flash points are considered mismatched. When the similarity value is greater than the threshold, the match is considered successful. After S matches, the flash point combination with the highest evaluation value is selected from the successfully matched combinations as the final selected flash point combination. If no match is successful each time, the set of flash points will be judged as not having 4 flash points; 2.2) Correction of flash points and pupils The pupil center P and the flash point g are separated from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of 2.3) Pupil and Flash Point Gaze Tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right.

5. The method for correcting non-contact sight tracking errors using flash point filling according to claim 1, characterized in that: In step 3), if the flash point set in step 2) is determined to not have 4 flash points, 3 flash points are matched, and the matched 3 flash points form a right triangle or an atypical right triangle: 3.1) Each time, three flash points are selected from the flash point set. The number of selections required can be obtained by formula (1). The point closest to the pupil is taken as the reference point. Assuming that the three points selected each time are flash points generated by the light source, the corresponding light source of each flash point can be found by the relative relationship between the coordinates. The reference point and the other two flash points (g1, g2, g3) respectively form a 1×2 vector, and the two vectors form a 2×2 matrix corresponding to the light source (l1, l2, l3). The matrix is ​​normalized as a whole to obtain the feature matrix of the flash point to be matched; using the same method, the feature matrix of the light source corresponding to these three flash points is obtained; the cosine similarity of these two feature matrices is used as the similarity value. The higher the similarity value, the better the matching point; a threshold is set for the similarity value. When the similarity value is lower than the threshold, the three flash points currently selected are considered unmatched. When the similarity value is greater than the threshold, the match is considered successful. The combination with the highest evaluation value is selected as the final selected combination among the successful matching combinations; the similarity value is calculated as follows: If the points selected each time are not matched successfully, the flash point set will be judged as having no three flash points; 3.2) For the three matched points (g1, g2, g3) and the corresponding light sources (l1, l2, l3), find the affine transformation matrix Matric. After obtaining the matrix, the coordinates of the missing light source (l4) are mapped into the eyeball plane as the missing flash point coordinates g4, and the missing point coordinates are obtained to fill in the flash point; 3.3) Correction of the flash point and pupil: Change the pupil center P and the flash point g from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of 3.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, and the vertical downward direction as the y-axis, the inverse ratio of the left and right eye's horizontal coordinates is used as the weight. Assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, the left eye weight is (L / (L+R)) and the right eye weight is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking. The left eye's gaze tracking is more accurate when looking to the left, while the right eye's gaze tracking is more accurate when looking to the right. 3.5) Head correction: Based on the average human pupil radius of 1.5mm, the conversion parameter S from image units to real units is obtained scalar , taking the calibrated left and right eye center coordinates P as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H: H=S scalar (P-P′) After the head error correction, the final error correction needs to be performed. The error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output; Gaussian process regression is used to model the errors.

6. The method for correcting non-contact sight tracking errors using flash point filling according to claim 1, characterized in that: The matching method for the two flash points in step 4) is as follows: 4.1) Each time, two flashing points are selected from the point set obtained in step 1). The number of selections required can be calculated using formula 4. The point closest to the pupil is used as the reference point. If the reference point is below the center of the pupil, it is considered that the reference point is generated by the lower left or lower right light source of the screen. If it is above the center of the pupil, the flashing point is considered to be generated by the upper left or upper right light source. The vector v is obtained from the reference point and another point. The cosine values ​​of v and the horizontal and vertical directions are calculated. The largest one is taken as the evaluation value for this time, and the one with the highest evaluation value is the best flash point. For the case of two flash points, if the angle corresponding to the cosine value is greater than 30 degrees, it is considered an unsuccessful match. If it is less than 30 degrees, it is considered a successful match. The combination with the highest evaluation value is selected as the final combination among the successful matches. If all combinations fail to match successfully, this flash point set will be judged as a case without two flash points. In this case, the frame will be discarded and the next frame will be used for eye tracking. 4.2) For known flash points, the corresponding light source position must be determined. For two points, a multilayer perceptron is used to predict all four points (g1, g2, g3, g4). The multilayer perceptron input corresponds to the two points and the pupil major and minor axes (a, b). The following formula is used to obtain four points to complete the flash point filling. (g1,g2,g3,g4)=MLPs(g1,g2,a,b) 4.3) Correction of the flash point and pupil: Change the pupil center P and the flash point g from V p and V g Projected into R p and R g ; When correcting pupil, click PR p V p The triangle △PR p V p , the pupil center P, the light source L, and the camera C form a triangle △PLC. The screen plane is parallel to the plane to be projected, so △PR p V p ∽△PLC, so the length of LC and V p The coordinates of R p coordinates; when correcting the flash point, point gR g V g The triangle △gR g V g , the flash point g, the light source point L, and the camera point C form a triangle △gLC. The screen plane is parallel to the plane to be projected, so we can get △gR g V g ∽△gLC, so the length of LC and V g The coordinates of R g coordinates of 4.4) Pupil and flash point gaze tracking: With the upper left corner of the screen as the origin coordinate (0,0), the horizontal rightward direction as the x-axis, the vertical downward direction as the y-axis, and the inverse ratio of the horizontal coordinates of the left and right eyes as the weights, assuming the horizontal coordinate of the left eye is L and the horizontal coordinate of the right eye is R, then the weight of the left eye is (L / (L+R)), and the weight of the right eye is (R / (L+R)). The left and right eye gaze tracking is multiplied by the weights respectively and then added together to obtain the final pupil and flash point gaze tracking; the left eye gaze tracking is more accurate when looking to the left, and the right eye gaze tracking is more accurate when looking to the right; assuming the screen resolution is W×H, the gaze tracking of both eyes is L(X l ,Y l ), R(Xr,Yr), the average eye tracking of both eyes is A(X a ,Y a ), and get the fused gaze tracking F: 4.5) Head correction: Take the average of the actual human pupil radius to obtain the conversion parameter S from image units to real units scalar , taking the left and right eye center coordinates P in the calibration phase as the reference, the left and right eye center coordinates obtained in the current image are P', and the head deflection is H: H=S scalar (P-P′) After head correction, the gaze tracking error is modeled using Gaussian process regression, with the gaze tracking result as input and the correction value as output; The kernel function used for modeling is as follows: the(x p ,x q )=k1(σ 2 +x p x q )+k2δ Among them, k1, k2 are parameters that need to be manually adjusted, δ is white noise,

Images