A spot detection method capable of compensating for line-of-sight estimation accuracy

Abstract

The embodiment of the application provides a spot detection method capable of compensating for line-of-sight estimation accuracy, comprising: acquiring an eye diagram with a spot, labeling the spot of the eye diagram, and making a training set; training a spot detection model through the training set, and inputting the eye diagram into the spot detection model to obtain a spot center and a spot serial number; detecting a weak spot in the eye diagram; and filtering a deformed spot. The embodiment of the application can strengthen the detection of the weak spot in the pupil, compensate for the insufficient detection of the weak spot in the pupil by the algorithm model, and reduce part of factors that cause negative effects in line-of-sight estimation in eye movement tracking through filtering of the deformed spot.

Description

Technical Field

[0001] The embodiments of this application belong to the field of eye-tracking technology, and in particular relate to a spot detection method that can compensate for the accuracy of gaze estimation. Background Technology

[0002] Eye tracking has a wide range of applications, including virtual reality, autonomous driving, and human factors engineering. For example, it can help people with disabilities control computers effectively; it can also support eye-controlled mouse pointers, allowing users to select focus more quickly; furthermore, the fusion of user gaze and facial information can improve the security of existing access control systems. The most common eye tracking technology currently is the pupillary-corneal reflex method, which uses low-power active near-infrared illumination of the eye, captures near-infrared images using a sensor array, processes the near-infrared signals to obtain eye features, and estimates the gaze direction and fixation point. Most eye tracking solutions on the market currently use this technology. Accurately detecting the light spot on the eyeball and matching it precisely with the infrared illumination source is crucial for accurately calculating the gaze direction. Currently, deep learning-based light spot detection algorithms are not ideal for detecting weak light spots within the pupil or filtering out distorted light spots, which affects the accuracy of gaze estimation. Summary of the Invention

[0003] To solve or alleviate the technical problems in the prior art, this application improves the detection accuracy of effective light spots through a post-processing algorithm for light spot detection, thereby compensating for the accuracy of line-of-sight estimation.

[0004] This application provides a spot detection method that can compensate for line-of-sight estimation accuracy, including:

[0005] Obtain an eye diagram with light spots, annotate the eye diagram with light spots, and create a training set;

[0006] The spot detection model is trained using the training set, and the eye diagram is input into the spot detection model to obtain the spot center and spot number.

[0007] Detect weak spots in the eye diagram;

[0008] Filter out distorted light spots.

[0009] As a preferred embodiment of this application, the step of marking light spots on the eye diagram includes:

[0010] Using the center of the pupil in each eye diagram as the center, a square eye diagram containing the entire iris region is cropped out;

[0011] Starting from the lower left spot of the square eye diagram, mark the center position of the spot clockwise;

[0012] The center positions of all the light spots are normalized to obtain the normalized coordinates of the center of each light spot in each eye diagram.

[0013] As a preferred embodiment of this application, before detecting the weak light spot in the eye image, the process includes:

[0014] Determine whether the number of light spots detected by the light spot detection model is equal to the number of infrared lamps that form the light spots;

[0015] If the number of light spots is equal to the number of infrared lamps that form the light spots, then the distorted light spots in the eye diagram are filtered out; if the number of light spots is less than the number of infrared lamps that form the light spots but greater than a first preset threshold, then the weak light spots in the eye diagram are detected.

[0016] As a preferred embodiment of this application, the detection of weak light spots in the eye image includes:

[0017] Determine whether the light spot in the eye diagram is inside the pupil fitting ellipse;

[0018] If the light spot in the eye diagram is not within the pupil fitting ellipse, then the distorted light spot in the eye diagram is filtered out; if the light spot in the eye diagram is within the pupil fitting ellipse, then the center coordinates of the weak light spot are further determined, and thus the weak light spot in the eye diagram is obtained.

[0019] As a preferred embodiment of this application, determining whether the light spot in the eye image is within the pupil fitting ellipse includes:

[0020] Calculate the distance between two adjacent light spots, determine whether the distance between all adjacent light spots is within a first preset range, and if the distance between all adjacent light spots is within the first preset range, then filter out the deformed light spots in the eye diagram;

[0021] If the distance between all adjacent light spots is not within the first preset range, then it is determined whether the distance between all adjacent light spots is within the second preset range. If the distance between all adjacent light spots is not within the second preset range, then the light spot does not belong to the light spot in the pupil of the eye diagram, and the deformed light spots in the eye diagram are filtered out.

[0022] If the distance between all adjacent light spots is within a second preset range, then the light spot belongs to the light spot within the pupil of the eye diagram.

[0023] As a preferred embodiment of this application, determining the coordinates of the weak light spot center includes:

[0024] Calculate the midpoint coordinates of the line connecting the centers of two adjacent light spots, and traverse all pixels with the midpoint coordinates as the center and the second preset threshold as the radius.

[0025] The number of pixels with values ​​greater than a second preset threshold that are within the pupil fitting ellipse is counted. If the number of pixels is zero, the distorted spots in the eye diagram are filtered out.

[0026] If the number of pixels is not equal to zero, the average coordinates of the coordinate set of each pixel are calculated, and the average coordinates are used as the center coordinates of the detected weak light spot.

[0027] As a preferred embodiment of this application, the step before filtering the deformed light spot includes:

[0028] The eye diagram is thresholded to obtain a binary image, and then a square eye diagram is cropped from the binary image with the pupil center as the center.

[0029] The square eye diagram is subjected to spot connectivity detection. It is determined whether each spot connectivity contains a spot center. If it does, the deformed spot detection is performed. If not, no processing is performed.

[0030] As a preferred embodiment of this application, the filtering of deformed light spots includes:

[0031] Detect whether the aspect ratio of the circumscribed moment of the connected domain of the light spot is within a third preset range. If the aspect ratio of the circumscribed moment of the connected domain of the light spot is within the third preset range, then the light spot is a non-distorted light spot.

[0032] If the aspect ratio of the circumscribed moment of the connected region of the light spot is not within the third preset range, it is determined that the area of ​​the connected region of the light spot is less than the third preset threshold. If the area of ​​the connected region of the light spot is less than the third preset threshold, then the light spot is a non-distorted light spot.

[0033] If the area of ​​the light spot connected region is not less than the third preset threshold, a sub-image is cropped from the square eye diagram with the centroid of the light spot connected region as the center, and then all pixels of the sub-image are traversed.

[0034] When the pixel value is a certain preset value, the number of pixels in the sub-image whose horizontal coordinate is less than or greater than the horizontal coordinate of the centroid of the connected region of the spot, and the number of pixels in the sub-image whose vertical coordinate is less than or greater than the vertical coordinate of the centroid of the connected region of the spot are counted respectively.

[0035] After traversing all pixels of the sub-image, calculate the first ratio between the number of pixels in the sub-image whose x-coordinate is less than and greater than the x-coordinate of the centroid of the connected region of the spot, and the second ratio between the number of pixels in the sub-image whose y-coordinate is less than and greater than the y-coordinate of the centroid of the connected region of the spot.

[0036] If the first ratio or the second ratio is greater than the fourth preset threshold, then the light spot is a deformed light spot; if the first ratio or the second ratio is not greater than the fourth preset threshold, then the light spot is a non-deformed light spot.

[0037] Compared with the prior art, the embodiments of this application provide a spot detection method that can compensate for the accuracy of gaze estimation. It can enhance the detection of weak spots in the pupil, make up for the shortcomings of the algorithm model in missing weak spots in the pupil, and reduce some of the factors that negatively affect gaze estimation in eye tracking by filtering out deformed spots. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. Some specific embodiments of this application will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings designate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0039] Figure 1 This is a flowchart of a spot detection method that can compensate for line-of-sight estimation accuracy, provided in an embodiment of this application.

[0040] Figure 2 This is an eye diagram provided in the embodiments of this application after completing weak light spots and readjusting the light spot sequence numbers. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0042] like Figure 1 As shown in the figure, this application provides a spot detection method that can compensate for line-of-sight estimation accuracy, including the following steps:

[0043] Step S101: Obtain an eye diagram with light spots, annotate the eye diagram with light spots, and create a training set.

[0044] The device is similar to a VR headset. It has a ring of infrared lights around the left and right eyes (eight infrared lights are used in this embodiment). An infrared camera is installed directly below the left and right eyes to capture images of the left and right eyes.

[0045] For the acquired eye images, all eye images are processed according to the right eye (the left eye image is processed after being horizontally flipped). First, a square eye image with a resolution of S1×S1 (156×156 in this embodiment) that can contain the entire iris region is cropped with the pupil center of each eye image as the center. Then, the center position of the light spot is manually marked clockwise with the light spot in the lower left corner of the eye image as the starting position. The center position of the light spot is then normalized. That is, assuming the coordinates of the center of the light spot are (x,y), the normalized position is (x / image width, y / image height).

[0046] The annotation results for each eye image are saved in a txt file with the corresponding image name.

[0047] For example, the annotation data in the txt file looks like the following (corresponding to the annotated eye diagram):

[0048] (0, 0)

[0049] (0, 0)

[0050] (0.30734358974358983, 0.5914692307692309)

[0051] (0.35065641025641014, 0.5099384615384616)

[0052] (0.5799589743589743, 0.47936410256410256)

[0053] (0.6258205128205125, 0.558346153846154)

[0054] (0.6130820512820512, 0.672997435897436)

[0055] (0, 0) 0 0 1 1 1 1 1 0

[0057] Starting from the lower left of the eye image, in a clockwise direction, the top 8 coordinates are the normalized coordinates of the centers of 8 light spots. The last 8 integers (0 0 1 1 1 1 1 0) indicate the presence or absence of a light spot at the corresponding position. 1 indicates the presence of a corresponding light spot, and 0 indicates the absence of a corresponding light spot. For example, the above means: there is no light spot at positions 1 and 2, there is a light spot at positions 3, 4, 5, 6, and 7, and there is no light spot at position 8. Moreover, the center coordinates of the light spot at the position without a light spot are (0,0).

[0058] Step S102: Train the spot detection model using the training set, and input the eye diagram into the spot detection model to obtain the spot center and spot number:

[0059] The training set obtained in step S101 is augmented with data, and a neural network model is constructed. A loss function is designed, and the model is iteratively trained using this loss function until it converges, resulting in the trained network model. Specific data augmentation includes geometric transformations and color transformations. Geometric transformations include rotation, flipping, scaling, translation, and cropping, etc., while color transformations include adjusting brightness, converting images to grayscale, and flipping image colors, etc.

[0060] The cropped eyeball image is then preprocessed and input into the trained network model for inference and post-processing to obtain the center and number of the light spot.

[0061] Determine whether the number of light spots detected in step 102 is equal to the number of infrared lamps (the number of infrared lamps in this embodiment is N=8). If yes, proceed to step S104; otherwise, if it is less than the number of infrared lamps but greater than 2, proceed to step S103.

[0062] Step S103: Detect weak light spots in the eye diagram:

[0063] During the calibration phase (user parameter calibration is required when users perform eye tracking using a head-mounted display), the distance Adjd between adjacent light spots is calculated and saved as user parameters.

[0064] Calculate the distance Dt between each adjacent light spot detected in step S102 above. If all adjacent light spot distances are within a predetermined range: If the above steps are not taken, proceed to step S104; otherwise, proceed to the following steps.

[0065] Determine whether the distance Dt between each adjacent light spot is within the pupil, wherein the distance Dt between each adjacent light spot satisfies the following condition: (where thr2 is a preset threshold), judgment method: During gaze tracking, it is necessary to extract key points of the pupil contour (usually 8 key points of the contour). Fit the key points of the contour into an ellipse. If the center of the light spot is inside the ellipse, it is an internal point of the pupil; otherwise, it is an external point. If two adjacent light spots are both inside the pupil, proceed to the following step to calculate the midpoint of the line connecting the centers of the two adjacent light spots; otherwise, proceed to step S104.

[0066] Calculate the midpoint of the line connecting the centers of two adjacent light spots. Using the midpoint as the center and R as the radius, traverse all pixels within the circle, count the number k pixels whose values ​​are greater than a set threshold (100 in this embodiment) and are located within the pupil fitting ellipse region, and record the coordinate values ​​of these pixels. If k is not equal to zero, then proceed to the following steps to sum and average the pixel coordinate sets respectively; otherwise, proceed to step S104.

[0067] Sum and average the pixel coordinates separately, i.e. Then (xx, yy) will be used as the center coordinates of the detected weak spot, and the coordinates and sequence number of the detected spot will be updated. That is, the sequence number of the weak spot is the sequence number of the first spot plus 1, the sequence number of the second spot is the sequence number of the weak spot plus 1, and the sequence number of the third spot is the sequence number of the second spot plus 1.

[0068] like Figure 2 As shown, the first eye diagram (from left to right) is the eye diagram after detecting and sorting the light spots using deep learning. A weak light spot with index 4 within one pupil was missed. The numbers 1, 2, 3, 4, 5, 6, and 7 in the first eye diagram represent the light spot indices. The second eye diagram (from left to right) is the eye diagram after the missed weak light spot index 4 is filled in using an algorithm. P1 and P2 in the second eye diagram represent the centers of two adjacent light spots. The third eye diagram (from left to right) is the eye diagram after readjusting the light spot indices. The numbers 1, 2, 3, 4, 5, 6, and 7 in the third eye diagram represent the light spot indices. From... Figure 2 It can be concluded that spot number 4 was initially missed, and the algorithm will complete the detection of the missed weak spot number 4.

[0069] Step S104, filtering out distorted light spots:

[0070] First, the eye image is segmented using the `threshold` function in OpenCV to obtain a binary image `binnary1`. Then, a square eye image `binnary2` with a resolution of S2×S2 (144×144 in this embodiment) is cropped from the binary image `binnary1`, centered on the pupil center. Next, the `connectedComponentsWithStats` function in OpenCV is used to perform connected component detection on the square eye image `binnary2`, sequentially determining whether a spot center is present within the connected component. If so, the following deformed spot detection is performed; otherwise, no processing is performed. The detection order and rules are as follows:

[0071] 1. Detect whether the aspect ratio of the external moments of the connected domain is within the threshold range (10.8~1.2 in this embodiment). If it is, it is considered to be a non-distorted spot; otherwise, proceed to rule 2 below for judgment.

[0072] 2. Abnormal light spots generally have a large area. If the area of ​​the connected region of the light spot is less than the set threshold (136 pixels in this embodiment), it is considered to be a non-abnormal light spot; otherwise, proceed to the third rule below for judgment.

[0073] 3. Using the centroid of the connected component as the center, extract a subgraph from the square eye diagram binnary2 with a width W equal to the width of the connected component's outer moment and a height H equal to the height of the connected component's outer moment. Then, iterate through all pixels in this subgraph. When the pixel value is 255, count the number of pixels whose x-coordinate is less than and greater than the centroid's x-coordinate (num_x1 and num_x2), and the number of pixels whose y-coordinate is less than and greater than the centroid's y-coordinate (num_y1 and num_y2). After iterating through all pixels in the subgraph, calculate the ratio of num_x1 to num_x2 (ratio_x = num_x1 / num_x2) and the ratio of num_y1 to num_y2 (ratio_y = num_x1 / num_x2). =num_y1 / num_y2, if ratio_x or ratio_y is greater than the set threshold (2 in this embodiment), then the spot is considered to be a deformed spot, and the relevant features of the spot are set to 0 (i.e. invalid, refer to the above rules on spot labeling), otherwise the spot is considered to be a non-deformed spot.

[0074] 4. If a deformed spot is detected, the deformed spot feature is set as an invalid feature, and other non-deformed spot features are substituted into the gaze tracking algorithm for gaze tracking calculation.

[0075] The spot detection method that can compensate for the accuracy of gaze estimation provided in this application can enhance the detection of weak spots in the pupil and make up for the shortcomings of the algorithm model in missing weak spots in the pupil. By filtering out deformed spots, it can reduce some of the factors that negatively affect gaze estimation in eye tracking.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A spot detection method that can compensate for line-of-sight estimation accuracy, characterized in that, include: Obtain an eye diagram with light spots, annotate the eye diagram with light spots, and create a training set; The spot detection model is trained using the training set, and the eye diagram is input into the spot detection model to obtain the spot center and spot number. Detect weak spots in the eye diagram; Filtering out distorted light spots; The detection of weak light spots in the eye image includes: Determine whether the light spot in the eye diagram is inside the pupil fitting ellipse; If the spot in the eye diagram is not within the pupil fitting ellipse, then the distorted spot in the eye diagram is filtered out; if the spot in the eye diagram is within the pupil fitting ellipse, then the center coordinates of the weak spot are further determined, and thus the weak spot in the eye diagram is obtained. The filtering of deformed light spots includes: The aspect ratio of the circumscribed moment of the connected domain of the light spot is detected to be within a third preset range. If the aspect ratio of the circumscribed moment of the connected domain of the light spot is within the third preset range, then the light spot is a non-distorted light spot. If the aspect ratio of the circumscribed moment of the connected region of the light spot is not within the third preset range, it is determined that the area of ​​the connected region of the light spot is less than the third preset threshold. If the area of ​​the connected region of the light spot is less than the third preset threshold, then the light spot is a non-distorted light spot. If the area of ​​the light spot connected region is not less than the third preset threshold, a sub-image is cropped from the square eye diagram with the centroid of the light spot connected region as the center, and then all pixels of the sub-image are traversed; the square eye diagram is obtained by thresholding the eye diagram to obtain a binary image, and then cropping it from the binary image with the pupil center as the center. When the pixel value is a certain preset value, the number of pixels in the sub-image whose horizontal coordinate is less than or greater than the horizontal coordinate of the centroid of the connected region of the spot, and the number of pixels in the sub-image whose vertical coordinate is less than or greater than the vertical coordinate of the centroid of the connected region of the spot are counted respectively. After traversing all pixels of the sub-image, calculate the first ratio between the number of pixels in the sub-image whose x-coordinate is less than and greater than the x-coordinate of the centroid of the connected region of the spot, and the second ratio between the number of pixels in the sub-image whose y-coordinate is less than and greater than the y-coordinate of the centroid of the connected region of the spot. If the first ratio or the second ratio is greater than the fourth preset threshold, then the light spot is a deformed light spot; if the first ratio or the second ratio is not greater than the fourth preset threshold, then the light spot is a non-deformed light spot.

2. The spot detection method that can compensate for line-of-sight estimation accuracy as described in claim 1, characterized in that, The step of marking light spots on the eye diagram includes: Using the center of the pupil in each eye diagram as the center, a square eye diagram containing the entire iris region is cropped out; Starting from the lower left spot of the square eye diagram, mark the center position of the spot clockwise; The center positions of all the light spots are normalized to obtain the normalized coordinates of the center of each light spot in each eye diagram.

3. The spot detection method that can compensate for line-of-sight estimation accuracy as described in claim 1, characterized in that, Before detecting the weak light spot in the eye diagram, the process includes: Determine whether the number of light spots detected by the light spot detection model is equal to the number of infrared lamps that form the light spots; If the number of light spots is equal to the number of infrared lamps that form the light spots, then the distorted light spots in the eye diagram are filtered out; if the number of light spots is less than the number of infrared lamps that form the light spots but greater than a first preset threshold, then the weak light spots in the eye diagram are detected.

4. The spot detection method that can compensate for line-of-sight estimation accuracy as described in claim 1, characterized in that, The step of determining whether the light spot in the eye image is within the pupil fitting ellipse includes: Calculate the distance between two adjacent light spots, determine whether the distance between all adjacent light spots is within a first preset range, and if the distance between all adjacent light spots is within the first preset range, then filter out the deformed light spots in the eye diagram; If the distance between all adjacent light spots is not within the first preset range, then it is determined whether the distance between all adjacent light spots is within the second preset range. If the distance between all adjacent light spots is not within the second preset range, then the light spot does not belong to the light spot in the pupil of the eye diagram, and the deformed light spots in the eye diagram are filtered out. If the distance between all adjacent light spots is within a second preset range, then the light spot belongs to the light spot within the pupil of the eye diagram.

5. The spot detection method that can compensate for line-of-sight estimation accuracy as described in claim 4, characterized in that, Determining the coordinates of the weak light spot center includes: Calculate the midpoint coordinates of the line connecting the centers of two adjacent light spots, and traverse all pixels with the midpoint coordinates as the center and the second preset threshold as the radius. The number of pixels with values ​​greater than a second preset threshold that are within the pupil fitting ellipse is counted. If the number of pixels is zero, the distorted spots in the eye diagram are filtered out. If the number of pixels is not equal to zero, the average coordinates of the coordinate set of each pixel are calculated, and the average coordinates are used as the center coordinates of the detected weak light spot.

6. The spot detection method that can compensate for line-of-sight estimation accuracy as described in claim 1, characterized in that, Before filtering out the deformed light spot, the following are included: The square eye diagram is subjected to spot connectivity detection. It is determined whether each spot connectivity contains a spot center. If it does, the deformed spot detection is performed. If not, no processing is performed.

Images