A 2D gaze point estimation method based on pupil-corneal reflection vector method

Abstract

This invention discloses a 2D gaze point estimation method based on the pupil-corneal reflection vector method. It utilizes the star ray method for pupil center localization and adds a classification pupil detection mechanism for error correction. Corneal feature extraction is performed to obtain the center coordinates of the corneal reflected light spot. Step 4: Establish a gaze mapping model, and assign the detected pupil-corneal reflection vector V(V) to each frame of the image. x V y The gaze coordinates G(G) mapped onto the screen x G y This invention enables 2D gaze point estimation based on the pupil-corneal reflection vector method. Compared with existing technologies, this invention can improve the accuracy of 2D gaze point estimation.

Description

Technical Field

[0001] This invention belongs to the field of image processing and eye tracking technology, and relates to 2D gaze point estimation, particularly a 2D gaze point estimation method based on the pupil-corneal reflection vector method. Background Technology

[0002] Assuming the human cornea is a regular sphere, regardless of its rotation, the absolute position of the corneal spot formed by the auxiliary light source on the sphere remains largely unchanged in space. In other words, when the eyeball rotates, the position of the corneal spot on the eyeball essentially does not change. When the gaze direction changes, the gaze information can be obtained by using the position of the corneal spot as a reference standard to calculate the intersection of the gaze point and the plane mapping. This method is called Pupil-Center Corneal Reflection (PCCR), a widely used technique in feature-based gaze estimation algorithms.

[0003] This technique consists of three parts: image preprocessing, feature extraction, and mapping function construction. Improvements to the pupil center and corneal feature extraction parts can significantly enhance the accuracy of fixation point estimation. Summary of the Invention

[0004] The purpose of this invention is to propose a 2D gaze point estimation method based on the pupil-corneal reflectance vector method. This method combines extracted corneal features to establish a gaze mapping model, and uses the detected pupil-corneal reflectance vector V(V) in each frame of the image to... x V y The gaze coordinates G(G) mapped onto the screen x G y This enables 2D gaze point estimation.

[0005] This invention is achieved using the following technical solution:

[0006] A 2D gaze point estimation method based on the pupil-corneal reflectance vector method includes the following steps:

[0007] Step S1: Perform noise reduction and edge detection on the image to be processed to achieve image preprocessing;

[0008] Step S2: Locate the pupil center using the star ray method;

[0009] Step S3: Perform corneal feature extraction. Based on the threshold corresponding to the extreme point of the error function, obtain the optimal threshold after adaptation. Using the optimal threshold as a limit, find the largest reflected light spot in the binarized eye area image, and then calculate the center coordinates (x, y) of the reflected light spot. pur ,y pur ), thus obtaining the coordinates of the center position of the corneal reflected light spot;

[0010] The error function formula is shown below:

[0011]

[0012] Among them, R ratio Let S be the error function. max S is the threshold for the area of ​​the reflected light spot. all The area of ​​the eye diagram;

[0013] Step S4: Establish a gaze mapping model, that is, establish a gaze mapping model based on the pupil center position obtained in step S2 and the corneal reflection spot center position obtained in step S3, and use the gaze mapping model to calculate the pupil-corneal reflection vector V(V) detected in each frame image. x V y The gaze coordinates G(G) mapped onto the screen x G y The mapping function model is:

[0014]

[0015]

[0016] Where a0-a7 and b0-b7 are undetermined coefficients;

[0017] Complete 2D fixation point estimation based on the pupil-corneal reflection vector method.

[0018] Compared with the prior art, the present invention can achieve the following beneficial technical effects:

[0019] 1) By adding a classification detection mechanism and an infrared light source, the accuracy of pupil center localization, the success rate of corneal feature extraction, and the accuracy of 2D fixation point estimation can be effectively improved.

[0020] 2) A complete 2D gaze point estimation process, which enables effective 2D gaze point estimation. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of a 2D gaze point estimation method based on the pupil-corneal reflection vector method;

[0022] Figure 2 This is a flowchart illustrating the implementation process of a 2D gaze point estimation method based on the pupil-corneal reflection vector method.

[0023] Figure 3 This is an example diagram illustrating the process of locating the pupil center using the star ray method;

[0024] Figure 4This is a schematic diagram of the eye diagram and spot position relationship generated by multiple infrared light sources in the corneal feature extraction algorithm; (a) example of eye diagram of multiple infrared light sources, (b) schematic diagram of the position relationship of 4 spots. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly described below with reference to the accompanying drawings.

[0026] like Figure 1 The diagram shows the overall flowchart of a 2D gaze point estimation method based on the pupil-corneal reflection vector method. The specific steps include:

[0027] Step S1: Median filtering and Sobel operator are used to denoise and detect edges in the image to be processed, thus achieving image preprocessing. This step effectively preserves edge characteristics while eliminating impact noise.

[0028] Step S2: Pupil center localization using the star ray method: In the preprocessed image to be processed, the boundary of the pupil region is fitted using the boundary fitting method, where the ellipse fitting formula is:

[0029]

[0030] Starting from the center of the first frame, 240 star rays are emitted to search for the pixel with the maximum gradient value as the pupil edge point. The radiation angle of the rays falls within two regions: -60° to 60° and 120° to 240°. These two regions can cover most of the pupil, and the optimal ellipse is fitted. A classification pupil detection mechanism is added for error correction, i.e., a correction function is added, and the current frame is estimated by averaging the values ​​of the previous two frames to reduce errors. Let the pupil center coordinates at time t be (x...). T(t) ,y T(t) The differences in the pupil center position in the x and y directions between time t and (t-1) are respectively & x1 =x T(t) -x T(t-1) & y1 =y T(t) -y T(t-1) Let the differences in the pupil center position in the x and y directions between time t and (t-2) be & x2 =x T(t) -x T(t-2) & y2 =y T(t) -y T(t-2) The mean A of the pupil coordinates in the x and y directions of the first two frames x =(x T(t-1) +x T(t-2) ) / 2, A y=(y T(t-1) +y T(t-1) The corrected expression for the pupil center coordinates of the current frame (time t) is as follows:

[0031]

[0032]

[0033] H and P are correction coefficients, which are empirical constants that are set according to the image resolution.

[0034] Step S3: Corneal feature extraction: The threshold corresponding to the extreme point of the error function is the optimal threshold after adaptation. Using the optimal threshold as a limit, the binarized eye region image is used to find the largest reflected spot, and then its center coordinates (x, y) are calculated. pur ,y pur ), thus obtaining the coordinates of the center position of the corneal reflected spot.

[0035] The underlying principle is as follows: the brightness of the corneal reflective spot gradually decreases from the center outwards, and the remaining spots are impurities. Generally, the center coordinates of the corneal reflective spot are extracted using a binarized eye diagram (eye area image). When the threshold is high, only brighter points can pass through. At this point, there are fewer impurities, but the corneal reflective spot area is also small, resulting in a larger error in the extracted center coordinates. As the threshold decreases, the number of passable spots gradually increases, leading to an increase in impurities. However, the area of ​​the corneal reflective spot also increases, thus reducing the error in extracting the center coordinates and increasing accuracy. When the threshold decreases further after reaching a certain level, the number of impurities continues to increase, but the area of ​​the corneal reflective spot does not increase significantly, and the error in extracting the center coordinates will increase again. Therefore, using a constant binarization threshold is not the optimal choice; an adaptive threshold needs to be provided based on the specific situation of each eye diagram frame. The relevant error function formula is shown below:

[0036]

[0037] Among them, R ratio Let S be the error function. max S is the threshold for the area of ​​the reflected light spot. all Let be the eye area. This formula describes the process of changing the aforementioned threshold and the accuracy of corneal reflective spot center coordinate extraction. The error function formula with the threshold as the independent variable is a function that first increases and then decreases.

[0038] Step S4: Establish a gaze mapping model, that is, establish a gaze mapping model based on the pupil center position and the corneal reflective spot center position, and use the gaze mapping model to calculate the pupil-corneal reflective vector V(V) detected in each frame of the image.x V y The gaze coordinates G(G) mapped onto the screen x G y ), where the pupil-corneal reflection vector is the difference vector between the coordinates of the pupil center and the coordinates of the corneal center, and the mapping function model is:

[0039]

[0040]

[0041] Where a0-a7 and b0-b7 are undetermined coefficients;

[0042] Thus far, steps S1 to S4 have completed the 2D gaze point estimation method based on the pupil-corneal reflection vector method.

[0043] In summary, the overall process of this invention requires calibrating the fixation point of the subject, obtaining the pupil center coordinates and corneal vector feature center coordinates at the fixation calibration point, thereby solving for the undetermined coefficients and establishing the gaze mapping function model, thus completing the 2D fixation point estimation based on the pupil-corneal reflection vector method.

[0044] Step S3 further includes a nine-point calibration process for the subject's gaze. This involves designing a calibration video and establishing a horizontal coordinate system with the lower left corner of the screen as the origin. The horizontal axis ranges from [0, 1920], and the vertical axis ranges from [0, 1080]. Calibration points are presented as small balls. During calibration, the subject visually tracks the small balls appearing at different positions on the screen. The calibration point balls move sequentially, and the subject maintains gaze while the balls remain stationary. Table 1 shows the coordinates of the nine calibration points.

[0045] Table 1

[0046]

[0047] like Figure 3 The image shown is an example of the process of locating the pupil center using the star ray method. A detailed description follows:

[0048] The first frame of the image is captured by observing the edge of the pupil, and 18 rays are emitted outwards from the center of that image, with an angle of 20° between each ray. Then, the image is moved forward along these 18 rays sequentially. Figure 3 As shown in (a). The ray stops when the difference between two adjacent pixels is greater than a set threshold φ. Here, only the case of increasing grayscale value is considered, and finally a set of candidate edge points is obtained, as shown in (a). Figure 3 As shown in (b).

[0049] Starting from each candidate edge point, rays are emitted again within ±50° on both sides of the ray passing through that candidate edge point. The interval between rays is 100 / n degrees, where n is the number of rays, n = 5 ∈ φ (minimum value is 5). Figure 3 (b) and Figure 3 As shown in (c).

[0050] The above steps complete the first iteration. The geometric center of all candidate edge points is then calculated as the starting point for the next iteration. Figure 3 As shown in (d); repeat steps 2.1-2.2 until the geometric centers of all candidate edge points converge to relatively stable positions, i.e., the distance between the geometric center positions calculated in two consecutive iterations is no more than 10 pixels, as shown in (d). Figure 3 As shown in (e), the geometric center that meets the conditions is located as the pupil center.

[0051] like Figure 4 The diagram shown is a schematic diagram of the eye diagram and the positional relationship of light spots generated by multiple infrared light sources in the corneal feature extraction algorithm.

[0052] By adding infrared light sources to increase the threshold, and using four near-infrared auxiliary light sources for each eye, four corneal light spots will be formed in the eye area image, such as... Figure 4 As shown in (a), due to the one-to-one correspondence between the infrared light source and its light spot, the position of the light spot is fixed and correlated. Even when there is a failure in light spot extraction and the number of light spots is only 3, the positions of the remaining light spots can still be determined through their vector correlation, such as... Figure 4 As shown in (b).

[0053] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the present invention is not limited to the above-described technical solutions. Any modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the scope defined by the appended claims.

Claims

1. A 2D gaze point estimation method based on the pupil-corneal reflection vector method, characterized in that, The specific steps are as follows: Step S1: carry out noise reduction and edge detection of the picture to be processed, and realize image preprocessing; Step S2: pupil center positioning is carried out by using the star ray method; Step S3: corneal feature extraction is performed, the adaptive optimal threshold is obtained according to the threshold corresponding to the pole of the error function, the eye area image after binarization processing is limited to find the largest area of the reflection spot with the optimal threshold, and then the center coordinate of the reflection spot is calculated to obtain the center position coordinate of the corneal reflection spot; The error function formula is as follows: wherein, is the error function, is a reflected light spot area threshold, is an eye diagram area; Step S4: establishing a line-of-sight mapping model, i.e. establishing a line-of-sight mapping model according to the pupil center position obtained in step S2 and the corneal reflection light spot center position obtained in step S3, and mapping the pupil-corneal reflection vector detected in each frame of image to the gaze point coordinate on the screen by using the line-of-sight mapping model mapping function model is: mapping function model is: wherein are to be determined coefficients; Complete 2D gaze point estimation based on the pupil-corneal reflection vector method.

2. The 2D gaze point estimation method based on the pupil-corneal reflection vector method of claim 1, wherein, The step 2 further includes the following processes: Step 2.1: fit the boundary of the pupil area, grab the first frame image through the pupil edge, take the center of the image as the starting point, and send out 18 rays around it with an interval angle of 20°; when the difference value of the pixel points of the edges of the adjacent two rays is greater than the set threshold value, the rays stop, and a set of candidate edge points is obtained; Step 2.2 successively takes each candidate edge point as a starting point, and again sends rays in the range of ±50° on both sides of the ray passing through the candidate edge point, with an interval angle between the rays of degrees, where n is the number of ray strips; Step 2.3: the above step completes the first iteration, calculates the geometric center of all candidate edge points as the starting point of the next iteration; repeat the steps 2.1-2.2, until the geometric center of all candidate edge points converges to a relatively stable position, that is, the distance between the geometric center positions calculated by the previous two iterations is not greater than 10 pixel points, and the geometric center position meeting the condition is positioned as the pupil center.

3. The 2D gaze point estimation method based on the pupil-corneal reflection vector method of claim 1, wherein, The step 2 further comprises increasing the classification pupil detection mechanism for error correction processing, estimating the current frame by the average of the previous two frames to reduce error, and recording the pupil center coordinates at time t as , The difference between the pupil center positions at time t and time in the x and y directions is respectively , , The difference between the pupil center positions at time t and time in the x and y directions is respectively , The average of the pupil coordinates of the previous two frames in the x and y directions is , The expression of the corrected pupil center coordinates of the current frame is as follows: wherein , is a correction factor, is the time.

4. The 2D gaze point estimation method based on the pupil-corneal reflection vector method of claim 1, wherein: After the corneal feature extraction in step 3, further including the step of nine-point calibration: designing a calibration video, establishing a horizontal coordinate system with the lower left corner of the screen as the origin; the horizontal axis interval is [0, 1920], and the vertical axis interval is [0, 1080]; the calibration points are in the form of small balls, and in the calibration process, the subject tracks the small balls appearing on the screen at different positions through vision, and the calibration point small balls move in turn and the subject keeps staring when the small ball stops.

5. The 2D gaze point estimation method based on the pupil-corneal reflection vector method of claim 1, wherein, The pupil-corneal reflection vector is the difference vector of the pupil center coordinates and the corneal center coordinates.

Images