Pupil trajectory generation method based on human eye video

Abstract

The embodiment of the present invention discloses a pupil trajectory generation method based on human eye video, including step 1: coarse filtering each frame image, performing the first morphological filtering, and eliminating the white spots formed by infrared light irradiating the pupil; step 2: calculating the minimum grayscale average value of each frame image, judging whether it is a valid frame according to the minimum grayscale average value, and taking the threshold if it is; step 3: performing binarization processing; step 4: performing the second morphological filtering again, removing eyebrows and small skin pigment areas, and filling the binarized pupil area; step 5: using the Laplace operator to perform edge detection to obtain the outline of the pupil; step 6: calculating the lower pole of the pupil. The present invention can effectively process frames with A>S×10%, and the frames that cannot be processed are basically closed eyes or blurred pupils; the present invention can generate more stable and accurate eye movement trajectories.

Description

Technical Field

[0001] The present invention relates to the technical field of eye tracking, and in particular to a method for generating pupil trajectories based on human eye videos. Background Art

[0002] Eye tracking is widely used. For example, in human-computer interaction applications, eye gestures are used to drive interaction events. In the medical and health fields, eye movements are used to analyze cognitive behaviors, and nystagmus is used to check BPPV (benign paroxysmal positional vertigo, also known as otolithic vertigo, which refers to short-term paroxysmal vertigo and nystagmus when the head moves rapidly to one or more specific head positions.) symptoms. In the field of user experience, eye movements are used to analyze users' reading habits and concentration. To quantitatively measure eye movements, computer vision is usually used to collect eye videos using imaging technology, analyze frame by frame, obtain the pupil position in each frame, and obtain the pupil movement trajectory.

[0003] Existing pupil trajectories are all expressed as curves showing the time-dependent changes of the pupil center point in the video. Therefore, the pupil trajectory tracking technology is reduced to the pupil center point positioning technology based on image processing.

[0004] Among the early pupil center positioning methods, the more typical methods include ellipse fitting and Hough circle detection, etc. The deep learning method developed in recent years can obtain the pupil radius and pupil center position and detect the blinking process.

[0005] The most similar method to the present invention is an improved ellipse fitting algorithm proposed by S. Wibirama. This method fits the ellipse by randomly selecting points. Compared with other existing methods, it can obtain higher accuracy when the pupil is occluded (not completely occluded), and the generated pupil trajectory is also smoother.

[0006] The opening and closing of the eye fissure is a frequent phenomenon in human eyes. When the eye fissure is opened and closed, especially when blinking, the observable pupil area becomes smaller. At present, the shortcomings of the existing technology are as follows:

[0007] (1) Assume that the maximum pupil area is S , the observable pupil area is A ,when A < S ×80%, it is difficult for existing methods to obtain accurate positioning results; when A < S ×60%, the deep learning method is prone to errors in estimating the center position. A < S ×60%, the accuracy of pupil localization obtained by S. Wibirama's method decreases significantly.

[0008] (2) For the existing method, it is necessary to add an additional method to determine the observable pupil area. A The calculation method of A < S ×60% of the frames are deleted, which reduces the number of available frames in the video and increases the processing overhead.

[0009] (3) When using traditional methods, the threshold for image binarization is difficult to calculate using a unified algorithm and requires additional manual adjustment.

[0010] (4) When using deep learning methods, high performance requirements are placed on the CPU and GPU, and a large amount of data is required for model training.

[0011] (5) When using existing technology, once the estimation is wrong, the generated pupil movement trajectory will have a large number of abnormal peaks. Summary of the invention

[0012] The technical problem to be solved by the embodiments of the present invention is to provide a method for generating pupil trajectories based on human eye videos to solve the problems of inaccurate pupil positioning and abnormal pupil movement trajectories.

[0013] In order to solve the above technical problems, an embodiment of the present invention proposes a method for generating pupil trajectories based on human eye videos, comprising:

[0014] Step 1: Perform a coarse filter on each frame of the human eye video, perform the first morphological filter, and eliminate the white spots formed by the infrared light irradiating the pupil;

[0015] Step 2: Calculate the minimum grayscale average of each frame image, and determine whether it is a valid frame based on the minimum grayscale average. If so, take the threshold E th =G min +ε, ε is the error compensation value, G min is the minimum grayscale average value;

[0016] Step 3: Using Eth as the threshold, perform binarization processing on the corresponding valid frame image;

[0017] Step 4: Perform a second morphological filter to remove eyebrows and small areas of skin pigment, and fill the binarized pupil area;

[0018] Step 5: Use the Laplacian operator to perform edge detection and obtain the outline of the pupil;

[0019] Step 6: Calculate the pupil lower point l t =(x t ,y t ), first find the bounding box of the pupil contour B={x left , xright , y bottom , y top},let x t = (x left + x right ) / 2, and the pupil central axis is x = x t . The intersection of the pupil central axis and the bottom edge y = y bottom of the bounding box is the lower pole of the pupil. Take y t = y bottom ; finally, output the pupil trajectory L = {l t , t = 1,..., T}.

[0020] Furthermore, in step 2, the following method is used to calculate the minimum gray average value G min :

[0021] Given a grayscale image I with size r rows and c columns; I(p) is the gray value at pixel point p; set a sliding window W of size n×n = {0, 1,..., n - 1}×{0, 1,..., n - 1}; first calculate the average gray value G(p) of the diagonal elements of the window W p = W + p, G(p) = å 0 ≤ k < nI (p+(k, k)) / n, p Î D = {1,..., r - n + 1}´{1,..., c - n + 1}, and then find the minimum gray average value G of I min = min pÎD G(p)

[0022] Furthermore, in step 2, the method of diagonal accumulation is used to quickly calculate the candidate threshold of the pupil,

[0023] establish a matrix s of size (r + 1) × (c + 1), set the first row and the first column of s to zero, and let

[0024] s(i + 1, j + 1) = I(i, j) + s(i, j), 1 ≤ i ≤ r, 1 ≤ j ≤ c,

[0025] then G(i, j) = [s(i + n, j + n) – s(i, j)] / n, (i, j) Î D.

[0026] The beneficial effects of the present invention are as follows:

[0027] (1) The present invention can improve the data availability. In the existing methods, it is difficult to obtain accurate results in the frames where the observable area A of the pupil < 60% of S. The present invention can effectively process the frames where A > 10% of S, and the cases that cannot be processed are basically the situations of closed eyes or blurred pupils.

[0028] (2) The pupil trajectory generated by the present invention is more stable. For infrared human eye videos, the algorithm can be adapted to human eye videos with different infrared light brightness, and the present invention can generate a more stable and accurate eye movement trajectory.

[0029] (3) The present invention is simple to implement and requires low computing power. However, deep learning methods require very high computing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a method for generating pupil trajectories based on human eye video according to an embodiment of the present invention.

[0031] Figure 2 This is a diagram showing the effect of positioning the lower pupil pole during the blinking process according to an embodiment of the present invention.

[0032] Figure 3 The pupil center of the embodiment of the present invention is located at X Schematic diagram of the displacement on the axis with the frame number.

[0033] Figure 4 The pupil center of the embodiment of the present invention is located at Y Schematic diagram of the displacement on the axis with the frame number.

[0034] Figure 5 The pupil center positioning method of the embodiment of the present invention is X Schematic diagram of the displacement on the axis with the frame number.

[0035] Figure 6 The pupil center positioning method of the embodiment of the present invention is Y Schematic diagram of the displacement on the axis with the frame number.

[0036] Figure 7 The pupil center positioning method of the embodiment of the present invention is X Comparison of displacement on the axis with the number of frames.

[0037] Figure 8 The pupil center positioning method of the embodiment of the present invention is Y Comparison of displacement on the axis with the number of frames.

[0038] Fig. 9 This is a diagram showing the offset effect of the existing pupil center location algorithm during blinking. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention is further described in detail below in conjunction with the drawings and specific embodiments.

[0040] Please refer to Figure 1 The pupil trajectory generation method based on human eye video in an embodiment of the present invention includes steps 1 to 6.

[0041] Step 1: Perform a coarse filter on each frame of the human eye video, perform the first morphological filter, and eliminate the white spots formed by the infrared light irradiating the pupil;

[0042] Step 2: Calculate the minimum grayscale average of each frame image, and determine whether it is a valid frame based on the minimum grayscale average. If so, take the threshold E th =G min +ε, ε is the error compensation value, G min is the minimum grayscale average value;

[0043] Step 3: Using Eth as the threshold, perform binarization processing on the corresponding valid frame image;

[0044] Step 4: Perform a second morphological filter to remove eyebrows and small areas of skin pigment, and fill the binarized pupil area;

[0045] Step 5: Use the Laplacian operator to perform edge detection to obtain the outline of the pupil;

[0046] Step 6: Calculate the pupil lower point l t =(x t ,y t ).

[0047] The data used in the present invention is a human eye video collected by a head-mounted device with a built-in infrared camera. Each frame is a grayscale image, and the pupil is at the part with the lowest grayscale value.

[0048] Input: Human eye video V . Assume that the acquired video is V ={ v t : t =1, ..., T}, v t For its t Frames, each frame size is r OK c List; v t ( p )for v t In pixels p The gray value at .

[0049] Output: pupil trajectory L .

[0050] Initialization: From video V The beginning NFrame acquisition upper threshold E up and lower threshold E low

[0051] For t=1 to N

[0052] (1) Yes v t Morphological filtering is performed to eliminate the white spots formed by infrared light irradiating the pupil.

[0053] (2) Find v t The minimum grayscale average G min ( t ).

[0054] End for

[0055] Pick G =min t=1，...,NGmin ( t ),make E up = G + , E low = G − , is a predefined deviation value. In practice, N = 10, = 30.

[0056] For t =1 to T

[0057] (1) Yes v t Perform a coarse filter. Perform the first morphological filter to eliminate the white spots formed by the infrared light shining on the pupil.

[0058] (2) Valid frame judgment and its threshold E th Calculate . Using Algorithm 1, we can find v t The minimum grayscale average G min If you are not satisfied E low < G min < E up , it is considered as an invalid frame and the next frame is processed. Otherwise, it is considered as a valid frame and the threshold is taken E th =G min + ε , ε is the error compensation value. ε =6.

[0059] (3) Image binarization. E th The image is binarized with .

[0060] (4) Fine filtering: Perform a second morphological filtering to remove small areas such as eyebrows and skin pigments, and fill in the binarized pupil area.

[0061] (5) Edge detection: Use the Laplacian operator to perform edge detection and obtain the pupil outline.

[0062] (6) Calculate the pupil subaperture l t = ( x t , y t ). First find the bounding box of the pupil contour B ={ x left , x right , y bottom , y top}, x left、 x right Refers to the x coordinate values ​​of the left and right sides of the bounding box, y bottom , y top Refers to the y coordinate values ​​of the upper and lower edges of the bounding box, x t =( x left + x right ) / 2, the pupil axis is x = x t , the pupil axis and the bottom edge of the bounding box y = y bottom The intersection of is the lower point of the pupil. y t = y bottom .

[0063] End for

[0064] Output pupil trajectory L = { lt , t =1, ..., T}.

[0065] like Figure 2 As shown, the effect of pupillary pole positioning method in the blink process. Five consecutive frames are shown, among which the area of ​​frames 2 and 3 A < S ×10%, considered as invalid frame.

[0066] The present invention uses the lower pupil point (rather than the pupil center point) to represent the pupil position, and obtains the lower pupil point of each frame in the video to obtain the pupil trajectory. However, the existing method uses the pupil center to represent the pupil position, and it is difficult to accurately obtain the pupil position when a large area of ​​the pupil is blocked by the upper eyelid.

[0067] As an implementation method, the minimum grayscale average value of the image G min calculate

[0068] Given a grayscale image I , the size is r OK c List; I ( p ) is a pixel p Set a size of n × n Sliding window W ={0, 1, ..., n -1} × {0, 1, ..., n -1}. Calculate the point first p Window W p = W + p The average gray value of the diagonal elements G ( p )=å 0≤ k <nI ( p +( k, k )) / n , p Î D ={1,..., r - n +1}´{1,..., c - n +1} and then find I The minimum grayscale average G min =min pÎD G ( p ).

[0069] The present invention adopts the method of sliding window diagonal mean to adaptively calculate the threshold of each frame, which can adapt to different contrasts and accurately segment the pupil area without manual adjustment.

[0070] As an implementation method, the fast average gray value G ( p ), the present invention proposes a diagonal accumulation algorithm. Establish a size of ( r+ 1) × ( c+ 1) Matrix s , set s The first row and first column of are both zero.

[0071] s ( i +1, j +1) = I ( i , j )+ s ( i , j ), 1≤ i ≤ r ,1≤ j ≤ c ,

[0072] Then the average gray value at point (i, j) is G ( i , j ) = [ s ( i + n , j + n ) –s ( i , j )] / n , ( i , j ) D . Thus, calculating the diagonal average of a frame only requires [( r– 1) × ( c– 1) +( r–n +1) × ( c–n +1)] operations, while the original algorithm requires ( r–n +1) × ( c–n +1) × ( n– 1) operation. s(i,j) and I(i,j) are the grayscale values ​​of matrix s and grayscale image I respectively.

[0073] The present invention uses a diagonal accumulation method to quickly calculate the threshold value to be selected for the pupil.

[0074] like Figure 3-Figure 4 As shown, the horizontal axis is the frame sequence, the vertical axis is the pixel position, and x-invalid and y-invalid represent the misalignment results estimated at the invalid frame. The method for locating the lower pupil pole proposed by the present invention. (The horizontal axis is the frame sequence, the vertical axis is the pixel position) The present invention can automatically identify invalid frames and will not include the data of invalid frames. Figure 5-Figure 6 The comparison between the two is Figure 7-Figure 8 , respectively Figure 3-4 The invalid frame data in is removed, and the two are normalized by z-score.

[0075] Because the center point positioning method is easy to be affected by the pupil observable area during blinking. A Therefore, the positioning results of the pupil center point positioning method in the densely distributed invalid frames will have a large deviation in the positioning results of the previous and next frames. Fig. 9 shown.

[0076] The present invention adopts computer vision to accurately capture the eye movement trajectory, so that the observable area meets the requirements. A > S ×10%. The problems solved include: ① When a large area of ​​the pupil is blocked, the pupil positioning is inaccurate; ② The threshold for binary conversion cannot be adaptively generated; ③ Eye-closed frames or other invalid frames cannot be automatically removed; ④ There are abnormal values ​​in the pupil movement trajectory.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating pupil trajectories based on human eye video, characterized in that: include: Step 1: Perform a coarse filter on each frame of the human eye video, perform the first morphological filter, and eliminate the white spots formed by the infrared light irradiating the pupil; Step 2: Calculate the minimum grayscale average of each frame image, and determine whether it is a valid frame based on the minimum grayscale average. If so, take the threshold E th =G min +ε, ε is the error compensation value, G min is the minimum grayscale average value; Step 3: Using Eth as the threshold, perform binarization processing on the corresponding valid frame image; Step 4: Perform a second morphological filter to remove eyebrows and small areas of skin pigment, and fill the binarized pupil area; Step 5: Use the Laplacian operator to perform edge detection and obtain the outline of the pupil; Step 6: Calculate the lower pupil point l t =(x t ,y t ), first find the bounding box B of the pupil contour = {x left ,x right ,y bottom ,y top }, let x t =(x left +x right ) / 2, the pupil axis is x=x t , the pupil centerline and the bottom edge of the bounding box y = y bottom The intersection of is the lower point of the pupil, and y t =y bottom ; Finally, output pupil trajectory L = {l t ,t=1,...,T}.

2. The method for generating pupil trajectories based on human eye video according to claim 1, characterized in that: In step 2, the following method is used to calculate the minimum grayscale average G min : Given a grayscale image I with r rows and c columns; I(p) is the grayscale value at pixel p; set a sliding window W = {0, 1, ..., n-1} × {0, 1, ..., n-1} of size n × n; first calculate the window W at point p p =W+p's average grayscale value of the diagonal elements G(p) =∑ 0≤k<n I(p+(k,k)) / n, p∈D={1,...,r-n+1}×{1,...,c-n+1} and then find the minimum grayscale average value G of I min =min p∈D G(p) 3. The method for generating pupil trajectories based on human eye video according to claim 1, characterized in that: In step 2, the diagonal accumulation method is used to quickly calculate the candidate threshold of the pupil. Create a matrix s of size (r+1)×(c+1), set the first row and first column of s to zero, and let s(i+1,j+1)=I(i,j)+s(i,j),1≤i≤r,1≤j≤c, Then G(i,j)=[s(i+n,j+n)–s(i,j)] / n,(i,j)∈D.

Images