A real-time eye localization method based on target detection and target tracking competition framework
Through the competition framework for object detection and target tracking, combined with eye classifiers and particle filtering algorithms, the problem of high cost and insufficient accuracy of eye tracking equipment is solved, and the real-time positioning of eyeballs with high accuracy and robustness is achieved, reducing the equipment Cost and suitable for a wide range of tracking applications.
Patent Information
- Application Number
- CN202111288976.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-02
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-11-02
AI Technical Summary
The existing eye movement tracking equipment is costly and has potential damage to the eyes. The convenience of the head-mounted equipment is limited, and the positioning accuracy of the visible eye movement tracking system is insufficient.
Using a method based on the competition framework of object detection and target tracking, face images are intercepted through object detection technology, binocular position information is obtained using eye classifiers, and eye tracking is combined with particle filtering algorithm to obtain accurate eye position information in real time.
Improves the accuracy and robustness of real-time eye positioning, reduces equipment costs, is suitable for any target detection and target tracking technology, and does not require external equipment, and is suitable for eye tracking and small target tracking.
Smart Images

Figure CN114005172B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the fields of computer vision and machine learning, and in particular to a real-time eye positioning method based on a target detection and target tracking competition framework. Background Art
[0002] The current eye tracking devices mainly use infrared tracking devices and most of them are head-mounted. This solution can achieve very high accuracy, but the equipment is expensive. And according to surveys, long-term exposure of infrared rays to the eyes can cause damage to the eyes, which poses a potential threat to human health, so it is difficult to popularize and promote to the public. Moreover, head-mounted devices have many limitations in terms of convenience, which greatly limits their usage scenarios. Most of the current visible light eye tracking systems use only target detection or target tracking, which cannot guarantee the accuracy of positioning. Summary of the invention
[0003] In view of the deficiencies in the prior art, the present invention provides a real-time eye positioning method based on a target detection and target tracking competition framework.
[0004] The objective of the present invention is achieved through the following technical scheme: first, the face part of the image is cut out by target detection technology; the center of the human eye is quickly located in the cut-out face image by target detection technology, the corresponding coordinate information is obtained, and the confirmation of the tracking target is completed; the position information of the eyeball is obtained by the simultaneous operation of target detection and target tracking, and then the competitive framework of target detection and target tracking is used to obtain accurate eyeball position information in real time.
[0005] The specific steps include:
[0006] Step 1: Get the face image: load the video information and capture the face image through the target detection technology; that is, when the object in the video looks directly at the camera in the first frame, the face position information is obtained through the face classifier, and the face image is captured based on the position information.
[0007] Step 2: Obtain the position information of both eyes through target detection: For the image of the face captured in step 1, use the eye classifier to obtain the position information of both eyes;
[0008] Step 3: Obtain binocular position histogram information: for the binocular position information obtained in step 2, capture binocular images and save the histogram features of the binocular images;
[0009] Step 4: Obtain eye position information through target tracking: Use particle filter algorithm to track the eyeballs and obtain eyeball position information in real time;
[0010] Step 5: Realize real-time positioning of the eyeball position: Perform competition judgment based on the eyeball position information obtained through target detection and target tracking, and obtain accurate eyeball position information in real time.
[0011] Furthermore, in the process of obtaining the position information of both eyes by using the eye classifier in step 2, the position information of the left and right eyes is obtained by using the eye classifier respectively. Since the feature search and matching is performed on the face part, the recognition accuracy and recognition speed of the eye classifier can be improved. According to the position of the left and right eyes on the face, the false detection results of the classifier are eliminated by geometric judgment.
[0012] Furthermore, the step 3 of obtaining the binocular position histogram information specifically includes the following steps:
[0013] Step 3.1: Format conversion: convert the original RGB space binocular image into an HSV space image;
[0014] Step 3.2: Use OpenCv's CalcHist function to obtain the histogram of the image;
[0015] Step 3.3: Use the Normalize function to normalize the histogram;
[0016] Step 3.4: Save the histogram feature.
[0017] Furthermore, the step 4 uses a particle filter algorithm to perform real-time eye tracking, including the following process: converting the original RGB image into an HSV space image, sprinkling a certain number of particles near the eye to be tracked, each particle corresponds to a rectangular area, and assigning an initial weight to each particle, the initial weight of each particle is the same, and the higher the weight value of the particle, the more important the particle is for determining the position of the eye. Then, the histogram features of the rectangular area corresponding to each particle are obtained respectively, and compared with the eye histogram features saved in the first frame, and the weight of each particle is updated according to the similarity between the two features, and high-similarity particles are assigned high weights, and low-similarity particles are assigned low weights. The particle with the highest weight is the prediction result of eye tracking. Then, the particles are relocated according to the weight of each particle, and the number of high-weight particles is increased and the number of low-weight particles is reduced in the next frame to ensure that there are more and more. Then, the eye tracking prediction of the next frame is performed, and the cycle is repeated until the end of the video.
[0018] Further, step 5 includes the following steps:
[0019] Step 5.1: Save the rectangle information of the eye detection result and the rectangle information of the eye tracking result of the previous frame;
[0020] Step 5.2: Calculate the rectangular overlap ratio of the eye detection and eye tracking results of the current frame. The formula is as follows:
[0021]
[0022] Among them, ROL is the rectangular overlap rate, RD is the rectangular area of the detection result, and RT is the rectangular area of the tracking result;
[0023] Step 5.3: Based on the results of step 5.2, the following judgments are made:
[0024] (1) If ROL>0.5, the actual coordinates of the eyeball
[0025] ROI(x,y)=(RD_X / 2+RT_X / 2, RD_Y / 2+RT_Y / 2)
[0026] (2) If ROL ≤ 0.5, calculate the overlap ratio R1 between the detection result and the previous frame, and the overlap ratio R2 between the tracking result and the previous frame.
[0027]
[0028]
[0029] Wherein, t-1 represents the result of the previous frame, t represents the result of the current frame, R1 is the rectangle overlap ratio of the frames before and after target detection, R2 is the rectangle overlap ratio of the frames before and after target tracking, and a threshold R is set. If at least one of R1 and R2 is greater than or equal to the threshold R, then the following judgment is made based on the size relationship between R1 and R2: If R1 ≥ R2, then the actual eye coordinate ROI (x, y) = (RD_X, RD_Y);
[0030] If R1<R2, then the actual eye coordinates ROI(x,y)=(RT_X,RT_Y);
[0031] Among them, RD_X is the horizontal coordinate of the eye center of the target detection result; RD_Y is the vertical coordinate of the eye center of the target detection result; RT_X is the horizontal coordinate of the eye center of the target tracking result; RT_Y is the vertical coordinate of the eye center of the target tracking result;
[0032] Step 5.4: If both R1 and R2 are less than the threshold R, it is determined that the current frame eye detection and eye tracking have lost the target at the same time, and lost recovery is required. The specific process is as follows:
[0033] Step 5.4.1: First, perform eye detection in the rectangular area formed around the eye positions obtained at time t-1. If the eye position is successfully detected, use this position coordinate information as the actual eye coordinates of the current frame and re-track particles at this position.
[0034] At this time, the actual eye coordinates ROI (x, y) = (RD t-1 _X,RD t-1 _Y);
[0035] Step 5.4.2: If the eye position is not detected in step 5.4.1, it is considered that the target object has made a rapid head tilt, such as Figure 3 As shown. At this time, it is necessary to perform ellipse fitting on the face part, and then use principal component analysis to calculate the ellipse center coordinates (cx, cy) and rotation angle θ.
[0036] Step 5.4.3: Based on the eyeball center coordinates (x1, y1) of the first frame, the ellipse center coordinates (cx, cy) calculated in step 5.4.2, and the rotation angle θ, the rotated eyeball center coordinates (RO_X, RO_Y) are calculated using the following formula. These coordinates are used as the actual eyeball coordinates of the current frame and particle tracking is performed again.
[0037] RO_X=(x1-cx)*cosθ+(y1-cy)*sinθ+cx;
[0038] RO_Y=(y1-cy)*cosθ-(x1-cx)*sinθ+cy;
[0039] At this time, the actual coordinates of the eyeball ROI (x, y) = (RO_X, RO_Y).
[0040] The beneficial effects of the present invention are as follows: the present invention adopts the above scheme and uses a competitive framework of target detection and target tracking, which combines the respective advantages of target detection and target tracking, improves the accuracy and robustness of real-time eye positioning, and Figure 5 The test results show that the MOTA value is basically maintained above 95%. The higher the MOTA value, the less false detection, missed detection and ID exchange in multi-target tracking. The present invention does not need to rely on any external equipment, which reduces the expensive cost and is also very simple and convenient to implement. The framework is applicable to any target detection and target tracking technology. It can not only perform eye tracking, but also can be used to track any small target. The method has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of the method of the present invention.
[0042] Figure 2 It is a flow chart of eye detection and eye tracking for eye localization through competition.
[0043] Figure 3 Schematic diagram of the failure of detection and tracking when the target tilts its head quickly.
[0044] Figure 4 These are 4 frames of video eye tracking effects.
[0045] Figure 5 This is the evaluation result of the method of the present invention. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0047] like Figure 1 The method flow chart of the present invention is shown below. Figure 1 , explaining each step in detail:
[0048] Step S110, wherein the loaded video can be a saved video file or a real-time video stream. After the system starts tracking, the system uses a face classifier to detect the face of the first frame and extract it. The classifier used here is obtained by using Adaboost cascade classifier + Haar-like features to classify and train 12,000 different face materials.
[0049] Step S120, using the eye classifier to obtain the position information of both eyes in the captured face image and save the histogram features of the eye images to complete the confirmation of the tracking target. The eye classifier used for eye detection is obtained by classifying and training 8000 different eye materials using Adaboost cascade classifier + LBP features.
[0050] Step S130: The subsequent target detection continues to detect and return the position information of the human eye, and at the same time, the particle filter algorithm is used to track the eye in real time, and then the competition framework of target detection and target tracking is used to obtain accurate eye position information.
[0051] The image of the face part uses the eye classifier to obtain the position information of both eyes, including the following steps:
[0052] Step 1: Use the eye classifier to obtain multiple position information of the left and right eyes respectively. Since the feature search and matching is performed on the face part, the recognition accuracy and recognition speed of the eye classifier can be improved.
[0053] Step 2: Based on the positions of the left and right eyes on the face, use geometric judgment to eliminate the false detection results of the classifier.
[0054] Saving the histogram features of binocular images includes the following steps:
[0055] Step 1: Convert the original binocular image in RGB space into an HSV space image.
[0056] Step 2: Use OpenCV's CalcHist function to obtain the histogram of the image.
[0057] Step 3: Use the Normalize function to normalize the histogram.
[0058] Step 4: Save the histogram feature.
[0059] Real-time eye tracking using particle filter algorithm includes the following process:
[0060] Convert the original RGB image into an HSV space image, and sprinkle a certain number of particles near the eyeball to be tracked. Each particle corresponds to a rectangular area, and an initial weight is assigned to each particle. The initial weight of each particle is the same. The higher the weight value of the particle, the more important the particle is for determining the position of the eyeball. Then obtain the histogram features of the rectangular area corresponding to each particle, and compare it with the eyeball histogram features saved in the first frame. Update the weight of each particle according to the similarity between the two features. High-similarity particles are assigned high weights, and low-similarity particles are assigned low weights. The particle with the highest weight is the prediction result of eyeball tracking. Then relocate the particles according to the weight of each particle, increase the number of high-weight particles in the next frame, and reduce the number of low-weight particles to ensure that there are more and more. Then make eyeball tracking predictions for the next frame, and repeat until the end of a piece of material.
[0061] The content in the dotted box is the entire competition framework. According to the competition framework, the entire competition process is: calculating the rectangle overlap rate of eye detection and eye tracking (the rectangle information can be obtained from the position information),
[0062]
[0063] Among them, ROL is the rectangular overlap rate, RD is the rectangular area of the detection result, and RT is the rectangular area of the tracking result;
[0064] If the rectangle overlap rate ROL>0.5, it means that the results of eye detection and eye tracking are highly consistent, so the final result is obtained by weighting the two position information, and this weight can be freely adjusted and optimized according to the results.
[0065] Then the eyeball center coordinates ROI (x, y) = (RD_X / 2+RT_X / 2, RD_Y / 2+RT_Y / 2)
[0066] Figure 2 ROI_X and ROI_Y in the figure represent the horizontal and vertical coordinates of the final result, respectively. RD represents the eye detection result, and RT represents the eye tracking result.
[0067] If the overlap rate ROL ≤ 0.5, it means that the results of eye detection and eye tracking conflict. At this time, which result to use mainly depends on whether there is a big change from the result of the previous frame.
[0068] Calculate the overlap rate R1 between the inspection result and the previous frame, and the overlap rate R2 between the tracking result and the previous frame respectively.
[0069]
[0070]
[0071] Among them, t-1 represents the result of the previous frame, and t represents the result of the current frame. Figure 2 R1 represents the rectangle overlap ratio of eye detection and the previous frame result, and R2 represents the rectangle overlap ratio of eye tracking and the previous frame result. If R1>R2, the final result is based on the eye detection result, and if R1<R2, the final result is based on the eye tracking result.
[0072] That is, if R1≥R2, the actual eye coordinates ROI(x,y)=(RD_X, RD_Y);
[0073] If R1<R2, then the actual eye coordinates ROI(x,y)=(RT_X,RT_Y);
[0074] Among them, RD_X is the horizontal coordinate of the eye center of the target detection result; RD_Y is the vertical coordinate of the eye center of the target detection result; RT_X is the horizontal coordinate of the eye center of the target tracking result; RT_Y is the vertical coordinate of the eye center of the target tracking result.
[0075] If both R1 and R2 are less than the threshold R, it is determined that the current frame eye detection and eye tracking have lost the target at the same time, and lost recovery is required. The specific process is as follows:
[0076] First, eye detection is performed in the rectangular area formed around the eye positions obtained at time t-1. If the eye position is successfully detected, the position coordinate information is used as the actual eye coordinates of the current frame, and particle tracking is performed again at this position.
[0077] At this time, the actual eye coordinates ROI (x, y) = (RD t-1 _X,RD t-1 _Y);
[0078] If the eye position is not detected, it is considered that the target object has a rapid head tilt, such as Figure 3 As shown. At this time, it is necessary to perform ellipse fitting on the face part, and then use principal component analysis to calculate the ellipse center coordinates (cx, cy) and rotation angle θ.
[0079] According to the eyeball center coordinates (x1, y1) of the first frame, the calculated ellipse center coordinates (cx, cy) and the rotation angle θ, the rotated eyeball center coordinates (RO_X, RO_Y) are calculated by the following formula, and these coordinates are used as the actual eyeball coordinates of the current frame and particle tracking is performed again.
[0080] RO_X=(x1-cx)*cosθ+(y1-cy)*sinθ+Cx;
[0081] RO_y=(y1-cy)*cosθ-(x1-cx)*sinθ+Cy;
[0082] At this time, the actual coordinates of the eyeball ROI (x, y) = (RO_X, RO_Y).
[0083] In the experiment, eye tracking tests were conducted on three subjects. A visible light camera was connected to a computer, and the target subject was asked to make eye movements facing the camera. At the same time, the program implementing the algorithm was run to analyze and process the real-time images sent back by the camera. The experiment showed that the algorithm can effectively track the eyeballs and achieve high accuracy and robustness. Figure 4 This is a 4-frame tracking image of one of the experimental objects. Figure 5 These are some of our experimental data test results, where each row represents the test data of an object, and the three materials are three test objects. num is the number of frames of each material. Through Precision, Recall, and Accuracy, we can see that the accuracy of the algorithm is very high, basically maintained above 90%, and the two data of MOTA and IDF1 indicate that the algorithm has no ID conversion, indicating that the algorithm is very robust.
Claims
1. A real-time eye positioning method based on a target detection and target tracking competition framework, characterized in that: The method comprises the following steps: Step 1: Get the face image: load the video information and capture the face image through target detection technology; Step 2: Obtain the position information of both eyes through target detection: For the image of the face captured in step 1, use the eye classifier to obtain the position information of both eyes; Step 3: Obtain binocular position histogram information: Based on the binocular position information obtained in step 2, capture binocular images and save the histogram features of the binocular images; Step 4: Obtain eye position information through target tracking: Use particle filter algorithm to track the eyeballs and obtain eyeball position information in real time; Step 5: Realize real-time positioning of the eyeball position: Perform competition judgment based on the eyeball position information obtained by target detection and target tracking, and obtain accurate eyeball position information in real time; The specific steps include: Step 5.1: Save the rectangle information of the eye detection result and the rectangle information of the eye tracking result of the previous frame; Step 5.2: Calculate the rectangular overlap ratio of the eye detection and eye tracking results of the current frame. The formula is as follows: , Among them, ROL is the rectangular overlap rate, RD is the rectangular area of the detection result, and RT is the rectangular area of the tracking result; Step 5.3: Based on the results of step 5.2, the following judgments are made: If ROL ≥ 0.5, the actual coordinates of the eyeball ROI (x,y) = ( RD_X / 2+ RT_X / 2, RD_Y / 2+ RT_Y / 2) If ROL < 0.5, calculate the overlap ratio R1 between the detection result and the previous frame, and the overlap ratio R2 between the tracking result and the previous frame respectively; , Among them, t-1 represents the result of the previous frame, t represents the result of the current frame, R1 is the rectangle overlap rate of the frames before and after target detection, and R2 is the rectangle overlap rate of the frames before and after target tracking. Set the threshold R. If at least one of R1 and R2 is greater than or equal to the threshold R, then the following judgment is made based on the size relationship between R1 and R2: If R1 ≥ R2, then the actual eye coordinate ROI (x, y) = ( RD_X , RD_Y ); If R1<R2, then the actual eye coordinate ROI(x,y)=( RT_X , RT_Y ); Among them, RD_X is the horizontal coordinate of the eye center of the target detection result; RD_Y is the vertical coordinate of the eye center of the target detection result; RT_X is the horizontal coordinate of the eye center of the target tracking result; RT_Y is the vertical coordinate of the eye center of the target tracking result; Step 5.4: If both R1 and R2 are less than the threshold R, it is determined that the current frame eye detection and eye tracking have lost the target at the same time, and lost recovery is required; the specific process is as follows: Step 5.4.1: First, perform eye detection in the rectangular area formed around the eye positions obtained at time t-1. If the eye position is successfully detected, use this position coordinate information as the actual eye coordinates of the current frame, and re-track particles at this position; At this time, the actual eye coordinate ROI (x, y) = ( RD t-1 _X , RD t-1 _Y ); Step 5.4.2: If the eyeball position is not detected in step 5.4.1, it is considered that the target object has a rapid head tilt. At this time, an ellipse fitting is required for the face part, and then the principal component analysis is used to calculate the ellipse center coordinates (cx, cy) and the rotation angle q. Step 5.4.3: According to the eyeball center coordinates (x1, y1) of the first frame, the ellipse center coordinates (cx, cy) calculated in step 5.4.2, and the rotation angle q, the rotated eyeball center coordinates (RO_X, RO_Y) are calculated by the following formula, and these coordinates are used as the actual eyeball coordinates of the current frame and particle tracking is performed again; , At this time, the actual eye coordinate ROI (x, y) = ( RO_X , RO_Y ).
2. The method for real-time eye positioning based on target detection and target tracking competition framework according to claim 1, characterized in that: In the process of obtaining the position information of both eyes by the eye classifier in step 2, the position information of the left and right eyes is obtained respectively by the eye classifier, and the false detection results of the classifier are eliminated by geometric judgment according to the positions of the left and right eyes on the face.
3. The method for real-time eye positioning based on target detection and target tracking competition framework according to claim 1, characterized in that: The step 3 obtains binocular position histogram information, which specifically includes the following steps: Step 3.1: Format conversion: convert the original RGB space binocular image into an HSV space image; Step 3.2: Use OpenCv's CalcHist function to obtain the histogram of the image; Step 3.3: Use the Normalize function to normalize the histogram; Step 3.4: Save the histogram feature.
4. The method for real-time eye positioning based on target detection and target tracking competition framework according to claim 1, characterized in that: The step 4 uses the particle filter algorithm to track the eyeball in real time, including the following process: converting the original RGB image into an HSV space image, sprinkling a certain number of particles near the eyeball to be tracked, each particle corresponds to a rectangular area, and assigning an initial weight to each particle, the initial weight of each particle is the same, and the weight value represents the importance of the particle to the eyeball positioning; then obtaining the histogram features of the rectangular area corresponding to each particle, By comparing with the eye histogram features saved in the first frame, the weight of each particle is updated according to the similarity between the two features. Particles with high similarity are assigned high weights, and particles with low similarity are assigned low weights. The particle with the highest weight is the prediction result of eye tracking. Then, the particles are relocated according to the weight of each particle. The number of high-weight particles increases in the next frame, and the number of low-weight particles decreases in the next frame. Then, eye tracking prediction is performed for the next frame, and the cycle repeats until the end of the video.
