A pupil detection and blink recognition method and system based on horizontal Haar features
By using multi-stage image processing based on horizontal Haar features, combined with pupil-iris contrast features and a preset scoring mechanism, the robustness and accuracy issues of existing pupil detection and blink recognition methods are solved, achieving efficient pupil detection and blink recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2025-07-29
- Publication Date
- 2026-05-12
AI Technical Summary
Existing pupil detection methods are easily affected by factors such as changes in lighting, reflection of near-infrared light sources, and pupil obstruction, resulting in insufficient robustness. Blink recognition methods have a high rate of false positives and false negatives.
A multi-stage image processing and feature enhancement method based on horizontal Haar features is adopted to obtain pupil-iris contrast features at multiple scales, and combined with a preset scoring mechanism and threshold judgment to achieve pupil detection and blink recognition.
It improves the robustness of pupil detection and the accuracy of blink recognition to over 99%, reduces computational overhead, and significantly optimizes video frame processing speed.
Smart Images
Figure CN120954076B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D gaze estimation technology, and in particular relates to a pupil detection and blink recognition method and system based on horizontal Haar features. Background Technology
[0002] Pupil detection and blink recognition are key preliminary steps in eye movement analysis. Pupil detection involves extracting the position and outline information of the pupil from an eye image that includes the pupil, iris, sclera, and eyelid regions. Blink recognition is often used to assess attention levels and determine the availability of eye movement signals.
[0003] However, existing pupil detection methods are susceptible to interference from factors such as changes in lighting, shooting angle, near-infrared light reflection, and pupil occlusion when lacking prior knowledge, resulting in insufficient robustness. At the same time, most existing blink recognition methods rely on the loss of pupil signals as the basis for blink judgment, leading to high false positive and false negative rates.
[0004] Existing technical solutions include: Pupil detection methods based on grayscale thresholds: This method utilizes the characteristic that the grayscale value of the pupil region is lower than that of the iris and sclera. By setting a fixed or adaptive grayscale threshold, the pupil region in the image is segmented from the surrounding structures to achieve preliminary pupil localization; Pupil detection methods based on Haar features: This method utilizes the brightness contrast characteristics between the pupil and the iris to construct a Haar feature operator and select the region with the largest feature value as the pupil region; Pupil detection methods based on deep learning: This method constructs a convolutional neural network (CNN) or Transformer structure to automatically learn the spatial distribution features of the pupil from eye images and predict the pupil center position or outline; Pupil detection methods based on eye movement events: This method uses high-resolution images captured by an event camera... Frame rate visual change information is used to detect pupils by identifying the edge contours of positive and negative events caused by pupil displacement; an electrooculogram (EOG) based blink recognition method: this method detects changes in bioelectrical signals generated by eye movement, utilizes the potential difference between the cornea and retina, deploys electrodes around the eye to sense voltage fluctuations caused by blinking, and thus judges blinking behavior; a magnetic search coil (MSC) based blink recognition method: this method attaches a miniature coil to the eyelid and tracks its inductive response changes in a magnetic field to estimate the three-dimensional movement trajectory of the eyelid in real time, thus judging blinking behavior; an eye key point detection based blink recognition method: this method obtains the geometric position of the upper and lower eyelids through a facial key point detection model, performs dynamic change analysis by calculating the eye aspect ratio (EAR), and determines a blinking state when the EAR is below a threshold; a pupil feature change based blink recognition method: this method monitors changes in pupil size (PS) in real time, and the loss of PS signal is regarded as eyelid occlusion caused by blinking behavior.
[0005] The shortcomings of existing technologies are as follows: Gray-scale threshold-based pupil detection methods require manual adjustment of the gray-scale threshold and are susceptible to interference from corneal reflection from near-infrared light sources; Haar feature-based pupil detection methods require manual adjustment of the scale of the Haar feature operator and are susceptible to interference from corneal reflection from near-infrared light sources; Deep learning-based pupil detection methods have weak generalization ability for eye images acquired under different environmental conditions, and the model training cost and inference computation are high; Eye-movement event-based pupil detection methods rely on high-frame-rate event camera hardware, resulting in high costs; Electrooculography (EOG)-based blink recognition methods have low signal-to-noise ratios and severe baseline drift, requiring frequent calibration; Magnetic Search Coil (MSC)-based blink recognition methods are highly invasive, have poor wearing experience, and are only suitable for laboratory research environments; Eye key point detection-based blink recognition methods have individual differences in the eye aspect ratio (EAR) threshold, requiring manual adjustment; Blink recognition methods based on pupil feature changes rely on pupil feature detection results, have high false positive and false negative rates, and high computational costs. Therefore, there is an urgent need to propose a pupil detection and blink recognition method and system based on horizontal Haar features. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention proposes a pupil detection and blink recognition method and system based on horizontal Haar features. Through multi-stage image processing and feature enhancement, it achieves accurate detection of the pupil in the eye region and efficient discrimination of blinking behavior.
[0007] To achieve the above objectives, this invention provides a pupil detection and blink recognition method based on horizontal Haar features, comprising:
[0008] Obtain the current eye image, and obtain the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image;
[0009] According to the preset scoring mechanism, the scoring results of the strongest local pupil-iris contrast features at each scale are obtained, and the highest score among the scoring results is obtained.
[0010] According to the preset scoring threshold, if the highest score is lower than the scoring threshold, it is determined to be a blinking state; otherwise, the pupil image is obtained by expanding the image region corresponding to the highest score.
[0011] Obtain the fitting result of the pupil contour in the pupil image, and select the pupil image and its fitting result as the pupil detection result.
[0012] On the other hand, to achieve the above objectives, the present invention also provides a pupil detection and blink recognition system based on horizontal Haar features, comprising:
[0013] The first acquisition module is used to acquire the current eye image and acquire the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image.
[0014] The second scoring module is used to obtain the scoring results of the strongest local pupil-iris contrast features at each scale according to a preset scoring mechanism, and to obtain the highest score among the scoring results.
[0015] The third determination module is used to determine the blinking state if the highest score is lower than the preset scoring threshold, and otherwise to obtain the pupil image based on the image region corresponding to the highest score.
[0016] The fourth fitting module is used to obtain the fitting result of the pupil contour in the pupil image, and select the pupil image and its fitting result as the pupil detection result.
[0017] Technical advantages of this invention: This invention discloses a pupil detection and blink recognition method and system based on horizontal Haar features, with an accuracy of not less than 99% for pupil detection and blink recognition; it has high robustness to eye images acquired under different environmental conditions and corneal reflections from near-infrared light sources; it uses pupil-iris contrast features to recognize blinks, resulting in high accuracy and low computational overhead; and it significantly optimizes the processing speed of continuous video frames. Attached Figure Description
[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0019] Figure 1 This is a flowchart illustrating the pupil detection and blink recognition method based on horizontal Haar features according to an embodiment of the present invention.
[0020] Figure 2 This is a schematic diagram of the pupil detection and blink recognition system based on horizontal Haar features according to an embodiment of the present invention;
[0021] Figure 3 This is a pupil-iris contrast feature used for pupil detection in an embodiment of the present invention. Detailed Implementation
[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0023] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0024] like Figure 1 As shown, this embodiment provides a pupil detection and blink recognition method based on horizontal Haar features, including:
[0025] Obtain the current eye image, and obtain the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image;
[0026] According to the preset scoring mechanism, the scoring results of the strongest local pupil-iris contrast features at each scale are obtained, and the highest score among the scoring results is obtained.
[0027] According to the preset scoring threshold, if the highest score is lower than the scoring threshold, it is determined to be a blinking state; otherwise, the pupil image is obtained by expanding the image region corresponding to the highest score.
[0028] Obtain the fitting result of the pupil contour in the pupil image, and select the pupil image and its fitting result as the pupil detection result.
[0029] Furthermore, the horizontal Haar feature operator template consists of three adjacent rectangular regions, where the middle rectangular region is used to extract pupil grayscale features, the two rectangular regions on both sides are used to extract iris grayscale features, and the three together construct the grayscale contrast structure between the pupil and the iris.
[0030] Furthermore, the pupil-iris contrast feature response is calculated as follows:
[0031] Response = L - (1 + D / L) × D;
[0032] Where D represents pupil features and L represents iris features, this calculation method combines pupil-iris contrast features with pupil grayscale features.
[0033] Furthermore, the pupil features and iris features are calculated using different methods, wherein:
[0034] Pupil features are represented by the average gray value D = dark_sum / dark_count of the included dark pixels;
[0035] Iris features are represented by the average gray value of the included pixels, L = light_sum / light_area.
[0036] Furthermore, the Haar feature scoring mechanism uses the following formula to score candidate image regions:
[0037] Quality = (LD) / (L+D);
[0038] Where L represents the average gray value of the iris region and D represents the average gray value of the pupil region, this calculation method effectively suppresses the score of iris-sclera contrast features, thereby avoiding misidentification.
[0039] Furthermore, the determination method uses the optimal Haar feature as the classification criterion, including:
[0040] If the optimal Haar feature score is lower than the preset threshold, it is determined to be a blinking state.
[0041] Furthermore, the methods for expanding the coarse pupil localization area include:
[0042] Centered on the candidate region with the highest score, the pupil diameter pd at the current scale is extended by 2pd in both the horizontal and vertical directions to obtain a rectangular region of 2pd×2pd as the coarse localization region of the pupil. This extension method can cover the pupil region as much as possible.
[0043] Furthermore, the pupil contour fitting employs a full-iteration iterative ellipse fitting method, including:
[0044] Under the premise that the number of sampling points is not less than 5, the least squares method is used for ellipse fitting;
[0045] Calculate the normalized distance from each point to the fitted ellipse, and remove points whose distances are outside a preset threshold range as outliers.
[0046] Refit the ellipse in the remaining point set and repeat the above process until there are no outliers or the number of sampling points is less than 5.
[0047] Furthermore, the method includes a continuous video frame processing performance optimization mechanism, the steps of which include:
[0048] If the previous frame was not identified as a blink and the ellipse fitting result is valid, then the unique scale of the first detection module in the next frame is set to the length of the minor axis of the ellipse.
[0049] Furthermore, this method uses ellipticity and area as evaluation criteria, including:
[0050] If the ratio of the minor axis to the major axis of the ellipse is less than 0.5, it is considered an invalid fitting result.
[0051] If the scale of the first detection module in the current frame is derived from the input of the previous frame, and the ratio of the product of the minor axis and the major axis to the square of the scale is less than 0.75, then it is determined to be an invalid fitting result.
[0052] Furthermore, after selecting the pupil image and its fitting result as the pupil detection and blink recognition results, the pupil detection method includes:
[0053] Obtain the display page, create a display window for the display page, and display the pupil image region, fitting results, and evaluation score through the display window.
[0054] Specifically, the implementation process of this embodiment includes:
[0055] 1. Reading eye images
[0056] 1.1 Dataset
[0057] The dataset used subsets Interval, Lamp, and Distance from the Chinese Academy of Sciences Iris Image Database (CASIA-Iris V4). The Interval dataset contains iris images captured by a close-range iris camera, the Lamp dataset contains iris images acquired by a handheld iris sensor, and the Distance dataset contains iris images acquired by a remote image acquisition system. All three datasets are frontal viewpoints, with acquisition distances ranging from near to far.
[0058] The LPW (Labelledpupils in the wild) dataset was used, which contains near-eye images of the eyes acquired in an outdoor environment using the Pupil Core head-mounted eye tracker.
[0059] 1.2 Reading Eye Images
[0060] The eye image is read in grayscale mode. The original data of the CASIA-Iris-Distance dataset is a face image. The eye region needs to be obtained by using the haarcascade_eye model in the CascadeClassifier class of the OpenCV library and then cropped to obtain the eye image.
[0061] like Figure 3 The diagram shows the use of the multi-scale horizontal Haar feature operator to detect pupil-iris contrast features.
[0062] 2.1 Image Preprocessing
[0063] The Otsu thresholding method is used to obtain the grayscale threshold of the bright area of the eye image. The lookup table method is used to set the grayscale values of pixels in the eye image that are not less than this threshold to 0, and the corresponding dark area mask is created.
[0064] 2.2 Multi-scale horizontal Haar feature operator
[0065] A set of scales that increase exponentially is constructed based on preset gradient coefficients. A horizontal Haar feature operator is then created for all scales in the set based on a preset template.
[0066] Wherein, pupil diameter pd = scale * eye image height, and the horizontal Haar feature operator consists of the central pupil region and the iris regions on the left and right sides.
[0067] 2.3 Pupil-Iris Contrast Feature Detection
[0068] Pupil features included Dark pixels The average grayscale value D is represented by the formula dark_sum / dark_count.
[0069] Iris features included Pixels The average grayscale value L = light_sum / light_area is represented by this value;
[0070] Integral images are used to accelerate the calculation of dark_sum, light_sum, and dark_count at different scales;
[0071] Response function of pupil-iris contrast features:
[0072] Response = L - (1 + D / L) * D;
[0073] The eye image region corresponding to the strongest pupil-iris contrast feature at each scale is selected as the candidate region for the current scale.
[0074] 3. Evaluate the results of the first round of testing
[0075] The following formula is used to evaluate candidate regions at each scale:
[0076] Quality = (LD) / (L+D);
[0077] In this context, L and D represent the pupil and iris features corresponding to the candidate regions, respectively. This scoring mechanism effectively suppresses the scoring of the iris-sclera region.
[0078] 4. Determine the blink recognition result
[0079] The candidate region with the highest score is selected. If the score of the candidate region is lower than the set threshold, it is determined to be a blinking state. Otherwise, the eye image region of 2pd*2pd at the current scale is expanded from this region as the coarse localization region of the pupil.
[0080] 5. Fitting the pupil contour
[0081] The pupil image width was normalized to 200px, and then a 5x5 median filter was applied to remove salt and pepper noise.
[0082] The OTSU thresholding method is used to obtain the grayscale threshold of the dark area of the pupil image, and the threshold is used to obtain the preliminary binary region of the pupil.
[0083] Perform morphological closing operations on the binary region of the pupil using a 5x5 elliptical structuring element convolution kernel;
[0084] All closed outer contours are extracted using a contour detection algorithm, and the contour with the largest area is selected as the pupil boundary.
[0085] Calculate the vector angle from the centroid of the contour to all contour points, and obtain a set of sampling points with equal angular distribution according to the preset sampling angle step size;
[0086] Calculate the direction vector between sampling points, and use the cross product of adjacent vectors to identify and remove concave points; repeat the calculation until there are no concave points.
[0087] The cv2.fitEllipse function is used to perform least-squares ellipse fitting on the current set of sampling points. The normalized ellipse distance of each sampling point relative to the fitted ellipse is calculated. If a sampling point falls outside the distance range [0.98, 1.05], it is considered an outlier and removed. The fitting is repeated until there are no outliers.
[0088] The coordinates of the ellipse center and its major and minor axes are restored according to the initial scaling ratio to obtain the pupil contour fitting result.
[0089] Alternative Solution 1: A pupil center detection method based on convolutional neural networks (CNN)
[0090] The first CNN model filters images with closed eyes;
[0091] The second CNN model estimates the pupil center.
[0092] Alternative Solution 2: First detect the pupil, then determine blinking based on the fitted ellipse (pupil features). First, apply the Canny edge detector and filter the detected edges through morphological operations.
[0093] Then, an ellipse is fitted to the retained edges, some edges are removed using empirically defined heuristic rules, and an ellipse is selected as the pupil based on the roundness of the ellipse and the gray value of the area it contains.
[0094] like Figure 2 As shown, this embodiment also provides a pupil detection and blink recognition system based on horizontal Haar features, including:
[0095] First acquisition module: acquires the current eye image, and acquires the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image;
[0096] The second scoring module: Based on the preset scoring mechanism, obtain the scoring results of the strongest local pupil-iris contrast features at each scale, and obtain the highest score among the scoring results;
[0097] The third determination module: According to the preset scoring threshold, if the highest score is lower than the scoring threshold, it is determined to be a blinking state; otherwise, the pupil image is obtained by expanding the image region corresponding to the highest score.
[0098] The fourth fitting module obtains the fitting result of the pupil contour in the pupil image, and selects the pupil image and its fitting result as the pupil detection result.
[0099] Specifically, the input of the first acquisition module is an eye image, and the output is the strongest pupil-iris contrast feature detected by the Haar feature operator at each scale level and its corresponding image region.
[0100] The second evaluation module takes as input the strongest pupil-iris contrast features detected by the Haar feature operator at each scale level, and outputs the highest score result based on the preset scoring mechanism.
[0101] The input to the third judgment module is the highest score result based on the preset scoring mechanism, and the output is the blink judgment result or pupil image;
[0102] The fourth fitting module takes a pupil image as input and outputs the pupil contour fitting result.
[0103] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A pupil detection and blink recognition method based on horizontal Haar features, characterized in that, include: Obtain the current eye image, and obtain the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image; The horizontal Haar feature operator consists of three adjacent rectangular regions, where the middle rectangular region is used to extract pupil grayscale features, and the two rectangular regions on both sides are used to extract iris grayscale features. Together, they construct the grayscale contrast structure between the pupil and the iris. The pupil-iris contrast feature is calculated by combining pupil features and iris features to enhance the contrast response between the pupil and iris. According to the preset scoring mechanism, the scoring results of the strongest local pupil-iris contrast features at each scale are obtained, and the highest score among the scoring results is obtained. According to the preset scoring threshold, if the highest score is lower than the scoring threshold, it is determined to be a blinking state; otherwise, the pupil image is obtained by expanding the image region corresponding to the highest score. Obtain the fitting result of the pupil contour in the pupil image, and select the pupil image and its fitting result as the pupil detection result.
2. The pupil detection and blink recognition method based on horizontal Haar features as described in claim 1, characterized in that, Pupil features and iris features are calculated using different methods. Pupil features are represented by the average gray value of the dark pixels contained within them, while iris features are represented by the average gray value of the pixels contained within them.
3. The pupil detection and blink recognition method based on horizontal Haar features as described in claim 1, characterized in that, The scoring mechanism uses the ratio of the average gray value of the iris region to the average gray value of the pupil region to suppress interference from iris-sclera contrast features.
4. The pupil detection and blink recognition method based on horizontal Haar features as described in claim 1, characterized in that, The process of determining the blinking state includes determining the blinking state if the optimal Haar feature score is lower than a preset threshold.
5. The pupil detection and blink recognition method based on horizontal Haar features as described in claim 1, characterized in that, The pupil contour was fitted using a full-iteration iterative ellipse fitting method, which included removing outliers and refitting until there were no outliers or the number of sampling points was insufficient.
6. A system for pupil detection and blink recognition based on horizontal Haar features according to any one of claims 1-5, characterized in that, include: The first acquisition module is used to acquire the current eye image and acquire the strongest pupil-iris contrast features and their corresponding image regions generated by horizontal Haar feature operators at multiple scales in the eye image. The second scoring module is used to obtain the scoring results of the strongest local pupil-iris contrast features at each scale according to a preset scoring mechanism, and to obtain the highest score among the scoring results. The third determination module is used to determine the blinking state if the highest score is lower than the preset scoring threshold, and otherwise to obtain the pupil image based on the image region corresponding to the highest score. The fourth fitting module is used to obtain the fitting result of the pupil contour in the pupil image, and select the pupil image and its fitting result as the pupil detection result.