Pupil contour fitting method and system based on contour point sampling recombination strategy

Abstract

The invention discloses a pupil contour fitting method and system based on a contour point sampling recombination strategy, and the method comprises the steps: obtaining an eye image, and carrying out the preprocessing of the eye image, so as to remove noise; carrying out binarization processing on the preprocessed eye image by adopting a self-adaptive thresholding method, and extracting a pupil region; calculating an initial center point based on the contour of the pupil region, and sampling contour points in the radial direction; grouping the contour points obtained by sampling, and eliminating interfered contour points; searching an optimal contour point group combination by adopting a pruning optimization strategy, and fitting a pupil contour based on an ellipse fitting method; and screening an optimal fitting result by using prior data, and outputting a final pupil contour ellipse. According to the method, the performance consumption is obviously optimized, and the batch processing speed of videos and images under the same experiment condition is improved.

Description

Technical Field

[0001] This invention belongs to the field of pupil contour fitting technology, and particularly relates to a pupil contour fitting method and system based on a contour point sampling and reconstruction strategy. Background Technology

[0002] Pupil contour fitting for near-eye images is a crucial intermediate step in eye movement behavior acquisition and analysis, providing the basis for generating secondary eye movement data such as pupil diameter and gaze vector direction. Near-eye images typically refer to visual representations of the human eye region acquired using wearable or external fixed devices. Pupil contour fitting involves representing the position and geometry of the pupil's outer contour using ellipses in near-eye images. However, existing pupil contour fitting methods for near-eye images, without prior data correction, are susceptible to reduced fitting accuracy due to factors such as near-infrared light reflection and pupil occlusion. Furthermore, existing high-quality pupil contour fitting methods involve complex steps in extracting and processing near-eye image elements, demanding high performance and image quality. Existing pupil detection methods based on grayscale thresholds utilize the characteristic that the grayscale value of the pupil region is lower than that of the iris and sclera. By setting a fixed or adaptive grayscale threshold, the pupil region in the image is segmented from the surrounding structures, achieving preliminary pupil localization. Pupil detection methods based on grayscale thresholds: The grayscale threshold needs to be manually adjusted and is easily affected by corneal reflection from near-infrared light sources. Therefore, there is an urgent need to propose a pupil contour fitting method and system based on a contour point sampling and reconstruction strategy. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a pupil contour fitting method and system based on a contour point sampling and reconstruction strategy. This method enhances and samples data from a roughly defined pupil region image, enabling precise pupil localization and contour fitting in an efficient, concise, and robust manner.

[0004] To achieve the above objectives, this invention provides a pupil contour fitting method based on a contour point sampling and reconstruction strategy, comprising:

[0005] Acquire an eye image and preprocess the eye image to remove noise;

[0006] An adaptive thresholding method is used to binarize the preprocessed eye image and extract the pupil region;

[0007] The initial center point is calculated based on the contour of the pupil region, and contour points are sampled along the radial direction;

[0008] The sampled contour points are grouped, and disturbed contour points are removed.

[0009] The optimal combination of contour points is searched using a pruning optimization strategy, and the pupil contour is fitted based on an ellipse fitting method.

[0010] The optimal fit is selected using prior data, and the final pupil contour ellipse is output.

[0011] On the other hand, to achieve the above objectives, the present invention also provides a pupil contour fitting system based on a contour point sampling and reconstruction strategy, comprising:

[0012] The binarization processing module is used to preprocess the input eye image and extract the pupil region using an adaptive thresholding method.

[0013] The contour point sampling module is used to calculate the initial center point based on the contour of the pupil region and sample contour points along the radial direction.

[0014] The contour point segmentation module is used to group the sampled contour points and remove the disturbed contour points.

[0015] The contour point combination evaluation module is used to search for the optimal contour point combination using a pruning optimization strategy;

[0016] The ellipse fitting module is used to fit the pupil contour based on the optimal combination of contour points.

[0017] The strong prior module is used to filter the best fit result using prior data and output the final pupil contour ellipse.

[0018] Technical advantages of this invention: This invention discloses a pupil contour fitting method and system based on a contour point sampling and reconstruction strategy, achieving a pupil localization and contour fitting accuracy of no less than 99%; it exhibits high robustness to eye images acquired under different optical environmental conditions and corneal reflection from near-infrared light sources; it has the ability to complete pupil center localization and contour completion even when the pupil is partially obscured by the eyelid or other parts; the pupil contour fitting results include quality assessment; it has strong prior-aided fitting capabilities for batch processing of continuous videos and images; and it significantly optimizes performance consumption, improving the batch processing speed of videos and images under the same experimental conditions. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart illustrating the pupil contour fitting method based on a contour point sampling and reconstruction strategy according to an embodiment of the present invention.

[0021] Figure 2This is a schematic diagram of the pupil contour fitting system based on a contour point sampling and reconstruction strategy according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a method for edge recognition and contour point sampling based on the binarization result of the pupil region according to an embodiment of the present invention;

[0023] Figure 4 This is a logical diagram illustrating the reorganization evaluation of contour point groups based on binary breadth-first search in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] like Figure 1 As shown, this embodiment provides a pupil contour fitting method based on a contour point sampling and reconstruction strategy, including:

[0027] Acquire pupil images and preprocess the images to remove noise;

[0028] An adaptive thresholding method based on dark area pixel statistics is used to binarize the image and extract the pupil region;

[0029] Radial sampling is performed based on the pupil center to obtain the pupil boundary contour point set;

[0030] The contour point set is grouped, and a pruning optimization strategy is used to search for the optimal combination of contour point groups;

[0031] The combinations are screened based on prior constraints, and ellipse fitting is performed to obtain the pupil contour fitting result.

[0032] Furthermore, the contour with the largest area is selected from all detected contours as the main contour of the pupil.

[0033] Calculate the centroid of the main contour area and use the centroid of the contour area as the initial center coordinates of the pupil.

[0034] Furthermore, delete contour points in the contour point set that do not meet the standards, including:

[0035] Remove contour points that may be affected by corneal reflection from infrared light sources;

[0036] Delete the outline points that are concave inward relative to the pupil area;

[0037] Furthermore, remove contour points within the highlight area, including:

[0038] Binarized highlight regions of pupil images are obtained based on thresholding methods;

[0039] Use morphological methods to expand the highlight area;

[0040] Delete the outline points within the highlight area.

[0041] Furthermore, delete contour points that are concave inward relative to the pupil area, including:

[0042] Calculate the concavity and convexity of all contour points and their adjacent contour points;

[0043] Delete all depressions;

[0044] Repeat the above process until no more depressions are detected.

[0045] Furthermore, the contour point grouping process intelligently segments the contour point sequence based on angle change features, and performs the segmentation operation when a preset angle change or polarity change is detected.

[0046] Further pruning optimization strategies include:

[0047] A breadth-first search is used to traverse the combinatorial space of the contour point group;

[0048] Establish an evaluation mechanism based on ellipse fit goodness of fit;

[0049] Implement subset pruning rules to exclude combined paths with poor fitting quality.

[0050] Furthermore, elliptic evaluation methods include:

[0051] Goodness-of-fit evaluation of contour point set;

[0052] Prior continuity assessment.

[0053] Furthermore, the ellipse fit goodness evaluation quantifies the fit quality by calculating the variance of the sum of the distances from the contour points to the foci of the fitted ellipse, and sets a preset threshold as a screening criterion for the combination scheme.

[0054] Furthermore, it also includes a scheme selection step based on prior fitting results, which selects the optimal contour point group combination scheme by comparing the similarity between the current candidate scheme and the fitting results of previous frames.

[0055] Furthermore, after selecting the fitting results, the pupil contour fitting method includes:

[0056] Obtain the display page, create a display window for the display page, and display the pupil contour fitting results and confidence level through the display window.

[0057] Specifically,

[0058] Read the region of interest (ROI):

[0059] The study used subsets of the Chinese Academy of Sciences Iris Image Database (CASIA-Iris V4): Interval, Lamp, and Distance. The Interval dataset contains iris images captured by a near-field iris camera, the Lamp dataset contains iris images acquired by a handheld iris sensor, and the Distance dataset contains iris images acquired by a remote image acquisition system. All three datasets are frontal viewpoints, with acquisition distances ranging from near to far. The LPW (Labelled pupils in the wild) dataset was also used, which contains near-eye viewpoint eye images acquired in outdoor environments using the Pupil Core head-mounted eye tracker.

[0060] The eye image is read in grayscale mode. The original data of the CASIA-Iris-Distance dataset is a face image. The eye region needs to be obtained by using the haarcascade_eye model in the CascadeClassifier class of the OpenCV library and then cropped to obtain the eye image.

[0061] The method obtains a rough outline of the pupil region in the eye image by means of grayscale thresholding, Haar feature detection, etc., and then further crops it to obtain the base image that can be used by this method.

[0062] Binarize the base image using the Otsu adaptive thresholding method:

[0063] The input region of interest (ROI) eye image is filtered using a 5×5 median filter kernel to remove salt-and-pepper noise and isolated noise points, resulting in a smooth eye image.

[0064] The distribution range of dark area pixels was determined based on percentile statistics, using the 100 / 1.96 percentile as the criterion for dark area pixels, where 1.96 is the critical value of the standard normal distribution corresponding to the 95% confidence interval. A dark area pixel mask was created to identify pixel regions in the image whose gray values ​​are lower than the percentile threshold. The Otsu adaptive thresholding method was applied to the dark area pixel regions to automatically calculate the optimal binarization threshold dark_thresh, which can effectively distinguish the pupil dark area from other regions.

[0065] The preprocessed eye image is reverse binarized using the calculated dark area threshold; this process yields a binary image where the pupil area is white and other areas are black.

[0066] A 7×7 elliptical structuring element is created as the convolution kernel for morphological operations. Two iterative morphological opening operations are performed on the binarized pupil region. First, an erosion operation is performed to remove small noisy connected regions and burrs. Then, a dilation operation is performed to restore the size of the main pupil region, resulting in a smooth and continuous pupil contour region.

[0067] Contour point sampling and segmentation:

[0068] The OpenCV contour detection algorithm was used to extract the contour of the binary pupil region. The RETR_EXTERNAL mode was used to detect only the outermost contour. The CHAIN_APPROX_SIMPLE approximation method was used to compress the contour chain code. The contour with the largest area was selected as the main pupil contour from all detected contours.

[0069] Calculate the centroid of the area of ​​the selected contour, and use the centroid of the contour area as the initial center coordinates of the pupil;

[0070] like Figure 3 As shown, the angle sampling step size angle_step is set to 15 degrees, and 24 sampling angles are generated at equal intervals within the range of 0 to 360 degrees; the angles are converted into unit direction vectors to construct a set of radial sampling directions, each direction vector containing x and y components;

[0071] Starting from the center of the pupil, search outwards in the current direction with a fixed step size (8 pixels) until the image boundary or pupil region boundary is encountered; record the coordinates of the valid point as the pupil boundary point in the current radial direction;

[0072] Concave point detection is performed on the sampled contour point sequence, and the direction vector between adjacent contour points is calculated; the cross product of the vectors is then used:

[0073] cross_products=vec1[:,0]*vec2[:,1]-vec1[:,1]*vec2[:,0];

[0074] Determine the convexity or concavity of the contour; when the cross product value is less than or equal to 0, the point is determined to be a concave point and needs to be removed from the contour point set.

[0075] The original region of interest image is subjected to high-brightness thresholding to obtain the spot region mask;

[0076] The 7×7 elliptical structuring element is used to perform an expansion operation on the spot area to obtain the expanded spot mask.

[0077] The size of the neighborhood inspection window is calculated based on the pupil diameter parameter pd. Spot interference is checked for each contour point obtained by radial sampling. If any pixel in the neighborhood is marked by a spot mask, the contour point is determined to be affected by spot interference.

[0078] The contour points that are not affected by the light spot are retained to form the optimized contour point set.

[0079] Intelligent segmentation of contour point sequences is performed based on angle threshold and polarity change. When a sharp corner is detected (the absolute value of the angle is less than the threshold) or the angle polarity changes, contour segmentation operation is performed.

[0080] Whether to merge the first and last contour segments depends on whether the starting point meets the segmentation criteria.

[0081] like Figure 4 The outline point group combination and evaluation are shown below:

[0082] An ellipse fit goodness evaluator is constructed to calculate the variance of the sum of the distances between each point in the contour point group and the focal points of the fitted ellipse, thereby evaluating the goodness of fit; the effectiveness of the combination scheme is determined based on a threshold.

[0083] A breadth-first search strategy is used to explore the optimal combination of segmented contour segments; an ellipse fitting goodness threshold of 2.8 is set as the evaluation criterion for combination schemes; the search queue is initialized starting from the longest contour point group.

[0084] Implement a pruning mechanism based on subset relationships. If a subset of a certain contour point group combination has been proven to have poor fitting effect, then skip all combinations containing that subset. Extract a contour point set for each candidate combination, requiring at least 5 points to meet the minimum constraint condition for ellipse fitting.

[0085] For contour point groups with good fitting results, recursively expand by adding new contour point groups; maintain the order of contour point group addition, ensure that the index of the newly added contour point group is greater than the maximum index in the current combination, and avoid repeated combinations;

[0086] The pruning set is dynamically maintained, and poorly fitting combination paths are added to the pruning list to improve subsequent search efficiency; all contour point combination schemes that meet the ellipse fitting requirements are returned, providing an optimized point set basis for subsequent ellipse fitting.

[0087] Solution selection and fitting based on prior data:

[0088] Redundancy checks are performed on all contour point combination schemes that meet the ellipse fitting requirements obtained by breadth-first search; a subset filtering algorithm is implemented to identify and eliminate redundant combinations that are completely included by other schemes.

[0089] Based on prior knowledge of pupil geometry, the filtered contour point group combination is further screened to remove extreme elliptical shapes that obviously do not conform to physiological characteristics; the center position of the ellipse is checked to ensure that it is within a reasonable range of the region of interest, so as to ensure the spatial consistency of the fitting results.

[0090] A fitting result with sufficient goodness of fit and a proportion of trusted contour point groups exceeding a certain threshold in the final fitted ellipse circumference is defined as excellent. In the case of multiple candidate contour point group combinations, the goodness of fit of each combination with the excellent fitting result of the previous frame is evaluated using an ellipse fitting evaluator, and the combination with the highest score is selected as the contour point group combination used for the final fitting.

[0091] Ellipse fitting:

[0092] The cv2.fitEllipse function is used to perform least-squares ellipse fitting on the current contour point group combination scheme to obtain the pupil center position and contour ellipse parameters.

[0093] Alternative Solution 1: A pupil contour fitting method based on the Canny edge detection and reconstruction strategy

[0094] First, the presence of reflection in the image is analyzed based on the peak values ​​in the intensity histogram. If the image is determined to have no reflection, it is then thresholded using adaptive parameters, and the pupil position is roughly estimated using an angle integral projection function; this position is subsequently fine-tuned based on the surrounding grayscale values.

[0095] If a reflection is detected, the Canny edge detector is used, and the resulting edges are filtered by morphological operations; then, an ellipse is fitted to the retained edges, and the ellipse containing the darkest gray value is selected as the pupil fitting result.

[0096] Alternative Solution 2: A pupil contour fitting method based on a contour sampling point mapping strategy

[0097] The pupil outline is divided into upper and lower parts at its maximum width. If the upper part of the pupil is obscured by the eyelid, the sampling points of the lower part are mapped to the upper part to complete the elliptical outline fitting.

[0098] like Figure 2 As shown, this embodiment also provides a pupil contour fitting system based on a contour point sampling and reconstruction strategy, including:

[0099] Binarization module: The threshold is obtained using the OTSU thresholding method to obtain a binary image of the pupil region based on threshold segmentation, and morphological opening operation is performed on the pupil region using an elliptical convolution kernel to make the contour of the binary image smoother.

[0100] Contour point sampling module: Based on the binarized eye image, the largest pupil area is selected, and then contour points are obtained by scanning and sampling at set angle intervals according to the center point obtained by circumscribing the pupil area contour. At the same time, contour points affected by near-eye infrared light reflection and relatively concave contour points are removed;

[0101] Contour point segmentation module: The sampled contour points are segmented according to the angle formed by three adjacent points to form several contour point groups;

[0102] Contour point group combination evaluation module: This module attempts different combinations of contour point groups using a breadth-first search of a binary search tree, and then uses the ellipse fitting module to perform ellipse fitting on each combination, evaluating the goodness of fit. Sub-objects of the binary tree for combinations with low goodness of fit are pruned, and the best contour point group combination is retained.

[0103] Ellipse Fitting Module: Fits an ellipse based on the received contour point set, and evaluates the quality of the ellipse and the goodness of fit of the contour point set.

[0104] Strong Prior Module: In the continuous eye image stream processing, based on the ellipse fitting result of the previous frame with high confidence, the most reasonable scheme is selected from multiple excellent contour point combination schemes.

[0105] The specific implementation process includes: the input of the binarization processing module is a roughly defined image of the pupil and its surrounding area, and the output is a binarized image of the pupil region based on threshold segmentation;

[0106] The input to the contour point sampling module is a binarized image of the pupil region, and the output is a continuous pupil contour point.

[0107] The input to the contour point segmentation module is continuous pupil contour points, and the output is several separate contour point groups;

[0108] The contour point combination evaluation module takes several contour point groups as input and outputs one or more contour point group combination schemes.

[0109] The ellipse fitting module takes a combination of contour points as input and outputs an ellipse and the goodness of fit.

[0110] The strong prior module takes the ellipse fitting result of the previous frame and several contour point combination schemes as input, and outputs an optimal contour point combination scheme.

[0111] This invention discloses a pupil contour fitting method and system based on a contour point sampling and reconstruction strategy, achieving a pupil localization and contour fitting accuracy of no less than 99%. It exhibits high robustness to eye images acquired under different optical conditions and corneal reflections from near-infrared light sources. It has the ability to complete pupil center localization and contour completion even when the pupil is partially obscured by the eyelid or other parts. The pupil contour fitting results include quality assessment. It possesses strong prior-aided fitting capabilities for batch processing of continuous videos and images. It significantly optimizes performance consumption and improves the batch processing speed of videos and images under the same experimental conditions.

[0112] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pupil contour fitting method based on a contour point sampling and reconstruction strategy, characterized in that, include: Acquire an eye image and preprocess the eye image to remove noise; An adaptive thresholding method is used to binarize the preprocessed eye image and extract the pupil region; The initial center point is calculated based on the contour of the pupil region, and contour points are sampled along the radial direction; The sampled contour points are grouped, and disturbed contour points are removed. The optimal combination of contour points is searched using a pruning optimization strategy, and the pupil contour is fitted based on an ellipse fitting method. The optimal fit is selected using prior data, and the final pupil contour ellipse is output.

2. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 1, characterized in that, The process of extracting the pupil area includes: Detect all contours from the binarized image and select the contour with the largest area as the main pupil contour; Calculate the centroid of the area of ​​the main contour and use it as the initial center coordinates of the pupil.

3. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 1, characterized in that, The process of removing disturbed contour points includes: Remove contour points affected by corneal reflection from infrared light sources; Delete the outline points that are concave inward relative to the pupil area.

4. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 3, characterized in that, The process of removing contour points affected by corneal reflection from an infrared light source includes: Highlight regions in eye images are obtained using a thresholding method; The highlight region was expanded using morphological methods; Delete the outline points located within the highlight area.

5. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 3, characterized in that, The process of deleting contour points that are concave inward relative to the pupil area includes: Calculate the concavity and convexity of all contour points and their adjacent contour points; Delete all depressions; Repeat the above process until no dents are detected.

6. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 1, characterized in that, The process of grouping the sampled contour points includes: Intelligent segmentation is performed based on the angular variation features between adjacent contour points; When a change in angle or polarity is detected under preset conditions, a segmentation operation is performed.

7. The pupil contour fitting method based on contour point sampling and reconstruction strategy as described in claim 1, characterized in that, The process of pruning optimization strategy includes: A breadth-first search is used to traverse the combinatorial space of the contour point group; Establish an evaluation mechanism based on ellipse fit goodness of fit; Implement subset pruning rules to exclude combined paths with poor fitting quality.

8. A system for pupil contour fitting based on a contour point sampling and reconstruction strategy according to any one of claims 1-7, characterized in that, include: The binarization processing module is used to preprocess the input eye image and extract the pupil region using an adaptive thresholding method. The contour point sampling module is used to calculate the initial center point based on the contour of the pupil region and sample contour points along the radial direction. The contour point segmentation module is used to group the sampled contour points and remove the disturbed contour points. The contour point combination evaluation module is used to search for the optimal contour point combination using a pruning optimization strategy; The ellipse fitting module is used to fit the pupil contour based on the optimal combination of contour points. The strong prior module is used to filter the best fit result using prior data and output the final pupil contour ellipse.

Images