A method for extracting the contour of the near-eye pupil

Abstract

This application belongs to the field of eye-tracking technology. This application provides a method for extracting the contour of the near-eye pupil. The embodiments of this disclosure convert eye images acquired by a high-speed eye camera into grayscale images, then perform coarse pupil localization to calculate the approximate position of the pupil, and finally perform edge detection on the pupil position region. The detected points are then fitted with an ellipse, and the parameters of the ellipse are output. This method features fast detection speed and high accuracy. By reducing the resolution for rapid pupil position detection and extracting the image of the region near the pupil center for edge detection, rapid extraction of the pupil contour is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of eye-tracking technology, and more particularly to a method for extracting the contour of the near-eye pupil. Background Technology

[0002] Eye-tracking algorithms are a technology that tracks eye movements by measuring eye movements and pupil changes, and analyzes information such as user attention and points of interest based on this data. The principle involves illuminating the human eye with low-power active infrared light, using a sensor array to capture near-infrared images, processing the near-infrared signals to obtain eye features, and thus estimating the gaze point and annotation direction.

[0003] Pupil contour extraction is a crucial part of eye-tracking algorithms. The speed and accuracy of contour extraction directly affect the performance and effectiveness of eye-tracking algorithms. However, due to image noise and infrared light spots, it is not easy to extract the pupil contour quickly and stably.

[0004] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0005] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] The purpose of this disclosure is to provide a method for extracting the contour of the near-eye pupil, thereby overcoming, to at least some extent, one or more problems caused by the limitations and defects of related technologies.

[0007] According to embodiments of this disclosure, a method for extracting the contour of the pupil near the eye is provided, comprising: The compressed video format image captured by the high-speed eye camera is decoded into YUV format, and the Y component is directly used as the input image img1; The resolution of the grayscale image is reduced to a preset low resolution to obtain image img2; After performing median filtering on the image img2, it is divided into multiple blocks. The minimum pixel value is calculated for each block, and a threshold is set based on the minimum value for binarization to obtain a binary image. Perform connected component analysis on the binary image, exclude connected components connected to the image edges, determine the pupil connected component based on the average gray value of the connected components, and obtain its coarse localization of the pupil center; Using the coarsely located pupil center as the center, a region of a predetermined size is cropped from image img1 to serve as pupil image img3; Using the coarsely located pupil center as the center, a range image of a predetermined size is extracted from image img1. The nearest neighbor interpolation algorithm is used to enlarge the range image to the same size as the pupil image img3. Then, the range image is fused with the pupil image img3 at the pixel level. Finally, the fused image is subjected to median filtering to obtain the fused image img4. After binarizing the fused image img4, edge detection is performed to obtain edge detection points; The edge detection points are fitted using the least squares method to calculate the ellipse parameters, which include the center coordinates, major and minor axes, and rotation angle. The ellipse parameters are validated; if the ellipse parameters do not conform to the pupil projection characteristics, the detection fails; if the ellipse parameters conform to the pupil projection characteristics, the ellipse parameters are output.

[0008] Furthermore, the resolution is reduced using the nearest neighbor interpolation algorithm, lowering the resolution from 640... 480 reduced to 160 120.

[0009] Furthermore, median filtering uses 11 11 filter cores.

[0010] Further, the steps of performing median filtering on image img2, dividing it into multiple blocks, calculating the minimum pixel value for each block, and performing binarization processing based on a threshold set according to the minimum value to obtain a binary image include: The median-filtered image img2 is segmented into 4 3 pieces, each piece is 40 cm in size 40; For each block, find the minimum value of the pixel, minValue; A threshold is set based on the minimum value minValue and a preset offset value, and then binarization is performed to obtain a binary image.

[0011] Further, the steps of performing connected component analysis on the binary image, excluding connected components connected to the image edges, determining the pupil connected component based on the average gray value of the connected components, and obtaining its coarse localization of the pupil center include: Perform connected component analysis on the binary image to obtain all connected components; Exclude connected components with connected edges, sort the remaining connected components according to their average gray value, and determine the connected component with the smallest average gray value as the pupil connected component. The center of the pupil's connected domain is taken as the coarse location of the pupil center.

[0012] Furthermore, the predetermined size of the region image is 200. 200 pixels.

[0013] Further, the steps of extracting a range image of a predetermined size from image img1, using the coarsely located pupil center as the center, enlarging the range image to the same size as pupil image img3 using a nearest neighbor interpolation algorithm, performing pixel-level fusion with pupil image img3, and then performing median filtering on the fused image to obtain fused image img4, include: Using the coarsely located pupil center as the center, extract 50 pixels from image img1. A 50-pixel range image; The range image was magnified to 200 using the nearest neighbor interpolation algorithm. 200 pixels; The minimum value of the corresponding pixels in the pupil image (img3) and the magnified range image is taken and then fused at the pixel level. Take advantage of 5 5. Median filtering is applied to the fused image to obtain the fused image img4.

[0014] Furthermore, the step of performing edge detection after binarizing the fused image img4 to obtain edge detection points includes: Binarize the fused image img4; the threshold is the minimum gray value + 10, and the largest connected component is taken as the pupil image img5; Edge detection was performed on the pupil image img5 using the Canny operator to obtain edge detection points; the low threshold of the Canny operator was 50, the high threshold was 150, and the aperture size of the Sobel operator was 3.

[0015] Furthermore, the steps for verifying the ellipse parameters include: If the ratio of the major axis to the minor axis of the ellipse is greater than 3 or the major axis is greater than 100 pixels, it is determined that it does not conform to the pupil projection feature and the detection fails. If the case is otherwise, it is determined to conform to the pupil projection characteristics, and the ellipse parameters are output.

[0016] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the above method, on the one hand, converts the eye image acquired by the high-speed eye camera into a grayscale image, then performs coarse pupil localization to calculate the approximate position of the pupil, and finally performs edge detection on the pupil position region, fitting the detected points to an ellipse and outputting the parameters of the ellipse. On the other hand, this method has the characteristics of fast detection speed and high accuracy. By reducing the resolution for rapid pupil position detection and extracting the image of the region near the pupil center for edge detection, rapid extraction of the pupil contour is achieved. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] Figure 1 The diagram illustrates the steps of a method for extracting the contour of the pupil near the eye in an exemplary embodiment of this disclosure. Figure 2 A detailed flowchart of the method for extracting the contour of the near-eye pupil in an exemplary embodiment of this disclosure is shown; Figure 3 A schematic diagram showing an eye image acquired in an exemplary embodiment of this disclosure; Figure 4 This diagram illustrates an image img2 obtained after reducing its resolution in an exemplary embodiment of this disclosure. Figure 5 A schematic diagram showing the median-filtered image img2 in an exemplary embodiment of this disclosure; Figure 6 This diagram illustrates the segmented binarized image img2 in an exemplary embodiment of this disclosure; Figure 7 This diagram illustrates the coarse positioning coordinate center detected by the pupil position in an exemplary embodiment of this disclosure. Figure 8 This diagram illustrates a pupil image (img3) extracted from a pupil center image in an exemplary embodiment of this disclosure. Figure 9 A schematic diagram showing the fused image img4 in an exemplary embodiment of this disclosure; Figure 10 This diagram illustrates the binarization of the fused image img4 in an exemplary embodiment of this disclosure. Figure 11 A schematic diagram of Canny edge detection in an exemplary embodiment of this disclosure is shown; Figure 12 This diagram illustrates the final recognition effect in an exemplary embodiment of the present disclosure. Detailed Implementation

[0019] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0020] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0021] This example implementation provides a method for extracting the contour of the near-eye pupil. (Reference) Figure 1 As shown, the method for extracting the near-eye pupil contour may include: Step S101: Decode the compressed video format image acquired by the high-speed eye camera into YUV format, and directly use the Y component as the input image img1; Step S102: Reduce the resolution of the grayscale image to a preset low resolution to obtain image img2; Step S103: After performing median filtering on image img2, it is divided into multiple blocks. The minimum pixel value is calculated for each block, and a threshold is set according to the minimum value for binarization to obtain a binary image. Step S104: Perform connected component analysis on the binary image, exclude connected components connected to the image edges, determine the pupil connected component based on the average gray value of the connected components, and obtain its coarse localization of the pupil center. Step S105: Using the coarsely located pupil center as the center, extract a region of a predetermined size from image img1 as pupil image img3; Step S106: Using the coarsely located pupil center as the center, extract a range image of a predetermined size from image img1. Use the nearest neighbor interpolation algorithm to enlarge the range image to the same size as the pupil image img3, then perform pixel-level fusion with the pupil image img3, and then perform median filtering on the fused image to obtain the fused image img4. Step S107: After binarizing the fused image img4, perform edge detection to obtain edge detection points; Step S108: Perform least squares fitting on the edge detection points to calculate the ellipse parameters; where the ellipse parameters include the center coordinates, major and minor axes, and rotation angle; Step S109: Verify the ellipse parameters; if the ellipse parameters do not conform to the pupil projection characteristics, output "detection failed"; if the ellipse parameters conform to the pupil projection characteristics, output the ellipse parameters.

[0022] The aforementioned method for extracting the pupil contour near the eye involves two main aspects. First, the eye image captured by a high-speed eye camera is converted into a grayscale image. Then, coarse pupil localization is performed to calculate the approximate pupil position. Finally, edge detection is conducted on the pupil region, and the detected points are fitted with an ellipse, outputting the ellipse parameters. Second, this method is characterized by high detection speed and high accuracy. By reducing the resolution for rapid pupil position detection and extracting the image near the pupil center for edge detection, rapid extraction of the pupil contour is achieved.

[0023] Below, we will refer to Figures 1 to 12 The steps of the above-described method for extracting the near-eye pupil contour in this example embodiment will be described in more detail.

[0024] In one embodiment, this embodiment discloses a method for extracting the contour of the near-eye pupil. For example... Figure 2 The diagram shows the flowchart of the method for extracting the contour of the near-eye pupil. It includes the following steps: Step S101: Image Acquisition. The raw data output by the high-speed eye camera is in MPEG compressed video format with a resolution of 640. 480, decoded using RK3588 hardware, the decoded image format is YUV420. Using the Y value directly as the input image, image img1 is obtained; Step S102: Reduce resolution. Use the nearest neighbor interpolation algorithm to reduce the resolution from the original 640. 480 reduced to 160 120, resulting in image img2; Step S103: Median filtering. Using 11... An 11-core filter is used to perform median filtering on img2 to remove noise and glare from the image. Piecewise binarization. The median-filtered image is divided into 4 segments. 3 pieces, each piece is 40 cm in size 40. For each block, find the minimum value minValue of the pixels, and use minValue+10 as the threshold for binarization. Step S104: Pupil position detection. After segmented binarization, connected component analysis is performed on the generated binary image to obtain all connected components. First, connected components with connecting edges are excluded. The remaining connected components are sorted according to their average gray level. The connected component with the smallest average gray level is the pupil, and its center position is taken as the coarse localization center. Step S105: Pupil Center Image Extraction. Based on the coarsely extracted pupil center position (x, y), calculate the pupil center coordinates (4x, 4y) in the original image img1. Using the (4x, 4y) coordinates as the center, crop a 200-degree image. A 200-pixel image is used as the pupil image (img3). Step S106: Spot noise removal. Coarsely locate the pupil center (x, y) 50 from image img2. A 50-pixel image was magnified to 200 using the nearest neighbor interpolation algorithm. 200, and merge with the pupil center image img3 (take the minimum pixel value of the two images), then use 5 Median filtering is performed using 5 filter kernels; Step S107: Edge detection. Edge detection is performed using the Canny operator with a low threshold of 50 and a high threshold of 150. The aperture size of the Sobel operator is 3, and edge detection points are obtained. Step S108: Ellipse Fitting. Binarize the fused image img4; the threshold is the minimum grayscale value + 10, and the largest connected component is taken as the pupil image img5. Use edge detection points to perform least-squares fitting to calculate the ellipse's center coordinates, major and minor axes, and rotation angle. Step S109: Output parameters. The calculated ellipse parameters are validated. If any parameter clearly does not conform to the pupil projection characteristics (major-minor axis ratio greater than 3, major axis greater than 100 pixels), the detection fails. Otherwise, the ellipse parameters are output.

[0025] In one specific embodiment, the images acquired by the high-speed eye camera are as follows: Figure 3 As shown.

[0026] The resolution was reduced using the nearest neighbor interpolation algorithm, from the original 640. 480 reduced to 160 120, the resulting image img2 is as follows Figure 4 As shown.

[0027] Use 11 An 11-kernel filter is used to perform median filtering on image img2 to remove noise and glare, resulting in the image shown below. Figure 5 As shown.

[0028] like Figure 6 As shown, the image after median filtering is divided into 4... 3 pieces, each piece is 40 cm in size 40. For each block, find the minimum pixel value minValue, and use minValue+10 as the threshold for binarization.

[0029] like Figure 7As shown, connected component analysis is performed on the generated binary image to obtain all connected components. First, connected components with connecting edges are excluded. The remaining connected components are sorted according to their average gray level, and the connected component with the smallest average gray level is the pupil. Its center position is taken as the coarse localization center.

[0030] Based on the pupil center (x, y) extracted from the coarse localization, the coordinates (4x, 4y) of the pupil center in the original image img1 are calculated. A 200-degree section is then cropped from the (4x, 4y) coordinates. The 200-pixel image is used as the pupil image (img3).

[0031] Obtain the coarsely located pupil center (x, y) 50 from image img2. A 50-pixel image was magnified to 200 using the nearest neighbor interpolation algorithm. 200, such as Figure 8 As shown; then it is fused with the pupil center image img3 (taking the minimum pixel value of the two images), the fused image is as follows. Figure 9 As shown. Then use 5 Median filtering is performed using 5 filter kernels.

[0032] Binarize the fused image img4; the threshold is the minimum gray value + 10, and the largest connected component is taken as the pupil image img5, as shown below. Figure 10 As shown.

[0033] like Figure 11 As shown, edge detection is performed using the Canny operator with a low threshold of 50 and a high threshold of 150. The aperture size of the Sobel operator is 3 to obtain edge detection points.

[0034] The parameters of the obtained ellipse are: rotation angle (angle between the minor axis and the X-axis, counterclockwise): 84.972; major axis: 60.768; minor axis: 48.42; center coordinates: (316.8, 225.526). The parameters meet the requirements. Figure 12 The image shown is the final recognition result.

[0035] The aforementioned method for extracting the pupil contour near the eye involves two main aspects. First, the eye image captured by a high-speed eye camera is converted into a grayscale image. Then, coarse pupil localization is performed to calculate the approximate pupil position. Finally, edge detection is conducted on the pupil region, and the detected points are fitted with an ellipse, outputting the ellipse parameters. Second, this method is characterized by high detection speed and high accuracy. By reducing the resolution for rapid pupil position detection and extracting the image near the pupil center for edge detection, rapid extraction of the pupil contour is achieved.

[0036] It should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise" in the above description indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0037] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0038] In the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0039] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0040] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0041] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for extracting the contour of the near-eye pupil, characterized in that, include: The compressed video format image captured by the high-speed eye camera is decoded into YUV format, and the Y component is directly used as the input image img1; The resolution of the grayscale image is reduced to a preset low resolution to obtain image img2; After performing median filtering on the image img2, it is divided into multiple blocks. The minimum pixel value is calculated for each block, and a threshold is set based on the minimum value for binarization to obtain a binary image. Connected component analysis is performed on the binary image to exclude connected components connected to image edges. The pupil connected component is determined based on the average gray value of the connected components, and its coarse pupil center is obtained. Specifically, this includes: performing connected component analysis on the binary image to obtain all connected components; excluding connected components connected to edges; sorting the remaining connected components by average gray value; determining the connected component with the smallest average gray value as the pupil connected component; and taking the center position of the pupil connected component as the coarse pupil center. Using the coarsely located pupil center as the center, a region of a predetermined size is cropped from image img1 to serve as pupil image img3; Using the coarsely located pupil center as the center, a range image of a predetermined size is extracted from image img1. The range image is then enlarged to the same size as pupil image img3 using a nearest neighbor interpolation algorithm. This enlarged range image is then fused pixel-wise with pupil image img3. Finally, median filtering is applied to the fused image img4. Specifically, this includes: using the coarsely located pupil center as the center, extracting a 50... A 50-pixel range image; the range image is then enlarged to 200 pixels using a nearest neighbor interpolation algorithm. 200 pixels; take the minimum value of corresponding pixels in the pupil image (img3) and the magnified range image, and perform pixel-level fusion; use 5 5. Median filtering is applied to the fused image to obtain the fused image img4; After binarizing the fused image img4, edge detection is performed to obtain edge detection points. Specifically, this includes: binarizing the fused image img4; where the threshold is the minimum gray value + 10, and the largest connected component is taken as the pupil image img5; and using the Canny operator to perform edge detection on the pupil image img5 to obtain edge detection points; where the low threshold of the Canny operator is 50 and the high threshold is 150. The edge detection points are fitted using the least squares method to calculate the ellipse parameters, which include the center coordinates, major and minor axes, and rotation angle. The ellipse parameters are validated; if the ellipse parameters do not conform to the pupil projection characteristics, the detection fails; if the ellipse parameters conform to the pupil projection characteristics, the ellipse parameters are output.

2. The method for extracting the near-eye pupil contour according to claim 1, characterized in that, The resolution was reduced using the nearest neighbor interpolation algorithm, from 640... 480 reduced to 160 120.

3. The method for extracting the near-eye pupil contour according to claim 2, characterized in that, Median filtering uses 11 11 filter cores.

4. The method for extracting the near-eye pupil contour according to claim 3, characterized in that, The steps of performing median filtering on image img2, dividing it into multiple blocks, calculating the minimum pixel value for each block, and then performing binarization based on a threshold set according to the minimum value to obtain a binary image include: The median-filtered image img2 is segmented into 4 3 pieces, each piece is 40 cm in size 40; For each block, find the minimum value of the pixel, minValue; A threshold is set based on the minimum value minValue and a preset offset value, and then binarization is performed to obtain a binary image.

5. The method for extracting the near-eye pupil contour according to claim 4, characterized in that, The area image of the predetermined size is 200. 200 pixels.

6. The method for extracting the near-eye pupil contour according to claim 5, characterized in that, The steps for verifying ellipse parameters include: If the ratio of the major axis to the minor axis of the ellipse is greater than 3 or the major axis is greater than 100 pixels, it is determined that it does not conform to the pupil projection feature and the detection fails. If the case is otherwise, it is determined to conform to the pupil projection characteristics, and the ellipse parameters are output.

Images