Pupil detection method, detection equipment and storage medium

Abstract

The invention provides a pupil detection method, detection equipment and a storage medium. The method comprises the following steps: acquiring a grayscale image for displaying an eye, and acquiring an eye contour from the grayscale image; calculating the length and / or the central point of each edge of the eye contour; determining a pupil contour from the eye contour according to the length and / or the central point; and extracting a plurality of feature points from the pupil contour, and performing fitting based on the plurality of feature points to obtain an ellipse corresponding to the pupil in the grayscale image. On the basis, non-pupil contour points can be better avoided, a very clean and simple eye contour is obtained, the number of edges and feature points used for fitting to obtain a pupil ellipse is small, and fitting precision and speed can be improved.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision image processing and eye tracking, and in particular to a pupil detection method, detection equipment and storage medium. Background Art

[0002] Eye interaction is to use the line of sight of the eyes to control and interact with interactive devices to meet user needs. Eye interaction has the advantages of natural, efficient and direct input, and can replace manual interaction in some control operations. Therefore, its application scenarios are becoming more and more extensive. One of the core of eye interaction is eye tracking, which is currently generally achieved through pupil detection.

[0003] The usual pupil detection method is: first obtain a grayscale image showing the eye, then calculate the gradient of each pixel based on the grayscale value of the image, so as to find the pixel with a larger gradient, and then obtain the eye contour with multiple edges, then fit an ellipse to each edge of the eye contour, and finally find the best fitting result as the pupil ellipse (that is, the ellipse corresponding to the pupil displayed in the image), or randomly select several feature points from all edges for ellipse fitting, repeat multiple times, until the best ellipse is finally found as the pupil ellipse. However, many of the pixel points with larger gradients found in this way do not belong to the points on the pupil contour, for example, many belong to the eyelids, eyelashes, etc., which not only affects the fitting accuracy, but also increases the program burden, wastes computing power, and thus affects the fitting speed. Summary of the invention

[0004] In view of this, the present application provides a pupil detection method, a detection device, and a storage medium, which can improve the problem that a large number of non-pupil contour points will affect the fitting accuracy and speed.

[0005] The present application provides a pupil detection method, comprising:

[0006] Acquire a grayscale image showing an eye, and acquire an eye contour from the grayscale image;

[0007] Calculating the length of each side and / or center point of the eye contour;

[0008] determining a pupil contour from the eye contour based on the length and / or center point;

[0009] A number of feature points are extracted from the pupil contour, and fitting is performed based on the number of feature points to obtain an ellipse corresponding to the pupil in the grayscale image.

[0010] Optionally, determining a pupil contour from the eye contour according to the length and / or the center point comprises:

[0011] Determining the curvature of each edge of the eye contour according to the center point;

[0012] Determine, from the eye contour, a first edge having a length less than a first length threshold and a curvature greater than a first curvature threshold;

[0013] An edge of the pupil outline is selected from among the first edges.

[0014] Optionally, determining the pupil contour from the eye contour according to the length and / or the center point further includes:

[0015] Determining a second edge from the eye contour having a length greater than a second length threshold;

[0016] The edge of the pupil outline is selected from among the second edges.

[0017] Optionally, the first length threshold is greater than the second length threshold.

[0018] Optionally, determining a pupil contour from the eye contour according to the length and / or the center point comprises:

[0019] Determining the curvature of each edge of the eye contour according to the center point;

[0020] Determining a third edge from the eye contour having a curvature greater than a second curvature threshold;

[0021] An edge of the pupil outline is selected from the third edges.

[0022] Optionally, the second curvature threshold is greater than or equal to 3 mm.

[0023] Optionally, determining the curvature of each edge of the eye contour according to the center point includes:

[0024] Selecting a plurality of sampling points from each edge of the eye contour;

[0025] Determine the two-dimensional coordinates of each sampling point;

[0026] Determine the two-dimensional coordinates of the center point of each edge according to the two-dimensional coordinates of each sampling point;

[0027] The curvature of each edge is determined based on the two-dimensional coordinates of the center point of each edge and the following relationship:

[0028]

[0029] Among them, c x is the horizontal coordinate of the center point of each edge, N is the number of selected sampling points, x i is the horizontal coordinate of the i-th sampling point, c y is the ordinate of the center point of each edge, yi is the ordinate of the i-th sampling point, and d is the curvature of each edge of the eye contour.

[0030] Optionally, determining the pupil contour from the eye contour according to the length and / or the center point further includes:

[0031] Determine a plurality of straight lines with the central point as the intersection point, wherein the angles between any two adjacent straight lines are equal and acute, so as to form a plurality of fan-shaped areas;

[0032] Determine a fourth side of the sector extending to a preset threshold from the eye contour;

[0033] An edge of the pupil outline is selected from the fourth edges.

[0034] A detection device provided in the present application includes a processor and a memory, wherein a pupil detection program is stored in the memory, and when the pupil detection program is executed by the processor, the steps of the pupil detection method as described in any one of the above items are implemented.

[0035] The present application provides a storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the pupil detection method as described in any one of the above items are implemented.

[0036] As described above, after obtaining the eye contour from the grayscale image, the present application calculates the length and / or center point of each side of the eye contour, and then determines the pupil contour from the eye contour according to the length and / or center point, and finally extracts a number of feature points from the pupil contour, and performs fitting based on the feature points to obtain the ellipse corresponding to the pupil in the grayscale image, that is, the pupil ellipse. In the scheme of the present application, the pupil contour is determined according to the length and / or center point of each side of the eye contour, which can better avoid non-pupil contour points, which is conducive to improving the fitting accuracy. At the same time, after eliminating these non-pupil contour points, a very clean and concise eye contour can be obtained, that is, the number of sides and feature points used to fit the pupil ellipse is small, which can reduce the program burden, save computing power, and improve the fitting speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of a pupil detection method provided in an embodiment of the present application;

[0038] Figure 2 is a schematic diagram of an eye tracker provided in an embodiment of the present application when being worn;

[0039] Figure 3 is based on Figure 2 A schematic diagram of a grayscale image captured by an eye tracker is shown;

[0040] Figure 4is a schematic diagram of determining a pupil contour based on a pupil detection method according to an embodiment of the present application;

[0041] Figure 5 is a schematic diagram comparing a pupil ellipse fitted by the pupil detection method according to an embodiment of the present application and a pupil ellipse fitted according to the prior art;

[0042] Figure 6 This is a schematic diagram of determining pupil contours using a pupil detection method based on another embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to solve the above problems existing in the prior art, the present application provides a pupil detection method and detection device, and a storage medium. These protected themes are based on the same concept, and the principles of solving the problems are basically the same or similar. The implementation methods of each protected theme can refer to each other, and the repeated parts will not be repeated.

[0044] In the scheme of the present application, the pupil contour is determined based on the length of each edge and / or center point of the eye contour, and non-pupil contour points are eliminated to obtain a very clean and concise eye contour. The number of edges and feature points finally used to fit the pupil ellipse is relatively small, thereby improving the fitting accuracy and speed.

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. In the absence of conflict, the following embodiments and their technical features can be combined with each other and also belong to the technical solutions of the present application.

[0046] Figure 1 : is a flowchart of a pupil detection method according to an embodiment of the present application. The pupil detection method may be referred to as a "detection method" or "method", and the execution subject of each step may be a detection device, an interactive device, or a storage medium, a processor, a controller, etc. having a pupil detection function. Figure 1 As shown, the method at least includes the following steps S1 to S4.

[0047] S1: Obtain a grayscale image showing an eye, and obtain an eye contour from the grayscale image.

[0048] In one example, combining Figure 2As shown, the present application can obtain the grayscale image of the eye in real time or periodically through the eye tracker 20. Specifically, the eye tracker 20 can include at least one or more eye cameras 21. The figure only shows one eye camera 21 as an example. The eye camera 21 can be set at the lower part of the human eye, and its lens is tilted upward to face the human eyeball. For example, the example in the figure sets the eye camera 21 at the lower right part of the right eye of the human. The eye camera 21 can capture the following Figure 3 and Figure 4 The grayscale image of the eye shown in the figure can capture the eye movement information by identifying the position change of the pupil in the grayscale image. In the actual scene, the eye camera 21 can directly capture the following Figure 3 and Figure 4 The grayscale image shown in FIG. 1 can also be directly photographed to obtain a color image. Then, the present application performs grayscale processing on the color image to obtain the following Figure 3 and Figure 4 Grayscale image shown.

[0049] The eye tracker 20 may also include other components, including but not limited to Figure 2 A world camera 22 is shown in FIG. The lens of the world camera 22 faces the front of the person, and is mainly used to capture the real physical scene in front of the person's eyes or within the field of view to obtain a real scene image. After calibration, the present application can map the eye information obtained based on pupil detection to the real scene image, for example, map the detected pupil as a point to the real scene image to display the gaze point of the human eye, and realize the gaze estimation commonly referred to in the art and the human-computer interaction related thereto.

[0050] Step S1 can use the gray gradient method to obtain the eye contour from the gray image, for example: first, calculate the gray value of each pixel in the gray image; then, use, for example, the Sobel algorithm or the Prewitt algorithm to calculate the difference between the gray value of each pixel in the gray image and the gray value of the surrounding pixels to obtain the gradient value of the pixel, and the gradient value reflects the degree of change of the gray value of the pixel; then, use, for example, the fixed threshold method or the adaptive threshold method to binarize the gradient value, mark the pixel with a gradient value greater than a preset threshold (which can be adaptively set according to actual needs) as an edge point, and mark other pixels as non-edge points; finally, use, for example, the Hough connection algorithm to connect adjacent edge points to form a complete edge line; the multiple edge lines finally obtained are the edges and textures in the gray image, as the edges of the eye contour. The principles and processes of the various algorithms involved can be referred to the prior art, and will not be repeated here.

[0051] S2: Calculate the length of each side and / or center point of the eye contour.

[0052] For any side that is a straight line, its center point is the midpoint of the line.

[0053] For any side of a curve, its center point can be called the center of curvature. A curve is formed by connecting several points. If we define any point as p i =(x i ,y i ), define the center point as c=(c x ,c y ), where c x is the horizontal coordinate of the center point, N is the number of edge points when the curve is formed in step S1, x i is the horizontal coordinate of the i-th sampling point, c y is the ordinate of the center point, y i is the ordinate of the i-th sampling point, then the center of curvature and any point on the curve satisfy the following relationship:

[0054]

[0055] S3: Determine the pupil outline from the eye outline based on length and / or center point.

[0056] In Example 1 of the present application, the method of determining the pupil contour includes steps S311 to S313:

[0057] S311, determining the curvature of each edge of the eye contour according to the central point;

[0058] S312, determining, from the eye contour, a first edge whose length is less than a first length threshold and whose curvature is greater than a first curvature threshold;

[0059] S313: Select an edge of the pupil contour from the first edges.

[0060] This example is equivalent to removing the edges with larger length and smaller curvature from the eye contour obtained in step S1, and using the remaining edges of the eye contour as the edges of the pupil contour. In the eye contour, the edges with larger length and smaller curvature can usually be the contour lines of the eyelids or the edges of the image.

[0061] Either the first length threshold and the first curvature threshold can be determined according to the adaptability of the length and width of the grayscale image. For example, the first length threshold and the length of the grayscale image can satisfy a first preset ratio threshold, and the width of the grayscale image can satisfy a second preset ratio threshold. On the premise that these two preset ratio thresholds are satisfied, the user can adjust the first length threshold by himself, thereby avoiding the detection and determination of non-pupil contour lines when the grayscale image size is large or small.

[0062] In the step S311, determining the curvature of each edge of the eye contour according to the center point may include: first, selecting a plurality of sampling points from each edge of the eye contour, and then determining the two-dimensional coordinates of each sampling point, for example, establishing a two-dimensional plane rectangular coordinate system based on the plane where the grayscale image is located, the origin of the two-dimensional plane rectangular coordinate system may be a vertex of the grayscale image, the horizontal coordinate may be the long side of the grayscale image, and the vertical coordinate may be the short side of the grayscale image, and the two-dimensional coordinates of each sampling point are the coordinates in the two-dimensional plane rectangular coordinate system; then, determining the two-dimensional coordinates of the center point of each edge of the eye contour according to the two-dimensional coordinates of each sampling point; finally, determining the curvature of each edge according to the two-dimensional coordinates of the center point of each edge and the following relationship:

[0063]

[0064] Among them, c x is the horizontal coordinate of the center point of each edge, N is the number of selected sampling points, x i is the horizontal coordinate of the i-th sampling point, c y is the ordinate of the center point of each edge, y i is the ordinate of the i-th sampling point, and d is the curvature of each edge of the eye contour.

[0065] For straight edges, the curvature d is zero, so they obviously need to be removed. The curvature d of curves such as eyelids is obviously smaller than the curvature d of the pupil, so they also need to be removed.

[0066] Combination Figure 4 As shown in the obtained eye contour, in the step S311, the edge L1 whose length is greater than the first length threshold and whose curvature is less than the first curvature threshold is deleted, which is usually the contour line of the eyelid, and the remaining line is the first edge, which can be used as the edge defining the pupil contour.

[0067] It should be noted that the present application may directly use the first edge as the edge of the pupil contour, or may continue to screen the first edge to determine a line that meets the conditions from the first edge as the edge of the pupil contour. For example, after the step S313, the step S3 also includes:

[0068] S314, determining a second side whose length is greater than a second length threshold from the eye contour; and

[0069] S315: Select an edge of the pupil contour from the second edges.

[0070] The first length threshold is greater than the second length threshold.

[0071] This example is equivalent to removing the edges with shorter lengths from the eye contour obtained in step S1, and using the remaining edges of the eye contour as the edges of the pupil contour. In the eye contour, the lines with a length less than the second length threshold can usually be regarded as random noise points obtained by the algorithm calculation in step S1. The line length of such edges is relatively short, which not only makes it difficult to reflect the contour features of the pupil, but also makes it difficult to apply the fitting calculation in the subsequent step S4. Figure 4 As shown in the obtained eye contour, in the steps S314 and S315, the edge L2 whose length is less than the second length threshold is deleted, and the remaining lines are the edges of the pupil contour.

[0072] In an actual scenario, the present application may also first execute steps S314 and S315, and then execute steps S311 to S313 on the second edge to obtain the first edge.

[0073] In Example 2 of the present application, the method of determining the pupil contour includes steps S321 to S323:

[0074] S321, determining the curvature of each edge of the eye contour according to the central point;

[0075] S322, determining a third side whose curvature is greater than a second curvature threshold from the eye contour;

[0076] S323, selecting an edge of the pupil contour from the third edges.

[0077] Compared with the above-mentioned Example 1, Example 2 can remove lines such as eyelashes (usually straight lines in the eye contour), small bright spots (usually displayed as particularly small circles in the eye contour) L3, and use the remaining edges as the edges of the pupil contour. Optionally, the second curvature threshold is greater than or equal to 3 mm. The second curvature threshold can be less than the first curvature threshold.

[0078] It should be noted that the present application includes multiple situations and multiple feasible implementation methods for a certain technical feature. Unless otherwise specified, it means that the corresponding technical feature can be implemented by combining any part of the methods. For example, the above example 1 can be combined with example 2 to screen out the edges that meet both example 1 and example 2 as the edges of the pupil contour. Through the combination scheme, the determination of the pupil contour can be implemented more accurately, thereby improving the accuracy of the pupil ellipse obtained by subsequent fitting.

[0079] S4: extracting a number of feature points from the pupil contour, and performing fitting based on the feature points to obtain an ellipse corresponding to the pupil in the grayscale image.

[0080] For ellipse fitting, the principle of step S4 is: from e.g. Figure 4Several feature points are extracted from the pupil contour shown as a feature point set, and then multiple feature points are randomly selected from the feature point set, for example, at least four feature points, because at least four non-collinear feature points can uniquely determine an ellipse, an ellipse is fitted through these four feature points, and then the distance from other feature points in the feature point set to the fitted ellipse is calculated, and whether these points are inner points, that is, whether they meet the points of the fitted ellipse, is determined based on whether the distance is within a set threshold; after multiple such random sampling and judgment processes, the fitted ellipse with the largest number of inner points is found, which is the final pupil ellipse.

[0081] In actual scenarios, the step S4 may use, including but not limited to, a RANSAC (Random sample consensus) algorithm to perform the fitting. The principle and specific process of the fitting can be found in the prior art in the art and will not be described in detail here.

[0082] Based on the above, in the solution of the present application, the pupil contour is determined according to the length of each side of the eye contour and / or the center point, which can better avoid non-pupil contour points, which is conducive to improving the fitting accuracy, so that the final fitting is as follows: Figure 5 The pupil ellipse 2 shown in the middle right figure is compared with Figure 5 The pupil ellipse 1 shown in the middle left figure, the pupil ellipse 2 fitted by the present application is more consistent with the actual pupil imaging; at the same time, after removing these non-pupil contour points, it can be obtained that Figure 4 The very clean and concise eye contour shown makes the number of edges and feature points used to fit the pupil ellipse in this application smaller, which can reduce the burden of related programs including the fitting program, save computing power, and improve the fitting speed.

[0083] In the aforementioned step S3, unlike the aforementioned examples 1 and 2 which both determine the pupil contour by calculating the curvature and / or length, the present application can also utilize the characteristic that the pupil is imaged as an ellipse or roughly an ellipse in the grayscale image to identify whether the edges of the eye contour belong to the pupil contour.

[0084] That is, in Example 3 of the present application, the method of determining the pupil contour includes steps S331 to S333:

[0085] S331, determining a plurality of straight lines with the center point as the intersection point, wherein the angles between any two adjacent straight lines are equal and acute angles, so as to form a plurality of fan-shaped areas;

[0086] S332, determining a fourth side of the fan-shaped area extending to a preset threshold from the eye contour;

[0087] S333. Select an edge of the pupil contour from the fourth edges.

[0088] Combination Figure 6 As shown, with the center point as the intersection point, at least three straight lines can be determined. In the figure, three straight lines are taken as an example. These three straight lines intersect to form six sectors. The angles corresponding to these six sectors are equal and acute, and the angle of each acute angle is 60°. In the step S332, the fourth side L4 extends to six sectors, and it can be determined that it belongs to the side of the pupil contour. The preset threshold value can be determined according to the adaptability of actual needs, for example, it can be determined according to the number of intersecting straight lines determined. In addition, in the obtained eye contour, the pupil contour may not be a completely closed ellipse, so the preset threshold value can be less than the total number of sectors formed by the intersection of several straight lines. The optional preset threshold value is greater than half of the total number of sectors.

[0089] This example 3 can also be combined with the above-mentioned example 1 or example 2 to select edges that satisfy both of these combined examples as edges of the pupil contour. Through the combined solution, the pupil contour can be determined more accurately, thereby improving the accuracy of the pupil ellipse obtained by subsequent fitting.

[0090] An embodiment of the present application also provides a storage medium on which a pupil detection program is stored. The pupil detection program is essentially a computer program, and when the pupil detection program is executed by a processor, the steps of the pupil detection method as in any example are implemented.

[0091] The storage medium includes but is not limited to any one of a read-only memory (ROM), a random access memory (RAM), a magnetic disk and an optical disk.

[0092] Since the program stored in the storage medium can execute the steps in the pupil detection method of any embodiment provided in the present application, the beneficial effects that can be achieved by the pupil detection method of any of the aforementioned embodiments can be achieved. Please refer to the aforementioned embodiments for details and will not be repeated here.

[0093] An embodiment of the present application also provides a detection device or chip, including a memory and a processor, wherein a pupil detection program is stored on the memory, and when the pupil detection program is executed by the processor, the steps of the pupil detection method of any of the aforementioned embodiments are implemented; and / or, the detection device or chip is provided with a storage medium as exemplified above, and the processor of the smart device or chip loads the storage medium to execute the steps of the pupil detection method, thereby achieving the beneficial effects that can be achieved by the pupil detection method of the corresponding embodiment.

[0094] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. For ordinary technicians in this field, all equivalent structural changes made using the contents of this specification and drawings are also included in the patent protection scope of the present application.

[0095] Although this article uses step codes such as S1, S2, etc., their purpose is to express the corresponding content more clearly and concisely, and does not constitute a substantial limitation on the order. Technical personnel in this field may execute S314 first and then S311, etc. during specific implementation, but these should all be within the scope of protection of this application.

[0096] Although the terms "first, second", etc. are used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. In addition, the singular forms "one", "an", and "the" are intended to include plural forms as well. The terms "or" and "and / or" are interpreted as inclusive, or mean any one or any combination. Only when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way, an exception to this definition will occur.

Claims

1. A pupil detection method, characterized in that: The method comprises: Acquire a grayscale image showing an eye, and acquire an eye contour from the grayscale image; Calculating the length of each side and / or center point of the eye contour; determining a pupil contour from the eye contour based on the length and / or center point; A number of feature points are extracted from the pupil contour, and fitting is performed based on the number of feature points to obtain an ellipse corresponding to the pupil in the grayscale image.

2. The pupil detection method according to claim 1, characterized in that: Determining a pupil contour from the eye contour based on the length and / or center point comprises: Determining the curvature of each edge of the eye contour according to the center point; Determine, from the eye contour, a first edge having a length less than a first length threshold and a curvature greater than a first curvature threshold; An edge of the pupil outline is selected from among the first edges.

3. The pupil detection method according to claim 2, characterized in that: Determining a pupil contour from the eye contour according to the length and / or center point, further comprising: Determining a second edge from the eye contour having a length greater than a second length threshold; The edge of the pupil outline is selected from among the second edges.

4. The pupil detection method according to claim 3, characterized in that: The first length threshold is greater than the second length threshold.

5. The pupil detection method according to claim 1, characterized in that: Determining a pupil contour from the eye contour based on the length and / or center point comprises: Determining the curvature of each edge of the eye contour according to the center point; Determining a third edge from the eye contour having a curvature greater than a second curvature threshold; An edge of the pupil outline is selected from the third edges.

6. The pupil detection method according to claim 5, characterized in that: The second curvature threshold is greater than or equal to 3 mm.

7. The pupil detection method according to claim 2 or 5, characterized in that: Determining the curvature of each side of the eye contour according to the center point includes: Selecting a plurality of sampling points from each edge of the eye contour; Determine the two-dimensional coordinates of each sampling point; Determine the two-dimensional coordinates of the center point of each edge according to the two-dimensional coordinates of each sampling point; The curvature of each edge is determined based on the two-dimensional coordinates of the center point of each edge and the following relationship: Among them, c x is the horizontal coordinate of the center point of each edge, N is the number of selected sampling points, x i is the horizontal coordinate of the i-th sampling point, c y is the ordinate of the center point of each edge, y i is the ordinate of the i-th sampling point, and d is the curvature of each edge of the eye contour.

8. The pupil detection method according to claim 2 or 5, characterized in that: Determining a pupil contour from the eye contour according to the length and / or center point, further comprising: Determine a plurality of straight lines with the central point as the intersection point, wherein the angles between any two adjacent straight lines are equal and acute, so as to form a plurality of fan-shaped areas; Determine a fourth side of the sector extending to a preset threshold from the eye contour; An edge of the pupil outline is selected from the fourth edges.

9. A detection device, characterized in that: The invention comprises a processor and a memory, wherein a pupil detection program is stored in the memory, and when the pupil detection program is executed by the processor, the steps of the pupil detection method according to any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the pupil detection method according to any one of claims 1 to 8 are implemented.

Images