Pupil marking method, pupil marking device and application thereof

Abstract

The invention provides a pupil labeling method, a pupil labeling device and application thereof. The pupil labeling method and the pupil labeling device can improve accuracy and real-time performance of pupil labeling. The pupil labeling method comprises the steps that pupil detection is conducted on an image to be labeled through a pupil detection method so as to obtain pupil feature parameters, and the pupil feature parameters comprise the long axis, the short axis, the center point and the angle of a pupil ellipse; on the basis of the pupil characteristic parameters, a plurality of equal division points of the pupil ellipse are calculated through an integral sampling method to serve as a plurality of key points of the pupil ellipse; and a plurality of key points of the pupil ellipse are adjusted online in real time, so that a fitting ellipse fitting the pupil is obtained, and pupil marking is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of eye movement tracking, and particularly to a pupil annotation method, a pupil annotation device, and their applications. Background Art

[0002] In recent years, with the rapid development of science and technology, the human-computer interaction method based on eye movement tracking has been importantly applied in the interaction of near-eye display devices (such as AR or VR glasses, etc.) due to its convenience and high efficiency. The existing eye movement tracking technology usually first uses an infrared camera to capture the human eye with an infrared LED lamp as a supplementary light source, and then establishes an eyeball model based on the characteristics of the human eye to calculate the gaze direction; since the pupil is the most important part of the human eye characteristics, the rapid and accurate extraction of pupil characteristics is the top priority of eye movement tracking.

[0003] Currently, in the eye movement tracking scheme for pupil detection using deep learning, it is usually necessary to extract the key points of the pupil through a deep learning model, but it requires a large number of sample annotations, with high annotation difficulty and extremely low time efficiency. For example, in an existing eye movement tracking method, it first roughly locates the pupil, and then obtains the elliptical parameter information through steps such as cropping, threshold segmentation, opening operation, contour finding, contour merging, concave hull operation, and ellipse fitting to achieve automatic annotation of the pupil position. However, the data obtained by this method may not necessarily be real and effective annotation data, and may possibly need further adjustment. Moreover, it is only pupil parameters. If it is to be further used as a deep learning training sample, further processing (such as binary processing of pupil key points and pupil classification, etc.) is required, resulting in poor accuracy and real-time performance. Summary of the Invention

[0004] An advantage of the present invention is to provide a pupil annotation method, a pupil annotation device, and their applications, which can improve the accuracy and real-time performance of pupil annotation.

[0005] Another advantage of the present invention is to provide a pupil annotation method, a pupil annotation device, and their applications. In one embodiment of the present invention, the pupil annotation method can automatically obtain pupil key points by using an integral sampling method, so as to improve the annotation efficiency while combining real-time adjustment of key points and feedback to further improve the accuracy.

[0006] Another advantage of the present invention is to provide a pupil annotation method, a pupil annotation device, and their applications. In one embodiment of the present invention, the pupil annotation method can refine the pupil contour after obtaining the pupil parameters, making the data more accurate.

[0007] Another advantage of the present invention lies in providing a pupil annotation method, a pupil annotation device and their applications. In one embodiment of the present invention, the pupil annotation method can combine subjective and objective evaluations by adjusting the annotation online, so that the annotation data can be corrected in real time to ensure the high accuracy of pupil annotation.

[0008] Another advantage of the present invention lies in providing a pupil annotation method, a pupil annotation device and their applications. To achieve the above object, a complex structure is not required in the present invention. Therefore, the present invention successfully and effectively provides a solution, not only providing a simple pupil annotation method, a pupil annotation device and their applications, but also increasing the practicability and reliability of the pupil annotation method, the pupil annotation device and their applications.

[0009] To achieve at least one of the above advantages or other advantages and objects of the present invention, the present invention provides a pupil annotation method, including the steps of:

[0010] Performing pupil detection on the image to be annotated by a pupil detection method to obtain pupil feature parameters, where the pupil feature parameters include the major axis, minor axis, center point and angle of the pupil ellipse;

[0011] Based on the pupil feature parameters, calculating a plurality of equally divided points of the pupil ellipse by an integral sampling method to be used as a plurality of key points of the pupil ellipse; and

[0012] Online real-time adjusting the plurality of key points of the pupil ellipse to obtain a fitting ellipse that fits the pupil, and completing pupil annotation.

[0013] According to an embodiment of the present application, the step of calculating a plurality of equally divided points of the pupil ellipse by an integral sampling method based on the pupil feature parameters to be used as a plurality of key points of the pupil ellipse includes the steps of:

[0014] Substituting the pupil feature parameters into an ellipse equation model to calculate the intersection points of the major and minor axes and the ellipse to be used as four vertex key parts of the pupil ellipse;

[0015] Substituting the major axis and minor axis in the pupil feature parameters into a perimeter calculation model to calculate the perimeter of the pupil ellipse; and

[0016] Based on the vertex key points and perimeter of the pupil ellipse, performing integral sampling on the pupil ellipse to obtain a plurality of equally divided points of the pupil ellipse as a plurality of equally divided key points of the pupil ellipse.

[0017] According to an embodiment of the present application, the step of performing integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain a plurality of equally divided points of the pupil ellipse as a plurality of equally divided key points of the pupil ellipse includes the steps of:

[0018] Starting from the four vertex key points respectively, perform co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance;

[0019] When the accumulated integral distance reaches the equal division length of the perimeter, calculate the end coordinates of the integral sampling to obtain four equal division key points; and

[0020] Starting from the four equal division key points respectively, loop and execute the above co-directional integral sampling steps until all equal division key points are obtained.

[0021] According to an embodiment of the present application, the step of performing integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain multiple equal division points of the pupil ellipse as multiple equal division key points of the pupil ellipse includes the steps of:

[0022] Starting from the four vertex key points respectively, perform co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance;

[0023] When the accumulated integral distance reaches the first equal division length of the perimeter, calculate the end coordinates of the integral sampling to obtain four equal division key points;

[0024] Starting from all vertex key points and equal division key points respectively, perform co-directional integral sampling on the pupil ellipse again to accumulate the integral sampling distance;

[0025] When the accumulated integral distance reaches the second equal division length of the perimeter, and the second equal division length is equal to half of the first equal division length, calculate the end coordinates of the integral sampling to obtain eight equal division key points; and

[0026] And so on until all equal division key points are obtained.

[0027] According to an embodiment of the present application, the step of online and real-time adjusting multiple key points of the pupil ellipse to obtain a fitting ellipse that fits the pupil to complete pupil annotation includes the steps of:

[0028] Perform visual fitting processing on multiple key points of the pupil ellipse to obtain multiple visual key points of the pupil ellipse;

[0029] Online adjust the positions of the visual key points to real-time fit a fitting ellipse that fits the pupil; and

[0030] Output the adjusted fitting ellipse as pupil annotation data to complete pupil annotation.

[0031] According to an embodiment of the present application, the step of online adjusting the position of the visible key point to fit an ellipse that fits the pupil in real time includes the steps of:

[0032] Perform ellipse fitting on the visible key point to obtain a fitting ellipse to be adjusted;

[0033] Subjectively judge the fitting degree between the fitting ellipse to be adjusted and the pupil edge on the image to be annotated; and

[0034] By correspondingly adjusting the position of the visible key point, fit an adjusted fitting ellipse in real time until an ellipse that fits the pupil is obtained to complete the adjustment.

[0035] According to an embodiment of the present application, the step of online adjusting the position of the visible key point to fit an ellipse that fits the pupil in real time further includes the steps of:

[0036] Objectively calculate the intersection over union between the adjusted fitting ellipse and the fitting ellipse to be adjusted as an objective adjustment reference to adjust the visible key point by combining subjective and objective evaluations.

[0037] According to an embodiment of the present application, the step of performing pupil detection on the image to be annotated by the pupil detection method to obtain pupil feature parameters, where the pupil feature parameters include the major axis, minor axis, center point, and angle of the pupil ellipse, includes the steps of:

[0038] Obtain an initial pupil ellipse in real time through a pupil rough positioning method or a pupil tracking method; and

[0039] Refine the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters.

[0040] According to an embodiment of the present application, the step of refining the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters includes the steps of:

[0041] Perform edge detection on the region of interest on the image to be annotated to obtain an edge image;

[0042] On the rays from the center point of the initial pupil ellipse to each edge point, respectively select a predetermined number of inner contraction points and outer expansion points as initial candidate points;

[0043] Solve the intersection of the initial candidate points and the edge image to obtain the final candidate points;

[0044] On the rays from the center point of the initial pupil ellipse to each final candidate point, solve for the gradient value of each final candidate point in the ray direction, so as to determine the final edge points according to the gradient magnitude;

[0045] Perform ellipse fitting on all the final edge points to obtain a first pupil ellipse;

[0046] Calculate the sum of the distances from each final edge point to the two foci on the first pupil ellipse, so as to remove abnormal edge points according to the fact that the sum of the distances from all points on the ellipse to the two foci is a constant value, and obtain the filtered edge points;

[0047] Perform ellipse fitting on all the filtered edge points to obtain a second pupil ellipse; and

[0048] Determine whether the second pupil ellipse is a pupil: if so, output the major axis, minor axis, center point, and angle of the second pupil ellipse as the pupil characteristic parameters; if not, determine it as a blink.

[0049] According to another aspect of the present application, the present application further provides a pupil annotation device, including components that are communicatively connected to each other:

[0050] A pupil detection module, configured to perform pupil detection on the image to be annotated by a pupil detection method to obtain pupil characteristic parameters, where the pupil characteristic parameters include the major axis, minor axis, center point, and angle of the pupil ellipse;

[0051] A key point acquisition module, configured to calculate multiple equally divided points of the pupil ellipse by an integral sampling method based on the pupil characteristic parameters as multiple key points of the pupil ellipse; and

[0052] A key point adjustment module, configured to online and real-time adjust the multiple key points of the pupil ellipse to obtain a fitting ellipse that fits the pupil, and complete pupil annotation.

[0053] According to an embodiment of the present application, the key point acquisition module includes components that are communicatively connected to each other: an intersection calculation module, configured to calculate the intersections of the major and minor axes with the ellipse by substituting the pupil characteristic parameters into the ellipse equation model as four vertex key parts of the pupil ellipse; a perimeter calculation module, configured to calculate the perimeter of the pupil ellipse by substituting the major axis and minor axis in the pupil characteristic parameters into the perimeter calculation model; and an integral sampling module, configured to perform integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse.

[0054] According to an embodiment of the present application, the key point adjustment module includes: a visualization module communicatively connected to each other, configured to perform visualization processing on multiple key points of the pupil ellipse to obtain multiple visible key points of the pupil ellipse; an online adjustment module, configured to online adjust the positions of the visible key points to fit an ellipse that conforms to the pupil in real time; and an output module, configured to output the adjusted fitting ellipse as pupil annotation data to complete pupil annotation.

[0055] According to an embodiment of the present application, the pupil detection module includes: a pupil ellipse acquisition module communicatively connected to each other, configured to obtain an initial pupil ellipse in real time by a pupil rough positioning method or a pupil tracking method; and a pupil ellipse refinement module, configured to perform refinement processing on the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters.

[0056] On the other hand of the present application, the present application further provides a method for training a pupil detection model, including the steps of:

[0057] Obtaining multiple pupil annotation samples by any one of the above-mentioned pupil annotation methods; and

[0058] Training the network in the pupil detection model based on the pupil annotation samples.

[0059] On the other hand of the present application, the present application further provides an electronic device, including:

[0060] A processor, configured to execute instructions; and

[0061] A storage medium communicatively connected to the processor, wherein the storage medium stores instructions, and the instructions are executed by the processor to enable the processor to execute the steps in any one of the above-mentioned pupil annotation methods.

[0062] On the other hand of the present application, the present application further provides an electronic device, including:

[0063] A near-eye display device; and

[0064] A pupil detection model trained by the above-mentioned method for training a pupil detection model, where the pupil detection model is configured in the near-eye display device to detect the pupil in an eye image collected by the near-eye display device. Description of the Drawings

[0065] Figure 1 is a schematic flowchart of a pupil annotation method according to an embodiment of the present application;

[0066] Figure 2Shows a schematic flowchart of the pupil detection step in the pupil annotation method according to the above embodiments of the present application;

[0067] Figure 3 Shows an example of the initial pupil ellipse obtained by pupil detection in the pupil annotation method according to the above embodiments of the present application;

[0068] Figure 4 Shows a schematic flowchart of the pupil ellipse refinement step in the pupil annotation method according to the above embodiments of the present application;

[0069] Figure 5 Shows an example of the refined pupil ellipse obtained by pupil ellipse refinement in the pupil annotation method according to the above embodiments of the present application;

[0070] Figure 6 Shows a schematic flowchart of the key point acquisition step in the pupil annotation method according to the above embodiments of the present application;

[0071] Figure 7 Shows a schematic diagram of the key point distribution obtained by the pupil annotation method according to the above embodiments of the present application;

[0072] Figure 8 Shows a first example of obtaining key points in the pupil annotation method according to the above embodiments of the present application;

[0073] Figure 9 Shows a second example of obtaining key points in the pupil annotation method according to the above embodiments of the present application;

[0074] Figure 10 Shows a schematic flowchart of the key point adjustment step in the pupil annotation method according to the above embodiments of the present application;

[0075] Figure 11 Shows an example of adjusting key points in the pupil annotation method according to the above embodiments of the present application;

[0076] Figure 12 Is a schematic block diagram of a pupil annotation device according to an embodiment of the present application;

[0077] Figure 13 Is a schematic block diagram of an electronic device according to an embodiment of the present application;

[0078] Figure 14 Is a schematic block diagram of another electronic device according to an embodiment of the present application.

[0079] Description of Main Component Symbols: 10. Pupil Marking Device; 11. Pupil Detection Module; 111. Pupil Ellipse Acquisition Module; 112. Pupil Ellipse Refinement Module; 12. Key Point Acquisition Module; 121. Intersection Calculation Module; 122. Perimeter Calculation Module; 123. Integral Sampling Module; 13. Key Point Adjustment Module; 131. Visualization Module; 132. Online Adjustment Module; 133. Output Module; 20. Near-Eye Display Device; 30. Pupil Detection Model; 41. Processor; 42. Storage Medium; 43. Input Device; 44. Output Device.

[0080] The above description of main component symbols further elaborates on this application in conjunction with the accompanying drawings and specific embodiments. Specific Embodiments

[0081] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and other obvious variations can be conceived by those skilled in the art. The basic principles defined in the following description can be applied to other implementation schemes, variant schemes, improvement schemes, equivalent schemes, and other technical schemes that do not depart from the spirit and scope of the present invention.

[0082] In the present invention, the term "a" in the claims and the specification should be understood as "one or more". That is, in one embodiment, the number of an element can be one, while in other embodiments, the number of this element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of this element is only one, the term "a" cannot be understood as being unique or single, and the term "a" cannot be understood as a limitation on the number.

[0083] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with this embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0084] Considering that although existing eye movement tracking methods can achieve automatic annotation of pupil positions, the data obtained may not necessarily be true and effective annotation data, and may need to be further adjusted. Moreover, it only provides pupil parameters. If it is to be further used as a deep learning training sample, further processing is required, resulting in poor accuracy and real-time performance. Therefore, this application creatively proposes a pupil annotation method, a pupil annotation device, and their applications, which can improve the accuracy and real-time performance of pupil annotation.

[0085] Specifically, referring to the accompanying drawings of the specification of this application Figures 1 to 11 , according to an embodiment of this application, a pupil annotation method is provided, which may include the steps:

[0086] S100: Through the pupil detection method, perform pupil detection on the image to be annotated to obtain pupil feature parameters, where the pupil feature parameters include the major axis, minor axis, center point, and angle of the pupil ellipse;

[0087] S200: Based on the pupil feature parameters, calculate multiple equally divided points of the pupil ellipse by the integral sampling method to serve as multiple key points of the pupil ellipse; and

[0088] S300: Online and real-time adjust the multiple key points of the pupil ellipse to obtain a fitting ellipse that fits the pupil, and complete pupil annotation.

[0089] It should be noted that the pupil annotation method of this application uses the integral sampling method to automatically calculate multiple equally divided points of the pupil ellipse as key points, which can improve the annotation efficiency and real-time performance; at the same time, the pupil annotation method of this application can also combine real-time adjustment of key points to obtain key point information with higher accuracy, so as to further improve the accuracy of pupil annotation data.

[0090] More specifically, as Figure 2 shown, step S100 of the pupil annotation method of this application may include the steps:

[0091] S110: Through the pupil rough positioning method or the pupil tracking method, obtain the initial pupil ellipse in real time; and

[0092] S120: Perform refinement processing on the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters.

[0093] It should be noted that as Figures 3 to 5As shown, the pupil annotation method of the present application will refine the pupil ellipse after obtaining the initial pupil ellipse, making the data of the refined pupil ellipse more accurate and helping to further improve the accuracy of subsequent pupil annotation. It can be understood that the pupil rough positioning or pupil tracking method in step S110 of the present application can be the pupil detection and tracking solution described in another patent "Pupil Detection and Tracking Method, Pupil Detection and Tracking System, and Near-Eye Display Device" filed by the applicant on the same day, or it can be a pupil detection solution in the prior art, as long as the initial pupil ellipse can be obtained. The present application will not elaborate on this further.

[0094] Optionally, as Figure 4 shown, step S120 of the pupil annotation method of the present application may include the steps:

[0095] S121: Perform edge detection on the region of interest on the image to be annotated to obtain an edge image;

[0096] S122: On the rays from the center point of the initial pupil ellipse to each edge point, respectively select a predetermined number of in-contracted points and out-expanded points as initial candidate points;

[0097] S123: Solve the intersection of the initial candidate points and the edge image to obtain final candidate points;

[0098] S124: On the rays from the center point of the initial pupil ellipse to each final candidate point, respectively solve the gradient value of each final candidate point in the ray direction to determine the final edge points according to the gradient magnitude;

[0099] S125: Perform ellipse fitting on all the final edge points to obtain a first pupil ellipse;

[0100] S126: Calculate the sum of the distances from each final edge point to the two foci on the first pupil ellipse, and remove the abnormal edge points according to the fact that the sum of the distances from all points on the ellipse to the two foci is a constant value to obtain the filtered edge points;

[0101] S127: Perform ellipse fitting on all the filtered edge points to obtain a second pupil ellipse; and

[0102] S128: Determine whether the second pupil ellipse is a pupil: If so, output the major axis, minor axis, center point, and angle of the second pupil ellipse as the pupil feature parameters; if not, determine it as a blink.

[0103] It can be understood that the predetermined number mentioned in the present application may not be limited to being implemented as two or more.

[0104] Exemplarily, step S121 of the pupil annotation method of the present application may include the steps:

[0105] Perform gamma transformation (i.e., gamma transform) on the image of the region of interest to increase the image contrast;

[0106] Perform median filtering on the transformed image of the region of interest to remove noise; and

[0107] Based on the image segmentation threshold (adaptive threshold) obtained by the OSTU algorithm (maximum inter-class variance method), perform canny edge detection on the filtered image of the region of interest to obtain the edge image.

[0108] According to the above embodiments of the present application, as Figure 6 and Figure 7 shown, step S200 of the pupil annotation method of the present application may include the steps of:

[0109] S210: Substitute the pupil feature parameters into the ellipse equation model to calculate the intersection points of the major and minor axes with the ellipse as the four vertex key points of the pupil ellipse;

[0110] S220: Substitute the major axis and minor axis in the pupil feature parameters into the perimeter calculation model to calculate the perimeter of the pupil ellipse; and

[0111] S230: Based on the vertex key points and perimeter of the pupil ellipse, perform integral sampling on the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse.

[0112] Exemplarily, in step S210 of the present application, the ellipse equation model can be but is not limited to being implemented as:

[0113] [(x - x0)cosθ+(y - y0)sinθ]2 / a2 + [(y - y0)cosθ - (x - x0)sinθ]2 / b2 = 0;

[0114] Where: x and y represent the point coordinates on the pupil ellipse; x0 and y0 represent the center point coordinates of the pupil ellipse; θ is the angle; a is the major axis; b is the minor axis. It can be understood that since the center point coordinates x0 and y0, the angle θ, and the major and minor axes a and b of the pupil ellipse in the pupil feature parameters are all known, that is, the slope and the center point are known. Therefore, substituting these pupil feature parameters into the above ellipse equation model can obtain the intersection points of the major and minor axes with the ellipse, that is, the four vertices of the pupil ellipse, and taking these four vertices as the key points of the pupil ellipse to obtain the four vertex key points of the pupil ellipse.

[0115] In step S220 of the present application, the perimeter calculation model can be but is not limited to being implemented as:

[0116]

[0117] Wherein: c is the perimeter of the pupil ellipse; a and b are the major axis and minor axis of the pupil ellipse, respectively. It can be understood that since the major axis a and minor axis b of the pupil ellipse in the pupil feature parameters are both known, substituting the major and minor axis data into the above perimeter calculation model can obtain the perimeter c of the pupil ellipse.

[0118] Optionally, in the first example of the present application, as Figure 8 shown, step S230 of the pupil annotation method of the present application may include the steps of:

[0119] S231: Starting from the four vertex key points respectively, perform co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance;

[0120] S232: When the integral cumulative distance reaches the equal division length of the perimeter, calculate the end coordinates of the integral sampling to obtain four equal division key points; and

[0121] S233: Starting from the four equal division key points respectively, repeatedly execute the above co-directional integral sampling step until all equal division key points are obtained.

[0122] For example, the equal division length of the perimeter c can be but is not limited to being implemented as one-sixteenth of the perimeter, i.e., c / 16: First, starting from the four vertex key points, perform clockwise integral sampling on the pupil ellipse. When the integral cumulative distance reaches c / 16, the calculated end coordinates are the coordinates of the equal division key points to obtain four equal division key points; then, starting from the four equal division key points, perform clockwise integral sampling on the pupil ellipse again. When the integral cumulative distance reaches c / 16, the calculated end coordinates are the coordinates of the new equal division key points to obtain four new equal division key points; finally, starting from the four new equal division key points, continue to perform clockwise integral sampling on the pupil ellipse. When the integral cumulative distance reaches c / 16, the calculated end coordinates are the coordinates of the latest equal division key points to obtain four latest equal division key points; then the twelve equal division key points obtained plus the four vertex key points are the sixteen equal division points of the pupil ellipse, that is, the sixteen key points of the pupil ellipse required to be obtained by the pupil annotation method of the present application.

[0123] It can be understood that the end coordinates after each step of integral sampling are x = a * cos(radians(angle)) and y = b * sin(radians(angle)). The distance of each step of integral sampling distance(x0, y0, x, y) can be calculated, where radians(angle) is the radian angle increased by each step of integral sampling, and x0 and y0 are the starting coordinates. Since the angle angle increased during each step of integral sampling is an extremely small value, the distance of each step of integral sampling distance(x0, y0, x, y) is also very small and can be approximated as the Euclidean distance. In addition, in other examples of this application, the equal division length of this perimeter can also be implemented as other ratios, as long as it is a multiple of four, and this application will not elaborate further on this.

[0124] It should be noted that in the second example of this application, as Figure 9 shown, step S230 of the pupil annotation method of this application may also include the steps:

[0125] S231’: Starting from the four vertex key points respectively, perform co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance;

[0126] S232’: When the integral accumulated distance reaches the first equal division length of the perimeter, calculate the end coordinates of the integral sampling to obtain four equal division key points; and

[0127] S233’: Starting from all vertex key points and equal division key points respectively, perform co-directional integral sampling on the pupil ellipse again to accumulate the integral sampling distance;

[0128] S234’: When the integral accumulated distance reaches the second equal division length of the perimeter, and the second equal division length is equal to half of the first equal division length, calculate the end coordinates of the integral sampling to obtain eight equal division key points; and

[0129] S235’: And so on until all equal division key points are obtained.

[0130] For example, the first equal division length of the perimeter c can be but is not limited to being implemented as one-eighth of the perimeter, i.e., c / 8; the second equal division length of the perimeter c can be but is not limited to being implemented as one-sixteenth of the perimeter, i.e., c / 16: First, starting from the four vertex key points, perform integral sampling on the pupil ellipse in a clockwise direction. When the cumulative integral distance reaches c / 8, the calculated end coordinates are the coordinates of the equal division key points to obtain four equal division key points; then, starting from the four vertex key points and the four equal division key points, perform integral sampling on the pupil ellipse again in a clockwise direction. When the cumulative integral distance reaches c / 16, the calculated end coordinates are the coordinates of the new equal division key points to obtain eight new equal division key points; then the twelve equal division key points obtained plus the four vertex key points are the sixteen equal division points of the pupil ellipse, that is, the sixteen key points of the pupil ellipse required to be obtained by the pupil annotation method of the present application. It can be understood that in the second example of the present application, in the integral sampling of the pupil annotation method of the present application, some equal division sampling points always start from the vertex key points, which helps to reduce the cumulative error and improve the accuracy of the equal division key points.

[0131] According to the above embodiments of the present application, as Figure 10 shown, step S300 of the pupil annotation method may include the steps:

[0132] S310: Perform visualization processing on multiple key points of the pupil ellipse to obtain multiple visible key points of the pupil ellipse;

[0133] S320: Online adjust the positions of the visible key points to fit an ellipse that fits the pupil in real time; and

[0134] S330: Output the adjusted fitted ellipse as pupil annotation data to complete pupil annotation.

[0135] It should be noted that in step S310 of the present application: convert the data of multiple key points of the pupil ellipse into a format, and then perform visualization on the key point data after the format conversion through a visualization tool to obtain multiple visible key points of the pupil ellipse.

[0136] Optionally, as Figure 11 shown, step S320 of the pupil annotation method includes the steps:

[0137] S321: Perform ellipse fitting on the visible key points to obtain a fitted ellipse to be adjusted;

[0138] S322: Subjectively judge the degree of fit between the fitted ellipse to be adjusted and the pupil edge on the image to be annotated; and

[0139] S323: By correspondingly adjusting the positions of the visible key points, a fitted ellipse after adjustment is fitted in real time until a fitted ellipse that fits the pupil is obtained, thus completing the adjustment.

[0140] It should be noted that this step S320 may further include the steps of objectively calculating the intersection over union ratio between the adjusted fitted ellipse and the fitted ellipse to be adjusted as an objective adjustment reference, so as to adjust the visible key points by combining subjective and objective evaluations. In this way, in this step S323, the pupil annotation method of the present application can combine subjective and objective evaluations to adjust the visible key points, and fit the ellipse in real time to give real-time evaluation feedback, which is convenient for improving the efficiency and accuracy of key point adjustment.

[0141] It can be understood that the key point adjustment of the present application can, but is not limited to, trigger the fitting ellipse through an interactive button and adjust the position of the visible key point on the ray from the center point of the fitting ellipse to the visible key point, so as to ensure that the radian angle of the visible key point remains unchanged before and after the position adjustment, and improve the reliability and accuracy of key point adjustment.

[0142] It is worth mentioning that, according to another aspect of the present application, as Figure 12 shown, an embodiment of the present application provides a pupil annotation device 10, which may include components that are communicatively connected to each other:

[0143] A pupil detection module 11, configured to perform pupil detection on the image to be annotated by a pupil detection method to obtain pupil feature parameters, where the pupil feature parameters include the major axis, minor axis, center point, and angle of the pupil ellipse;

[0144] A key point acquisition module 12, configured to calculate a plurality of equally divided points of the pupil ellipse by an integral sampling method based on the pupil feature parameters as a plurality of key points of the pupil ellipse; and

[0145] A key point adjustment module 13, configured to online and real-time adjust a plurality of key points of the pupil ellipse to obtain a fitted ellipse that fits the pupil, thus completing pupil annotation.

[0146] It should be noted that, in an example of the present application, as Figure 12As shown, the key point acquisition module 12 may include the following components communicatively connected to each other: an intersection calculation module 121, configured to calculate the intersections of the major and minor axes with the ellipse by substituting the pupil feature parameters into the ellipse equation model, so as to obtain the four vertex key points of the pupil ellipse; a perimeter calculation module 122, configured to calculate the perimeter of the pupil ellipse by substituting the major and minor axes in the pupil feature parameters into the perimeter calculation model; and an integral sampling module 123, configured to perform integral sampling on the pupil ellipse based on the vertex key points and the perimeter of the pupil ellipse, so as to obtain a plurality of equally divided points of the pupil ellipse as a plurality of equally divided key points of the pupil ellipse.

[0147] In an example of the present application, as Figure 12 shown, the key point adjustment module 13 may include the following components communicatively connected to each other: a visualization module 131, configured to perform visualization processing on the plurality of key points of the pupil ellipse to obtain a plurality of visible key points of the pupil ellipse; an online adjustment module 132, configured to online adjust the positions of the visible key points to fit an ellipse that conforms to the pupil in real time; and an output module 133, configured to output the adjusted fitted ellipse as pupil annotation data to complete pupil annotation.

[0148] In an example of the present application, as Figure 12 shown, the pupil detection module 11 includes the following components communicatively connected to each other: a pupil ellipse acquisition module 111, configured to obtain an initial pupil ellipse in real time by using a pupil rough positioning method or a pupil tracking method; and a pupil ellipse refinement module 112, configured to perform refinement processing on the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters.

[0149] Optionally, the pupil ellipse refinement module 112 is specifically configured to perform edge detection on the region of interest on the image to be annotated to obtain an edge image; respectively select a predetermined number of contracted points and expanded points on the rays from the center point of the initial pupil ellipse to each edge point as initial candidate points; solve the intersection of the initial candidate points and the edge image to obtain final candidate points; respectively solve the gradient value of each final candidate point in the ray direction on the rays from the center point of the initial pupil ellipse to each final candidate point, so as to determine the final edge points according to the gradient magnitude; perform ellipse fitting on all the final edge points to obtain a first pupil ellipse; calculate the sum of the distances from each final edge point to the two foci on the first pupil ellipse, so as to remove abnormal edge points according to the fact that the sum of the distances from all points on the ellipse to the two foci is a constant value, to obtain filtered edge points; perform ellipse fitting on all the filtered edge points to obtain a second pupil ellipse; and determine whether the second pupil ellipse is a pupil: if so, output the major axis, minor axis, center point, and angle of the second pupil ellipse as the pupil feature parameters; if not, determine that it is a blink.

[0150] It is worth mentioning that, according to another aspect of the present application, an embodiment of the present application provides a method for training a pupil detection model, which may include the steps of: obtaining a plurality of pupil annotation samples through the above-mentioned pupil annotation method; and training the network in the pupil detection model based on the pupil annotation samples.

[0151] In addition, according to another aspect of the present application, as Figure 13 shown, an embodiment of the present application provides an electronic device, which may include: a near-eye display device 20; and a pupil detection model 30 trained by the above-mentioned method for training a pupil detection model, where the pupil detection model 30 is configured in the near-eye display device for detecting pupils in an eye image collected through the near-eye display device 20.

[0152] Schematic electronic product

[0153] Next, the electronic device according to an embodiment of the present invention will be described with reference to Figure 14 . As Figure 14 shown, the electronic device includes one or more processors 41 and a storage medium 42.

[0154] The processor 41 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0155] The storage medium 42 may include one or more computing program products, and the computing program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more instructions may be stored on the computer-readable storage medium, and the processor 41 may run the instructions to implement the methods of the various embodiments of the present invention described above and / or other desired functions.

[0156] In one example, as Figure 14 shown, the electronic device may further include: an input device 43 and an output device 44, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0157] The output device 44 may output various information to the outside, including a display screen, etc. The output device 44 may include, for example, a display and a communication network and its connected remote output devices, etc.

[0158] Of course, for simplicity,Figure 14 Only some of the components related to the present invention in the electronic device are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device may further include any other appropriate components.

[0159] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0160] The above embodiments only represent several implementation manners of the present invention, and the description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can be made, and these all belong to the protection scope of the present invention.

Claims

1. A pupil annotation method, characterized in that, Including the steps: Performing pupil detection on the image to be annotated by the pupil detection method to obtain pupil feature parameters, where the pupil feature parameters include the major axis, minor axis, center point, and angle of the pupil ellipse; Based on the pupil feature parameters, calculating multiple equally divided points of the pupil ellipse by the integral sampling method to serve as multiple key points of the pupil ellipse; And Online and in real-time adjusting the multiple key points of the pupil ellipse to obtain a fitted ellipse that fits the pupil, thereby completing pupil annotation.

2. The pupil annotation method according to claim 1, characterized in that, The step of calculating multiple equally divided points of the pupil ellipse by the integral sampling method based on the pupil feature parameters to serve as multiple key points of the pupil ellipse includes the steps: Substituting the pupil feature parameters into the ellipse equation model to calculate the intersection points of the major and minor axes with the ellipse to serve as the four vertex key points of the pupil ellipse; Substituting the major axis and minor axis in the pupil feature parameters into the perimeter calculation model to calculate the perimeter of the pupil ellipse; And Based on the vertex key points and perimeter of the pupil ellipse, performing integral sampling on the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse.

3. The pupil annotation method according to claim 2, characterized in that, The step of performing integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse includes the steps: Starting from the four vertex key points respectively, performing co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance; When the accumulated integral distance reaches the equally divided length of the perimeter, calculating the end coordinates of the integral sampling to obtain four equally divided key points; And Starting from the four equally divided key points respectively, repeatedly executing the above co-directional integral sampling steps until all equally divided key points are obtained.

4. The pupil annotation method according to claim 2, characterized in that, The step of performing integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse includes the steps: Starting from the four vertex key points respectively, performing co-directional integral sampling on the pupil ellipse to accumulate the integral sampling distance; When the accumulated integral distance reaches the first equally divided length of the perimeter, calculating the end coordinates of the integral sampling to obtain four equally divided key points; Starting from all vertex key points and equally divided key points respectively, performing co-directional integral sampling on the pupil ellipse again to accumulate the integral sampling distance; When the accumulated integral distance reaches the second equally divided length of the perimeter and the second equally divided length is equal to half of the first equally divided length, calculating the end coordinates of the integral sampling to obtain eight equally divided key points; And And so on until all equally divided key points are obtained.

5. The pupil annotation method according to any one of claims 1 to 4, characterized in that, The step of online and in real-time adjusting the multiple key points of the pupil ellipse to obtain a fitted ellipse that fits the pupil and completing pupil annotation includes the steps: Performing visual fitting processing on the multiple key points of the pupil ellipse to obtain multiple visual key points of the pupil ellipse; Online adjusting the positions of the visual key points to real-time fit a fitted ellipse that fits the pupil; And Outputting the adjusted fitted ellipse as pupil annotation data to complete pupil annotation.

6. The pupil annotation method according to claim 5, characterized in that, The steps of online adjusting the position of the visible key points to fit an ellipse that fits the pupil in real time include the steps of: Performing ellipse fitting on the visible key points to obtain a fitting ellipse to be adjusted; Subjectively judging the fitting degree between the fitting ellipse to be adjusted and the pupil edge on the image to be annotated; And By correspondingly adjusting the position of the visible key points, fitting an adjusted fitting ellipse in real time until an ellipse that fits the pupil is obtained to complete the adjustment.

7. The pupil annotation method according to claim 6, characterized in that, The steps of online adjusting the position of the visible key points to fit an ellipse that fits the pupil in real time further include the steps of: Objectively calculating the intersection over union ratio between the adjusted fitting ellipse and the fitting ellipse to be adjusted as an objective adjustment reference to adjust the visible key points by combining subjective and objective evaluations.

8. The pupil annotation method according to any one of claims 1 to 4, characterized in that, The steps of performing pupil detection on the image to be annotated by the pupil detection method to obtain pupil characteristic parameters, where the pupil characteristic parameters include the major axis, minor axis, center point, and angle of the pupil ellipse, include the steps of: Real-time obtaining an initial pupil ellipse by the pupil rough positioning method or the pupil tracking method; And Performing refinement processing on the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil characteristic parameters.

9. The pupil annotation method according to claim 8, characterized in that, The steps of performing refinement processing on the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil characteristic parameters include the steps of: Performing edge detection on the region of interest on the image to be annotated to obtain an edge image; On the rays from the center point of the initial pupil ellipse to each edge point, respectively selecting a predetermined number of in-contracted points and out-expanded points as initial candidate points; Solving the intersection of the initial candidate points and the edge image to obtain final candidate points; Respectively on the rays from the center point of the initial pupil ellipse to each final candidate point, solving the gradient value of each final candidate point in the ray direction to determine the final edge points according to the gradient magnitude; Performing ellipse fitting on all the final edge points to obtain a first pupil ellipse; Calculating the sum of the distances from each final edge point to the two foci on the first pupil ellipse to remove abnormal edge points according to the fact that the sum of the distances from all points on the ellipse to the two foci is a constant value to obtain filtered edge points; Performing ellipse fitting on all the filtered edge points to obtain a second pupil ellipse; And Judging whether the second pupil ellipse is a pupil: if so, outputting the major axis, minor axis, center point, and angle of the second pupil ellipse as the pupil characteristic parameters; If not, it is determined as a blink.

10. A pupil annotation device, characterized in that, Including communicatively connected to each other: A pupil detection module for performing pupil detection on the image to be annotated by the pupil detection method to obtain pupil characteristic parameters, where the pupil characteristic parameters include the major axis, minor axis, center point, and angle of the pupil ellipse; A key point acquisition module for calculating multiple equally divided points of the pupil ellipse by the integral sampling method based on the pupil characteristic parameters as multiple key points of the pupil ellipse; And A key point adjustment module, which is used to adjust multiple key points of the pupil ellipse online in real time to obtain a fitting ellipse that fits the pupil, so as to complete pupil annotation.

11. The pupil annotation device according to claim 10, characterized in that, The key point acquisition module includes: an intersection calculation module that is communicatively connected to each other, which is used to substitute the pupil feature parameters into the ellipse equation model to calculate the intersections of the major and minor axes with the ellipse as the four vertex key parts of the pupil ellipse; A perimeter calculation module, which is used to substitute the major axis and minor axis in the pupil feature parameters into the perimeter calculation model to calculate the perimeter of the pupil ellipse; And an integral sampling module, which is used to perform integral sampling on the pupil ellipse based on the vertex key points and perimeter of the pupil ellipse to obtain multiple equally divided points of the pupil ellipse as multiple equally divided key points of the pupil ellipse.

12. The pupil annotation device according to claim 10, characterized in that, The key point adjustment module includes: a visualization module that is communicatively connected to each other, which is used to perform visualization processing on multiple key points of the pupil ellipse to obtain multiple visible key points of the pupil ellipse; An online adjustment module, which is used to adjust the positions of the visible key points online to fit a fitting ellipse that fits the pupil in real time; And an output module, which is used to output the adjusted fitting ellipse as pupil annotation data to complete pupil annotation.

13. The pupil annotation device according to any one of claims 10 to 12, characterized in that, The pupil detection module includes: a pupil ellipse acquisition module that is communicatively connected to each other, which is used to obtain an initial pupil ellipse in real time through a pupil rough positioning method or a pupil tracking method; And a pupil ellipse refinement module, which is used to refine the contour of the initial pupil ellipse to obtain the major axis, minor axis, center point, and angle of the refined pupil ellipse as the pupil feature parameters.

14. A method for training a pupil detection model, characterized in that, Including the steps: Obtain multiple pupil annotation samples through the pupil annotation method described in any one of claims 1 to 9; And Train the network in the pupil detection model based on the pupil annotation samples.

15. An electronic device, characterized in that, Including: A processor for executing instructions; And A storage medium communicatively connected to the processor, wherein the storage medium stores instructions that are executed by the processor to cause the processor to execute the steps in the pupil annotation method described in any one of claims 1 to 9.

16. An electronic device, characterized in that,Including: A near-eye display device; And A pupil detection model trained by the training method of the pupil detection model described in claim 14, where the pupil detection model is configured in the near-eye display device to detect the pupil in the eye image collected by the near-eye display device.

Images