Pupil movement tracking method, apparatus, device, and storage medium

Abstract

The present disclosure provides a pupil movement tracking method, device, equipment and storage medium. The method comprises: preprocessing an N+1th frame image in an eye video to obtain a corresponding binary image; performing class circle region detection on the binary image to determine position information of a suspected pupil region in the binary image; performing mask processing on the binary image based on the position information to obtain a binary image after mask processing; performing pupil contour extraction on the binary image after mask processing to obtain corresponding suspected pupil contour feature points, and screening the suspected pupil contour feature points based on the position information to obtain pupil contour feature points; determining a coefficient of variation of the pupil contour feature points in the N+1th frame image based on the pupil contour feature points and a pupil center point in an Nth frame image in the eye video; and determining a pupil center point in the N+1th frame image based on the coefficient of variation, to be used for pupil movement tracking.

Description

Technical Field

[0001] This disclosure relates to the field of ophthalmic OCT (Optical Coherence Tomography) technology, and more particularly to a pupil movement tracking method, apparatus, device, and storage medium. Background Technology

[0002] In OCT-based ocular blood flow imaging systems, eye movements can cause deviations in the amplitude, phase, and spatial position and orientation of the OCT signal, resulting in artifacts in the generated OCT images. Therefore, it is necessary to track eye movements in real time and compensate the imaging system based on these movements to help acquire high-quality OCT blood flow images.

[0003] With the rapid development of eye-tracking technology, video-based eye-tracking technology has become the mainstream of current development due to its many advantages, such as convenient equipment, low cost, and relatively simple engineering implementation.

[0004] Currently, video-based eye-tracking technology often relies on active infrared light sources and is easily affected by interference from eyelids, eyelashes, and purchin reflections, leading to inaccurate pupil center positioning and reducing the stability and robustness of pupil movement tracking. Summary of the Invention

[0005] This disclosure provides a pupil motion tracking method, apparatus, device, and storage medium to improve the accuracy of pupil center positioning, thereby improving the stability and robustness of pupil motion tracking.

[0006] In a first aspect, embodiments of this disclosure provide a pupil motion tracking method, including:

[0007] Preprocess the N+1th frame image in the eye video to obtain the corresponding binary image;

[0008] Perform circular region detection on the binary image to determine the location information of the suspected pupil region in the binary image;

[0009] The binary image is masked based on the location information of the suspected pupil region to obtain a masked binary image.

[0010] Pupil contour extraction is performed on the binary image after masking to obtain the corresponding suspected pupil contour feature points, and the suspected pupil contour feature points are filtered based on the position information of the suspected pupil region to obtain the pupil contour feature points among the suspected pupil contour feature points.

[0011] Based on the pupil contour feature points and the pupil center point in the Nth frame of the eye video, determine the coefficient of variation of the pupil contour feature points in the (N+1)th frame; and

[0012] Based on the coefficient of variation, the pupil center point in the (N+1)th frame image is determined for pupil motion tracking.

[0013] Secondly, embodiments of this disclosure also provide a pupil movement tracking device, comprising:

[0014] The image preprocessing module is configured to preprocess the N+1th frame image in the eye video to obtain the corresponding binary image;

[0015] A circular region detection module is configured to perform circular region detection on the binary image to determine the location information of the suspected pupil region in the binary image;

[0016] The image masking module is configured to perform masking processing on the binary image based on the location information of the suspected pupil region to obtain a masked binary image.

[0017] The pupil contour feature point acquisition module is configured to extract the pupil contour from the binary image behind the mask to obtain the corresponding suspected pupil contour feature points, and to filter the suspected pupil contour feature points based on the position information of the suspected pupil region to obtain the pupil contour feature points among the suspected pupil contour feature points.

[0018] The coefficient of variation determination module is configured to determine the coefficient of variation of the pupil contour feature points in the (N+1)th frame image based on the pupil contour feature points and the pupil center point in the Nth frame image of the eye video; and

[0019] The pupil center point determination module is configured to determine the pupil center point in the (N+1)th frame image based on the coefficient of variation for pupil motion tracking.

[0020] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0021] One or more processors;

[0022] Storage device, configured to store one or more programs,

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the pupil motion tracking method as described in any embodiment of this disclosure.

[0024] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the pupil motion tracking method as described in any embodiment of this disclosure.

[0025] The technical solution provided in this disclosure uses the coefficient of variation of pupil contour feature points to determine whether the pupil position in the (N+1)th frame image has changed compared to the pupil position in the Nth frame image. Based on this, the position of the pupil center point in the (N+1)th frame image is further determined. Since this technical solution eliminates interference from eyelids, eyelashes, and purchin reflection spots, it can avoid positioning deviations caused by these interferences. Thus, it can accurately locate the position information of the pupil center point in the (N+1)th frame image with lower detection parameter setting requirements, thereby improving the stability and robustness of pupil motion tracking. Attached Figure Description

[0026] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0027] Figure 1 A flowchart of a pupil motion tracking method according to an embodiment of the present disclosure is shown as an example;

[0028] Figure 2A and Figure 2B An example diagram of the original image of the N+1th frame and its binary image according to an embodiment of the present disclosure is shown;

[0029] Figure 2C An example is shown based on Figure 2B The resulting binary image after masking;

[0030] Figure 3 An exemplary diagram of pupil contour feature points extracted according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A flowchart of a pupil motion tracking method according to another embodiment of the present disclosure is shown as an example;

[0032] Figure 5A A flowchart of a pupil motion tracking method according to yet another embodiment of the present disclosure is shown as an example;

[0033] Figure 5B , Figure 5C and Figure 5D Example diagrams are shown illustrating three methods for filtering pupil contour feature points according to embodiments of the present disclosure;

[0034] Figure 5E and Figure 5F Example diagrams of polygons with and without depressions in convex hull detection according to embodiments of the present disclosure are shown in sequence.

[0035] Figure 6 A schematic diagram of a pupil motion tracking device according to an embodiment of the present disclosure is shown as an example;

[0036] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown as an example. Detailed Implementation

[0037] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0038] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0039] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0040] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] Figure 1A flowchart of a pupil motion tracking method according to an embodiment of this disclosure is illustrated. This disclosure is suitable for situations where eye tracking is achieved through pupil motion tracking based on eye video, without requiring an additional infrared light source. The method can be executed by a pupil motion tracking device, which can be implemented in software and / or hardware. Optionally, the device can be implemented using an electronic device. This electronic device may include, but is not limited to, ophthalmic OCT equipment.

[0043] like Figure 1 As shown, the pupil motion tracking method may include steps S110-S160.

[0044] In step S110, the N+1th frame image in the eye video is preprocessed to obtain the corresponding binary image.

[0045] For example, in an OCT imaging scenario, the eye video mentioned in step S110 can be an eye video captured for the patient's left or right eye. During OCT imaging, the patient's pupil movement can be located and tracked based on the video captured for the patient's eyes, thereby compensating the imaging system in OCT or OCTA imaging to assist in obtaining high-quality OCT or OCTA imaging. An eye video can contain multiple frames of raw images, where the (N+1)th frame can be the current eye image to be processed, where N is an integer and N≥0.

[0046] In this embodiment of the disclosure, the eye video mentioned in step S110 can be acquired in real time or pre-acquired.

[0047] For example, the preprocessing of the N+1th frame image mentioned in step S110 may include, but is not limited to: first, performing Gaussian filtering on the original N+1th frame image to remove image noise and initially eliminate interference factors such as eyelashes; then, using an image adaptive thresholding method to perform image binarization on the Gaussian-filtered image to obtain a preprocessed initial binary image; and then, by performing small-area denoising operations on the initial binary image, a cleaner preprocessed binary image can be obtained. This embodiment of the present disclosure can remove noise from the original image and initially filter out related interference through image preprocessing operations.

[0048] In step S120, circular region detection is performed on the binary image to determine the location information of the suspected pupil region in the binary image.

[0049] In step S120, for example, the Hough circle detection algorithm can be used to detect circular regions in the binary image obtained in step S110. It should be understood that, due to interference from eyelids, Purkinje spots, etc., when performing circular region detection on the binary image in step S120, typically two, three, or even more circular regions can be detected. For example, refer to... Figure 2A and Figure 2B , Figure 2A It is the original image. Figure 2B Based on Figure 2A The binary image obtained through preprocessing is then processed. Figure 2B The binary image shown is used for circular region detection. Circular regions formed by the upper and lower eyelids, the pupil, and the Purkinje reflection spot can be detected. Therefore, after obtaining the circular region detection results, the results can be rearranged to locate the region suspected to be the pupil (referred to as the suspected pupil region in this paper). The suspected pupil region is the most likely circular region among the detected circular regions to be the pupil region.

[0050] In one embodiment, for the first frame image (i.e., N=0) in an eye video, the suspected pupil region can be determined by utilizing the image grayscale values ​​of various circular regions in the binary image or the original image corresponding to the binary image. This is because the first frame image has no reference image, and in the imaging of the external eye structure, most of the pupil area appears black, meaning the image feature of the pupil is characterized by a small image grayscale value. Therefore, the suspected pupil region among several circular regions can be determined based on the image grayscale value.

[0051] In another embodiment, for non-first frame images (N>0) in eye-tracking videos, one approach is to determine the suspected pupil regions within several circular regions based on image grayscale values, similar to how the first frame image is processed. However, this method places strict requirements on the setting of detection parameters. Therefore, for non-first frame images in eye-tracking videos, another approach is to determine the suspected pupil regions within several circular regions based on the Euclidean distance between the center position of each circular region in the current frame and the center position of the pupil in the adjacent previous frame. The suspected pupil regions obtained using this method rely on previous pupil motion tracking results, thus the requirements on the setting of detection parameters are not very strict.

[0052] It should also be understood that, in this embodiment of the disclosure, determining the location information of the suspected pupil region includes determining its center point and radius, among other feature information. It should be noted that the center point and radius of the suspected pupil region may only be preliminary center point and radius of the pupil region, while the center point and radius of the actual pupil region need to be further determined based on this feature information.

[0053] In step S130, the binary image is masked based on the location information of the suspected pupil region to obtain the masked binary image.

[0054] After obtaining the location information of the suspected pupil area through step S120, in step S130, it is also necessary to mask the binary image obtained through step S110 based on the location information to eliminate the influence of interference factors such as eyelids on pupil motion tracking, and retain only the pupil and the surrounding area of ​​the pupil that the user is interested in for continued use in pupil motion tracking.

[0055] In one embodiment, during the masking process in step S130, a mask template with the same size as the binary image can be generated first. Then, the generated mask template is multiplied by the binary image, thereby filtering out areas of no interest to the user and retaining only the areas of interest. It should be understood that in this embodiment, the area of ​​interest to the user should be the pupil and its surrounding area, i.e., the aforementioned suspected pupil area.

[0056] For example, the process of generating the above-mentioned mask template may include: obtaining the location information of the suspected pupil region according to step S120, first generating a rectangular mask template with the center point of the suspected pupil region as the center and the side length being twice the radius of the suspected pupil region (or (radius × (1+x%)) or (radius × (1-x%))); then using an image dilation algorithm to dilate the rectangular mask template to obtain a rectangular mask template with rounded corners, which is the mask template with the same size as the binary image.

[0057] For example, refer to Figure 2B and Figure 2C , Figure 2B It is the binary image before the mask. Figure 2C This is the binary image after masking. As you can see, the binary image after masking eliminates interference from eyebrows, lower eyelids, and part of the upper eyelids, retaining only the pupil and the area around the pupil that the user is interested in.

[0058] In step S140, pupil contours are extracted from the masked binary image to obtain corresponding suspected pupil contour feature points. Based on the location information of the suspected pupil region, the suspected pupil contour feature points are filtered to obtain pupil contour feature points among the suspected pupil contour feature points.

[0059] Return to continue reading Figure 2CAs can be seen, although image preprocessing and masking can eliminate interference from elements such as eyebrows, lower eyelids, and parts of the upper eyelids, some interference from the upper eyelids and eyelashes still exists in the remaining suspected pupil area and surrounding pupil area, which can still affect pupil motion tracking. Therefore, in step S140, pupil contour feature points can be extracted and filtered to obtain cleaner and more realistic pupil contour feature points, thereby providing a data foundation and support for accurate pupil motion tracking.

[0060] For example, in step S140, an edge feature point extraction algorithm can be used to extract pupil contour feature points from the masked binary image, and the extracted pupil contour feature points are marked to obtain contour feature points about the white area in the masked binary image, i.e., feature points of suspected pupil contours. The extracted suspected pupil contour feature points can be further filtered to identify the true pupil contour feature points.

[0061] For example, pupil contour feature points can be filtered using both distance filtering and angle filtering algorithms. In some embodiments, only distance filtering may be performed; in some embodiments, only angle filtering may be performed; in some embodiments, distance filtering may be performed first, followed by angle filtering; in some embodiments, angle filtering may be performed first, followed by distance filtering; this disclosure does not limit the scope of the embodiments.

[0062] refer to Figure 3 After distance and angle filtering, interference from the upper eyelid, upper eyelashes, and stray light spots can be eliminated, leaving only the true pupil outline feature points.

[0063] In step S150, the coefficient of variation of the pupil contour feature points in the N+1th frame image is determined based on the pupil contour feature points and the pupil center point in the Nth frame image of the eye video.

[0064] It should be understood that the Nth frame refers to the previous image frame adjacent to the (N+1)th frame in the same eye video. The coefficient of variation, also known as the coefficient of dispersion, is a normalized measure of the dispersion of a probability distribution. In this embodiment, the coefficient of variation can represent the degree of change between the pupil center point in the (N+1)th frame and the pupil center point in the Nth frame. Based on the coefficient of variation, it can be determined whether the pupil center point in the (N+1)th frame has actually changed compared to the pupil center point in the Nth frame.

[0065] In this embodiment of the disclosure, the Euclidean distance between each pupil contour feature point in the N+1th frame image and the pupil center point in the Nth frame image can be calculated, and the coefficient of variation of the pupil contour feature points in the N+1th frame image relative to the pupil center point in the Nth frame image can be determined based on these Euclidean distances.

[0066] For example, the process of determining the coefficient of variation in step S150 may include, but is not limited to: determining the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature points and the position information of the pupil center point in the Nth frame image of the eye video; determining the standard deviation and average value corresponding to each distance; and determining the coefficient of variation of the pupil contour feature points in the N+1th frame image based on the determined standard deviation and average value.

[0067] In practical terms, the distance between each pupil contour feature point in the (N+1)th frame and the pupil center point in the Nth frame can be understood as Euclidean distance. Using the Euclidean distance calculation formula, based on the position coordinates of each pupil contour feature point in the (N+1)th frame and the position coordinates of the pupil center point in the Nth frame, the Euclidean distance between these two points can be calculated. All calculated distances are then averaged to obtain the mean, and the square root of the variance of each distance is taken to obtain the standard deviation. Finally, the standard deviation of each distance is divided by the mean to obtain the coefficient of variation of the pupil contour feature point in the (N+1)th frame relative to the pupil center point in the Nth frame.

[0068] It should be noted that since only non-first frame images in eye videos have an adjacent previous frame image, and the first frame image in eye videos does not have an adjacent previous frame image, the first frame image does not have a coefficient of variation. Only when the N+1th frame image is a non-first frame image can the coefficient of variation corresponding to the N+1th frame image be determined through step S150.

[0069] In step S160, the pupil center point in the N+1th frame image is determined based on the coefficient of variation for pupil motion tracking.

[0070] After obtaining the coefficient of variation of the pupil contour feature points in the (N+1)th frame image relative to the pupil center point in the Nth frame image in step S150, step S160 can detect this coefficient of variation to determine whether the degree of pupil change corresponding to this coefficient of variation has reached the minimum degree of pupil change. If the detection result shows that the degree of pupil change indicated by the current coefficient of variation has reached the minimum degree of pupil change, it indicates that the pupil center point in the (N+1)th frame image has indeed moved relative to the pupil center point in the Nth frame image. At this time, all the pupil contour feature points identified in the (N+1)th frame image are accurate pupil contour feature points, so a new pupil center point can be re-determined based on all the identified pupil contour feature points as the pupil center point in the (N+1)th frame image. For example, an ellipse fitting can be performed based on all the pupil contour feature points identified in the (N+1)th frame image, and the center of the fitted ellipse can be used as the pupil center point in the (N+1)th frame image. If the pupil change corresponding to the coefficient of variation does not reach the minimum pupil change, it indicates that the pupil center point in the (N+1)th frame image has not moved relative to the pupil center point in the Nth frame image, or although it has moved, the amount of movement is very small and can be ignored. In this case, the pupil center point in the Nth frame image can be directly determined as the pupil center point corresponding to the N+1th frame image. This embodiment uses the coefficient of variation to accurately locate the position information of the pupil center point in the N+1th frame image, thereby improving the accuracy of pupil motion tracking.

[0071] It should be noted that since the first frame of the eye video does not have a previous frame adjacent to it, when determining the pupil center point in the first frame image, after obtaining the pupil contour feature points in the first frame image through step S140, ellipse fitting can be directly performed on all pupil contour feature points in the first frame image, and the center point of the fitted ellipse can be used as the pupil center point in the first frame image.

[0072] For example, for non-first frame images in an eye video, the process of determining the pupil center point in the N+1th frame image based on the coefficient of variation in step S160 may include, but is not limited to: in response to the coefficient of variation being greater than or equal to a preset coefficient, performing ellipse fitting based on pupil contour feature points to determine the pupil center point in the N+1th frame image; and / or in response to the coefficient of variation being less than a preset coefficient, determining that the pupil center point in the N+1th frame image coincides with the pupil center point in the Nth frame image.

[0073] It should be understood that the preset coefficient is the minimum coefficient of variation corresponding to pupil movement as predetermined. By detecting whether the coefficient of variation of the pupil contour feature points in the (N+1)th frame image relative to the pupil center point in the Nth frame image is greater than or equal to this preset coefficient, it can be determined whether the pupil center point in the (N+1)th frame image has moved relative to the pupil center point in the Nth frame image. For example, if the coefficient of variation is greater than or equal to the preset coefficient, it indicates that the pupil center point in the (N+1)th frame image has moved relative to the pupil center point in the Nth frame image. In this case, based on the least squares method, an ellipse can be fitted to all pupil contour feature points in the (N+1)th frame image, and the center point of the fitted ellipse can be determined as the position of the pupil center point in the (N+1)th frame image. When the coefficient of variation is less than a preset coefficient, it indicates that the pupil center point in the (N+1)th frame image has not moved relative to the pupil center point in the Nth frame image, or although it has moved, the amount of movement is very small and can be ignored. In this case, it can be directly determined that the pupil center point in the Nth frame image is the same as the pupil center point in the (N+1)th frame image. In other words, in this case, it is determined that the pupil center point in the Nth frame image coincides with the pupil center point in the (N+1)th frame image. This embodiment of the present disclosure can accurately locate the position information of the pupil center point in non-first frame images using the coefficient of variation, improving the accuracy of pupil motion tracking.

[0074] In this embodiment, by preprocessing the (N+1)th frame of the eye video to obtain the corresponding binary image, noise in the original image can be removed, and interference factors such as eyelashes can be initially eliminated. Then, circular region detection is performed on the binary image to determine the location information of the suspected pupil region. Based on the location information of the suspected pupil region, the binary image is masked to obtain a masked binary image, which can eliminate the influence of interference factors such as eyelids on pupil motion tracking. Finally, pupil contour extraction is performed on the masked binary image to obtain the corresponding suspected pupil contour feature points, and based on the suspected pupil region... Location information is used to filter suspected pupil contour feature points to obtain the pupil contour feature points among the suspected pupil contour feature points, which can further eliminate interference from the upper eyelid, upper eyelashes, and stray spots. Based on the pupil contour feature points and the pupil center point in the Nth frame of the eye video, the coefficient of variation of the pupil contour feature points in the N+1th frame is determined, and based on the coefficient of variation, the pupil center point in the N+1th frame is determined. With lower detection parameter settings, the position information of the pupil center point in the N+1th frame can be accurately located, improving the stability and robustness of pupil motion tracking.

[0075] As an optional embodiment, step S110 may include: performing Gaussian filtering on the (N+1)th frame image in the eye video to obtain the filtered (N+1)th frame image; performing binarization processing on the filtered (N+1)th frame image based on an adaptive threshold processing method to obtain an initial binary image; determining the connected area corresponding to each connected region in the initial binary image, and filtering out connected regions whose connected area is less than or equal to a preset area to obtain the filtered binary image.

[0076] It should be understood that a connected region can refer to an image region in the initial binary image composed of foreground pixels with the same pixel value and adjacent positions, such as a connected region composed of white pixels. The preset area can refer to the maximum area of ​​a pre-defined interference region.

[0077] For example, the Gaussian filtering process may include, but is not limited to: for each pixel in the (N+1)th frame image, a weighted average of the original pixel value of that pixel and the original pixel values ​​of its neighboring pixels can be performed, and the result is determined as the pixel value of that pixel in the filtered (N+1)th frame image, thus obtaining the filtered (N+1)th frame image. Gaussian filtering can remove image noise and initially eliminate influencing factors such as eyelash interference. The adaptive thresholding process may include, but is not limited to: dividing the filtered (N+1)th frame image into multiple regions, and obtaining the threshold for each region by weighted averaging the pixel values ​​of each pixel in each region, comparing the pixel value of each pixel in each region with the threshold of that region, setting the pixel value of pixels with values ​​higher than the threshold to 1 in the binary image, thus becoming white pixels, and setting the pixel value of pixels with values ​​lower than the threshold to 0 in the binary image, thus becoming black pixels, thereby obtaining an initial binary image with significant differences in brightness. After obtaining the initial binary image, each connected region in the initial binary image can be identified, and the connected area corresponding to each connected region can be calculated. The connected area corresponding to each connected region is compared with a preset area. If the connected area corresponding to a certain connected region is less than or equal to the preset area, it indicates that the connected region is an interference region. At this time, the connected region can be removed, for example, by setting the pixel value of each pixel in the connected region to 0. This removes small interference regions in the initial binary image, resulting in a cleaner binary image, further improving the preprocessing effect, and thus improving the accuracy of pupil motion tracking.

[0078] Figure 4 A flowchart of a pupil motion tracking method according to another embodiment of the present disclosure is shown as an example.

[0079] like Figure 4 As shown, the pupil motion tracking method may include the following steps:

[0080] In step S110, the N+1th frame image in the eye video is preprocessed to obtain the corresponding binary image.

[0081] This step and Figure 1 The steps S110 shown are the same or similar, and will not be described again in this embodiment.

[0082] In step S410, circular region detection is performed on the binary image to obtain the position information of each circular region in the binary image.

[0083] In step S410, all circular regions in the binary image can be detected using the Hough circle detection method to obtain the positional information of each detected circular region. For example, the positional information of the detected circular regions may include feature information such as the center point coordinates (a, b) and radius r of the circular region. The Hough circle detection process may include, but is not limited to, using a circle expression (xa). 2 +(yb) 2 =r 2 By obtaining the curve expressions x = a + rcos(θ) and y = b + rsin(θ), intersection detection can be performed on the curves corresponding to the location information of each circular region. Based on the curve intersection points, a = x - rcos(θ) and b = y - rsin(θ), the location information of each circular region in the binary image can be obtained. Using the Hough circle detection method, all circular regions can be detected more accurately and quickly, further improving the accuracy and efficiency of pupil localization.

[0084] In step S420, based on the position information of each circular region, a suspected pupil region is determined from each circular region, and the position information corresponding to the suspected pupil region is obtained.

[0085] In step S420, since multiple circular regions can be detected in the binary image, it is necessary to rearrange all the detected circular regions to locate the most likely circular region of the pupil, i.e., the suspected pupil region, and determine the location information of the located circular region as the location information corresponding to the suspected pupil region.

[0086] When the (N+1)th frame is not the first frame in the eye video, as one implementation, the process of determining the suspected pupil region in step S420 can be as follows: based on the image grayscale value corresponding to each circular region, the suspected pupil region is determined from each circular region. Here, the image grayscale value can refer to the sum of the pixel grayscale values ​​of the circular region in the binary image, or it can refer to the sum of the pixel grayscale values ​​of the circular region in the original image corresponding to the binary image. For example, the circular region with the smallest image grayscale value can be determined as the suspected pupil region.

[0087] Alternatively, if the N+1th frame image is not the first frame image in the eye video, as another implementation, the process of determining the suspected pupil region in step S420 can also be as follows: based on the center point of each circular region in each circular region and the center point of the pupil region in the Nth frame image, determine the Euclidean distance between each circular region and the pupil region; and based on the Euclidean distance between each circular region and the pupil region, determine the suspected pupil region from each circular region.

[0088] In this embodiment, when the (N+1)th frame is not the first frame in the eye video, the Euclidean distance between the center point of each circular region and the center point of the pupil region in the Nth frame can be determined based on the center point location information of each circular region and the center point location information of the pupil region in the Nth frame. The circular region with the smallest Euclidean distance can be identified as the suspected pupil region. This is because pupil movement is continuous, and for adjacent frames, the distance deviation between the two pupil center points in the two images will not be too large after pupil movement. Therefore, the circular region with the smallest Euclidean distance can be identified as the suspected pupil region. For non-first frame images, by relying on the pupil region in the previous adjacent image frame, the suspected pupil region in the non-first frame image can be located more accurately and conveniently, improving the efficiency and accuracy of pupil localization.

[0089] When the N+1th frame image is the first frame image in the eye video, the process of determining the suspected pupil region in step S420 can be as follows: determine the image grayscale value of each circular region based on the position information of each circular region; and determine the suspected pupil region from each circular region based on the image grayscale value of each circular region.

[0090] In this embodiment, determining the image grayscale value of each circular region may include: determining the image grayscale value of each circular region in the binary image; or, determining the image grayscale value of each circular region in the binary image in the corresponding N+1th frame of the original image. In the first frame image, the circular region with the smallest image grayscale value can be identified as the suspected pupil region, thereby ensuring the overall effect of pupil motion tracking.

[0091] In step S130, the binary image is masked based on the location information of the suspected pupil region to obtain the masked binary image.

[0092] In step S140, pupil contours are extracted from the masked binary image to obtain corresponding suspected pupil contour feature points. Based on the location information of the suspected pupil region, the suspected pupil contour feature points are filtered to obtain pupil contour feature points among the suspected pupil contour feature points.

[0093] In step S150, the coefficient of variation of the pupil contour feature points in the N+1th frame image is determined based on the pupil contour feature points and the pupil center point in the Nth frame image of the eye video.

[0094] In step S160, the pupil center point in the N+1th frame image is determined based on the coefficient of variation for pupil motion tracking.

[0095] In this embodiment, steps S130 to S160 are... Figure 1 The steps S130 to S160 shown are the same or similar, and will not be described again in this embodiment.

[0096] In this embodiment, by performing circular region detection on a binary image, the position information of each circular region in the binary image is obtained, and based on the position information of each circular region, a suspected pupil region is determined from each circular region. This can eliminate the influence of circular regions formed by interference from eyelids, Purkinje spots, etc., and screen out the most likely suspected pupil region, thereby further improving the accuracy of pupil movement tracking.

[0097] Figure 5A A flowchart of a pupil motion tracking method according to yet another embodiment of the present disclosure is shown as an example.

[0098] like Figure 5A As shown, the pupil motion tracking method specifically includes the following steps:

[0099] In step S110, the N+1th frame image in the eye video is preprocessed to obtain the corresponding binary image.

[0100] In step S120, circular region detection is performed on the binary image to determine the location information of the suspected pupil region in the binary image.

[0101] In step S130, the binary image is masked based on the location information of the suspected pupil region to obtain the masked binary image.

[0102] In this embodiment, steps S110 to S130 and Figure 1 The steps S110 to S130 shown are the same or similar, and will not be described again in this embodiment.

[0103] In step S510, pupil contours are extracted from the masked binary image to obtain the corresponding suspected pupil contour feature points.

[0104] In step S520, based on the location information of the suspected pupil region and the location information of each suspected pupil contour feature point, distance filtering is performed on each suspected pupil contour feature point to obtain the first candidate pupil contour feature point.

[0105] It should be understood that extracting pupil contour feature points from the masked binary image may result in impurity feature points due to interference from parts of the upper eyelid, upper eyelashes, stray spots, etc. Therefore, to obtain cleaner pupil contour feature points, it is necessary to filter out these impurity feature points. The first candidate pupil contour feature point is a feature point whose distance matches the pupil contour feature after initial screening. When performing distance filtering in step S520, the Euclidean distance between each suspected pupil contour feature point and the center point of the suspected pupil region obtained in step S120 can be determined first. Then, it can be determined whether the difference between each distance value and the radius of the suspected pupil region obtained in step S120 is within a preset error range. If the difference is within the preset error range, it indicates that the corresponding suspected pupil contour feature point may be a real pupil contour feature point. This suspected pupil contour feature point can be determined as the first candidate pupil contour feature point and retained. Otherwise, if the difference exceeds the preset error range, it indicates that the corresponding suspected pupil contour feature point is an interfering feature point such as the upper eyelid, upper eyelashes, or stray light spots. In this case, the suspected pupil contour feature point can be screened out. By using distance filtering, potentially real pupil contour feature points can be initially selected from the suspected pupil contour feature points, and interference from related impurity feature points can be filtered out.

[0106] As an optional embodiment, step S520 may include: determining the distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil region based on the pupil center point location information and the location information of each suspected pupil contour feature point; and determining the difference between each distance and the pupil radius, and determining the suspected pupil contour feature points whose absolute difference is less than or equal to a preset difference as the first candidate pupil contour feature points.

[0107] The location information of the suspected pupil region includes: the center point of the suspected pupil region (hereinafter referred to as the suspected pupil center point) and the radius r of the suspected pupil, etc. The preset difference can be (r×a%). The distance filtering process is as follows: Based on the location information of the pupil center point of the suspected pupil region and the location information of each suspected pupil contour feature point, the Euclidean distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil region is determined; the Euclidean distance corresponding to each suspected pupil contour feature point is subtracted from the pupil radius of the suspected pupil region to obtain the corresponding difference, and each suspected pupil contour feature point whose absolute difference is less than or equal to the preset difference is determined as the first candidate pupil contour feature point. Suspected pupil contour feature points whose absolute difference is greater than the preset difference are not true pupil contour feature points, but may be interfering feature points caused by eyelids, eyelashes, etc. Distance filtering can initially eliminate the influence of interfering feature points such as eyelids and eyelashes, and improve the accuracy of pupil movement tracking.

[0108] In step S530, the first candidate pupil contour feature points are filtered by angle to obtain the second candidate pupil contour feature points.

[0109] It should be understood that eyelashes vary in length, especially for those with longer eyelashes. In some cases, distance-based filtering alone may not be sufficient to completely eliminate interference from eyelashes. Therefore, angle-based filtering can be used to further filter out impurity feature points caused by eyelashes. The second candidate pupil contour feature points are feature points whose angles also match the pupil contour features, selected from the first candidate pupil contour feature points whose distance matches the pupil contour features.

[0110] After initial screening of suspected pupil contour feature points, interference from the lower eyelid and lower eyelashes is largely eliminated. Therefore, in the secondary screening, only impurity feature points caused by interference from the upper eyelid and upper eyelashes need to be excluded. Thus, in step S530, when performing angle screening, a horizontal line passing through the center point of the suspected pupil region can be used as the horizontal boundary line, and only the first candidate pupil contour feature points (hereinafter referred to as target candidate pupil contour feature points) located above this horizontal boundary line can be subjected to secondary screening. This embodiment of the present disclosure can achieve secondary screening through the following implementation method.

[0111] One implementation method is to establish a rectangular coordinate system with the center point of the suspected pupil as the origin, connect each target candidate pupil contour feature point to the center point of the suspected pupil area, and use the acute angle ∠α (∠α is between 0° and 90°) formed by each line and the horizontal dividing line (X coordinate axis) or the angle ∠β (∠β is between 0° and 180°) formed by each line and the positive X-axis (positive X-axis direction) for angle screening.

[0112] For example, for an acute angle ∠α, it can be determined whether ∠α is less than or equal to a first preset angle. If ∠α is less than or equal to the first preset angle, the corresponding target candidate pupil contour feature point is retained; otherwise, if ∠α is greater than the first preset angle, the corresponding target candidate pupil contour feature point is filtered out.

[0113] For example, for an acute angle ∠α, if the corresponding target candidate pupil contour feature point is a coordinate point in the first quadrant, it can be determined whether ∠α is less than or equal to a second preset angle. If ∠α is less than or equal to the second preset angle, the corresponding target candidate pupil contour feature point is retained; otherwise, if ∠α is greater than the second preset angle, the corresponding target candidate pupil contour feature point is filtered out. If the corresponding target candidate pupil contour feature point is a coordinate point in the second quadrant, it can be determined whether ∠α is less than or equal to a third preset angle. If ∠α is less than or equal to the third preset angle, the corresponding target candidate pupil contour feature point is retained; otherwise, if ∠α is greater than the third preset angle, the corresponding target candidate pupil contour feature point is filtered out. The second preset angle and the third preset angle are different.

[0114] For example, for the included angle ∠β, it can be determined whether ∠β is less than or equal to a second preset angle, or whether ∠β is greater than or equal to a fourth preset angle but less than 180°. If ∠β is less than or equal to the second preset angle, the corresponding target candidate pupil contour feature point is retained; if ∠β is greater than or equal to the fourth preset angle and less than 180°, the corresponding target candidate pupil contour feature point is also retained; otherwise, if ∠α is greater than the second preset angle and less than the fourth preset angle, the corresponding target candidate pupil contour feature point is filtered out. The fourth preset angle is complementary to the aforementioned third preset angle.

[0115] refer to Figure 5B A rectangular coordinate system is established with the pupil center point as the origin, the horizontal line passing through the pupil center point as the x-axis, and the vertical line passing through the pupil center point as the y-axis. If a first candidate pupil contour feature point is located above the x-axis and within the first quadrant, then the angle formed by the line connecting the first candidate pupil contour feature point and the pupil center point as one side and the positive x-axis as the other side is ∠θ. If ∠θ satisfies: 0°<∠θ≤∠θ1, then the first candidate pupil contour feature point is retained. Otherwise, if ∠θ1<∠θ≤∠90°, then the first candidate pupil contour feature point is discarded.

[0116] See also Figure 5B If a first candidate pupil contour feature point is located above the x-axis and within the second quadrant, then the angle ∠θ is formed by the line connecting the first candidate pupil contour feature point and the pupil center point as one side and the negative x-axis as the other side. If ∠θ satisfies 0°<∠θ≤∠θ2, then the first candidate pupil contour feature point is retained. Otherwise, if ∠θ2<∠θ≤∠90°, then the first candidate pupil contour feature point is discarded.

[0117] refer to Figure 5CUsing the horizontal line passing through the center of the pupil as the x-axis, if a first candidate pupil contour feature point is located above the x-axis, and the angle ∠γ (∠γ ranging from 0° to 180°) formed by the line connecting it to the center of the pupil as one side and the positive direction of the x-axis as the other side satisfies: 0°<∠γ≤∠θ1 or 180°>∠γ≥∠θ3, where the angles corresponding to ∠θ1 and ∠θ3 are preset angles, then the first candidate pupil contour feature point is considered a true pupil contour feature point and is retained. Otherwise, if the angle ∠γ satisfies: ∠θ1<∠γ<∠θ3, then the first candidate pupil contour feature point may be an impurity feature point caused by interference from the upper eyelid, upper eyelashes, or stray light spots, and such a feature point can be directly discarded.

[0118] refer to Figure 5D A rectangular coordinate system is established with the center of the pupil as the origin, the horizontal line passing through the center of the pupil as the x-axis, and the vertical line passing through the center of the pupil as the y-axis. Since the pupil is usually circular or elliptical, it is essentially symmetrical about the y-axis, and some people's eyes may also be symmetrical about the y-axis. In this case, if the upper eyelid and upper eyelashes obscure the pupil, this obscuring is also essentially symmetrical about the y-axis. Therefore, for example, when performing angle filtering, only the first candidate pupil contour feature points located in the first quadrant can be filtered, and then the symmetry between the first candidate pupil contour feature points in the first quadrant and the first candidate pupil contour feature points in the second quadrant can be used to filter the first candidate pupil contour feature points in the second quadrant. Alternatively, conversely, only the first candidate pupil contour feature points located in the second quadrant can be filtered, and then the symmetry between the first candidate pupil contour feature points in the first quadrant and the first candidate pupil contour feature points in the second quadrant can be used to filter the first candidate pupil contour feature points in the first quadrant. For example, first, take the line connecting the first candidate pupil contour feature point located in the first quadrant and the pupil center point as one side, and the positive x-axis as the other side, forming an angle ∠δ. If ∠δ satisfies: 0°<∠δ≤∠θ1, then retain the first candidate pupil contour feature point and record it as the second candidate pupil contour feature point. Otherwise, if ∠θ1<∠δ≤∠90°, then discard the first candidate pupil contour feature point. After filtering out the second candidate pupil contour feature points located in the first quadrant, utilize the symmetry between the first candidate pupil contour feature points in the first quadrant and the first candidate pupil contour feature points in the second quadrant to determine the first candidate pupil contour feature point in the second quadrant corresponding to each second candidate pupil contour feature point located in the first quadrant, and record it as the second candidate pupil contour feature point, thus filtering out the second candidate pupil contour feature points located in the second quadrant.

[0119] In step S540, pupil contour feature points are determined based on the second candidate pupil contour feature points.

[0120] As mentioned above, based on the external structure of the eye, the pupil region located below the horizontal dividing line is not significantly affected by factors such as the lower eyelid, lower eyelashes, and stray light spots. Therefore, the first candidate pupil contour feature point located below the horizontal dividing line can be directly determined as the true pupil contour feature point. Simultaneously, in this embodiment, the second candidate pupil contour feature point can also be directly determined as the true pupil contour feature point.

[0121] In other embodiments of this disclosure, the second candidate pupil contour feature point and the first candidate pupil contour feature point located below the horizontal dividing line can be treated as a whole. Using convex hull detection, the candidate pupil contour feature points forming the convex hull of the second candidate pupil contour feature point and the first candidate pupil contour feature point located below the horizontal dividing line can be determined as the true pupil contour feature points. In these embodiments, through the above-described secondary screening, the problem of positioning deviations in pupil movement tracking caused by partial obstruction of the pupil by the upper eyelid and upper eyelashes can be further eliminated, thereby further improving the accuracy and stability of pupil movement tracking.

[0122] refer to Figure 5E In this embodiment, the convex hull detection process may include, but is not limited to: establishing a rectangular coordinate system with the center point of the suspected pupil region as the origin, the horizontal line passing through the center point as the x-axis, and the vertical line passing through the center point as the y-axis. A convex hull detection point M is set on the positive half-axis of the y-axis. The convex hull detection point M can be determined based on the point in the real pupil region that is farthest from the center point of the real pupil region. For example, the distance L between the point in the real pupil region that is farthest from the center point of the real pupil region and the center point can be directly used as the distance between the convex hull detection point M and the aforementioned coordinate origin, or L / 2 can be used as the distance between the convex hull detection point M and the aforementioned coordinate origin, etc. The convex hull detection point M only needs to be located on the positive half-axis of the y-axis; the specific location of it is not limited in this embodiment.

[0123] Continue to refer to Figure 5E The process involves connecting all the second candidate pupil contour feature points selected in step S530, all the first candidate pupil contour feature points located below the horizontal dividing line, and the pairs of adjacent points in the convex hull detection points M to construct a polygon. The polygon is then tested using convex hull detection to determine if any depressions exist within it. Figure 5E As shown, the polygon constructed in the figure has 4 depressions (as indicated by the dashed circles). The feature points at each depression location are deleted. Then, based on all the remaining points, the adjacent points are connected to each other to construct a new polygon. Convex hull detection and feature point deletion at the depression locations are performed again. The above operation is repeated until the constructed polygon no longer has depressions. Figure 5E After deleting the feature points at each concave location from the polygon constructed in the image, the polygon is reconstructed as follows: Figure 5F As shown. Figure 5F The polygons in the image no longer have indentations, and can be... Figure 5F All feature points except for the convex hull detection point M are used as the true pupil contour feature points.

[0124] In this embodiment of the disclosure, convex hull detection can further eliminate impurity feature points caused by interference from bright spots such as stray light spots, thereby further improving the accuracy of pupil center positioning.

[0125] In step S150, the coefficient of variation of the pupil contour feature points in the N+1th frame image is determined based on the pupil contour feature points and the pupil center point in the Nth frame image of the eye video.

[0126] In step S160, the pupil center point in the N+1th frame image is determined based on the coefficient of variation for pupil motion tracking.

[0127] In this embodiment, steps S150 to S160 are... Figure 1 The steps S150 to S160 shown are the same or similar, and will not be described again in this embodiment.

[0128] The technical solution of this disclosure embodiment, by performing distance filtering on each suspected pupil contour feature point based on the location information of the suspected pupil region and the location information of each suspected pupil contour feature point, obtains a first candidate pupil contour feature point; and performs angle filtering on the first candidate pupil contour feature point to obtain a second candidate pupil contour feature point. Thus, by combining distance filtering and angle filtering, impurity feature points caused by interference from eyelids and eyelashes are removed, further improving the accuracy of pupil center positioning.

[0129] Figure 6 An exemplary schematic diagram of a pupil motion tracking device according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, the pupil motion tracking device 600 specifically includes: an image preprocessing module 610, a near-circular region detection module 620, an image mask processing module 630, a pupil contour feature point acquisition module 640, a coefficient of variation determination module 650, and a pupil center point determination module 660.

[0130] The image preprocessing module 610 is configured to preprocess the N+1th frame image in the eye video to obtain the corresponding binary image.

[0131] The circular region detection module 620 is configured to perform circular region detection on a binary image to determine the location information of a suspected pupil region in the binary image;

[0132] The image masking processing module 630 is configured to perform masking processing on the binary image based on the location information of the suspected pupil region to obtain the masked binary image.

[0133] The pupil contour feature point acquisition module 640 is configured to extract the pupil contour from the masked binary image to obtain the corresponding suspected pupil contour feature points, and to filter the suspected pupil contour feature points based on the location information of the suspected pupil region to obtain the pupil contour feature points among the suspected pupil contour feature points.

[0134] The coefficient of variation determination module 650 is configured to determine the coefficient of variation of the pupil contour feature points in the (N+1)th frame image based on the pupil contour feature points and the pupil center point in the Nth frame image of the eye video; and

[0135] The pupil center point determination module 660 is configured to determine the pupil center point in the N+1th frame image based on the coefficient of variation for pupil motion tracking.

[0136] As an optional embodiment, the circular region detection module 620 may include:

[0137] The first location information acquisition submodule is configured to perform circular region detection on the binary image to obtain the location information of each circular region in the binary image; and

[0138] The second location information acquisition submodule is configured to determine the suspected pupil region from each of the circular regions based on the location information of each circular region, and obtain the location information corresponding to the suspected pupil region.

[0139] As an optional embodiment, the second location information acquisition submodule is configured to: determine the Euclidean distance between each circular region and the pupil region based on the location information of each circular region and the location information of the pupil region in the Nth frame image; and determine the suspected pupil region from each circular region based on the Euclidean distance between each circular region and the pupil region.

[0140] As an optional embodiment, the pupil contour feature point acquisition module 640 may include:

[0141] The first candidate pupil contour feature point acquisition submodule is configured to perform distance filtering on each suspected pupil contour feature point based on the location information of the suspected pupil region and the location information of each suspected pupil contour feature point to obtain the first candidate pupil contour feature point.

[0142] The submodule for obtaining the second candidate pupil contour feature points is configured to perform angle filtering on the first candidate pupil contour feature points to obtain the second candidate pupil contour feature points; and

[0143] The pupil contour feature point determination submodule is configured to determine pupil contour feature points based on the second candidate pupil contour feature points.

[0144] As an optional embodiment, the pupil contour feature point determination submodule is specifically configured to: perform convex hull detection based on the second candidate pupil contour feature points, and determine the candidate pupil contour feature points that form the convex hull among the second candidate pupil contour feature points as pupil contour feature points.

[0145] As an optional embodiment, the location information of the suspected pupil region includes: the location information of the pupil center point and the pupil radius;

[0146] The first candidate pupil contour feature point acquisition submodule is configured to: determine the distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil region based on the pupil center point location information and the location information of each suspected pupil contour feature point; and determine the difference between each distance and the pupil radius, and determine the suspected pupil contour feature points whose absolute difference is less than or equal to the preset difference as the first candidate pupil contour feature points.

[0147] As an optional embodiment, the second candidate pupil contour feature point acquisition submodule is configured to: determine the horizontal angle with the line connecting each first candidate pupil contour feature point and the pupil center point of the suspected pupil area as a side based on the pupil center point location information and the location information of each first candidate pupil contour feature point; and determine the first candidate pupil contour feature points with the horizontal angle less than or equal to a preset angle as the second candidate pupil contour feature points.

[0148] As an optional embodiment, the coefficient of variation determination module 650 is configured to: determine the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature points and the position information of the pupil center point in the Nth frame image of the eye video; and determine the standard deviation and mean value corresponding to each distance, and determine the coefficient of variation of the pupil contour feature points in the N+1th frame image based on the determined standard deviation and mean value.

[0149] As an optional embodiment, the pupil center point determination module 660 is configured to: in response to a coefficient of variation being greater than or equal to a preset coefficient, perform ellipse fitting based on pupil contour feature points to determine the pupil center point in the N+1th frame image; and / or in response to a coefficient of variation being less than a preset coefficient, determine that the pupil center point in the N+1th frame image coincides with the pupil center point in the Nth frame image.

[0150] As an optional embodiment, when the N+1th frame image is the first frame image in the eye video, the second position information acquisition submodule is further configured to: determine the image grayscale value of each circular region based on the position information of each circular region; and determine the suspected pupil region from each circular region based on the image grayscale value of each circular region.

[0151] The pupil movement tracking device provided in this disclosure can execute the pupil movement tracking method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0152] The pupil motion tracking device provided in this embodiment addresses the same or similar technical problems and achieves the same technical effects as the pupil motion tracking method provided in the foregoing embodiments, and will not be described again in this embodiment.

[0153] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0154] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown as an example. Reference is made below. Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 7 The diagram shows the structure of an ophthalmic OCT device 500. The electronic device in this embodiment can be an ophthalmic OCT device, etc. The electronic device in this embodiment is merely an example and should not be construed as limiting the functionality or scope of use of this embodiment.

[0155] like Figure 7 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0156] The following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. However, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0157] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0158] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0159] The electronic device provided in this embodiment and the pupil movement tracking method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0160] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the pupil movement tracking method provided in the above embodiments.

[0161] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0162] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0163] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0164] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0167] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0168] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0169] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A pupil motion tracking method, comprising: Preprocess the N+1th frame image in the eye video to obtain the corresponding binary image; Perform circular region detection on the binary image to determine the location information of the suspected pupil region in the binary image; The binary image is masked based on the location information of the suspected pupil region to obtain a masked binary image. Pupil contour extraction is performed on the binary image after masking to obtain the corresponding suspected pupil contour feature points, and the suspected pupil contour feature points are filtered based on the position information of the suspected pupil region to obtain the pupil contour feature points among the suspected pupil contour feature points. Based on the position information of the pupil contour feature points and the position information of the pupil center point in the Nth frame image of the eye video, the distance between each pupil contour feature point and the pupil center point is determined. Determine the standard deviation and mean value corresponding to each distance, and based on the determined standard deviation and mean value, determine the coefficient of variation of the pupil contour feature points in the N+1th frame image; In response to the coefficient of variation being greater than or equal to a preset coefficient, ellipse fitting is performed based on the pupil contour feature points to determine the pupil center point in the N+1th frame image. and / or In response to the coefficient of variation being less than the preset coefficient, it is determined that the pupil center point in the (N+1)th frame image coincides with the pupil center point in the Nth frame image.

2. The method according to claim 1, wherein, Perform circular region detection on the binary image to determine the location information of the suspected pupil region in the binary image, including: Perform circular region detection on the binary image to obtain the positional information of each circular region in the binary image; and Based on the location information of each of the circular regions, a suspected pupil region is determined from each of the circular regions, and the location information corresponding to the suspected pupil region is obtained.

3. The method according to claim 2, wherein, Based on the location information of each of the circular regions, a suspected pupil region is determined from each of the circular regions, including: Based on the position information of each of the circular regions and the position information of the pupil region in the Nth frame image, the Euclidean distance between each circular region and the pupil region is determined; and Based on the Euclidean distance between each of the circular regions and the pupil region, a suspected pupil region is determined from each of the circular regions.

4. The method according to claim 1, wherein, Based on the location information of the suspected pupil region, the suspected pupil contour feature points are filtered to obtain the pupil contour feature points among the suspected pupil contour feature points, including: Based on the location information of the suspected pupil region and the location information of each suspected pupil contour feature point, distance filtering is performed on each suspected pupil contour feature point to obtain the first candidate pupil contour feature point. Angle filtering is performed on the first candidate pupil contour feature points to obtain the second candidate pupil contour feature points; and Based on the second candidate pupil contour feature points, pupil contour feature points are determined.

5. The method according to claim 4, wherein, Based on the second candidate pupil contour feature points, pupil contour feature points are determined, including: Convex hull detection is performed based on the second candidate pupil contour feature points, and the candidate pupil contour feature points that form the convex hull among the second candidate pupil contour feature points are determined as the pupil contour feature points.

6. The method according to claim 4, wherein, The location information of the suspected pupil area includes: the location information of the pupil center point and the pupil radius; Based on the location information of the suspected pupil region and the location information of each suspected pupil contour feature point, distance filtering is performed on each suspected pupil contour feature point to obtain a first candidate pupil contour feature point, including: Based on the pupil center point location information and the location information of each suspected pupil contour feature point, determine the distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil region; and The difference between each distance and the pupil radius is determined, and suspected pupil contour feature points whose absolute difference is less than or equal to a preset difference are determined as first candidate pupil contour feature points.

7. The method according to claim 6, wherein, Angle filtering is performed on the first candidate pupil contour feature points to obtain the second candidate pupil contour feature points, including: Based on the pupil center point location information and the location information of each first candidate pupil contour feature point, a horizontal angle is determined with the line connecting each first candidate pupil contour feature point to the pupil center point of the suspected pupil region as its side; and The first candidate pupil contour feature point with a horizontal angle less than or equal to a preset angle is determined as the second candidate pupil contour feature point.

8. The method according to claim 2, wherein, In the case that the (N+1)th frame image is the first frame image in the eye video. Based on the location information of each of the circular regions, a suspected pupil region is determined from each of the circular regions, including: Based on the location information of each type of circular region, the image grayscale value of each type of circular region is determined; and Based on the image grayscale values ​​of each of the circular regions, the suspected pupil region is determined from each of the circular regions.

9. A pupil motion tracking device, comprising: The image preprocessing module is configured to preprocess the N+1th frame image in the eye video to obtain the corresponding binary image; A circular region detection module is configured to perform circular region detection on the binary image to determine the location information of the suspected pupil region in the binary image; The image masking module is configured to perform masking processing on the binary image based on the location information of the suspected pupil region to obtain a masked binary image. The pupil contour feature point acquisition module is configured to extract the pupil contour from the binary image behind the mask to obtain the corresponding suspected pupil contour feature points, and to filter the suspected pupil contour feature points based on the position information of the suspected pupil region to obtain the pupil contour feature points among the suspected pupil contour feature points. The coefficient of variation determination module is configured to determine the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature points and the position information of the pupil center point in the Nth frame image of the eye video; and to determine the standard deviation and mean value corresponding to each distance, and to determine the coefficient of variation of the pupil contour feature points in the N+1th frame image based on the determined standard deviation and mean value. as well as The pupil center point determination module is configured to perform ellipse fitting based on the pupil contour feature points in response to the coefficient of variation being greater than or equal to a preset coefficient, so as to determine the pupil center point in the N+1th frame image. And / or in response to the coefficient of variation being less than the preset coefficient, determine that the pupil center point in the (N+1)th frame image coincides with the pupil center point in the Nth frame image.

10. An electronic device, comprising: One or more processors; Storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the pupil motion tracking method as described in any one of claims 1-8.

11. A storage medium containing computer-executable instructions, wherein, The computer-executable instructions, when executed by a computer processor, are used to perform the pupil movement tracking method as described in any one of claims 1-8.

Images