Pupil motion tracking method and apparatus, and device and storage medium

Abstract

Provided in the present application are a pupil motion tracking method and apparatus, and a device and a storage medium. The pupil motion tracking method comprises: performing pre-processing on an (N+1)th frame of image in an eye video, so as to obtain a corresponding binary image (S110); performing quasi-circular region detection on the binary image, so as to determine position information of a suspected pupil region in the binary image (S120); performing masking processing on the binary image on the basis of the position information of the suspected pupil region, so as to obtain a masked binary image (S130); performing pupil contour extraction on the masked binary image, so as to obtain corresponding suspected pupil contour feature points, and screening the suspected pupil contour feature points on the basis of the position information of the suspected pupil region, so as to obtain pupil contour feature points among the suspected pupil contour feature points (S140); on the basis of the pupil contour feature points and a pupil center point in an Nth frame of image in the eye video, determining a variation coefficient of the pupil contour feature points in the (N+1)th frame of image (S150); and on the basis of the variation coefficient, determining a pupil center point in the (N+1)th frame of image for pupil motion tracking (S160). By means of the present application, pupil center positioning accuracy is improved, thereby improving the stability and robustness of pupil motion tracking.

Description

Pupil movement tracking method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 31, 2023, with application number 202310638468.2, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of optical coherence tomography (OCT) in ophthalmology, for example, to pupil movement tracking methods, devices, equipment and storage media. Background Art

[0003] In OCT-based human eye blood flow imaging systems, eye movement can cause deviations in the OCT signal amplitude, phase, and spatial position and orientation, leading to artifacts in the resulting OCT images. Therefore, it is necessary to track eye movement in real time and compensate the imaging system accordingly to assist in acquiring high-quality OCT blood flow images.

[0004] With the rapid development of eye tracking technology, video-based eye tracking technology has become the mainstream of development due to its many advantages such as convenient equipment, low cost, and relatively simple engineering implementation.

[0005] Video-based eye tracking technology often relies on active infrared light sources and is easily interfered by eyelids, eyelashes, and Purkinje reflex spots, resulting in inaccurate pupil center positioning and reducing the stability and robustness of pupil movement tracking.

[0006] Summary of the Invention

[0007] The present application provides a pupil movement tracking method, apparatus, device and storage medium to improve the accuracy of pupil center positioning, thereby improving the stability and robustness of pupil movement tracking.

[0008] In a first aspect, an embodiment of the present application provides a pupil movement tracking method, comprising:

[0009] Preprocess the N+1th frame image in the eye video to obtain the corresponding binary image;

[0010] Performing quasi-circular area detection on the binary image to determine position information of a suspected pupil area in the binary image; performing masking on the binary image based on the position information of the suspected pupil area to obtain a masked binary image; performing pupil contour extraction on the masked binary image to obtain corresponding suspected pupil contour feature points, and screening the suspected pupil contour feature points based on the position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points; determining a coefficient of variation of the pupil contour feature points in the (N+1)th frame image based on the pupil contour feature points and the pupil center point in the (N+1)th frame image in the eye video; and determining the pupil center point in the (N+1)th frame image based on the coefficient of variation for pupil movement tracking.

[0011] In a second aspect, an embodiment of the present application further provides a pupil movement tracking device, comprising:

[0012] The image preprocessing module is configured to preprocess the N+1th frame image in the eye video to obtain a corresponding binary image; the quasi-circular area detection module is configured to perform quasi-circular area detection on the binary image to determine the position information of the suspected pupil area in the binary image; the image mask processing module is configured to perform mask processing on the binary image based on the position information of the suspected pupil area to obtain a masked binary image; the pupil contour feature point acquisition module is configured to perform pupil contour extraction on the masked binary image to obtain the corresponding Suspected pupil contour feature points, and screening the suspected pupil contour feature points based on the position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points; a variation coefficient determination module is configured to determine the variation coefficient of the pupil contour feature points in the N+1th frame image based on the pupil contour feature points and the pupil center point in the Nth frame image in the eye video; and a pupil center point determination module is configured to determine the pupil center point in the N+1th frame image based on the variation coefficient for pupil movement tracking.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising:

[0014] One or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the pupil movement tracking method described in any embodiment of the present application.

[0015] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the pupil movement tracking method described in any embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a flow chart of a pupil movement tracking method provided in an embodiment of the present application;

[0017] FIG2A is an N+1th frame of an original image provided by an embodiment of the present application;

[0018] FIG2B is an example diagram of a binary image provided by an embodiment of the present application;

[0019] FIG2C is a binary image after mask processing provided by an embodiment of the present application;

[0020] FIG3 is a rendering of pupil contour feature points provided by an embodiment of the present application;

[0021] FIG4 is a flow chart of another pupil movement tracking method provided in an embodiment of the present application;

[0022] FIG5A is a flow chart of another pupil movement tracking method provided by an embodiment of the present application;

[0023] FIG5B is an example diagram of an embodiment of the present application providing an example of using an angle screening method to screen pupil contour feature points;

[0024] FIG5C is another example diagram of screening pupil contour feature points using an angle screening method according to an embodiment of the present application;

[0025] FIG5D is another example diagram of screening pupil contour feature points using an angle screening method according to an embodiment of the present application;

[0026] FIG5E is an example diagram of a concave polygon in a convex hull detection according to an embodiment of the present application;

[0027] FIG5F is an example diagram of a polygon without a concave portion in a convex hull detection according to an embodiment of the present application;

[0028] FIG6 is a schematic diagram of a pupil movement tracking device provided in an embodiment of the present application;

[0029] FIG7 is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although some embodiments of the present application are shown in the accompanying drawings, the present application can be implemented in various forms. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. The accompanying drawings and embodiments of the present application are for illustrative purposes only.

[0031] The multiple steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional operations and / or omit the operations shown. The scope of the present application is not limited in this respect.

[0032] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0033] The concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] The modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0035] FIG1 is a flow chart of a pupil movement tracking method provided by an embodiment of the present application. The embodiment of the present application is suitable for implementing eye tracking through pupil movement tracking based on eye video, without the need for an additional infrared light source. The method can be performed by a pupil movement tracking device, which can be implemented in the form of software and / or hardware. The device can be implemented by an electronic device. The electronic device can include an ophthalmic OCT device.

[0036] As shown in FIG. 1 , the pupil movement tracking method may include S110 - S160 .

[0037] In S110 , the N+1th frame image in the eye video is preprocessed to obtain a corresponding binary image.

[0038] For example, in an OCT imaging scenario, the eye video mentioned in S110 may be an eye video collected for the patient's left eye or right eye. During the OCT imaging process, the patient's pupil movement can be located and tracked based on the video collected for the patient's eye, so as to compensate the imaging system in OCT or optical coherence tomography angiography (OCTA) imaging to assist in obtaining high-quality OCT imaging or OCTA. An eye video may contain multiple frames of original images, wherein the N+1th frame image may be the eye image currently to be processed, where N is an integer and N≥0.

[0039] In the embodiment of the present application, the eye video mentioned in S110 can be acquired in real time or in advance.

[0040] Exemplarily, the preprocessing of the N+1th frame image mentioned in S110 may include: first performing Gaussian filtering on the original image of the N+1th frame to remove image noise and preliminarily eliminate interference factors such as eyelashes; then using an image adaptive threshold method to binarize the image obtained after Gaussian filtering to obtain a preprocessed initial binary image; and then performing a small area denoising operation on the initial binary image to obtain a cleaner preprocessed binary image. The embodiment of the present application can remove noise from the original image and preliminarily filter out related interference through image preprocessing operations.

[0041] At S120 , circular-like region detection is performed on the binary image to determine position information of the suspected pupil region in the binary image.

[0042] In S120, illustratively, the Hough circle detection algorithm can be used to perform circular-like area detection on the binary image obtained by S110. Due to interference from the eyelids, Purkinje reflection spots, etc., when the circular-like area detection is performed on the binary image in S120, two or three or even more circular-like areas can usually be detected. For example, with reference to Figures 2A and 2B, Figure 2A is the original image, and Figure 2B is the binary image obtained by preprocessing based on Figure 2A. The circular-like area detection is performed on the binary image shown in Figure 2B, and the circular-like area composed of the upper eyelid and the lower eyelid, the circular-like area composed of the pupil, and the circular-like area composed of the Purkinje reflection spots can be detected. Therefore, after obtaining the circular-like area detection result, the detection result can also be rearranged to locate the area where the pupil is suspected (referred to as the suspected pupil area in this article). The suspected pupil area is a circular-like area that is most likely to be the pupil area in the detected circular-like area.

[0043] In one embodiment, for the first frame of an eye video (i.e., N=0), the image grayscale values ​​of multiple quasi-circular regions in the binary image or the original image corresponding to the binary image can be used to determine the suspected pupil area. This is because the first frame does not have a reference image, and in the imaging of the external eye structure, most of the pupil area appears black, that is, the image characteristic of the pupil is manifested by a small image grayscale value. Therefore, the suspected pupil area can be determined from the image grayscale values ​​of several quasi-circular regions.

[0044] In another embodiment, for non-first frame images (N>0) in the eye video, one method is to determine the suspected pupil area in several quasi-circular areas based on the image grayscale value, just like processing the first frame image, but this method has stricter requirements for setting the detection parameters. Therefore, for non-first frame images in the eye video, another method is to determine the suspected pupil area in several quasi-circular areas based on the Euclidean distance between the center position of each quasi-circular area in the current frame image and the center position of the pupil in the previous adjacent frame image. The suspected pupil area obtained by this method depends on the previous pupil motion tracking results, so the requirements for setting the detection parameters are not very strict.

[0045] In this embodiment of the present application, determining the location information of the suspected pupil region includes determining characteristic information such as the center point and radius of the region. The center point and radius of the suspected pupil region may only be preliminary center point and radius of the pupil region, and the center point and radius of the actual pupil region need to be further determined based on this characteristic information.

[0046] At S130 , masking is performed on the binary image based on the position information of the suspected pupil region to obtain a masked binary image.

[0047] After obtaining the position information of the suspected pupil area through S120, in S130, it is also necessary to mask the binary image obtained through S110 based on the position information to eliminate the influence of interference factors such as eyelids on pupil movement tracking, and only retain the pupil and the surrounding area of ​​the pupil that the user is interested in to continue to be used for pupil movement tracking.

[0048] In one embodiment, during masking in S130, a mask template of the same size as the binary image may be generated, and then the generated mask template may be multiplied by the binary image. This can filter out areas of the binary image that the user is not interested in, retaining only the areas of interest to the user. In this embodiment of the present application, the area of ​​interest to the user should be the pupil and its surrounding area, i.e., the suspected pupil area mentioned above.

[0049] Exemplarily, the process of generating the above-mentioned mask template may include: obtaining the position information of the suspected pupil area according to S120, first generating a rectangular mask template with the center point of the suspected pupil area as the center and twice the radius of the suspected pupil area (or (radius × (1+x%)), or (radius × (1-x%))) as the side length; then using the image dilation algorithm to expand the rectangular mask template to obtain a rectangular mask template with rounded corners, which is a mask template with the same size as the binary image.

[0050] For example, referring to Figures 2B and 2C, Figure 2B shows the binary image before masking, and Figure 2C shows the binary image after masking. As can be seen, the masked binary image eliminates interference from the eyebrows, lower eyelids, and part of the upper eyelid, retaining only the pupil and surrounding area of ​​interest to the user.

[0051] In S140 , pupil contour extraction is performed on the masked binary image to obtain corresponding suspected pupil contour feature points, and the suspected pupil contour feature points are screened based on position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points.

[0052] Continuing with Figure 2C, it can be seen that although interference such as eyebrows, lower eyelids, and parts of the upper eyelid can be eliminated after image preprocessing and masking, interference such as parts of the upper eyelid and upper eyelashes still exist in the remaining suspected pupil area and the area surrounding the pupil, which still affects pupil motion tracking. Therefore, pupil outline feature points can also be extracted and filtered in S140 to obtain cleaner and more realistic pupil outline feature points, thereby providing data foundation and data support for accurate pupil motion tracking.

[0053] For example, in S140, an edge feature point extraction algorithm can be used to extract pupil outline feature points from the masked binary image. The extracted pupil outline feature points are then marked to obtain outline feature points for the white area in the masked binary image, i.e., suspected pupil outline feature points. The extracted suspected pupil outline feature points can be further screened to identify the true pupil outline feature points.

[0054] For example, pupil contour feature point screening can be performed using a distance screening algorithm and an angle screening algorithm. In some embodiments, only distance screening can be performed; in some embodiments, only angle screening can be performed; in some embodiments, distance screening can be performed first, followed by angle screening; in some embodiments, angle screening can be performed first, followed by distance screening; this is not limited in the present application.

[0055] Referring to Figure 3, after distance screening and angle screening, the interference of the upper eyelid, upper eyelashes, and stray light spots can be eliminated, and only the true pupil contour feature points are retained.

[0056] At S150 , a coefficient of variation of pupil contour feature points in the (N+1)th frame image is determined based on the pupil contour feature points and the pupil center point in the (N)th frame image of the eye video.

[0057] The Nth frame image refers to the previous image frame adjacent to the N+1th frame image in the same eye video. The coefficient of variation, also known as the dispersion coefficient, is a normalized measure of the degree of dispersion of a probability distribution. In an embodiment of the present application, the coefficient of variation can represent the degree of change between the pupil center point in the N+1th frame image and the pupil center point in the Nth frame image. Based on the coefficient of variation, it can be determined whether the pupil center point in the N+1th frame image has actually changed compared to the pupil center point in the Nth frame image.

[0058] In an embodiment of the present application, the Euclidean distance between each pupil contour feature point in the N+1 frame image and the pupil center point in the N frame image can be calculated, and based on these Euclidean distances, the coefficient of variation of the pupil contour feature point in the N+1 frame image relative to the pupil center point corresponding to the N frame image can be determined.

[0059] Exemplarily, the variation coefficient determination process in S150 may include: determining the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature point and the position information of the pupil center point in the Nth frame image in the eye video; and determining the standard deviation and average value corresponding to multiple distances, and determining the variation coefficient of the pupil contour feature point in the N+1th frame image based on the determined standard deviation and average value.

[0060] The distance between each pupil contour feature point in the N+1 frame image and the pupil center point in the N frame image can refer to the Euclidean distance. Using the Euclidean distance calculation formula, based on the position coordinates of each pupil contour feature point in the N+1 frame image and the position coordinates of the pupil center point in the N frame image, the Euclidean distance between each pupil contour feature point in the N+1 frame image and the pupil center point in the N frame image can be calculated, and all calculated distances are averaged to obtain the average value of all distances, and the variances of all distances are squared to obtain the standard deviation of all distances. The standard deviations corresponding to all distances are divided by the average value to obtain the coefficient of variation of the pupil contour feature points in the N+1 frame image relative to the pupil center point corresponding to the N frame image.

[0061] Since only the non-first frame image in the eye video has an adjacent previous frame image, the first frame image in the eye video does not have an adjacent previous frame image, so there is no coefficient of variation for the first frame image. Only when the N+1th frame image is not the first frame image, the coefficient of variation corresponding to the N+1th frame image can be determined through S150.

[0062] At S160 , based on the coefficient of variation, the pupil center point in the (N+1)th frame image is determined for pupil movement tracking.

[0063] After obtaining the coefficient of variation of the pupil contour feature points in the N+1th frame image relative to the pupil center point in the Nth frame image through S150, the coefficient of variation can be tested in S160 to determine whether the degree of pupil change corresponding to the coefficient of variation has reached the minimum degree of pupil change. If the test result shows that the degree of pupil change indicated by the current coefficient of variation has reached the minimum degree of pupil change, it indicates that the pupil center point in the N+1th frame image has indeed moved relative to the pupil center point in the Nth frame image. At this time, all the pupil contour feature points identified in the N+1th frame image are accurate pupil contour feature points, and thus a new pupil center point can be re-determined based on all the identified pupil contour feature points as the pupil center point in the N+1th frame image. For example, an ellipse fitting can be performed based on all the pupil contour feature points identified for the N+1th frame image, and the center of the fitted ellipse can be used as the pupil center point in the N+1th frame image. If the degree of pupil change corresponding to the coefficient of variation does not reach the minimum degree of pupil change, it indicates that the pupil center in the N+1th frame image has not moved relative to the pupil center in the Nth frame image, or although it has moved, the movement is very small and can be ignored. In this case, the pupil center in the Nth frame image can be directly determined as the pupil center corresponding to the N+1th frame image. This embodiment uses the coefficient of variation to accurately locate the position information of the pupil center in the N+1th frame image, thereby improving the accuracy of pupil movement tracking.

[0064] Since the first frame image in the eye video does not have an adjacent previous frame image, when determining the pupil center point in the first frame image, after obtaining the pupil contour feature points in the first frame image through S140, ellipse fitting can be directly performed on all pupil contour feature points in the first frame image, and the ellipse center point obtained by fitting can be used as the pupil center point in the first frame image.

[0065] Exemplarily, for non-first frame images in the eye video, the process of determining the pupil center point in the N+1 frame image based on the coefficient of variation in S160 may include: in response to the coefficient of variation being greater than or equal to a preset coefficient, performing ellipse fitting based on pupil contour feature points to determine the pupil center point in the N+1 frame image; and / or in response to the coefficient of variation being less than a preset coefficient, determining that the pupil center point in the N+1 frame image coincides with the pupil center point in the N frame image.

[0066] The preset coefficient is a preset minimum coefficient of variation corresponding to pupil movement. By detecting whether the coefficient of variation of the pupil contour feature points in the N+1th frame image relative to the pupil center point in the Nth frame image is greater than or equal to the preset coefficient, it can be determined whether the pupil center point in the N+1th frame image has moved relative to the pupil center point in the Nth frame image. For example, when the coefficient of variation is greater than or equal to the preset coefficient, it indicates that the pupil center point in the N+1th frame image has moved relative to the pupil center point in the Nth frame image. At this time, an ellipse fitting can be performed on all pupil contour feature points in the N+1th frame image based on the least squares method, and the center point of the fitted ellipse is determined as the position of the pupil center point in the N+1th frame image. When the coefficient of variation is less than a preset coefficient, it indicates that the pupil center in the N+1th frame image has not moved relative to the pupil center in the Nth frame image, or that although it has moved, the movement is very small and can be ignored. In this case, the pupil center in the Nth frame image can be directly determined to be the pupil center in the N+1th frame image. In other words, in this case, the pupil center in the Nth frame image is determined to coincide with the pupil center in the N+1th frame image. The embodiment of the present application can accurately locate the position information of the pupil center in non-first frame images through the coefficient of variation, thereby improving the accuracy of pupil movement tracking.

[0067] In the embodiment of the present application, by pre-processing the N+1 frame image in the eye video to obtain the corresponding binary image, the noise in the original image can be removed and the interference factors such as eyelashes can be preliminarily eliminated; the circular area detection is performed on the binary image to determine the position information of the suspected pupil area in the binary image, and the binary image is masked based on the position information of the suspected pupil area to obtain the masked binary image, which can eliminate the influence of interference factors such as eyelids on pupil movement tracking; the pupil contour is extracted from the masked binary image to obtain the corresponding suspected pupil contour feature points, and the suspected pupil circle is extracted based on the position information of the suspected pupil area. The pupil contour feature points are screened to obtain the pupil contour feature points among the suspected pupil contour feature points, which can further eliminate the interference of the upper eyelid, upper eyelashes and stray light spots; thereby, based on the pupil contour feature points and the pupil center point in the N-th frame image in the eye video, the variation coefficient of the pupil contour feature points in the N+1-th frame image is determined, and based on the variation coefficient, the pupil center point in the N+1-th frame image is determined, which can avoid the positioning deviation caused by these interferences, and thus can accurately locate the position information of the pupil center point in the N+1-th frame image on the basis of lower detection parameter setting requirements, thereby improving the stability and robustness of pupil movement tracking.

[0068] As an embodiment, S110 may include: performing Gaussian filtering on the N+1th frame image in the eye video to obtain the filtered N+1th frame image; binarizing the filtered N+1th frame image based on an adaptive threshold processing method to obtain an initial binary image; determining the connected area corresponding to each connected region in the initial binary image, and filtering out the connected regions whose connected area is less than or equal to a preset area to obtain a filtered binary image.

[0069] The connected region may refer to an image region composed of adjacent foreground pixels having the same pixel value in the initial binary image, such as a connected region composed of white pixels. The preset area may refer to a preset maximum area of ​​the interference region.

[0070] Exemplarily, the Gaussian filtering process may include: for each pixel in the N+1th frame image, performing a weighted average of the original pixel value of the pixel and the original pixel values ​​of neighboring pixels in the pixel's neighborhood, and determining the result of the process as the pixel value of the pixel in the filtered N+1th frame image, thereby obtaining the filtered N+1th frame image. Gaussian filtering can remove image noise and preliminarily eliminate factors such as eyelash interference. The adaptive thresholding process may include: dividing the filtered N+1th frame image into multiple regions, performing a weighted average of the pixel values ​​of multiple pixels in each region to obtain a threshold for each region, comparing the pixel values ​​of the pixels in each region with the threshold for that region, setting the pixel values ​​of pixels above the threshold in the binary image to 1, thereby converting them to white pixels, and setting the pixel values ​​of pixels below the threshold to 0, thereby converting them to black pixels, thereby obtaining an initial binary image with a significant difference in brightness. After obtaining the initial binary image, multiple connected regions in the initial binary image can be identified, and the connected area corresponding to each connected region can be calculated. The connected area corresponding to each connected region is compared with a preset area. If the connected area corresponding to a connected region is less than or equal to the preset area, it indicates that the connected region is an interference region. In this case, the connected region can be removed, for example, by setting the pixel values ​​of multiple pixels in the connected region to 0. This can remove the small interference regions in the initial binary image, obtain a cleaner binary image, further improve the preprocessing effect, and thus improve the accuracy of pupil movement tracking.

[0071] FIG4 is a flowchart of another pupil movement tracking method provided in an embodiment of the present application.

[0072] As shown in FIG4 , the pupil movement tracking method may include the following operations:

[0073] In S110 , the N+1th frame image in the eye video is preprocessed to obtain a corresponding binary image.

[0074] This operation is the same as or similar to S110 shown in FIG1 , and will not be described in detail in this embodiment of the present application.

[0075] In S410 , circular-like regions are detected on the binary image to obtain position information of a plurality of circular-like regions in the binary image.

[0076] In S410, all circular-like regions in the binary image can be detected based on the Hough circle detection method, and the position information of each detected circular-like region can be obtained. For example, the position information of the detected circular-like region can include feature information such as the center point coordinates (a, b) and radius r of the circular-like region. The Hough circle detection process can include: based on the circle expression (xa)2 +(yb) 2 =r 2 , we can obtain the curve expressions x = a + rcos(θ) and y = b + rsin(θ), thereby performing intersection detection on the curves corresponding to the location information of each quasi-circular area. Solving the equations based on the curve intersections, a = x - rcos(θ) and b = y - rsin(θ), we can obtain the location information of each quasi-circular area in the binary image. Using the Hough circle detection method, we can more accurately and quickly detect all quasi-circular areas, further improving the accuracy and efficiency of pupil localization.

[0077] In S420 , based on the position information of the plurality of quasi-circular areas, a suspected pupil area is determined from the plurality of quasi-circular areas, and position information corresponding to the suspected pupil area is obtained.

[0078] In S420, since multiple circular-like areas can be detected in the binary image, it is necessary to rearrange all the detected circular-like areas, locate the circular-like area that is most likely to be the pupil, that is, the suspected pupil area, and determine the position information of the located circular-like area as the position information corresponding to the suspected pupil area.

[0079] In the case where the (N+1)th frame image is not the first frame image in the eye video, as an implementation method, the process of determining the suspected pupil area in S420 may be: based on the image grayscale value corresponding to each quasi-circular area, the suspected pupil area is determined from multiple quasi-circular areas. The image grayscale value may refer to the sum of the grayscale values ​​of the pixels of the quasi-circular area in the binary image, or may refer to the sum of the grayscale values ​​of the pixels of the quasi-circular area in the original image corresponding to the binary image. For example, the quasi-circular area with the smallest image grayscale value may be determined as the suspected pupil area.

[0080] Alternatively, in the case where the N+1th frame image is a non-first frame image in the eye video, as another implementation method, the process of determining the suspected pupil area in S420 can also be: based on the center point of each quasi-circular area in the multiple quasi-circular areas and the center point of the pupil area in the Nth frame image, determine the Euclidean distance between each quasi-circular area and the pupil area; and based on the Euclidean distance between each quasi-circular area and the pupil area, determine the suspected pupil area from the multiple quasi-circular areas.

[0081] In this embodiment, when the N+1 frame image is a non-first frame image in the eye video, the Euclidean distance between the center point of each quasi-circular area and the center point of the pupil area in the N-th frame image can be determined based on the center point position information of each quasi-circular area and the center point position information of the pupil area in the N-th frame image. The quasi-circular area with the smallest Euclidean distance can be determined as the suspected pupil area. This is because pupil movement is continuous. For adjacent frame images, after the pupil moves, the distance deviation between the two pupil center points in the two images will not be too large. Therefore, the quasi-circular area with the smallest Euclidean distance can be determined as the suspected pupil area. For non-first frame images, relying on the pupil area in the previous adjacent image frame, the suspected pupil area in the non-first frame image can be located more accurately and conveniently, thereby improving the efficiency and accuracy of pupil positioning.

[0082] When the N+1th frame image is the first frame image in the eye video, the process of determining the suspected pupil area in S420 can be: determining the image grayscale value of each quasi-circular area based on the position information of each quasi-circular area; and determining the suspected pupil area from multiple quasi-circular areas based on the image grayscale value of each quasi-circular area.

[0083] In this embodiment, determining the image grayscale value of each quasi-circular region may include determining the image grayscale value of each quasi-circular region in the binary image; or determining the image grayscale value of each quasi-circular region in the binary image in the corresponding (N+1)th frame of the original image. In the first frame, the quasi-circular region with the smallest image grayscale value may be determined as the suspected pupil region, thereby ensuring the overall effect of pupil movement tracking.

[0084] At S130 , masking is performed on the binary image based on the position information of the suspected pupil region to obtain a masked binary image.

[0085] In S140 , pupil contour extraction is performed on the masked binary image to obtain corresponding suspected pupil contour feature points, and the suspected pupil contour feature points are screened based on position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points.

[0086] At S150 , a coefficient of variation of pupil contour feature points in the (N+1)th frame image is determined based on the pupil contour feature points and the pupil center point in the (N)th frame image of the eye video.

[0087] At S160 , based on the coefficient of variation, the pupil center point in the (N+1)th frame image is determined for pupil movement tracking.

[0088] In this embodiment, S130 to S160 correspond to or are similar to S130 to S160 shown in FIG1 , and are not described in detail herein.

[0089] In an embodiment of the present application, circular-like area detection is performed on a binary image to obtain position information of multiple circular-like areas in the binary image. Based on the position information of each circular-like area, a suspected pupil area is determined from the multiple circular-like areas. This can eliminate the influence of the circular-like areas formed by interference such as eyelids and Purkinje reflex spots, and screen out the most likely suspected pupil area, thereby further improving the accuracy of pupil movement tracking.

[0090] FIG5A is a flowchart of another pupil movement tracking method provided in an embodiment of the present application.

[0091] As shown in FIG5A , the pupil movement tracking method includes the following operations:

[0092] In S110 , the N+1th frame image in the eye video is preprocessed to obtain a corresponding binary image.

[0093] At S120 , circular-like region detection is performed on the binary image to determine position information of the suspected pupil region in the binary image.

[0094] At S130 , masking is performed on the binary image based on the position information of the suspected pupil region to obtain a masked binary image.

[0095] In this embodiment, S110 to S130 correspond to or are similar to S110 to S130 shown in FIG1 , and are not described in detail herein.

[0096] At S510 , pupil contour extraction is performed on the masked binary image to obtain corresponding suspected pupil contour feature points.

[0097] At S520 , based on the position information of the suspected pupil area and the position information of each suspected pupil outline feature point, distance screening is performed on each suspected pupil outline feature point to obtain a first candidate pupil outline feature point.

[0098] When extracting pupil outline feature points from the masked binary image, there may be impurity feature points due to interference from parts of the upper eyelid, upper eyelashes, stray light spots, etc. Therefore, to obtain cleaner pupil outline feature points, these impurity feature points need to be screened out. The first candidate pupil outline feature point is a preliminarily screened feature point whose distance meets the pupil outline characteristics. During distance screening in S520, the Euclidean distance between each suspected pupil outline feature point and the center point of the suspected pupil region obtained in S120 can be determined. Then, a determination is made as to whether the difference between each distance value and the radius of the suspected pupil region obtained in S120 is within a preset error range. If the difference is within the preset error range, the corresponding suspected pupil outline feature point is likely to be a true pupil outline feature point. The suspected pupil outline feature point can be determined as the first candidate pupil outline feature point and retained. If the difference is outside the preset error range, the corresponding suspected pupil outline feature point is an interfering feature point from the upper eyelid, upper eyelashes, or stray light spots, and in this case, the suspected pupil outline feature point can be screened out. Through distance screening, the pupil contour feature points that may be real can be preliminarily screened out from the suspected pupil contour feature points, and the interference of related impurity feature points can be filtered out.

[0099] As an embodiment, S520 may include: determining the distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil area based on the pupil center point position information and the position information of each suspected pupil contour feature point; and determining the difference between each distance and the pupil radius, and determining the suspected pupil contour feature point whose absolute value of the difference is less than or equal to the preset difference as the first candidate pupil contour feature point.

[0100] The position information of the suspected pupil area includes: the center point of the suspected pupil area (referred to as the suspected pupil center point) and the suspected pupil radius r and other feature information. The preset difference can be (r×a%). The process of distance screening is as follows: based on the position information of the pupil center point of the suspected pupil area and the position information of each suspected pupil contour feature point, the Euclidean distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil area is determined; the Euclidean distance corresponding to each suspected pupil contour feature point is subtracted from the pupil radius of the suspected pupil area to obtain the corresponding difference, and each suspected pupil contour feature point whose absolute value of the difference is less than or equal to the preset difference is determined as the first candidate pupil contour feature point. Suspected pupil contour feature points whose absolute value of the difference is greater than the preset difference are not true pupil contour feature points, and may be interference feature points caused by eyelids, eyelashes, etc. Distance screening can preliminarily eliminate the influence of interference feature points such as eyelids and eyelashes, thereby improving the accuracy of pupil movement tracking.

[0101] At S530 , the first candidate pupil contour feature points are screened by angle to obtain second candidate pupil contour feature points.

[0102] Eyelashes vary in length, and for those with longer eyelashes, distance alone may not be enough to completely eliminate interference from eyelashes. Therefore, angle filtering can be used to further filter out impurity feature points caused by eyelashes. Second candidate pupil outline feature points are selected from the first candidate pupil outline feature points whose distances match the pupil outline feature, and whose angles also match the pupil outline feature.

[0103] After the initial screening of the suspected pupil contour feature points, since the interference caused by the lower eyelid and lower eyelashes is basically eliminated, only the impurity feature points caused by the upper eyelid and upper eyelashes can be eliminated during the secondary screening. Therefore, when performing angle screening in S530, the horizontal line passing through the center point of the suspected pupil area can be used as the horizontal dividing line, and only the first candidate pupil contour feature points (hereinafter referred to as target candidate pupil contour feature points) located above the horizontal dividing line can be secondary screened. The embodiment of the present application can achieve secondary screening through the following implementation method.

[0104] One implementation method may be to establish a rectangular coordinate system with the suspected pupil center point as the origin, connect each target candidate pupil contour feature point with the center point of the suspected pupil area, and use an acute angle ∠α (∠α is between 0° and 90°) formed by each connecting line and the horizontal dividing line (X coordinate axis) or use the angle ∠β (∠β is between 0° and 180°) between each connecting line and the positive X semi-axis (positive X axis) to perform angle screening.

[0105] For example, for the acute angle ∠α, it can be determined whether ∠α is less than or equal to a first preset angle. If ∠α is less than or equal to the first preset angle, the corresponding target candidate pupil contour feature point is retained; if ∠α is greater than the first preset angle, the corresponding target candidate pupil contour feature point is filtered out.

[0106] For example, for an acute angle ∠α, when the corresponding target candidate pupil contour feature point is a coordinate point in the first quadrant, it can be determined whether ∠α is less than or equal to a second preset angle. If ∠α is less than or equal to the second preset angle, the corresponding target candidate pupil contour feature point is retained; if ∠α is greater than the second preset angle, the corresponding target candidate pupil contour feature point is filtered out. When the corresponding target candidate pupil contour feature point is a coordinate point in the second quadrant, it can be determined whether ∠α is less than or equal to a third preset angle. If ∠α is less than or equal to the third preset angle, the corresponding target candidate pupil contour feature point is retained; if ∠α is greater than the third preset angle, the corresponding target candidate pupil contour feature point is filtered out. The second preset angle is different from the third preset angle.

[0107] For example, for the angle ∠β, it can be determined whether ∠β is less than or equal to the second preset angle, or whether ∠β is greater than or equal to the fourth preset angle but less than 180°. If ∠β is less than or equal to the second preset angle, the corresponding target candidate pupil contour feature point is retained; if ∠β is greater than or equal to the fourth preset angle and less than 180°, the corresponding target candidate pupil contour feature point is also retained; if ∠α is greater than the second preset angle and less than the fourth preset angle, the corresponding target candidate pupil contour feature point is filtered out. The fourth preset angle is complementary to the third preset angle.

[0108] Referring to Figure 5B , a rectangular coordinate system is established with the pupil center as the origin, the horizontal line through the pupil center as the x-axis, and the vertical line through the pupil center as the y-axis. If a first candidate pupil contour feature point is located above the x-axis and within the first quadrant, then the angle formed by the line between the first candidate pupil contour feature point and the pupil center as one side and the positive x-axis as the other side is ∠θ. If ∠θ satisfies: 0°<∠θ≤∠θ1, then the first candidate pupil contour feature point is retained. If ∠θ1<∠θ≤∠90°, then the first candidate pupil contour feature point is discarded.

[0109] Continuing with FIG5B , if a first candidate pupil outline feature point is located above the x-axis and within the second quadrant, the angle ∠θ formed by the line connecting the first candidate pupil outline feature point and the pupil center point is defined as one side and the negative x-axis is defined as the other side. If ∠θ satisfies: 0°<∠θ≤∠θ2, the first candidate pupil outline feature point is retained. If ∠θ2<∠θ≤∠90°, the first candidate pupil outline feature point is discarded.

[0110] Referring to Figure 5C, with the horizontal line passing through the pupil center as the x-axis, if a first candidate pupil contour feature point is located above the x-axis, and the angle ∠γ (∠γ is between 0° and 180°) formed by the line connecting it and the pupil center as one side and the positive direction of the x-axis as the other side satisfies: 0°<∠γ≤∠θ1 or 180°>∠γ≥∠θ3, and the angles corresponding to ∠θ1 and ∠θ3 are preset angles, then the first candidate pupil contour feature point is a true pupil contour feature point and is retained. If the angle ∠γ satisfies: ∠θ1<∠γ<∠θ3, then the first candidate pupil contour feature point may be an impurity feature point caused by interference from the upper eyelid, upper eyelashes, and stray light spots, and such feature point can be directly discarded.

[0111] Referring to Figure 5D , a rectangular coordinate system is established with the pupil center as the origin, a horizontal line passing through the pupil center as the x-axis, and a vertical line passing through the pupil center as the y-axis. Because the pupil is typically circular or elliptical, it is generally symmetrical about the y-axis. Some eyes may also be symmetrical about the y-axis. In this case, if the upper eyelid and upper eyelashes obstruct the pupil, this obstruction is also generally symmetrical about the y-axis. Therefore, for example, when performing angle screening, it is also possible to screen only the first candidate pupil outline feature points located in the first quadrant, and then use the symmetric relationship between the first candidate pupil outline feature points in the first quadrant and the first candidate pupil outline feature points in the second quadrant to screen the first candidate pupil outline feature points in the second quadrant. Alternatively, it is also possible to screen only the first candidate pupil outline feature points located in the second quadrant, and then use the symmetric relationship between the first candidate pupil outline feature points in the first quadrant and the first candidate pupil outline feature points in the second quadrant to screen the first candidate pupil outline feature points in the first quadrant. For example, first, the line between the first candidate pupil contour feature point located in the first quadrant and the pupil center point is taken as one side, and the angle formed by the positive x-axis is taken as the other side as ∠δ. If ∠δ satisfies: 0°<∠δ≤∠θ1, then the first candidate pupil contour feature point is retained and recorded as the second candidate pupil contour feature point. Otherwise, if ∠θ1<∠δ≤∠90°, then the first candidate pupil contour feature point is discarded. After the second candidate pupil contour feature points located in the first quadrant are screened out, the symmetric relationship between the first candidate pupil contour feature points in the first quadrant and the first candidate pupil contour feature points in the second quadrant is used to determine the first candidate pupil contour feature points in the second quadrant corresponding to each second candidate pupil contour feature point located in the first quadrant, and recorded as second candidate pupil contour feature points, thereby screening out the second candidate pupil contour feature points located in the second quadrant.

[0112] At S540 , pupil outline feature points are determined based on the second candidate pupil outline feature points.

[0113] As previously mentioned, the external structure of the eye indicates that the pupil region below the horizontal dividing line is not significantly affected by factors such as the lower eyelid, lower eyelashes, and stray light. Therefore, the first candidate pupil outline feature point below the horizontal dividing line can be directly determined as the true pupil outline feature point. Furthermore, in this embodiment, the second candidate pupil outline feature point can also be directly determined as the true pupil outline feature point.

[0114] In other embodiments of the present application, the second candidate pupil outline feature points and the first candidate pupil outline feature points located below the horizontal dividing line can be taken as a whole, and convex hull detection can be used to determine the candidate pupil outline feature points that form the convex hull of the second candidate pupil outline feature points and the first candidate pupil outline feature points located below the horizontal dividing line as the true pupil outline feature points. In the embodiments of the present application, the above-mentioned secondary screening can further eliminate the problem of positioning deviation in pupil movement tracking caused by partial occlusion of the pupil by the upper eyelid and upper eyelashes, thereby further improving the accuracy and stability of pupil movement tracking.

[0115] Referring to Figure 5E, in an embodiment of the present application, the process of convex hull detection may include: establishing a rectangular coordinate system with the center point of the suspected pupil area as the origin, the horizontal line passing through the center point as the x-axis, and the vertical line passing through the center point as the y-axis. A convex hull detection point M is set on the positive half axis of the y-axis. The convex hull detection point M can be determined based on the point in the real pupil area that is farthest from the center point of the real pupil area. For example, the distance L between the point in the real pupil area that is farthest from the center point of the real pupil area and the center point can be directly used as the distance between the convex hull detection point M and the above-mentioned coordinate origin, or L / 2 can be used as the distance between the convex hull detection point M and the above-mentioned coordinate origin. As long as the convex hull detection point M is located on the positive half axis of the y-axis, the embodiment of the present application does not limit its position.

[0116] Continuing with FIG5E , all second candidate pupil outline feature points screened by S530, all first candidate pupil outline feature points located below the horizontal dividing line, and the convex hull detection point M are connected to form a polygon. The constructed polygon is tested based on the convex hull detection method to determine whether there are any depressions in the polygon. As shown in FIG5E , the polygon constructed in the figure has four depressions (as indicated by dotted circles). The feature points at each depression are deleted, and then, based on all the remaining points, the adjacent points are connected to each other again to construct a new polygon. The convex hull detection and the feature points at the depressions are deleted again. The above operations are repeated until the constructed polygon no longer has depressions. After deleting the feature points at each depression, the polygon constructed in FIG5E is reconstructed as shown in FIG5F . The polygon in FIG5F no longer has depressions, and all feature points in FIG5F except the convex hull detection point M can be regarded as the true pupil outline feature points.

[0117] In the embodiment of the present application, convex hull detection can be used to eliminate impurity feature points caused by interference from bright spots such as stray light spots, thereby further improving the accuracy of pupil center positioning.

[0118] At S150 , a coefficient of variation of pupil contour feature points in the (N+1)th frame image is determined based on the pupil contour feature points and the pupil center point in the (N)th frame image of the eye video.

[0119] At S160 , based on the coefficient of variation, the pupil center point in the (N+1)th frame image is determined for pupil movement tracking.

[0120] In this embodiment, S150 to S160 correspond to or are similar to S150 to S160 shown in FIG1 , and are not described in detail herein.

[0121] The technical solution of the embodiment of the present application performs distance screening on each suspected pupil contour feature point based on the position information of the suspected pupil area and the position information of each suspected pupil contour feature point to obtain a first candidate pupil contour feature point; and performs angle screening on the first candidate pupil contour feature point to obtain a second candidate pupil contour feature point, thereby removing impurity feature points caused by interference such as eyelids and eyelashes through distance screening and angle screening, further improving the accuracy of pupil center positioning.

[0122] Figure 6 is a schematic diagram of a pupil movement tracking device provided in an embodiment of the present application. As shown in Figure 6, the pupil movement tracking device 600 includes: an image preprocessing module 610, a quasi-circular area detection module 620, an image mask processing module 630, a pupil contour feature point acquisition module 640, a variation coefficient determination module 650 and a pupil center point determination module 660.

[0123] The image preprocessing module 610 is configured to preprocess the N+1th frame image in the eye video to obtain a corresponding binary image; the quasi-circular region detection module 620 is configured to perform quasi-circular region detection on the binary image to determine the position information of the suspected pupil region in the binary image; the image mask processing module 630 is configured to perform mask processing on the binary image based on the position information of the suspected pupil region to obtain a masked binary image; the pupil contour feature point acquisition module 640 is configured to extract the pupil contour from the masked binary image. To obtain the corresponding suspected pupil contour feature points, and screen the suspected pupil contour feature points based on the position information of the suspected pupil area to obtain the pupil contour feature points among the suspected pupil contour feature points; the variation coefficient determination module 650 is configured to determine the variation coefficient of the pupil contour feature points in the N+1 frame image based on the pupil contour feature points and the pupil center point in the N-th frame image in the eye video; and the pupil center point determination module 660 is configured to determine the pupil center point in the N+1 frame image based on the variation coefficient for pupil movement tracking.

[0124] As an embodiment, the quasi-circular area detection module 620 may include:

[0125] The first position information obtaining submodule is configured to perform circular-like area detection on the binary image and obtain the position information of multiple circular-like areas in the binary image; and the second position information obtaining submodule is configured to determine the suspected pupil area from the multiple circular-like areas based on the position information of the multiple circular-like areas, and obtain the position information corresponding to the suspected pupil area.

[0126] As an embodiment, the second position information obtaining submodule is configured to: determine the Euclidean distance between each quasi-circular area and the pupil area based on the position information of each quasi-circular area in the multiple quasi-circular areas and the position information of the pupil area in the Nth frame image; and determine the suspected pupil area from the multiple quasi-circular areas based on the Euclidean distance between each quasi-circular area and the pupil area.

[0127] As an embodiment, the pupil contour feature point acquisition module 640 may include:

[0128] The first candidate pupil contour feature point acquisition submodule is configured to perform distance screening on each suspected pupil contour feature point based on the position information of the suspected pupil area and the position information of each suspected pupil contour feature point to obtain the first candidate pupil contour feature point; the second candidate pupil contour feature point acquisition submodule is configured to perform angle screening on the first candidate pupil contour feature point to obtain the second candidate pupil contour feature point; and the pupil contour feature point determination submodule is configured to determine the pupil contour feature point based on the second candidate pupil contour feature point.

[0129] As an embodiment, the pupil contour feature point determination submodule is configured to: perform convex hull detection based on the second candidate pupil contour feature points, and determine the candidate pupil contour feature points that constitute the convex hull in the second candidate pupil contour feature points as pupil contour feature points.

[0130] As an embodiment, the position information of the suspected pupil area includes: pupil center position information and pupil radius;

[0131] The first candidate pupil contour feature point acquisition submodule is configured to: determine the distance between each suspected pupil contour feature point and the pupil center point of the suspected pupil area based on the pupil center point position information and the position information of each suspected pupil contour feature point; and determine the difference between each distance and the pupil radius, and determine the suspected pupil contour feature point whose absolute value of the difference is less than or equal to the preset difference as the first candidate pupil contour feature point.

[0132] As an embodiment, the second candidate pupil contour feature point acquisition submodule is configured to: determine the horizontal angle with the line between each first candidate pupil contour feature point and the pupil center point of the suspected pupil area as the edge based on the pupil center point position information and the position information of each first candidate pupil contour feature point; and determine the first candidate pupil contour feature point whose horizontal angle is less than or equal to the preset angle as the second candidate pupil contour feature point.

[0133] As an embodiment, the coefficient of variation determination module 650 is configured to: determine the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature point and the position information of the pupil center point in the Nth frame image in the eye video; and determine the standard deviation and average value corresponding to multiple distances, and determine the coefficient of variation of the pupil contour feature point in the N+1th frame image based on the determined standard deviation and average value.

[0134] As an embodiment, the pupil center point determination module 660 is configured to: in response to the coefficient of variation being greater than or equal to a preset coefficient, perform ellipse fitting based on the pupil contour feature points to determine the pupil center point in the N+1th frame image; and / or in response to the coefficient of variation being less than a preset coefficient, determine that the pupil center point in the N+1th frame image coincides with the pupil center point in the Nth frame image.

[0135] As an embodiment, when the N+1th frame image is the first frame image in the eye video, the second position information obtaining submodule is further configured to: determine the image grayscale value of each quasi-circular area based on the position information of each quasi-circular area; and determine the suspected pupil area from multiple quasi-circular areas based on the image grayscale value of each quasi-circular area.

[0136] The pupil movement tracking device provided in the embodiments of the present application can execute the pupil movement tracking method provided in any embodiment of the present application, and has the corresponding functional modules and effects of the execution method.

[0137] The pupil movement tracking device provided in the embodiment of the present application solves the same or similar technical problems and achieves the same or similar technical effects as the pupil movement tracking method provided in the aforementioned embodiment, and will not be described in detail in this embodiment.

[0138] The multiple units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the names of the multiple functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of this application.

[0139] FIG7 is a schematic diagram of an electronic device provided in an embodiment of the present application. Referring now to FIG7 , a schematic diagram of the structure of an electronic device (e.g., the ophthalmic OCT device in FIG7 ) 500 suitable for implementing an embodiment of the present application is shown. The electronic device in the embodiment of the present application may be, for example, an ophthalmic OCT device. The electronic device in this embodiment is merely an example and should not limit the functionality or scope of use of the embodiment of the present application.

[0140] As shown in Figure 7, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0141] The following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. However, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may be implemented or present instead.

[0142] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0143] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0144] The electronic device provided in the embodiment of the present application and the pupil movement tracking method provided in the above embodiment belong to the same concept. For technical details not described in detail in this embodiment, please refer to the above embodiment, and this embodiment has the same effect as the above embodiment.

[0145] An embodiment of the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the pupil movement tracking method provided in the above embodiment.

[0146] The computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. Examples of computer-readable storage media can include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0147] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as the Hyper Text Transfer Protocol (HTTP), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0148] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0149] The computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer over any type of network, including a LAN or WAN, or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to the various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The units involved in the embodiments described in this application may be implemented in software or hardware. The name of a unit does not necessarily limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0152] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on a Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0153] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. Examples of machine-readable storage media can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] The various forms of processes shown above can be used to reorder, add, or delete operations. For example, the multiple operations described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This document is not limited here.

Claims

1. A pupil movement tracking method, include: Preprocess the N+1th frame image in the eye video to obtain the corresponding binary image; Performing circular-like region detection on the binary image to determine position information of a suspected pupil region in the binary image; Performing mask processing on the binary image based on the position information of the suspected pupil area to obtain a masked binary image; Extracting pupil contours from the masked binary image to obtain corresponding suspected pupil contour feature points, and screening the suspected pupil contour feature points based on position information of the suspected pupil area to obtain pupil contour feature points from the suspected pupil contour feature points; Determine the coefficient of variation of the pupil contour feature points in the (N+1)th frame image based on the pupil contour feature points and the pupil center point in the (N)th frame image in the eye video; as well as Based on the coefficient of variation, the pupil center point in the N+1th frame image is determined for pupil movement tracking.

2. The method according to claim 1, in, The performing circular region detection on the binary image to determine the position information of the suspected pupil region in the binary image includes: Performing circular-like region detection on the binary image to obtain position information of a plurality of circular-like regions in the binary image; and Based on the position information of the plurality of quasi-circular areas, a suspected pupil area is determined from the plurality of quasi-circular areas, and position information corresponding to the suspected pupil area is obtained.

3. The method according to claim 2, in, The determining of the suspected pupil area from the plurality of quasi-circular areas based on the position information of the plurality of quasi-circular areas comprises: Determine the Euclidean distance between each quasi-circular area and the pupil area based on the position information of each quasi-circular area among the plurality of quasi-circular areas and the position information of the pupil area in the Nth frame image; and The suspected pupil area is determined from the plurality of circular-like areas based on the Euclidean distance between each circular-like area and the pupil area.

4. The method according to claim 1, in, The screening of the suspected pupil contour feature points based on the position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points includes: Based on the position information of the suspected pupil area and the position information of each suspected pupil contour feature point, performing distance screening on each suspected pupil contour feature point to obtain a first candidate pupil contour feature point; The first candidate pupil contour feature points are screened by angle to obtain the second candidate pupil contour feature points. collection points; and The pupil outline feature point is determined based on the second candidate pupil outline feature point.

5. The method according to claim 4, in, The step of determining the pupil contour feature point based on the second candidate pupil contour feature point comprises: Convex hull detection is performed based on the second candidate pupil contour feature points, and candidate pupil contour feature points constituting the convex hull among the second candidate pupil contour feature points are determined as the pupil contour feature points.

6. The method according to claim 4, in, The position information of the suspected pupil area includes: pupil center position information and pupil radius; Based on the position information of the suspected pupil area and the position information of each suspected pupil contour feature point, distance screening is performed on each suspected pupil contour feature point to obtain a first candidate pupil contour feature point, including: Based on the pupil center position information and the position information of each suspected pupil outline feature point, determining the distance between each suspected pupil outline feature point and the pupil center point of the suspected pupil area; and The difference between each distance and the pupil radius is determined, and the suspected pupil contour feature point whose absolute value of the difference is less than or equal to the preset difference is determined as the first candidate pupil contour feature point.

7. The method according to claim 6, in, The step of performing angle screening on the first candidate pupil contour feature points to obtain second candidate pupil contour feature points includes: Based on the pupil center position information and the position information of each first candidate pupil outline feature point, determining a horizontal angle with a line between each first candidate pupil outline feature point and the pupil center point of the suspected pupil area as an edge; and A first candidate pupil outline feature point whose horizontal angle is less than or equal to a preset angle is determined as the second candidate pupil outline feature point.

8. The method according to claim 1, in, The determining the coefficient of variation of the pupil contour feature points in the N+1th frame image based on the pupil contour feature points and the pupil center point in the Nth frame image in the eye video comprises: Determining the distance between each pupil contour feature point and the pupil center point based on the position information of the pupil contour feature point and the position information of the pupil center point in the Nth frame image in the eye video; and Determine the standard deviation and the average value corresponding to the multiple distances, and determine the coefficient of variation of the pupil contour feature points in the N+1th frame image based on the determined standard deviation and the average value.

9. The method according to any one of claims 1 to 8, in, Determining the pupil center point in the N+1th frame image based on the coefficient of variation includes at least one of the following: In response to the coefficient of variation being greater than or equal to a preset coefficient, performing ellipse fitting based on the pupil contour feature points to determine the pupil center point in the N+1th frame image; In response to the coefficient of variation being less than a preset coefficient, it is determined that the pupil center point in the (N+1)th frame image coincides with the pupil center point in the (N)th frame image.

10. The method according to claim 2, in, In the case where the N+1th frame image is the first frame image in the eye video, Based on the position information of the plurality of quasi-circular areas, determining a suspected pupil area from the plurality of quasi-circular areas comprises: Determining the image grayscale value of each quasi-circular area according to the position information of each quasi-circular area; and A suspected pupil area is determined from the plurality of circular-like areas according to the image grayscale value of each circular-like area.

11. A pupil movement tracking device, include: An image preprocessing module is configured to preprocess the N+1th frame image in the eye video to obtain a corresponding binary image; a quasi-circular region detection module, configured to perform quasi-circular region detection on the binary image to determine position information of a suspected pupil region in the binary image; An image mask processing module is configured to perform mask processing on the binary image based on the position information of the suspected pupil area to obtain a masked binary image; a pupil contour feature point acquisition module, configured to perform pupil contour extraction on the masked binary image to obtain corresponding suspected pupil contour feature points, and screen the suspected pupil contour feature points based on position information of the suspected pupil area to obtain pupil contour feature points among the suspected pupil contour feature points; A variation coefficient determination module is configured to determine the variation coefficient of the pupil contour feature points in the N+1th frame image based on the pupil contour feature points and the pupil center point in the Nth frame image in the eye video; as well as The pupil center point determination module is configured to determine the pupil center point in the N+1th frame image based on the coefficient of variation for pupil movement tracking.

12. An electronic device, include: at least one processor; a storage device configured to store at least one program, When the at least one program is executed by the at least one processor, the at least one processor implements the pupil movement tracking method as described in any one of claims 1-10.

13. A storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform the pupil movement tracking method according to any one of claims 1 to 10.

Images