System and method for eye tracking during eye treatment

Abstract

An example system for tracking eye movement during eye treatment includes an image capture device configured to capture a plurality of images of an eye. In one embodiment, a system includes a controller including a processor that receives a plurality of images from an image capture device. The processor implements a plurality of trackers. Each tracker is configured to detect a respective feature in the plurality of images and to provide a respective data set related to eye motion based on the respective feature. The respective features detected by the plurality of trackers are orthogonal to each other, and the respective data sets provided by the plurality of trackers are independent of each other. The processor merges the data sets from the plurality of trackers and determines an indicator of eye movement based on the merged data sets.

Description

[0001] This application is a divisional application of Chinese invention patent application No. 201980071362.0, whose application date is September 19, 2019 and whose invention name is “System and method for eye tracking during eye treatment”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 733,620, filed on September 19, 2018, the entire contents of which are incorporated herein by reference. background

[0005] field

[0006] The present disclosure relates to systems and methods for ocular treatment, and more particularly to systems and methods for tracking movement of the eye to deliver treatment to a desired ocular region.

[0007] Description of Related Technology

[0008] Cross-linking therapy can be used to treat eyes with conditions such as keratoconus. In particular, keratoconus is a degenerative eye disease in which structural changes within the cornea cause the cornea to weaken and become abnormally cone-shaped. Cross-linking therapy can strengthen and stabilize areas weakened by keratoconus and prevent undesired shape changes.

[0009] Cross-linking therapy can also be used after surgical procedures such as laser-assisted in situ keratomileusis (LASIK) surgery. For example, due to the thinning and weakening of the cornea caused by LASIK surgery, a complication known as post-LASIK ectasia can occur. In post-LASIK ectasia, the cornea undergoes progressive steepening (bulging). Accordingly, cross-linking therapy can strengthen and stabilize the corneal structure after LASIK surgery and prevent post-LASIK ectasia.

[0010] Cross-linking therapy can also be used to induce corneal refractive changes to correct conditions such as myopia, hyperopia, astigmatism, irregular astigmatism, presbyopia, and the like. Summary of the invention

[0011] Because the cross-linking process may require exposing the cornea to the photoactivating light for at least several minutes (e.g., one to thirty minutes), some eye movement is likely to occur during the process. To account for the occurrence of eye movement, the systems and methods may employ an eye tracking system to determine any changes in corneal position and in response adjust the illumination system so that the photoactivating light is precisely applied to the designated corneal area.

[0012] An example system for tracking movement of an eye during eye treatment includes an image capture device configured to capture multiple images of an eye. The system includes one or more controllers, the one or more controllers including one or more processors configured to execute program instructions stored on one or more computer-readable media. The one or more processors receive multiple images from the image capture device. The one or more processors implement multiple trackers. Each tracker is configured to detect corresponding features in the multiple images and provide corresponding data sets related to movement of the eye based on the corresponding features. The corresponding features detected by the multiple trackers are orthogonal to each other, and the corresponding data sets provided by the multiple trackers are independent of each other. The one or more processors merge the data sets from the multiple trackers and determine an indicator of movement of the eye based on the merged data set.

[0013] Because errors in eye tracking can result in ineffective treatment and / or injury / harm to the patient, eye tracking in medical treatments such as cross-linking therapy should be robust and accurate. Because each tracker acquires information on orthogonal (non-overlapping) image features in multiple images and provides independent estimates of eye movement, the example systems and methods are highly robust. These independent estimates are analyzed against each other, which serves to reduce error variance and allow smooth estimates of eye movement to be obtained with greater accuracy. While trackers sometimes fail due to reflections and other obstructions in the image, the systems and methods account for errors and missing information caused by such failures, thereby allowing tracking to continue. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example system is illustrated that delivers a crosslinking agent and photoactivating light to the cornea of ​​an eye to thereby generate crosslinks of corneal collagen, in accordance with aspects of the present disclosure.

[0015] Figure 2 An example therapeutic system with an eye tracking system according to aspects of the present disclosure is illustrated.

[0016] Figure 3 Illustrate features that may be detected in a pixelated image of an eye captured by an example eye tracking system, according to aspects of the present disclosure.

[0017] Figure 4 An example method employing an ensemble tracker to track and process multiple features in a captured eye image for an eye tracking system is illustrated in accordance with aspects of the present disclosure.

[0018] Figure 5A-B illustrates an example method according to aspects of the present disclosure that employs a higher-level meta-heuristic to combine data from three trackers to produce a final estimate related to eye movements.

[0019] Figure 6 FIG. 1 illustrates an example frame with image features, each image feature being employed by a corresponding tracker to track the image based on the image features according to aspects of the present disclosure. Figure 5A The example method illustrated in -B determines the consistency / consensus for the pupil center location.

[0020] Figure 7 FIG. 1 illustrates an example eye tracking process according to aspects of the present disclosure that produces a track that follows a time series of frames F i A trace that is processed and progresses through different states.

[0021] Figure 8 FIG. 1 is a diagram illustrating a method for performing a Figure 7 The mature track shown in Figure 2 processes a series of frames F i Example method of .

[0022] Fig.9A An example image is illustrated that captures an irregularly shaped pupil or changes in pupil shape as a phenomenon that may cause errors.

[0023] Fig. 9B An example image is illustrated that captures shadows and reflections caused by a mask worn by a patient for treatment or by an eye being partially blocked by an eyelid as phenomena that may cause errors.

[0024] Fig. 9C An example image is illustrated that captures a dropper and a speculum used for treatment as a phenomenon that may cause errors.

[0025] Fig.9D An example image is illustrated that captures the graying of the pupil caused by an intraocular implant as a possible error-inducing phenomenon.

[0026] Fig.9E An example image is illustrated that captures an irregular reflection pattern due to an implanted intraocular lens as a possible error-inducing phenomenon.

[0027] Fig.9F An example image is illustrated that captures an occlusion such as a finger or eye dropper as a phenomenon that may cause errors.

[0028] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments of the present disclosure have been shown in the drawings by way of example and will be described in detail herein. However, it should be understood that it is not intended to limit the present disclosure to the specific forms disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternatives that fall within the spirit of the present disclosure. DETAILED DESCRIPTION

[0029] Figure 1 An example treatment system 100 for generating collagen crosslinks in a cornea 2 of an eye 1 is shown. The treatment system 100 includes an applicator 132 for applying a crosslinking agent 130 to the cornea 2. In an example embodiment, the applicator 132 may be an eye dropper, a syringe, or the like that applies the photosensitizer 130 as drops to the cornea 2. Example systems and methods for applying crosslinking agents are described in U.S. Pat. No. 10,342,697, filed on April 13, 2017, and entitled “Systems and Methods for Delivering Drugs to an Eye,” the entire contents of which are incorporated herein by reference.

[0030] The crosslinking agent 130 can be provided in a formulation that allows the crosslinking agent 130 to pass through the corneal epithelium 2a and reach the underlying areas in the corneal stroma 2b. Alternatively, the corneal epithelium 2a can be removed or otherwise incised to allow the crosslinking agent 130 to be applied more directly to the underlying tissue.

[0031] The treatment system 100 includes an irradiation system having a light source 110 and an optical element 112 for directing light to the cornea 2. The light causes photoactivation of the crosslinking agent 130 to generate crosslinking activity in the cornea 2. For example, the crosslinking agent may include riboflavin, and the photoactivating light may include ultraviolet A (UVA) (e.g., approximately 365 nm) light. Alternatively, the photoactivating light may also include another wavelength, such as a visible light wavelength (e.g., approximately 452 nm). As further described below, corneal crosslinking improves the strength of the cornea by creating chemical bonds within the corneal tissue according to a photochemically powered reaction system. For example, riboflavin and photoactivating light may be applied to stabilize and / or strengthen corneal tissue to address diseases such as keratoconus or post-LASIK ectasia.

[0032] The treatment system 100 includes one or more controllers 120 that control aspects of the system 100 including the light source 110 and / or the optical element 112. In embodiments, the cornea 2 may be more extensively treated with the crosslinking agent 130 (e.g., using eye drops, a syringe, etc.), and the photoactivating light from the light source 110 may be selectively directed to areas of the treated cornea 2 following a particular pattern.

[0033] The optical element 112 may include one or more reflectors or lenses for directing and focusing the photoactivation light emitted by the light source 110 to a specific pattern on the cornea 2. The optical element 112 may also include a filter for partially blocking the wavelength of light emitted by the light source 110 and for selecting a specific wavelength of light to be directed to the cornea 2 to photoactivate the crosslinking agent 130. In addition, the optical element 112 may include one or more beam splitters for splitting the light beam emitted by the light source 110, and the optical element may include one or more heat sinks for absorbing the light emitted by the light source 110. The optical element 112 may also accurately and precisely focus the photoactivation light to a specific focal plane within the cornea 2, such as a specific depth in the underlying region 2b where crosslinking activity is desired.

[0034] Furthermore, the specific regime of the photoactivating light can be modulated to achieve a desired degree of crosslinking in selected areas of the cornea 2. One or more controllers 120 can be used to control the operation of the light source 110 and / or the optical element 112 to precisely deliver the photoactivating light according to any combination of wavelength, bandwidth, intensity, power, location, penetration depth, and / or treatment duration (duration of light periods, dark periods, and ratio of light periods to dark period duration).

[0035] The parameters for light activation of the crosslinker 130 can be adjusted, for example, to reduce the amount of time required to achieve the desired crosslinking. In an example embodiment, the time can be reduced from minutes to seconds. Although some configurations may be 5 mW / cm 2 The photoactivation light is applied at an irradiance of 5 mW / cm , however, a higher irradiance of photoactivation light (e.g., 5 mW / cm 2 The total dose of energy absorbed in the cornea 2 can be described as the effective dose, which is the amount of energy absorbed by the area of ​​the corneal epithelium 2a. For example, the effective dose for the corneal surface 2A area can be, for example, 5 J / cm 2 or up to 20J / cm 2 or 30J / cm 2 The described effective doses may be delivered from a single application of energy or from repeated applications of energy.

[0036] The optical elements 112 of the treatment system 100 may include microelectromechanical systems (MEMS) devices, such as digital micromirror devices (DMDs), for modulating the application of photoactivating light in space and time. Using DMD technology, photoactivating light from the light source 110 is projected in a precise spatial pattern, which is created by microscopic mirrors arranged in an array on a semiconductor chip. Each mirror represents one or more pixels in the projected light pattern. Using the DMD, topography-guided crosslinking can be implemented. The topography-based control of the DMD can use several different spatial and temporal irradiance and dose distributions. These spatial and temporal dose distributions can be created using continuous wave irradiation, but can also be modulated by pulsed irradiation, which is achieved by pulsing the irradiation source at varying frequencies and duty cycles. Alternatively, the DMD can modulate different frequencies and duty cycles on a pixel-by-pixel basis, providing the ultimate flexibility of using continuous wave irradiation. Or alternatively, both pulsed irradiation and modulated DMD frequency and duty cycle combinations can also be combined. This allows for spatially determined corneal cross-linking for targeted amounts. The spatially determined cross-linking can be combined with dosimetry, interferometry, optical coherence tomography (OCT), corneal topography, etc. for pre-treatment planning and / or for real-time monitoring and modulation of corneal cross-linking during treatment. Aspects of the dosimetry system are described in further detail below. In addition, preclinical patient information can be combined with finite element biomechanical computer modeling to create a patient-specific pre-treatment plan.

[0037] To control aspects of the delivery of photoactivated light, embodiments may also employ aspects of multiphoton excitation microscopy. In particular, rather than delivering a single photon of a specific wavelength to the cornea 2, the therapeutic system 100 may deliver multiple photons of longer wavelengths (i.e., lower energy) that combine to initiate crosslinking. Advantageously, longer wavelengths are less scattered within the cornea 2 than shorter wavelengths, which allows longer wavelength light to penetrate the cornea 2 more efficiently than shorter wavelength light. Since the photosensitizer absorbs much less light at longer wavelengths, shielding of incident radiation at deeper depths within the cornea is also reduced compared to conventional short wavelength irradiation. This allows for enhanced control of deep targeted crosslinking. For example, in some embodiments, two photons may be employed, each of which carries approximately half of the energy required to excite a molecule in the crosslinker 130 to generate the photochemical kinetic reaction described further below. When a crosslinker molecule absorbs two photons simultaneously, it absorbs enough energy to release active free radicals in the corneal tissue. Embodiments may also use lower energy photons, so that the crosslinker molecule must absorb, for example, three, four, or five photons simultaneously to release the active free radicals. Nearly simultaneous absorption of multiple photons is unlikely, so a high flux of excitation photons would be required, and the high flux can be delivered by a femtosecond laser.

[0038] A number of conditions and parameters affect the cross-linking of corneal collagen with the cross-linking agent 130. For example, the irradiance and dosage of the photoactivating light affects the amount and rate of cross-linking.

[0039] When the cross-linking agent 130 is, in particular, riboflavin, the UVA light may be applied continuously (continuous wave (CW)) or as pulsed light, and this choice has an impact on the amount, rate, and extent of cross-linking. If the UVA light is applied as pulsed light, the duration of the exposure period, the dark period, and the ratio of the exposure period to the dark period duration have an impact on the resulting corneal stiffening. Pulsed light irradiation can be used to produce greater or less corneal tissue stiffening than can be achieved with continuous wave irradiation, with the same amount or dose of energy delivered. More optimized chemical amplification can be achieved using light pulses of appropriate length and frequency. For pulsed light therapy, the on / off duty cycle can be between about 1000 / 1 and about 1 / 1000; the irradiance can be at about 1 mW / cm 2 About 1000mW / cm 2 The average irradiance is between about 0.01 Hz and about 1000 Hz, and the pulse rate can be between about 1000 Hz and about 100,000 Hz.

[0040] The treatment system 100 can generate pulsed light by using a DMD, electronically switching the light source 110 on and off, and / or using a mechanical shutter or an optoelectronic (e.g., Pockels cell) shutter or a mechanical chopper or a rotating aperture. Due to the pixel-specific modulation capability of the DMD and the subsequent stiffening based on the modulation frequency, duty cycle, irradiance, and dose delivered to the cornea, complex biomechanical stiffness patterns can be imparted to the cornea to allow for various amounts of refractive correction. These refractive corrections, for example, can involve correction of myopia, hyperopia, astigmatism, irregular astigmatism, hyperopia, and a combination of corrections of complex corneal refractive surfaces caused by ophthalmic symptoms such as keratoconus, clear edge disease, ectasia after LASIK, and other symptoms of corneal biomechanical changes / degeneration. A particular advantage of the DMD system and method is that it allows for random asynchronous pulse profile patterning, creating non-periodic and uniformly appearing illumination that eliminates the possibility of triggering photosensitive epileptic seizures or flicker sickness at pulse frequencies between 2 Hz and 84 Hz.

[0041] Although example embodiments may employ a stepped on / off pulsed light function, it is understood that other functions for applying light to the cornea may be employed to achieve similar effects. For example, light may be applied to the cornea according to a sinusoidal function, a sawtooth function, or other complex function or curve, or any combination of functions or curves. In fact, it is understood that the function may be generally stepped, with a smoother transition between on / off values. Further, it is understood that the irradiance does not necessarily have to be reduced to a zero value during the off period, and may be above zero during the off period. The desired effect may be achieved by applying light to the cornea according to a curve that varies the irradiance between two or more values.

[0042] For example, examples of systems and methods for delivering photoactivated light are described in U.S. patent application publication No. 2011 / 0237999, filed on March 18, 2011 and entitled “Systems and Methods for Applying and Monitoring Eye Therapy,” U.S. patent application publication No. 2012 / 0215155, filed on April 3, 2012 and entitled “Systems and Methods for Applying and Monitoring Eye Therapy,” and U.S. patent application publication No. 2013 / 0245536, filed on March 15, 2013 and entitled “Systems and Methods for Corneal Cross-Linking with Pulsed Light,” the entire contents of which are incorporated herein by reference.

[0043] The addition of oxygen also affects the amount of corneal stiffening. In human tissue, the O2 content is very low compared to the atmosphere. However, the rate of cross-linking in the cornea is related to the O2 concentration when the cornea is irradiated with photoactivating light. Therefore, it would be advantageous to actively increase or decrease the concentration of O2 during irradiation to control the rate of cross-linking until the desired number of cross-links is achieved. During the cross-linking treatment, oxygen can be applied in a variety of different ways. One method involves supersaturating riboflavin with O2. Thus, when riboflavin is applied to the eye, a higher concentration of O2 is delivered directly into the cornea with the riboflavin and affects the reaction involving O2 when the riboflavin is exposed to the photoactivating light. According to another method, a steady state of O2 (at a selected concentration) can be maintained at the surface of the cornea to expose the cornea to a selected amount of O2 and promote the entry of O2 into the cornea. As Figure 1As shown in, for example, the treatment system 100 also includes an oxygen source 140 and an oxygen delivery device 142, which optionally delivers a selected concentration of oxygen to the cornea 2. For example, U.S. Patent No. 8,574,277, entitled “Eye Therapy” filed on October 21, 2010, and U.S. Patent No. 9,707,126, entitled “Systems and Methods for Corneal Cross-Linking with Pulsed Light” filed on October 31, 2012, describe example systems and methods for applying oxygen during cross-linking treatment, the entire contents of which are incorporated herein by reference. In addition, an example mask device for delivering oxygen concentration and photoactivated light in eye treatment is described in U.S. Patent Application Publication No. 2017 / 0156926, entitled “Systems and Methods for Treating an Eye with a Mask Device” filed on December 5, 2016, the entire contents of which are incorporated herein by reference. For example, a mask may be placed over the eyes to produce a consistent and known oxygen concentration above the surface.

[0044] When riboflavin absorbs radiant energy, especially light, riboflavin undergoes photoactivation. There are two photochemical kinetic pathways for riboflavin photoactivation, namely Type I and Type II. Reactions involved in both Type I and Type II mechanisms and other aspects of photochemical kinetic reactions that generate cross-linking activity are described in U.S. Patent Application Publication No. 10,350,111, filed on April 27, 2016 and entitled “Systems and Methods for Cross-Linking Treatments of an Eye,” the entire contents of which are incorporated herein by reference.

[0045] In order to treat keratoconus or, for example, to achieve refractive correction, an efficient cross-linking process applies the photoactivating light as precisely as possible to a specific area of ​​the cornea treated with the cross-linking agent. Application of the photoactivating light outside the specific area may cause undesirable structural changes or damage in the cornea and may have a negative impact on the treatment outcome. However, due to the eye movements that may occur during the procedure, precise application of the photoactivating light may be difficult to achieve. For example, such eye movements may include moving along the cornea, such as Figure 1 , changes in gaze angle, and / or bulk head movement. Figure 1), the depth of the cornea 2 is measured along the z-axis, and the pattern of photoactivating light can be projected in the lateral xy plane. ) Because the cross-linking procedure may require exposure of the cornea to the photoactivating light for at least several minutes (e.g., one to thirty minutes), some movement of the eye is likely to occur during the procedure.

[0046] To account for the occurrence of eye movement, embodiments may employ an eye tracking system to determine any changes in corneal position and in response adjust the illumination system so that photoactivating light is precisely applied to a designated area of ​​the cornea. Figure 2 An example treatment system 200 with an eye tracking system 250 is shown. The treatment system 200 includes an illumination system for directing photoactivation light to the cornea 2 of the eye 1. The illumination system includes a light source 110 and an optical element 112 as described above. The light source 110 may, for example, include one or more LEDs that emit UV light to photoactivate riboflavin that has been applied to the cornea 2. The optical element 112 projects the photoactivation light onto the cornea 2 in a precise spatial pattern along the xy plane. In addition, the treatment system 200 includes one or more controllers 120 to control various aspects of the treatment system 200.

[0047] The eye tracking system 250 includes a camera 252 (image capture device) that dynamically captures multiple images 20 of the eye 1 during the process. Each image 20 may correspond to one of a series of frames in a video of the eye 1 in motion. In some embodiments, the camera 252 may be a high-speed infrared camera, and the image 20 may be a pixelated digital image. Basically, the controller 120 processes the image 20 to detect the position of one or more geometric features of the eye 1 relative to the camera 252 and therefore (relative to) the treatment system 200. Using the position of the one or more features as a reference, the controller 120 can determine the position of a designated area of ​​the cornea 2. Therefore, the controller 120 can adjust the treatment system 200 to deliver photoactivation light to the location of the designated area. The eye tracking system 250 also includes software (e.g., computer-readable instructions stored on a non-transitory medium) used by the controller 120 to process the image 20.

[0048] As described above, eye tracking in medical treatments such as cross-linking therapy should be robust and accurate because errors in eye tracking can result in ineffective treatment and / or injury / harm to the patient. Some eye tracking systems rely on light reflections from the cornea captured in each image of the eye. For example, Figure 3 An example reflection 28 is shown. However, the reflection pattern may provide an unreliable basis for eye tracking. Fig.9EAs shown in , an intraocular lens (IOL) implanted in the eye can generate irregular reflection patterns. In addition, when the eye is kept open for a long time through a speculum used to limit blinking during treatment, the tear film on the cornea may break up. The dry and broken tear film forms a reflective surface that produces bright patterns in the image and brings more challenges to reflection-based tracking systems. Therefore, in order to achieve robust and accurate eye tracking, an embodiment adopts a method that does not rely on light reflection from the cornea.

[0049] Figure 3 An example pixelated image 20 of an eye captured by camera 252 is shown. In particular, image 20 includes features 22, 24, 26. Image feature 22 corresponds to low-level anatomical structures in iris region 3 formed by the iris sphincter and pigmented vascular matrix. These anatomical structures appear as textures in image 20, particularly when the image is captured by a high-speed infrared camera. Image feature 24 corresponds to a dark, generally circular or circular-like shape defined by the contrast between iris region 3 and pupil region 4. Image feature 26 corresponds to a generally circular boundary between iris region 3 and pupil region 4. Controller 120 may detect image features 22, 24, 26 and determine changes in the shape and position of the image features 22, 24, 26 over a time series of images 20.

[0050] The eye tracking system 250 may include Figure 4 . In particular, the integrated tracker 454 (as implemented by the controller 120) uses trackers A, B, and C to process the infrared image 20a captured by the high-speed infrared camera. All three trackers A, B, and C operate very quickly using local pixel information and can work concurrently. Frame-to-frame motion estimates for the eye can be obtained very quickly (e.g., within a few milliseconds) from any of the trackers A, B, and C.

[0051] Tracker A is specifically adapted to image features 22 (i.e., texture in the iris region). Tracker A may use a variation of the Lucas Kanade Tomasi (LKT) feature tracker to estimate the multi-scale optical flow of a set of feature points in the iris region 3, which are essentially extreme points with high spatial frequency. These feature points are detected at the beginning of tracking and can be automatically supplemented when feature points are lost due to scene changes.

[0052] Tracker B is specifically adapted to the image feature 24, i.e., the contrast between the iris and the pupil. Using the fact that the pupil appears darker than the iris in the infrared image 20a, tracker B may use optimization techniques to find a darker shape formed by a series of pixels in the pupil region 4 surrounded by lighter-colored pixels in the iris region 3. Tracker B may employ robust statistics to ignore bright and saturated pixels that appear due to reflections, such as using a Huber-type M estimator.

[0053] Tracker C is specifically adapted to image feature 26, namely, the iris-pupil boundary. Tracker C can detect the circular pupil-iris boundary by fitting a circular or elliptical model to a higher-scale edge map obtained from infrared image 20a.

[0054] like Figure 4 As shown in , the example method 400 employs an integrated tracker 454 that combines data from trackers A, B, and C to provide a more robust, accurate, and efficient estimate of eye motion from infrared image 20a. For example, using an integrated average motion based on information associated with all three image features 22, 24, and 26, the integrated tracker 454 can estimate parameters for the pupil as an indicator of overall eye motion.

[0055] The integrated tracker 454 is highly robust because the trackers A, B, and C are designed to capture information from orthogonal (non-overlapping) image features in image 20a and provide independent estimates of eye motion. These independent estimates are compared to each other and analyzed using advanced meta-heuristic procedures, thereby reducing error variance and allowing smooth estimates of eye motion to be obtained with higher accuracy. Sometimes, when tracking is affected by reflections and other occlusions (such as Fig.9A -E), trackers A, B, C may fail. However, high-level metaheuristics can take into account errors and missing information caused by such failures, thereby allowing tracking to continue.

[0056] The integrated tracker 454 efficiently models the characteristic motions occurring in the eye 1. The motions of the eye 1 range from high-speed, volatile movements known as saccadic movements to slow, smooth, follow-up movements used to track slow-moving objects. The eye 1 may also undergo vestibulo-ocular and optokinetic reflexes when the gaze angle is repositioned. In addition, when moving along an object in depth (e.g., along Figure 1 The eye 1 undergoes vergence movements when the eye is moved about the z-axis (shown in FIG. 1 ) to maintain an object in the center of the binocular field of view. The integrated tracker 454 is fast and robust enough to account for anatomical changes in eye geometry, changes in iris reflectivity, and changes in pupil contrast under these typical movements.

[0057] The integrated tracker 454 takes into account tracking errors that may be introduced due to reflections of illumination present during the capture of the image 20. Advantageously, the example method 400 is independent of the geometry of the illumination. For example, the example method 400 provides efficient eye tracking regardless of whether the illumination is provided by a point light source or an ambient / diffuse light source, or whether the light source is on-axis or off-axis. In particular, the example method 400 can reject retro-reflections of the light source from the optical surfaces of the eye (i.e., the anterior and posterior surfaces of the cornea and lens). By using an advanced meta-heuristic algorithm to combine tracking data from three independent trackers A, B, and C, the error variance of the motion-related estimates can be reduced and the accuracy of the trackers can be improved, and the tracking speed can be maintained under super real-time (i.e., 60Hz) conditions.

[0058] As described above, trackers A, B, and C are designed to acquire information from orthogonal image features in image 20a and provide estimates related to eye movement, and the integrated tracker 454 uses an advanced metaheuristic algorithm to manage trackers A, B, and C and merge the data from trackers A, B, and C to produce a final estimate related to eye movement that minimizes the net estimation error. Figure 5A FIG. 5B illustrates an example method 500 that employs an advanced meta-heuristic algorithm to produce a final estimate related to eye motion. Eye motion is represented by changes in eye pupil parameters (i.e., pupil center position and pupil radius). A time series of infrared images 20a produces frames F that capture eye motion. i In the example shown in the figure, the previous frame F n-1 The pupil parameters in the image (shown as data 502) are known. Tracker A uses a multi-scale Lucas Kanade Tomasi (LKT) feature tracker in action 504 to determine the image corresponding to the current frame F. n The net motion of the pupil is related to the motion of these feature points and can be determined by combining the motion vectors of the individual feature points using random sample consensus (RANSAC). In act 506, the resulting iris (feature point) motion (i.e., the net motion of the pupil) is applied to the previous frame F. n-1 The pupil center position is thus generated for the current frame F n A first estimate 508 of pupil parameters.

[0059] Tracker B and tracker C may be initialized using the first estimate 508 of the pupil center location as an initial guess. In act 510, tracker B generates a second estimate 512 of pupil parameters by solving an optimization problem using a gradient ascent method. In particular, the contrast between the pixel intensity in the pupil region and the pixel intensity in the iris region is maximized to determine the unknown pupil center location and radius. Meanwhile, in act 514, tracker C generates a third estimate 516 of pupil parameters by fitting a circular pupil-iris boundary to the edge map to determine the unknown pupil center location and radius.

[0060] In decision 518, the estimates 508, 512, 516 of pupil parameters are evaluated to determine whether they are consistent with each other. The advanced metaheuristic algorithm measures the deviation between the estimates of pupil parameters produced by trackers A, B, C and ranks them based on their consistency. If the least consistent estimate deviates from the other two estimates by an amount greater than an empirical threshold, then the inconsistent estimate is considered to be erroneous and is discarded, and the remaining two mutually consistent estimates are averaged in act 520 to produce the estimate for frame F. n Conversely, if the least consistent estimate is within the empirical bounds, the estimates 508, 512, 516 are considered to be consistent with each other and are then combined in act 522 to produce the average estimate 508, 512, 516 for frame F. n Final estimate of pupil parameters in 524. Figure 6 The figure shows an example frame with image features 22 , 24 , 26 employed by trackers A, B, C respectively, and the consensus on the pupil center location based on the final estimate 524 .

[0061] In summary, trackers A, B, and C are designed so that most of the useful information in the image is used. Each tracker targets a specific image feature that is mutually exclusive and non-overlapping with the image features used by the other trackers. The method thus obtains multiple measurements using orthogonal pieces of information and then combines (merges) these measurements in a robust way, thereby reducing the error introduced by a bad measurement of any one feature. In this way, the failure of any one feature will not affect the tracker at all, and the error variance is always reduced by averaging.

[0062] Fig.9A FIG. 20A shows various example phenomena that may be captured in image 20a and may produce aberrations, noise, distortion, occlusion, etc., which may affect the estimates 508, 512, 516 produced by trackers A, B, C. Such phenomena may cause errors and lead to inconsistencies between estimates 508, 512, 516 as described above. Specifically, Fig.9AAn image 90a is shown that captures an irregularly shaped pupil or a change in pupil shape. Fig. 9B Image 90b is shown which captures shadows and reflections caused by a mask worn by a patient undergoing treatment or by an eye being partially blocked by an eyelid. Fig. 9C An image 90c is shown which captures the dropper and speculum used for treatment. Fig.9D An image 90d is shown which captures the graying of the pupil caused by the intraocular implant. Fig.9E Image 90e is shown, which captures the unstable reflection pattern resulting from the implanted IOL. Fig.9F An image 90d is shown that captures an occlusion such as a finger or an eye dropper.

[0063] The high-level meta-heuristic algorithm (as shown in FIG. 5 ) determines the degree / amount of consistency between the estimates 508 , 512 , 516 from trackers A, B, C, respectively, and attempts to reduce the error by analyzing the estimates against each other. n If the error at F is small, the estimates 508, 512, 516 will be approximately the same, with very small mutual deviations, and should therefore provide an indication of the actual pupil parameters (i.e., center position and radius). n If the error at is not negligible, the estimates 508, 512, 516 will be inconsistent with each other and will reveal deviations from the actual pupil parameters. The inconsistency occurs because trackers A, B, C are designed to encode orthogonal information by measuring non-overlapping image features, and the same error may be manifested in different ways by trackers A, B, C.

[0064] like Figure 7 As shown in FIG. 1 , the above eye tracking process may generate a track 702, which is a time series of frames F. i is processed and progresses through different states. Figure 7, when the eye tracking process begins, the pupil cannot be found in the images of the initial frames F1, F2, F3, and F4. Therefore, these initial frames fall into a state 704 labeled "Track Not Found". After the pupil can be found in the image, the track 702 is considered to be in a state 706 labeled "Infant Track" until certain criteria described below are met. Frames F5, F6, and F7 correspond to a time period when the track 702 is in the infant track state. After the criteria are met, the track 702 is considered to be in a state 708 labeled "mature track". Frame F8 and subsequent frames correspond to a time period when the track 702 is in the mature track state.

[0065] When the track 702 is in the early tracking state, the processing of the corresponding frames is slower and more detailed to ensure that the track 702 is truly established and to avoid false starts. In particular, the entire image of each frame is independently searched (global search) to obtain separate estimates of pupil parameters. In this way, in the early stages of the track 702, the estimates of consecutive frames can be analyzed against each other to verify the temporal consistency across the frames. Each frame undergoes this slower, more detailed processing until temporal consistency is established for an empirical threshold number (N) of frames. After the temporal consistency is established, the track 702 enters the mature tracking state. If the consistency check fails at any time, the process starts over, and the track 702 cannot be considered to be in the mature tracking state until N consecutive frames are found to be consistent.

[0066] After the track 702 enters the mature tracking state, the frame can be processed according to the example method 500. As described above, in the previous frame F n-1 The pupil estimate 502 at is used in the current frame F n As an initial guess in the processing of F, a relatively narrow search for pupil parameters based on trackers A, B, and C is performed. In the mature tracking state, due to the previous frame F n-1 The pupil parameters in the current frame F are used as n The initial guess in , so no temporal consistency check is done. Since no temporal consistency check is done and the search is local, the pursuit process in this state is faster.

[0067] Figure 8 The diagram shows a sequence of frames F for processing when the tracking is in the mature tracking state. i In particular, for frame F n-1 , pupil parameters 502 are determined based on the consensus generated by applying a high-level meta-heuristic algorithm to trackers A, B, and C. As described above, for frame F n , using the n-1Starting with pupil parameters 502 of trackers A, B, and C, pupil parameters 524 are determined based on another consensus generated by applying a high-level metaheuristic algorithm to trackers A, B, and C. For subsequent frames, repeat the process for frame F. n For example, for frame F n+1 , using the n Starting at pupil parameters 524 , pupil parameters are determined based on a further consensus generated by applying a high-level meta-heuristic algorithm to trackers A, B, and C. Method 800 is robust and fast enough to process frames at a rate of 60 Hz.

[0068] As described above, according to some aspects of the present disclosure, some or all of the steps in the above-described and illustrated processes may be automatically performed or directed under the control of a controller (e.g., controller 120). Generally, a controller may be implemented as a combination of hardware elements and software elements. Hardware aspects may include a combination of operationally coupled hardware components, including a microprocessor, a logic circuit, a communication / network port, a digital filter, a memory, or a logic circuit. A controller may be adapted to implement operations specified by a computer executable code, which may be stored on a computer readable medium.

[0069] As described above, the controller may be a programmable processing device that executes software or stored program instructions, such as an external conventional computer or an onboard field programmable gate array (FPGA) or digital signal processor (DSP). In general, the physical processor and / or machine used by the embodiments of the present disclosure for any processing or evaluation may include one or more networked or non-networked general-purpose computer systems, microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), microcontrollers, etc., programmed according to the teachings of the exemplary embodiments of the present disclosure, as understood by those skilled in the art of computer and software technology. The physical processor and / or machine may be networked externally to the image capture device, or may also be integrated to reside within the image capture device. Suitable software may be easily prepared by a programmer with ordinary skills based on the teachings of the exemplary embodiments, as understood by those skilled in the art of software technology. In addition, the devices and subsystems of the exemplary embodiments may be implemented by preparing application-specific integrated circuits or by interconnecting suitable networks of conventional component circuits, as understood by those skilled in the art of electrical technology. Therefore, the exemplary embodiments are not limited to any specific combination of hardware circuits and / or software.

[0070] Stored on any computer-readable medium or a combination of computer-readable media, embodiments of the present disclosure may include software or stored program instructions for controlling the devices and subsystems of the example embodiments, for driving the devices and subsystems of the example embodiments, for enabling the devices and subsystems of the example embodiments to interact with human users. Such software may include, but is not limited to, device drivers, firmware, operating systems, development tools, application software, and the like. Such computer-readable media may additionally include a computer program product of an embodiment of the present disclosure, which is used to implement all or part of the processing implemented in each embodiment (if the processing is distributed). The computer code device of the example embodiments of the present disclosure may include any suitable interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), Java classes and applets, complete executable programs, and the like. In addition, some of the processing of the example embodiments of the present disclosure may be distributed to have better performance, reliability, cost, and the like.

[0071] Common forms of computer-readable media may include, for example, floppy disks, diskettes, hard disks, magnetic tape, any other suitable magnetic media, CD-ROMs, CDRWs, DVDs, any other suitable optical media, punch cards, paper tape, optical marker sheets, any other suitable physical media with patterns of holes or other optically identifiable markings, RAM, PROM, EPROM, FLASH-EPROM, any other suitable memory chip or cartridge, a carrier wave, or any other suitable medium from which a computer can read.

[0072] Although the present disclosure has been described with reference to one or more specific embodiments, it will be appreciated by those skilled in the art that many changes may be made to these embodiments without departing from the spirit and scope of the present disclosure. Each of these embodiments and their obvious variations are considered to fall within the spirit and scope of the present disclosure. It is also contemplated that additional embodiments according to aspects of the present disclosure may be combined with any number of features from any embodiment described herein.

Claims

1. A system for tracking eye movement during eye treatment, comprising: an image capture device configured to capture a plurality of images of an eye; as well as an integrated tracker, the integrated tracker using a first tracker, a second tracker, and a third tracker to process a plurality of images, wherein the first tracker is configured to detect a first feature in the plurality of images and provide a first data set related to the movement of the eye based on the first feature, the first feature comprising an anatomical structure in an iris region of the eye; wherein the second tracker is configured to detect a second feature in the plurality of images and provide a second data set related to the movement of the eye based on the second feature, the second feature comprising a shape defined by a contrast between an iris region and a pupil region of the eye; and The third tracker is configured to detect a third feature in the plurality of images and provide a third data set related to the movement of the eye based on the third feature, wherein the third feature includes a boundary between an iris region and a pupil region of the eye.

2. The system of claim 1 , further comprising one or more controllers, the one or more controllers comprising one or more processors configured to execute program instructions stored on one or more computer-readable media, the program instructions causing the one or more processors to identify, based on the tracking state, a temporal consistency state for a threshold number of frames in a temporal sequence of frames corresponding to the plurality of images.

3. The system of claim 2, wherein: To determine the indicator of eye movement, the program instructions cause the one or more processors to process frames corresponding to a temporal sequence of the plurality of images in a manner based on (i) for frame F n The consensus of the combined dataset from the first, second and third trackers and (ii) for the previous frame F n-1 The determined eye position is iteratively determined in frame F n The position of the eyes in the.

4. The system of claim 1, wherein: The first tracker uses a multi-scale Lucas Kanade Tomasi (LKT) feature tracker to determine the optical flow of a set of feature points.

5. The system of claim 4, wherein: The first tracker generates a first estimate of pupil parameters by applying the motion of the iris resulting from the optical flow of the set of feature points.

6. The system of claim 5, wherein: The second tracker generates a second estimate of the pupil parameters by solving an optimization problem using a gradient ascent method.

7. The system of claim 6, wherein: The third tracker produces a third estimate of pupil parameters by fitting a boundary determined from the resulting motion to the edge map.

8. The system of claim 7, wherein: The first estimate, the second estimate and the third estimate are combined to generate a final estimate of the pupil parameter.

9. The system of claim 1, wherein: The multiple images are pixelated, and each tracker detects corresponding features based on local pixel information.

10. The system of claim 1, wherein: The image capture device includes a high-speed infrared camera, and the plurality of images are infrared images.

Images