System and method for eye tracking during eye treatment

Abstract

We provide a system for tracking eye movements during eye treatment. [Solution] One exemplary system for tracking eye movements during eye treatment includes an image capture device configured to capture multiple images of the eye. The system includes a controller which includes a processor that receives multiple images from the image capture device. The processor implements multiple trackers. Each tracker is configured to detect a respective feature in the multiple images and, based on the respective feature, provides a respective dataset related to eye movement. The respective features detected by the multiple trackers are orthogonal to each other, and the respective datasets provided by the multiple trackers are independent of each other. The processor combines the datasets from the multiple trackers and determines an index of the eye movement based on the dataset.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 733,620 (filed September 19, 2018), the contents of which are fully incorporated herein by reference.

[0002] Technical field This disclosure relates to a system and method for treating the eye, more specifically, a system and method for tracking eye movements to treat a desired area of ​​the eye. [Background technology]

[0003] Cross-linking therapy can be used to treat eyes affected by diseases (e.g., keratoconus). In particular, keratoconus is a degenerative eye disease in which structural changes within the cornea weaken the cornea and cause it to deform into an abnormal cone shape. Cross-linking therapy can strengthen and stabilize the area weakened by keratoconus and prevent undesirable shape changes.

[0004] Cross-linking treatment may also be used after surgical procedures such as LASIK (Laser-Assisted in-site Keratomileusis). For example, a complication known as post-LASIK ectasia can occur due to corneal thinning and weakening caused by LASIK surgery. In post-LASIK ectasia, a progressive steepening (bulging) of the cornea occurs. Therefore, cross-linking treatment can strengthen and stabilize the corneal structure after LASIK surgery and prevent post-LASIK ectasia.

[0005] Cross-linking therapy can also be used to induce refractive changes in the cornea to correct disorders such as nearsightedness, farsightedness, astigmatism, irregular astigmatism, and presbyopia. [Overview of the project]

[0006] In cross-linking surgery, the cornea needs to be exposed to photoactivating light for at least several minutes (e.g., 1 to 30 minutes), making it highly likely that some eye movement will occur during the procedure. To address the occurrence of eye movement, the system and method can use an eye-tracking system to determine any changes in corneal position and, accordingly, adjust the illumination system to precisely irradiate specific areas of the cornea with photoactivating light.

[0007] An exemplary system for tracking eye movements during eye treatment includes an image capture device configured to capture multiple images of the eye. The system includes one or more controllers, each including one or more processors configured to execute program instructions stored in one or more computer-readable media. The one or more processors receive the multiple images from the image capture device. The one or more processors implement multiple trackers. Each tracker is configured to detect a feature in each of the multiple images and, based on that feature, provide a corresponding dataset related to the eye movements. The features detected by the multiple trackers are orthogonal to each other, and the corresponding datasets provided by the multiple trackers are independent of each other. The one or more processors combine the datasets from the multiple trackers and determine an index of the eye movements based on the combined dataset.

[0008] Eye tracking in medical treatments (e.g., cross-linking therapy) must be robust and accurate, because errors in eye tracking can lead to ineffective treatment and / or injury to the patient. The exemplary system and method are highly robust because the tracker acquires information about orthogonal (non-overlapping) image features in the multiple images and provides independent estimates of eye movement. These independent estimates are analyzed against each other using a sophisticated level of metaheuristic processes to reduce error variance and enable more accurate acquisition of smoother estimates of eye movement. If the tracker occasionally malfunctions due to reflections or other interferences in the images, the system and method can continue tracking because it can compensate for the erroneous or missing information from such malfunctions. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an exemplary system for applying a crosslinking agent and photoactivating light to the cornea of ​​the eye in order to generate crosslinks of corneal collagen according to an aspect of the present disclosure. [Figure 2] This figure shows an exemplary therapeutic system comprising an eye-tracking system according to an aspect of the present disclosure. [Figure 3] This figure shows features that may be detected in a pixelated image of an eye captured by an exemplary eye-tracking system according to an aspect of the present disclosure. [Figure 4] This figure shows an exemplary solution for an eye tracking system according to an aspect of the present disclosure, which uses a cluster tracker to track and process multiple features in an image of a captured eye. [Figure 5A] This figure illustrates an exemplary method, according to an aspect of the present disclosure, that uses a high level of metaheuristics (general heuristic problem-solving method) to combine data from three trackers in order to generate a final estimate related to eye movement. [Figure 5B]A diagram showing an exemplary method of using high-level metaheuristics to combine data from three trackers to generate a final estimate related to eye movement according to an aspect of the present disclosure. [Figure 6] A diagram showing an exemplary frame with features of images used by respective trackers to determine a consensus of pupil center positions based on the exemplary method shown in FIGS. 5A-5B according to an aspect of the present disclosure. [Figure 7] A diagram showing an exemplary eye tracking process of generating a track that progresses through various states as a time series of frames Fi is processed according to an aspect of the present disclosure. [Figure 8] A diagram showing an exemplary approach for processing a series of frames Fi in a "mature track" as shown in FIG. 7 according to an aspect of the present disclosure. [Figure 9A] An exemplary image capturing an irregularly shaped pupil or a change in pupil shape as an error-inducing phenomenon that can occur. [Figure 9B] An exemplary image capturing shadows and reflections from partial occlusion of the eye by a mask or eyelid worn by the patient for treatment as an error-inducing phenomenon that can occur. [Figure 9C] An exemplary image capturing an eye dropper and an ophthalmoscope used in treatment as an error-inducing phenomenon that can occur. [Figure 9D] An example of an image capturing graying of the pupil caused by an intraocular implant as an error-inducing phenomenon that can occur. [Figure 9E] An exemplary image capturing an irregular reflection pattern due to an implanted intraocular lens as an error-inducing phenomenon that can occur. [Figure 9F] An exemplary image capturing an obstruction (e.g., a finger or an eye dropper) as an error-inducing phenomenon that can occur.

Mode for Carrying Out the Invention

[0010] While various modifications and alternative forms are possible for this disclosure, specific embodiments are shown in the drawings as examples and will be described in detail herein. However, it should be understood that this disclosure is not intended to limit itself to any specific form disclosed, but rather to encompass all modifications, equivalents, and alternatives that are in line with the spirit of this disclosure.

[0011] Figure 1 shows one exemplary therapeutic system 100 for generating collagen cross-linking in the cornea 2 of an eye 1. The therapeutic system 100 includes an applicator 132 for administering a cross-linking agent 130 to the cornea 2. In the exemplary embodiment, the applicator 132 may be an eye dropper, syringe, etc., for administering the photosensitizer 130 as a droplet to the cornea 2. Exemplary systems and methods for administering the cross-linking agent are described in U.S. Patent No. 10,342,697, filed April 13, 2017, entitled "Systems and Methods for Delivering Drugs to an Eye," which is incorporated herein by reference in its entirety.

[0012] The crosslinking agent 130 may be provided in a formulation that allows the crosslinking agent 130 to pass through the corneal epithelium 2a to the lower region within the corneal matrix 2b. Alternatively, the corneal epithelium 2a may be removed or otherwise incised to allow the crosslinking agent 130 to be administered more directly to the lower tissue.

[0013] The treatment system 100 includes an irradiation system comprising a light source 110 and an optical element 112 for directing the light onto the cornea 2. This light causes photoactivation of the crosslinking agent 130, generating crosslinking activity in the cornea 2. For example, the crosslinking agent may include riboflavin, and the light for photoactivation (hereinafter referred to as photoactivating light) may include ultraviolet A (UVA) light (e.g., about 365 nm). Alternatively, the photoactivating light may include another wavelength, for example, a visible wavelength (e.g., about 452 nm). As will be further described below, corneal crosslinking improves corneal strength by generating chemical bonds within the corneal tissue according to a photochemical reaction mechanism. For example, riboflavin and photoactivating light may be applied to stabilize and / or strengthen the corneal tissue to address a disease (e.g., keratoconus or post-LASIK ectasia).

[0014] The treatment system 100 includes one or more controllers 120 that control the configuration of the system 100 (including a light source 110 and / or an optical element 112). In one embodiment, the cornea 2 can be broadly treated with a crosslinking agent 130 (e.g., using an eye dropper, syringe, etc.), and the photoactivating light from the light source 110 can be selectively directed to areas of the cornea 2 to be treated according to a specific pattern.

[0015] The optical element 112 may include one or more mirrors or lenses for directing and focusing the photoactivating light emitted by the light source 110 onto a specific pattern on the cornea 2. The optical element 112 may further include filters for partially blocking wavelengths of light emitted by the light source 110 and for selecting specific wavelengths of light directed onto the cornea 2 to photoactivate the crosslinking agent 130. Furthermore, the optical element 112 may include one or more beam splitters for splitting the beam of light emitted by the light source 110, and one or more heat sinks for absorbing light emitted by the light source 110. The optical element 112 can also accurately and precisely focus the photoactivating light onto a specific focal plane in the cornea 2 (for example, at a specific depth in the lower region 2b where crosslinking activity is desired).

[0016] Furthermore, the inherent properties of the photoactivating light can be modified to achieve a desired degree of crosslinking in a selected area of ​​the cornea 2. One or more controllers 120 can control the operation of the light source 110 and / or optical element 112 to precisely deliver the photoactivating light according to any combination of wavelength, bandwidth, intensity, output, position, penetration depth, and / or treatment duration (exposure cycle duration, dark cycle duration, and ratio of exposure cycle to dark cycle duration).

[0017] The parameters for photoactivation of the crosslinking agent 130 can be adjusted, for example, to shorten the time required to achieve the desired crosslinking. In one implementation example, the time can be reduced from minutes to seconds. Some configurations use 5 mW / cm². 2 While photoactivating light can be irradiated at a certain irradiance, to shorten the time required to achieve the necessary crosslinking, the photoactivating light can be irradiated at a higher irradiance (e.g., 5 mW / cm²). 2 It can be irradiated at multiples of . The total dose of energy absorbed by the cornea 2 can be described as the effective dose, which is the amount of energy absorbed by passing through the region of the corneal epithelium 2a. For example, the effective dose for the region of the corneal surface 2a is, for example, 5 J / cm². 2 , or 20 J / cm 2 , or 30 J / cm 2 It can be of a certain magnitude. The stated effective dose can be obtained from a single energy delivery or from repeated energy deliveries.

[0018] The optical elements 112 of this treatment system 100 may include a microelectromechanical system (MEMS), such as a digital micro-mirror device (DMD), to spatially and temporally modulate the irradiation of photoactivating light. Using DMD technology, photoactivating light from the light source 110 is projected in a precise spatial pattern created by an array of extremely small mirrors on a semiconductor chip. Each mirror represents one or more pixels of the projected light pattern. Using DMD, topographic-guided bridging can be performed. Topographic control of the DMD can use several different spatial and temporal illuminance and dose profiles. These spatial and temporal dose profiles can be created using continuous wave irradiation, but can also be adjusted via pulsed irradiation by pulsed irradiation, which is achieved by changing the frequency and duty cycle of the irradiation source. Alternatively, the DMD can be adjusted so that the frequency and duty cycle of each pixel differ, providing ultimate flexibility using continuous wave irradiation. Alternatively, both pulsed irradiation and a combination of adjusted DMD frequencies and duty cycles may be used. This spatially determined crosslinking can be combined with dosimetry, interferometry, optical coherence tomography (OCT), corneal topography, etc., for pretreatment planning and / or real-time monitoring and adjustment of corneal crosslinking during treatment. The embodiments of the dosimetry system will be described in more detail below. Furthermore, preclinical patient information can be combined with finite-element biomechanical computer modeling to create patient-specific pretreatment plans.

[0019] To control the projection of photoactivating light, embodiments may also employ multiphoton excitation microscopy techniques. Specifically, rather than irradiating the cornea 2 with a single photon of a specific wavelength, the treatment system 100 may irradiate with multiple photons of longer wavelengths (i.e., lower energy) that bind to initiate crosslinking. Advantageously, longer wavelengths are scattered less within the cornea 2 than shorter wavelengths, which allows longer wavelength light to penetrate the cornea 2 more efficiently than shorter wavelength light. The shielding effect of incident irradiation at deeper points in the cornea is also reduced compared to conventional short-wavelength irradiation, because light absorption by the photosensitizer is much less at longer wavelengths. This allows for enhanced control over depth-specific crosslinking. For example, in some embodiments, two photons may be used, each carrying approximately half the energy required to excite molecules in the crosslinking agent 130 to produce the photochemical reactions described further below. If the crosslinking molecule absorbs both photons simultaneously, it absorbs enough energy to release reactive radicals in the corneal tissue. Embodiments may also utilize lower-energy photons, such that the crosslinking molecule must simultaneously absorb, for example, three, four, or five photons, in order to release reactive radicals. The likelihood of multiple photons being absorbed nearly simultaneously is low, and therefore high-flux excitation photons are required, which can be provided via a femtosecond laser.

[0020] Numerous conditions and parameters affect the crosslinking of corneal collagen by the crosslinking agent 130. For example, the illuminance and dose of photoactivating light affect the amount and rate of crosslinking.

[0021] When the crosslinking agent 130 is specifically riboflavin, UVA light may be irradiated continuously (continuous wave (CW)) or as pulsed light, and this choice affects the amount, speed, and degree of crosslinking. When UVA light is irradiated as pulsed light, the exposure cycle, the duration of the dark cycle, and the ratio of the exposure cycle to the dark cycle duration affect corneal hardening as a result. Pulsed light irradiation can be used to strengthen or weaken corneal tissue hardening more than can be achieved with continuous wave irradiation applying the same amount or dose of energy. Light pulses of appropriate length and frequency can be used to achieve more optimal chemical amplification. For pulsed light therapy, the on / off duty cycle can be between approximately 1 / 1000 and approximately 1 / 1000. The illuminance is an average illuminance of approximately 1 mW / cm². 2 From approximately 1000 mW / cm² 2 It can be between approximately 0.01 Hz and approximately 1000 Hz, or between approximately 1000 Hz and approximately 100,000 Hz.

[0022] The treatment system 100 can generate pulsed light by employing a DMD (electronically turning the light source 110 on and off) and / or by using a mechanical or photoelectronic (e.g., Pockels cell) shutter or a mechanical chopper or rotating aperture. Due to the modulation capabilities inherent in the pixels of the DMD, and subsequent stiffening based on the modulated frequency, duty cycle, illuminance, and dose irradiated to the cornea, complex biomechanical stiffness patterns are imparted to the cornea, enabling refractive correction of varying degrees. These refractive corrections may include combinations of myopia, hyperopia, astigmatism, irregular astigmatism, presbyopia, and complex corneal refractive corrections due to eye conditions (e.g., keratoconus, clear peripheral disease, post-LASIK ectasia) and other conditions such as biomechanical changes / degeneration of the cornea. A specific advantage of the DMD system and method is that it allows for the creation of non-periodic and uniform-looking illumination for random asynchronous pulse topography patterning (this illumination eliminates the possibility of causing photosensitive epileptic seizures or flicker vertigo to pulse frequencies between 2 Hz and 84 Hz).

[0023] Although exemplary embodiments may utilize a stepped on / off pulsed light function, it is understood that other functions for irradiating the cornea with light may be used to achieve a similar effect. For example, light may be irradiated onto the cornea according to a sinusoidal function, a sawtooth wave function, or other complex functions or curves, or any combination of functions or curves. In fact, it will be understood that the function may be substantially stepped even if it is a more gradual transition between on / off values. Furthermore, it will be understood that the illuminance does not need to decrease to zero during the off-cycle, and may even be above zero during the off-cycle. The desired effect may be achieved by irradiating the cornea with light according to a curve that varies the illuminance between two or more values.

[0024] Examples of systems and methods for delivering photoactivating light are described, for example, in U.S. Patent Application Publication No. 2011 / 0237999 (filed March 18, 2011), entitled "Systems and Methods for Applying and Monitoring Eye Therapy," U.S. Patent Application Publication No. 2012 / 0215155 (filed April 3, 2012), entitled "Systems and Methods for Applying and Monitoring Eye Therapy," and U.S. Patent Application Publication No. 2013 / 0245536 (filed March 15, 2013), entitled "Systems and Methods for Corneal Cross-Linking with Pulsed Light." The contents of these applications are incorporated herein by reference in their entirety.

[0025] The addition of oxygen also affects the degree of corneal hardening. In human tissue, the O2 content is extremely low compared to air. However, the rate of crosslinking in the cornea is related to the concentration of O2 when irradiated with photoactivating light. Therefore, it may be advantageous to actively increase or decrease the O2 concentration during irradiation to control the crosslinking rate until the desired degree of crosslinking is achieved. Oxygen may be administered during crosslinking therapy in several different ways. One approach is to supersaturate riboflavin with O2. In this way, when riboflavin is administered to the eye, a high concentration of O2 is directly administered into the cornea along with the riboflavin, influencing the O2-involved reaction when riboflavin is exposed to photoactivating light. According to another approach, a steady state of O2 (at a selected concentration) is maintained on the surface of the cornea, and the cornea is exposed to a selected amount of O2, allowing the O2 to enter the cornea. As shown in Figure 1, for example, the treatment system 100 also includes an oxygen source 140 and an oxygen supply device 142 that optionally supplies oxygen to the cornea 2 at an optionally selected concentration. Exemplary systems and methods for administering oxygen during cross-linking therapy are described, for example, in U.S. Patent No. 8,574,277, titled "Eye Therapy" (filed October 21, 2010) and U.S. Patent No. 9,707,126, titled "Systems and Methods for Corneal Cross-Linking with Pulsed Light" (filed October 31, 2012). The contents of these applications are incorporated herein by reference in their entirety. Furthermore, exemplary mask devices for administering oxygen concentration and photoactivating light in eye treatment are described in U.S. Patent Application Publication No. 2017 / 0156926 (filed December 3, 2016), entitled "Systems and Methods for Treating an Eye with a Mask Device," the contents of which are incorporated herein by reference in their entirety. For example, a mask may be positioned to cover the eye in order to provide a consistent and known oxygen concentration on the surface of the eye.

[0026] Riboflavin undergoes photoactivation upon absorbing irradiation energy (particularly light). There are two photochemical dynamic pathways for the photoactivation of riboflavin: Type I and Type II. The reactions involved in both Type I and Type II mechanisms, as well as another aspect of the photochemical dynamic reactions that generate crosslinking activity, are described in U.S. Patent No. 10,350,111 (filed April 27, 2016), entitled "Systems and Methods for Cross-Linking Treatments of an Eye," which is incorporated herein by reference in its entirety.

[0027] For example, to treat keratoconus or to achieve refractive correction, effective cross-linking surgery involves irradiating a specific area of ​​the cornea to be treated with a crosslinking agent with photoactivating light as precisely as possible. Irradiating areas outside the designated region with photoactivating light can cause undesirable structural changes or damage to the cornea, potentially negatively impacting the treatment outcome. However, precise irradiation of the photoactivating light can be difficult to achieve due to eye movements that may occur during the procedure. Such eye movements may include, for example, translation along the xy plane, changes in line of sight angle, and / or septal movement, as shown in Figure 1. (In Figure 1, the depth of the cornea 2 is measured along the z axis, and the pattern of photoactivating light may be projected onto the lateral xy plane.) Since cross-linking surgery requires exposing the cornea to photoactivating light for at least several minutes, e.g., 1 to 30 minutes, there is a very high probability that some eye movement will occur during the procedure.

[0028] To address the occurrence of eye movements, embodiments may utilize an eye tracking system to determine changes in corneal position and, in response, adjust the irradiation system to precisely irradiate a specific area of ​​the cornea with photoactivating light. Figure 2 shows an exemplary treatment system 200 with an eye tracking system 250. The treatment system 200 includes an irradiation system for directing photoactivating light onto the cornea 2 of eye 1. The irradiation system includes a light source 110 and optical elements 112, as described above. For example, the light source 110 may include one or more LEDs that emit UV light to photoactivate riboflavin applied to the cornea 2. The optical elements 112 irradiate the cornea 2 with photoactivating light along the xy plane in a precise spatial pattern. Furthermore, the treatment system 200 includes one or more controllers 120 for controlling embodiments of the treatment system 200.

[0029] The eye-tracking system 250 includes a camera 252 (image capture device) that dynamically captures (also called "capturing") multiple images 20 of the eye 1 during surgery. Each image 20 may correspond to one of a series of frames in a video of the moving eye 1. In some embodiments, the camera 252 may be a high-speed infrared camera, and the images 20 may be pixelated digital images. Generally, the controller 120 can process the images 20 to detect the position of one or more geometric features of the eye 1 relative to the camera 252 and, consequently, the treatment system 200. Using the position of one or more features as a reference, the controller 120 can determine the position of a specified region of the cornea 2. In this way, the controller 120 can adjust the treatment system 200 to deliver photoactivating light to the location of the specified region. The eye-tracking system 250 also includes software (e.g., computer-readable instructions stored in a non-temporary medium) used by the controller 120 to process the images 20.

[0030] As mentioned above, eye tracking in medical treatments (e.g., cross-linking therapy) must be robust and accurate, as errors in eye tracking can lead to ineffective treatment and / or injury to the patient. Some eye tracking systems rely on the reflection of light from the cornea captured in the image of the eye. Figure 3 shows, for example, an exemplary reflection 28. However, reflection patterns can provide an unreliable basis for eye tracking. As shown in Figure 9E, for example, an intraocular lens (IOL) implanted in the eye can produce an irregular reflection pattern. Furthermore, if the eye is kept open for an extended period by microscopy used to restrict blinking during treatment, the tear film on the cornea can be damaged. A dried and broken tear film creates reflective surfaces that create a shimmering pattern in the image, presenting further challenges to reflection-based tracking systems. Therefore, to achieve robust and accurate eye tracking, embodiments employ an approach that does not rely on the reflection of light from the cornea.

[0031] Figure 3 shows an exemplary pixelated image 20 of an eye captured by camera 252. In particular, image 20 includes features 22, 24, and 26. Image feature 22 corresponds to low-level anatomical structures in the iris region 3 formed by the pupillary sphincter muscle and pigmented fibrous interstitium. These anatomical structures appear as texture (shading) in image 20, especially when captured by a high-speed infrared camera. Image feature 24 corresponds to a dark, substantially circular or circular shape defined by the contrast between the iris region 3 and the pupillary region 4. Image feature 26 corresponds to a substantially circular boundary between the iris region 3 and the pupillary region 4. Controller 120 can detect image features 22, 24, and 26 and determine the changes in shape and position of image features 22, 24, and 26 over time in image 20.

[0032] The eye tracking system 250 may include a collective tracker 454 as shown in Figure 4. In particular, the collective tracker 454 (implemented by one or more controllers 120) uses trackers A, B, and C to process infrared images 20a captured by a high-speed infrared camera. All three trackers A, B, and C operate very quickly using local pixel information and can operate simultaneously in parallel. Estimates of eye movement between frames can be obtained very quickly (e.g., within a few milliseconds) from any of the trackers A, B, or C.

[0033] Tracker A is specifically tuned for image features 22 (i.e., the texture of the iris region). Tracker A may use a variation of the Lucas Kanade Tomasi (LKT) feature tracker to estimate the multiscale optical flow of a set of feature points within the iris region 3 (these are essential extremum points with high spatial frequencies). These feature points are detected at the start of the tracking and can be automatically replenished if lost due to changes in the scene.

[0034] Tracker B is specifically tuned to image feature 24 (i.e., the contrast between the iris and the pupil). Using the fact that the pupil appears darker than the iris in the infrared image 20a, tracker B can use optimization techniques to look for darker shapes formed by the set of pixels in the pupil region 4 surrounded by brighter colored pixels in the iris region 3. Tracker B can use robust statistics to ignore bright, luminous, and saturated pixels caused by reflection, for example, using Huber's M-estimators.

[0035] Tracker C is specifically tuned to image feature 26 (i.e., the iris-pupil boundary). Tracker C can detect a circular pupil-iris boundary by fitting a circular or elliptical model to a higher-scale boundary map obtained from the infrared image 20a.

[0036] As shown in Figure 4, the exemplary approach 400 uses a collective tracker 454 that combines data from trackers A, B, and C to provide a more robust, accurate, and efficient estimation of eye movements from infrared images 20a. For example, using collective mean motion based on information relevant to all three image features 22, 24, and 26, the collective tracker 454 can estimate pupillary parameters as an indicator of overall eye movement.

[0037] The aggregate tracker 454 is highly robust because trackers A, B, and C are designed to acquire information about orthogonal (non-overlapping) image features within image 20a and provide independent estimates of eye movements. These independent estimates are analyzed against each other using a more advanced level of meta-heuristic process to reduce error variance and obtain more accurate, smoother estimates of eye movements. Trackers A, B, and C may fail when tracking becomes difficult due to reflections and other obstructions, as shown in Figures 9A-9E. However, the advanced meta-heuristic process can compensate for the erroneous and missing information caused by such obstructions, allowing tracking to continue.

[0038] The aggregate tracker 454 efficiently models characteristic movements occurring in eye 1. Eye 1 movements range from high-speed ballistic motions, known as saccadic movements, to slow, smooth tracking movements used to track slow-moving objects. Eye 1 may also undergo vestibulo-ocular and optokinetic reflexes when the angle of line of sight is repositioned. Furthermore, eye 1 may undergo binocular transfer motion when an object moves in the depth direction (e.g., along the z-axis shown in Figure 1) while the object is maintained in the center of both eyes' visual fields. The aggregate tracker 454 is fast enough and robust to handle anatomical changes in eye shape, changes in iris reflectivity, and changes in pupil contrast under these typical movements.

[0039] The aggregate tracker 454 addresses errors in tracking that may be introduced by reflections of illumination present during the capture of the image 20. Advantageously, the exemplary approach 400 is independent of the illumination arrangement. For example, the exemplary approach 400 provides effective eye tracking regardless of whether the illumination is provided by a point source or an ambient / diffuse source, or whether the light source is on-axis or off-axis. In particular, the exemplary approach 400 can reject the retroreflection of the light source from the optical surfaces of the eye (i.e., the front and back surfaces of the cornea and the lens). Using a higher level of metaheuristics to combine the tracking data from the three independent trackers A, B, and C, the error variance of the estimated values related to movement is reduced, the accuracy of the tracker is improved, and the tracking speed can be maintained at super real-time (i.e., 60 Hz).

[0040] As described above, the trackers A, B, and C are designed to obtain information regarding the features of orthogonal images within the image 20a and provide an estimated value regarding the movement of the eye. The aggregate tracker 454 uses a higher level of metaheuristics to manage the trackers A, B, and C and combine the data from the trackers A, B, and C to generate a final estimated value regarding the movement of the eye (the net estimation error is minimized). FIGS. 5A and B show an exemplary method 500 of using a higher level of metaheuristics to generate a final estimated value regarding the movement of the eye. The movement of the eye is represented by changes in the parameters of the pupil of the eye (i.e., the center position of the pupil and the pupil radius). A time series of the infrared image 20a generates a frame F i to capture the movement of the eye. In the illustrated example, the pupil parameters of the previous frame F n-1 (shown as data 502) are known. The tracker A in operation 504 uses a multi-scale Lucas-Kanade tomasi (LKT) feature tracker to process the current frame F nThe optical flow of a set of feature points corresponding to the shading in the iris region is determined. The net pupillary motion correlates with the motion of these feature points and can be determined by combining the motion vectors of the individual feature points using Random Sample Consensus (RANSAC). In motion 506, the resulting motion of the iris (feature points), i.e., the net pupillary motion, is determined in the previous frame F n-1 Applied to the center position of the pupil in the current frame F n Generate a first estimate of 508 for the pupillary parameter.

[0041] Trackers B and C may be initialized with a first estimate 508 of the pupil center position as an initial estimate. In operation 510, tracker B generates a second estimate 512 of the pupil parameters by solving an optimization problem using gradient ascent. In particular, the contrast between the pixel intensity in the pupil region and the iris region is maximized to determine the unknown pupil center position and radius. Meanwhile, in operation 514, tracker C generates a third estimate 516 of the pupil parameters by fitting a circular pupil-iris boundary to a boundary map in order to determine the unknown pupil center position and radius.

[0042] In decision 518, the pupillary parameter estimates 508, 512, and 516 are evaluated to determine whether they are consistent with each other. A higher level of metaheuristic measures the deviations between the pupillary parameter estimates generated by trackers A, B, and C and ranks them based on their consistency. If the least consistent estimate deviates from the other two estimates by an amount greater than the empirical threshold, this one inconsistent estimate is considered wrong and rejected, and the remaining two consistent estimates are averaged in action 520, and frame F n This generates the final estimate of the pupillary parameter 524. If the least consistent estimate is within the empirical limits, estimates 508, 512, and 516 are considered consistent with each other and are then combined by averaging estimates 508, 512, and 516 in operation 522, resulting in frame Fn This generates the final estimate of the pupillary parameter. Figure 6 shows an exemplary frame with a consensus on the central pupillary position based on image features 22, 24, and 26 adopted by trackers A, B, and C, respectively, and the final estimate 524.

[0043] In summary, trackers A, B, and C are designed to utilize most of the useful information within an image. Each tracker targets a specific image feature that is mutually exclusive and non-overlapping with the features used by the other trackers. This approach acquires multiple measurements using the orthogonal parts of the information and robustly combines these measurements to reduce errors caused by an inappropriate measurement of any one feature. In this way, a failure of any one feature has no effect on the trackers at all, and the error variance is always reduced by averaging.

[0044] Figures 9A–F show various exemplary phenomena that may be captured in image 20a and generate aberrations, noise, distortion, interference, etc., which may affect the estimates 508, 512, and 516 generated by trackers A, B, and C. Such phenomena may induce errors and lead to inconsistencies between the estimates 508, 512, and 516 as described above. Specifically, Figure 9A shows image 90a capturing an irregularly shaped pupil or a change in pupil shape. Figure 9B shows image 90b capturing shadows and reflections from partial closure of the eye by a mask or eyelid worn by the patient for treatment. Figure 9C shows image 90c capturing eye droppers and a microscope used for treatment. Figure 9D shows image 90d capturing pupillary graying caused by an intraocular implant. Figure 9E shows image 90e capturing an irregular reflection pattern caused by an implanted IOL. Figure 9F shows image 90f, which captures an obstruction (e.g., a finger or eye dropper).

[0045] A more advanced level of metaheuristics, as shown in Figure 5, attempts to reduce errors by determining the amount of consistency between the estimates 508, 512, and 516 from trackers A, B, and C, respectively, and analyzing them against each other. Given frame F n If the error is small, the estimates 508, 512, and 516 have very small mutual deviations and are substantially identical, and therefore should provide an indicator of the actual pupil parameters (i.e., central position and radius). On the other hand, given frame F n If the error is not negligible, the estimates 508, 512, and 516 will be inconsistent with each other, revealing a deviation from the actual pupillary parameters. Trackers A, B, and C, by design, encode orthogonal information by measuring non-overlapping image features, thus inconsistencies can occur, and the same error may be shown differently by trackers A, B, and C.

[0046] As shown in Figure 7, the above eye tracking process is performed in frame F i As the time series is processed, a track 702 can be generated that progresses through various states. Figure 7 shows that when the eye-tracking process begins, no pupil is found in the images of the initial frames F1, F2, F3, and F4. Therefore, these initial frames are classified into state 704, designated as "track not found". Once a pupil is found in the image, track 702 is considered to be in state 706, designated as "initial track", until certain criteria described below are met. Frames F5, F6, and F7 correspond to the period when track 702 is in the "initial track" state. Once the above criteria are met, track 702 is considered to be in state 708, designated as "mature track". Frame F8 and subsequent frames correspond to the period when track 702 is in the "mature track" state.

[0047] When track 702 is in the “initial track” state, the processing of corresponding frames is slower and more comprehensive to ensure that track 702 is truly established and to avoid false starts. In particular, the entire image of each frame is independently searched (global search) to obtain individual estimates of the pupil parameter. Thus, estimates of consecutive frames can be analyzed against each other to verify the temporal consistency between frames in the initial stages of track 702. Frames undergo this slower and more comprehensive processing until temporal consistency is established for an empirical threshold number (N) of frames. Once this temporal consistency is established, track 702 enters the “mature track” state. If the consistency check fails at any point, the above process is restarted, and track 702 cannot be considered in the “mature track” state until it is found that N consecutive frames are consistent.

[0048] Once track 702 enters the “mature track” state, the frame may be processed according to the exemplary method 500. As described above, the previous frame F n-1 The pupil estimate of 502 is an initial estimate for a relatively narrow search for pupil parameters by trackers A, B, and C, for the current frame F n Used in the processing of the "Maturity Track" state. n-1 The pupil parameter in the current frame F n Since it is used as an initial estimate, temporal consistency is not checked. Because temporal consistency is not checked and the search is local, the tracking process in this state is faster.

[0049] Figure 8 shows a series of frames F when the track is in the "mature track" state. i A general approach 800 for processing is shown. Specifically, the pupillary parameter 502 is determined based on a consensus made by applying a more advanced level of metaheuristics to trackers A, B, and C, for frame F n-1This is determined. As mentioned above, the pupil parameter 524 is determined for frame F n-1 Starting with pupil parameter 502, frame F is based on another consensus created by applying a more advanced level of metaheuristics to trackers A, B, and C. n This will be determined regarding Frame F. n The processing applied to the pupil parameter is repeated for subsequent frames. For example, the pupil parameter is applied to frame F n Starting with pupil parameter 524, and based on yet another consensus created by applying a more advanced level of metaheuristics to trackers A, B, and C, frame F n+1 This will be determined. Approach 800 is robust and fast enough to process frames at a speed of 60 Hz.

[0050] As described above, according to some aspects of this disclosure, some or all of the steps of the above-described and illustrated procedures may be automated or guided under the control of a controller (e.g., controller 120). Generally, the controller may be implemented as a combination of hardware and software elements. The hardware aspects may include a combination of operable-coupled hardware components, such as a microprocessor, logic circuits, communication / network ports, digital filters, memory, or logic circuits. The controller may be adapted to perform operations specified by computer-executable code (which may be stored in a computer-readable medium).

[0051] As described above, the controller may be a programmable processing device that executes software or stored program instructions (e.g., an external conventional computer, or an onboard field-programmable gate array (FPGA), or digital signal processor (DSP)). Generally, the physical processor and / or machine used by embodiments of the present disclosure for any processing or evaluation may include, as will be understood by those skilled in computer and software technology, one or more networked or unnetworked general-purpose computer systems, microprocessors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), microcontrollers, etc., programmed in accordance with the teachings of the exemplary embodiments of the present disclosure. The physical processor and / or machine may be networked externally with the image capture device, or may be integrated to reside within the image capture device. As will be understood by those skilled in software technology, appropriate software can be readily prepared by a programmer of ordinary skill based on the teachings of the exemplary embodiments. Furthermore, the devices and subsystems of the exemplary embodiments may be implemented by preparing application-specific integrated circuits or by interconnecting multiple conventional component circuits in a suitable network, as will be understood by those skilled in electrical technology. Thus, the exemplary embodiments are not limited to any particular combination of hardware circuits and / or software.

[0052] The exemplary embodiments of this disclosure may include software or stored program instructions stored on any one or combination of computer-readable media for controlling the devices and subsystems of the exemplary embodiments, for driving the devices and subsystems of the exemplary embodiments, and for enabling the devices and subsystems of the exemplary embodiments to interact with a human user or the like. Such software may include, but is not limited to, device drivers, firmware, operating systems, development tools, and application software. Such computer-readable media may further include computer program products of the embodiments of this disclosure for performing all or part of the processing performed in the implementation (if the processing is distributed). The computer code devices of the exemplary embodiments of this disclosure may include, but are not limited to, any suitable interpretable or executable code mechanism, including, but not limited to, scripts, interpretable programs, dynamic link libraries (DLLs), Java classes and applets, and complete executable programs. Furthermore, some of the processing of the exemplary embodiments of this disclosure may be distributed for better performance, reliability, cost, etc.

[0053] Common forms of computer-readable media may include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, and other suitable magnetic media; CD-ROMs, CDRWs, DVDs, and other suitable optical media; punch cards, paper tapes, optical mark sheets, other suitable physical media with hole patterns or other optically recognizable markings; RAM, PROMs, EPROMs, FLASH®-EPROMs, and other suitable memory chips or cartridges; computer-readable carrier waves or other suitable media.

[0054] While this disclosure has been described with reference to one or more specific embodiments, those skilled in the art will recognize that many modifications can be made without departing from the spirit and scope of this disclosure. Each of these embodiments and their obvious variations is considered to fall within the spirit and scope of this disclosure. Additional embodiments according to aspects of this disclosure are also envisioned to be able to combine any number of features of any of the embodiments described herein.

Claims

1. A system for tracking eye movements during eye treatment, An image capture device configured to capture multiple images of the eye, and One or more controllers including one or more processors configured to execute program instructions stored in one or more computer-readable media, Equipped with, The program instruction is given to one or more processors: The image capture device receives the plurality of images, Multiple trackers are implemented, each tracker is configured to detect a feature in the multiple images and, based on that feature, provide a dataset related to the eye movements, the features detected by the multiple trackers are orthogonal to each other, and the datasets provided by the multiple trackers are independent of each other. The datasets from the aforementioned multiple trackers are combined, and Based on the combined dataset, the indicators of eye movement are determined. The above system.

2. The system according to claim 1, wherein each dataset provided by each tracker shows the changes in shape and / or position of each feature over time in a series of frames corresponding to the plurality of images.

3. The system according to claim 1, wherein the indicator of eye movement indicates the movement of the pupil of the eye.

4. The system according to claim 1, wherein the plurality of images are pixelated, and each tracker detects the respective features based on local pixel information.

5. The system according to claim 1, wherein the image capture device includes a high-speed infrared camera, and the plurality of images are infrared images.

6. The system according to claim 1, wherein when combining the datasets from the plurality of trackers, the program instruction causes one or more processors to analyze the datasets from the plurality of trackers with respect to each other, identify inconsistencies between the datasets, and correct the inconsistencies in the datasets.

7. The system according to claim 6, wherein the inconsistency is caused at least by reflections of lighting captured by the plurality of images and / or by obstacles.

8. The aforementioned plurality of trackers A first tracker is configured to detect a first feature, including the anatomical structure of the iris region of the eye, in the aforementioned plurality of images, and to provide a first dataset related to the movement of the eye based on the first feature. A second tracker configured to detect a second feature in the plurality of images, including a shape defined by the contrast between the iris region and the pupil region of the eye, and to provide a second dataset related to the movement of the eye based on the second feature, and A third tracker is configured to detect a third feature in the plurality of images, including the boundary between the iris region and the pupil region of the eye, and to provide a third dataset related to the movement of the eye based on the third feature. The system according to claim 1, including the following:

9. The system according to claim 1, wherein the program instruction causes one or more processors to identify the state of temporal consistency with respect to a threshold for the number of frames in a time series of frames corresponding to the plurality of images.

10. In order to determine the index of the eye movement, the program instruction instructs one or more processors to (i) frame F ｎ (ii) the consensus from the combined dataset from the multiple trackers and the previous frame F ｎ－１ Based on the eye position determined for the frame F ｎ The system according to claim 1, wherein the time series of frames corresponding to the plurality of images is processed by repeatedly determining the position of the eye in the plurality of images.

11. A method for tracking eye movements during eye treatment, Acquiring multiple images of the eye using an image capture device, and The present invention involves implementing multiple trackers using one or more processors, each tracker being configured to detect a feature in the plurality of images and to provide a dataset related to the eye movements based on that feature, wherein the features detected by the plurality of trackers are orthogonal to each other, and the datasets provided by the plurality of trackers are independent of each other. Using the one or more processors mentioned above, the data sets from the multiple trackers are combined, and Using the one or more processors mentioned above, an index of eye movement is determined based on the combined dataset. to include, The above method.

12. The method according to claim 11, wherein each dataset provided by each tracker shows the changes in shape and / or position of each feature over time in a series of frames corresponding to the plurality of images.

13. The method according to claim 11, wherein the indicator of the eye movement indicates the movement of the pupil of the eye.

14. The method according to claim 11, wherein the plurality of images are pixelated, and each tracker detects the respective features based on local pixel information.

15. The method according to claim 11, wherein the image capture device includes a high-speed infrared camera, and the plurality of images are infrared images.

16. The method according to claim 11, wherein the merging of the datasets from the plurality of trackers includes analyzing the datasets from the plurality of trackers against each other in order to identify and correct inconsistencies between the datasets.

17. The method according to claim 16, wherein the inconsistency is caused at least by reflections of lighting captured by the plurality of images and / or by obstacles.

18. The aforementioned plurality of trackers A first tracker is configured to detect a first feature, including the anatomical structure of the iris region of the eye, in the plurality of images, and to provide a first dataset related to the movement of the eye based on the first feature. A second tracker configured to detect a second feature in the plurality of images, including a shape defined by the contrast between the iris region and the pupil region of the eye, and to provide a second dataset related to the movement of the eye based on the second feature, and A third tracker configured to detect a third feature in the plurality of images, including the boundary between the iris region and the pupil region of the eye, and to provide a third dataset related to the movement of the eye based on the third feature, The method according to claim 11, including the method described in claim 11.

19. The method according to claim 11, further comprising identifying the state of temporal consistency with respect to a threshold for the number of frames in a time series of frames corresponding to the plurality of images.

20. Determining the aforementioned index of eye movement is (i) frame F ｎ (ii) the consensus from the combined dataset from the multiple trackers for (ii) the previous frame F ｎ－１ Based on the eye position determined for the frame F ｎ The method according to claim 11, comprising processing a time series of frames corresponding to the plurality of images by repeatedly determining the position of the eye in the plurality of images.

Images