Patient guiding trigger of measurement for ophthalmologic diagnostic device

Abstract

To provide a patient guiding trigger of measurement for an ophthalmic diagnostic device.SOLUTION: A system and method for tracking a position and state of an eye during an ophthalmic procedure include: an ophthalmic device configured to measure characteristics of the eye; an eye tracker configured to capture a stream of images of the eye; and a logic device configured to analyze the stream of images to determine whether or not the eye is fixating on a target object, detect a predetermined blink sequence in a first stream of images, delay a predetermined tear stabilization period, initiate a stable tear film interval, and capture at least one measurement result of the eye with the ophthalmic device when the eye is fixating during the stable tear film interval. The blink sequence may include a plurality of continuous blinks, and detection of the blink sequence may include processing of the image through a neural network trained to detect open and / or closed eyes.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates generally to ophthalmic diagnostic systems and methods, and more particularly to systems and methods for tracking the position, orientation and / or state of the eye, for example in imaging, diagnostic and / or surgical systems. [Background technology]

[0002] A wide variety of ophthalmic devices are used to image, measure, diagnose, track, surgically correct, and / or surgically repair a patient's eye. The operation of an ophthalmic device, such as a topography device, keratometry device, wavefront analyzer, or another device that measures an aspect of the eye (e.g., optical, geometric, etc.), is often based on the assumption that the eye is maintained in a defined position and orientation relative to the diagnostic device. The patient may be positioned by a human operator of the ophthalmic device and instructed, for example, to look into the device and view a target object (e.g., a fixation light) and align the patient's line of sight (e.g., the axis along which one looks) with the optical axis of the ophthalmic device. If the patient does not fixate properly, readings may be inaccurate and / or the system may not function properly.

[0003] To ensure accurate data acquisition, a human operator of an ophthalmic device is often tasked with monitoring the patient, explaining the initialization procedure to the patient, and / or monitoring feedback from the device during data acquisition to determine whether the patient is properly fixating on the target object to align their eyes. One known technique involves relying on the patient's cooperation in fixating on the target object as instructed by the device operator. However, existing approaches have many drawbacks, including human error of the patient attempting fixation (e.g., elderly patients may not be able to maintain eye position, the patient may lack sufficient concentration to fixate, the patient may not be able to look directly at the target object, etc.) and human error and variability due to the operator monitoring the patient during the procedure. In another approach, retinal scanning and imaging analysis may be used to track the position and orientation of the patient's eyes, but the operation of the retinal imaging system may interfere with the diagnostic procedure. As a result, retinal scanning and imaging systems are often shut down or disabled for use in eye tracking during diagnostic procedures performed using an ophthalmic device.

[0004] Other drawbacks of conventional systems include that the patient may not know when the measurement has begun and may lose fixation, blink, or move in other ways that affect the reliability of the measurement. The patient may be asked to fixate for an extended period of time, which may be uncomfortable for the patient and may lead to a suboptimal state of the eye. The operator may also be tasked with determining the optimal state of the eye to perform the measurement. For example, the eye may become dry over the course of the measurement procedure, and moisture may return each time the patient blinks, which may lead to constant changes in the eye's reflectance. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the foregoing, there is a continuing need in the art for improved techniques for determining and / or tracking the position, orientation and state of a patient's eye during an ophthalmic procedure. [Means for solving the problem]

[0006] FIELD OF THE DISCLOSURE The present disclosure relates generally to systems and methods involving patient control of ophthalmic diagnostic data acquisition. The systems and methods provided herein can be used to determine the optimal time for preparing an eye for a measurement.

[0007] In one or more embodiments, the system includes an ophthalmic device configured to measure a characteristic of the eye, an eye tracker configured to capture a first stream of images of the eye, and a logic device configured to analyze the first stream of images to determine whether the eye is fixating a target object, detect a predetermined blink sequence in the first stream of images, initiate a stable tear film interval after a predetermined tear stabilization period, and capture at least one measurement of the eye with the ophthalmic device while the eye is fixating the target object during the stable tear film interval. The blink sequence may include multiple consecutive blinks, and detecting the blink sequence may include processing the images through a neural network trained to detect open and / or closed eyes.

[0008] In some embodiments, the eye tracker is configured to capture a first image of the eye from a first position and a second image of the eye from a second position, and the logic device is further configured to detect a first plurality of eye characteristics from the first image, the eye characteristics having first corresponding image coordinates, detect a second plurality of eye characteristics from the second image, the eye characteristics having second corresponding image coordinates, and determine a calibration offset and a calibration gain based at least in part on the first corresponding image coordinates, the second corresponding image coordinates, the first position, and the second position. The logic device may be further configured to determine an eye fixation position and orientation relative to an optical axis of the eye tracker based at least in part on the first corresponding image coordinates and / or the second corresponding image coordinates.

[0009] In some embodiments, the logic device is configured to estimate eye fixation parameters based at least in part on the determined eye fixation position and orientation, and track a current eye position and orientation by receiving a first stream of images from the eye tracker and analyzing at least one image from the first stream of images to determine the current eye position and orientation relative to the eye fixation parameters, wherein the eye fixation parameters include a reference position and orientation of the eye at the time of fixation.

[0010] The logic device may be further configured to determine a fixation position relative to an optical axis of the eye tracker by constructing and analyzing a histogram of the detected eye positions and orientations, wherein analyzing the histogram further includes determining whether a coordinate of the relative maximum value includes the fixation position and orientation, and wherein determining whether the coordinate of the relative maximum value includes the fixation position and orientation further includes comparing the relative maximum value to a threshold and / or an average coordinate value of the histogram.

[0011] In some embodiments, the system further comprises a retinal imaging system including an optical coherence tomography (OCT) scanner configured to perform a retinal scan, wherein the eye tracker is further configured to capture a stream of images of the eye during the retinal scan, wherein the retinal imaging system is further configured to capture multiple retinal images of the eye, detect whether a fovea is present in one or more of the multiple retinal images of the eye, and identify a first retinal image from the multiple retinal images of the eye having the detected fovea, and wherein the logic device is further configured to determine a corresponding image from the stream of images having temporal proximity to the first retinal image, and analyze the corresponding image to determine eye fixation parameters.

[0012] In some embodiments, the logic device is configured to track the position and orientation of the eye, calculate an offset from the eye fixation parameter, and determine whether the offset is less than a threshold, when the offset is less than the threshold the eye is determined to be fixating and the control processor generates an indication of fixation, and when the offset is greater than the threshold the eye is determined to be out of alignment and the control processor generates an indication of not fixating.

[0013] The logic device may be further configured to perform an ocular diagnostic procedure and track the position of the eye with an eye tracker during the ocular diagnostic procedure. The system may further include a diagnostic device configured to perform the ocular diagnostic procedure while tracking the position and orientation of the eye with the eye tracker, the diagnostic device configured to modify the ocular diagnostic procedure based at least in part on data representing the ocular fixation parameters and the tracked eye position.

[0014] In various embodiments, a method includes capturing a first stream of images of an eye using an eye tracker, analyzing the first stream of images to determine whether the eye is fixating a target object, detecting a predetermined blink sequence in the first stream of images, tracking a stable tear film interval after a predetermined tear stabilization period, and capturing at least one measurement of the eye with an ophthalmic device while the eye is fixating the target object during the stable tear film interval. The blink sequence may include multiple consecutive blinks, and detecting the predetermined blink sequence in the first stream of images may include processing the images through a neural network trained to detect open and / or closed eyes.

[0015] The method may further include capturing a first image of the eye from a first position, capturing a second image of the eye from a second position different from the first position, detecting a first plurality of eye characteristics from the first image, the eye characteristics having first corresponding image coordinates, detecting a second plurality of eye characteristics from the second image, the eye characteristics having second corresponding image coordinates, and determining a calibration offset and a calibration gain based at least in part on the first corresponding image coordinates, the second corresponding image coordinates, the first position, and the second position.

[0016] The method may further include capturing a stream of images of the eye, detecting an eye position and orientation in the stream of images based at least in part on coordinates of the detected eye characteristic, a calibration offset, and a calibration gain, and determining an eye fixation position and orientation relative to an optical axis. The method may further include estimating an eye fixation parameter based at least in part on the determined eye fixation position and orientation, and tracking the eye position and orientation by analyzing one or more images from the stream of images to determine the eye position and orientation relative to the eye fixation parameter, the eye fixation parameter including a reference position and orientation of the eye at the time of fixation. The method may further include training a neural network to receive the stream of images and output a determination of the eye position and / or tear film state.

[0017] In some embodiments, the method further includes detecting the fixation position relative to the optical axis by constructing and analyzing a histogram of the detected eye positions and orientations, wherein analyzing the histogram further includes determining a relative maximum value.

[0018] The method may further include performing a retinal imaging scan of the eye using an optical coherence tomography (OCT) scanner, capturing multiple retinal images of the eye from the retinal imaging scan, capturing a stream of images using an imaging device configured to image the surface of the eye, detecting whether a fovea is present in one or more of the multiple retinal images, identifying a first retinal image from the multiple retinal images having the detected fovea, determining corresponding images from the stream of images having temporal proximity to the first retinal image, and analyzing the corresponding images to determine eye fixation parameters.

[0019] In some embodiments, the method may further include tracking the position and orientation of the eye and calculating an offset from the eye fixation parameter and determining whether the offset is less than a threshold, where when the offset is less than the threshold the eye is determined to be fixating and the control processor generates an indication of fixation, and when the offset is greater than the threshold the eye is determined to be out of alignment and the control processor generates an indication of not fixating.

[0020] The method may further include performing an ocular diagnostic procedure and tracking the position and orientation of the eye using the image capture device, and modifying the ocular diagnostic procedure based at least in part on data representing the ocular fixation parameters and the tracked eye position.

[0021] The scope of the present disclosure is defined by the claims, which are incorporated into this section by reference. A more detailed understanding, together with a realization of additional advantages thereof, will be afforded to those skilled in the art by consideration of the following detailed description of one or more embodiments. Reference will first be made to the accompanying drawings, which will be briefly described.

[0022] Aspects of the present disclosure and their advantages may be better understood with reference to the following drawings and detailed description below. Like reference numerals are used to identify like elements shown in one or more of the drawings, and it should be understood that what is shown in the drawings is for the purpose of illustrating, and not limiting, embodiments of the present disclosure. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. [Brief explanation of the drawings]

[0023] [Figure 1A-1B] 1A-1B illustrate an example eye tracking and imaging system in accordance with one or more embodiments of the present disclosure. [Figure 2] FIG. 2 illustrates an example eye tracking and imaging system with automatic initialization and calibration, in accordance with one or more embodiments of the present disclosure. [Figure 3] FIG. 3 illustrates an example eye tracking and imaging system in accordance with one or more embodiments of the present disclosure. [Figure 4] FIG. 4 illustrates an example neural network, according to one or more embodiments of the present disclosure. [Figure 5] FIG. 5 illustrates an example computing system in accordance with one or more embodiments of the present disclosure. [Figure 6A] FIG. 6A illustrates an example operation of an automatic initialization and calibration system in accordance with one or more embodiments of the present disclosure. [Figure 6B] FIG. 6B illustrates an example operation of an eye tracker system in accordance with one or more embodiments of the present disclosure. [Figure 7] FIG. 7 illustrates a method for estimating absolute eye position according to one or more embodiments of the present disclosure. [Figure 8] FIG. 8 shows an example heat map of eye position and orientation detected using an eye tracker, in accordance with one or more embodiments of the present disclosure. [Figure 9]FIG. 9 shows an example histogram constructed from eye position and orientation data detected using an eye tracker, in accordance with one or more embodiments of the present disclosure. [Figure 10] FIG. 10 illustrates an example system for implementing the method of FIG. 7 in accordance with one or more embodiments of the present disclosure. [Figure 11] FIG. 11 illustrates an example measurement process according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] The present disclosure provides systems and methods for tracking the position, orientation, and / or state of an eye during an ophthalmic procedure.

[0025] To obtain high-quality diagnostic data for ophthalmic diagnoses, the eye should be in a well-defined position and condition during measurements. For example, for many ophthalmic devices, measurement sequences are performed when the patient's eye is fixating along the optical axis of the target device (e.g., along an axis and / or offset within an acceptable error range for the measurement) and the patient's eye has an intact tear film. In various embodiments, improved systems and methods include automated calibration of eye-tracking devices, accurate eye position and fixation determination, improved eye-tracking procedures, determination of absolute and estimated absolute fixation positions, and improved timing of measurement data acquisition based on fixation and / or tear film conditions.

[0026] An intact tear film is often essential for reflectance-based diagnostic devices, such as keratometers or topographers, which operate using reflection from the corneal surface. In many applications, dry areas of the cornea do not allow for optimal reflectance-based measurements. The tear film is also a refractive surface that can be used by certain diagnostic devices, such as wavefront measurement devices. Once the eyelids distribute the tears over the eye, the tear film can rehydrate each time the patient blinks. The tear film stabilizes after t1 seconds (e.g., 0.5–2 seconds), remains intact for t2 seconds (e.g., 1–3 seconds), and then dries until the next blink. These periods (t1 and t2) may vary from patient to patient and may be estimated for a patient pool, for example, through clinical studies.

[0027] In addition to an intact tear film, a stable fixation of the eye ensures that the patient's visual axis is aligned with the optical axis of the diagnostic device. If the patient does not fixate or fixates poorly, the results may include inaccurate measurements, unreliable measurements, or the device's inability to perform measurements. It may be difficult for a patient to fixate on a stable fixation target for an extended period of time. The length of time a patient can fixate adequately also varies from patient to patient. In various embodiments disclosed herein, systems and methods track the position and orientation of the eye (e.g., whether the eye is properly fixating), the state of the eye (e.g., whether the tear film is intact), and identify time intervals during which accurate and reliable measurements can be made.

[0028] In some ophthalmic systems, the quality of acquired data may depend on the skill and awareness of the operator. In these systems, the operator may determine when the eye's position, orientation, and condition are appropriate for measurement, resulting in variability in measurements performed by different operators. In systems using automated measurements, measurements may occur regardless of whether the patient is ready for measurement, which may lead to acquisitions when the patient is not fixating and / or when the tear film is not stable (e.g., outside of t2). The patient may also be required to fixate for long measurement sequences without knowing the exact point at which the measurement will begin. For example, in one approach, the operator may position the patient to align their eye with the optical axis of the diagnostic system. The patient may be instructed to fixate a known target point to align the patient's gaze until the operator and device are ready to measure.

[0029] After determining that the patient is fixating, the operator may instruct the patient to blink to establish a tear film. The patient then attempts to maintain fixation on the target point throughout the procedure. The operator and / or device may then capture measurements of the patient's eye. However, measurements may be captured before the tear film stabilizes (e.g., during time period t1), during a period when the retina is stable (e.g., during time period t2), or after the tear film has begun to deteriorate. In this manner, captured measurements may occur during an unknown tear film state, leading to unreliable measurements. The improvements disclosed herein enable data acquisition once the tear film stabilizes, thereby improving diagnostic accuracy. In some embodiments, the improved system may function independently of the operator, reducing the amount of time the patient needs to fixate.

[0030] The systems and methods disclosed herein further include improved initialization and calibration of ophthalmic systems relative to a patient's own eye, improved eye tracking, improved determination of absolute fixation position and orientation, and other improvements and advantages over conventional systems. The improved initialization and calibration techniques disclosed herein enable more accurate measurements of a patient's eye and can be used in diagnostic systems to determine whether the patient's line of sight (also referred to herein as the patient's visual axis) is aligned with the optical axis of the diagnostic system. The patient's line of sight / visual axis can be the axis along which the patient's eye is oriented to view an object. The systems and methods disclosed herein enable simpler, shorter, and more accurate system initialization and calibration and more accurate fixation determination. Diagnostic data obtained according to the systems and methods disclosed herein is more meaningful and accurate than data obtained through conventional techniques. If the patient does not fixate properly during measurement and this is not taken into account, the accuracy of the readings can be significantly compromised. In many embodiments, the accuracy with which a person can fixate (actively control their gaze on a stationary target) may be on the order of one degree, but may be significantly worse depending on the condition of the eye (e.g., severe cataracts). The systems and methods disclosed herein improve accuracy by determining and using the gaze profile of the patient's eye during measurement. The use of the patient's gaze profile can remove measurement noise from the readings caused by gaze movement and the inability to fixate steadily.

[0031] 1A and 1B, an exemplary eye tracking system for use with an ophthalmic device is now described in accordance with one or more embodiments. One method for tracking gaze is by analyzing camera images and comparing the position of the pupil in the image to the position of the corneal reflection generated from an illumination source that is fixed in space relative to the observation camera. The system shown in FIGS. 1A and 1B includes a calibration procedure in which the patient is instructed to fixate on a known fixation point, allowing the system to calibrate the details of the observed eye. As shown, eye tracking system 100 includes an image capture component (e.g., a visible light camera) and an illumination source 112A (e.g., one or more light-emitting diodes (LEDs)) in a known, fixed position relative to the image capture component. Eye tracking system 100 is configured to image and track eye 102A by capturing and analyzing a stream of images of eye 102A, such as exemplary camera image 120A.

[0032] The camera image 120A is analyzed to identify one or more characteristics of the eye 102A, such as an image of the cornea 122A, the pupil 124A, and the reflection 126A of the illumination source 112A. By identifying one or more eye characteristics in the camera image 120A, information about the position and orientation of the eye 102A, such as the gaze azimuth angle GA and the gaze elevation angle GE, can be determined. The camera image 120A can be analyzed to determine coordinates of the alignment and / or offset position of the eye during the procedure. For example, the camera image 120A can be analyzed to determine the image coordinates [CRx, CRy] of the corneal reflection 126A (CR) of the illumination and / or the image coordinates [PCx, PCy] of the pupil 124A (e.g., the center of the pupil PC).

[0033] The image coordinate difference between the corneal reflection CR and the pupil center PC can be calculated as follows: Dx=CRx-PCx Dy = CRy - PCy These image coordinate differences are proportional to the azimuth angle (GA) and elevation angle (GE) of the line of sight. Dx~GA Dy~GE

[0034] To more accurately derive the viewing azimuth angle GA and the viewing elevation angle GE from Dx and Dy, an offset (ax, ay) and a gain (bx, by) can be applied to each image coordinate x and y, respectively. GA=ax+bx*Dx GE=ay+by*Dy

[0035] The variables a and b may depend on a variety of factors, including, for example, the particular ocular anatomy being imaged, the camera settings, and the camera's illumination source and optics. In some embodiments, determining a and b may include an initialization procedure in which the tracked patient is asked to fixate a set of targets (e.g., a grid of fixation points) that stimulate a defined gaze on the eye. For example, FIG. 1A illustrates a scenario in which the patient is asked to focus on a first known fixation point, such as a point near or aligned with the optical axis of the eye tracking system 100. FIG. 1B illustrates a scenario in which the eye 102B is observing a second known fixation point, such as a point next to the camera. The camera image 120B may include images of the cornea 122B, pupil 124B, and reflection 126B of the illumination source 112A. Because the camera position and orientation, eye position and orientation, and fixation point position are known during fixation, the gaze azimuth angle GA and gaze elevation angle GE are known or can be estimated for each fixation point. The two camera images 120A and 120B are each analyzed to determine the x and y coordinates of one or more eye characteristics in each image (e.g., the center of the pupil in the image, the location of the reflection in the image). A simultaneous equation can then be solved for a and b to initialize and calibrate the system for eye tracking.

[0036] 1A and 1B may be laborious to implement and / or prone to error for some patients. The patient may be instructed to fixate, for example, a grid of five or more fixation points separately to calculate the values ​​of a and b by statistical or other mathematical analysis (e.g., least-squares analysis). Asking a patient to fixate several targets requires significant patient cooperation, and the patient's gaze directed at any one point is subject to error.

[0037] Further embodiments of the present disclosure will now be described with reference to FIG. 2, which illustrates an eye-tracking system 200 including automated initialization and calibration components and procedures that enable accurate gaze tracking in diagnostic systems for keratometry, corneal topography, aberrometry, and other uses. While the system and method illustrated in FIG. 2 may be fully automated and reduce / eliminate the need for patients to run through time-consuming initialization procedures, various aspects may be used with manual and / or other automated eye-tracking initialization and calibration procedures, including procedures that involve an operator guiding the patient to fixate on a series of known fixation points.

[0038] The eye tracking system 200 can be implemented in any device that uses accurate eye fixation. For example, many ophthalmic devices, such as keratometers, topographers, and aberrometers, rely on accurate eye fixation during diagnostic procedures. Having accurate information about the actual gaze during acquisition of diagnostic data can enable more accurate comparison of diagnostic readings (e.g., corneal topography maps) taken at different times and taking into account differences in gaze when comparing readings, such as filtering out readings with poor fixation, compensating for readings with poor fixation by taking into account the actual gaze direction, and / or more accurate comparison of diagnostic readings (e.g., corneal topography maps) taken at different times and taking into account differences in gaze when comparing readings.

[0039] The eye tracking system 200 includes a first image capture device 201 having a first illumination source 202 and a second image capture device 210 having a second illumination source 212. The first image capture device 201 and the first illumination source 202 may be configured, for example, as a single camera eye tracker adapted to capture images of a patient's eye 220. The first image capture device 201 may include a visible spectrum image capture component configured to capture images of the surface of the patient's eye 220 along the optical axis of the ophthalmic device. The second image capture device 210 may include a visible spectrum image capture component configured to capture images of the surface of the patient's eye 220 from a known angle α (e.g., 20 degrees from the first image capture device 210). In some embodiments, the second image capture device 210 is the same type of imaging device as the first image capture device 210 (e.g., composed of the same similar components, the same device model number, etc.) and is positioned at approximately the same distance from the eye 220 to produce a second camera image 214 having similar image characteristics as the first camera image 204.

[0040] The processing system 230 controls the operation of the eye tracking system 200 and may include a control component 232, an image capture processing component 234, an eye tracking component 236, and a system initialization component 238. The processing system 230 may include one or more systems or devices implemented through a combination of hardware, firmware, and / or software. In some embodiments, the control component 232 is configured to manage the operation of the first image capture device 201 and the second image capture device 210, including providing instructions to synchronize the image capture operation of the image capture devices 201 and 210. Depending on the system configuration, the first image capture device 201 and the second image capture device 210 may be instructed to capture images simultaneously and / or sequentially with a short interval between images (e.g., timed to capture two images of the eye 220 at the same position). The image processing component 234 is configured to analyze the captured images to determine one or more eye characteristics, such as the center of the pupil, the position of the cornea, and / or the position of a reflection within the image. The eye tracking component 236 is configured to track the position of the eye based on calibrated measurements of eye characteristics identified in one or more images.

[0041] The system initialization component 238 is configured to initialize the measurement equations to accurately calculate the gaze azimuth angle GA and gaze elevation angle GE from the captured image data. In various embodiments, the patient is instructed to fixate a known fixation point that approximates the optical axis of the diagnostic device. The operator may interact with the eye tracking system 200 using a user interface to initiate the eye tracking procedure and guide the user through the initialization process. In some embodiments, images are captured from each of the image capture devices 201 and 210 while the patient fixates on the known point. The first camera image 204 captured by the first image capture device 201 and the second camera image 214 captured by the second image capture device 210 are used in the system initialization routine. Eye fixation may be determined, for example, based on the operator's judgment, through image analysis of the position of the reflection relative to the center of the pupil using a retinal imaging system that detects the fovea, through statistical analysis of multiple images captured over time, and / or through other techniques.

[0042] The two images 204 and 214 are processed through image processing component 234 to determine eye characteristics for each image. The two sets of eye characteristics represent two different measurements taken while the eye 220 was fixating a known fixation point. The two sets of equations may then be used to solve for the calibration offset a and gain b, which are used to determine the gaze azimuth angle GA and gaze elevation angle GE from the image data. By using a second camera to image the eye from a second angle, two images and measurements of the eye may be taken relative to a single fixation point, thereby allowing the offset and gain to be determined without the tedious multiple fixation point initialization procedure. In other embodiments, one or more additional cameras may be provided at other angles and / or more than one fixation point may be used as needed to further minimize errors.

[0043] The calibration offsets and gains may be used in the process of determining the position and orientation of the eye based on the captured images. In some embodiments, the calibration offsets and gains may be immediately available for use by the eye tracking system 200. In some embodiments, the calibration offsets and gains are stored in a storage device 242 (e.g., random access memory, a hard drive, flash memory, cloud storage, etc.) in a database or in a lookup table 244. For example, the lookup table may store calibration offset and gain values ​​associated with a patient identifier and the patient's eye. Other information, such as camera type, camera position, and measurement date, may also be stored. During operation, the eye tracking system 200 may use the lookup table to determine the absolute orientation of the eye from pixel locations of the pupil center and corneal reflection measured from images acquired by the eye tracker 200.

[0044] Processing system 230 may also include a tear film status component 240 configured to analyze the captured images to detect eye-open and eye-close events and track the status of the tear film, including whether the patient has blinked recently, whether the tear film is stable for measurement, and / or whether the eye is dry and needs to be restored. The use of tear film status, along with the calibration and initialization processes and absolute eye position determination and fixation tracking disclosed herein, enables more accurate eye diagnosis.

[0045] Various example embodiments of the present disclosure will now be described in further detail with reference to FIGS. 3-11 . Referring to FIG. 3 , a system 300 according to one or more embodiments includes an eye tracking module 310 (also referred to herein as an “eye tracker”) and an optional retinal imaging system 330, which are communicatively coupled. The eye tracking module 310 is configured to track the orientation of the eye 302 and may include a first imaging device 312, a second imaging device 313, and one or more illumination components 314. In some embodiments, the first imaging device 312 and the second imaging device 313 are digital cameras or other digital imaging devices configured to image certain features of the eye, such as the pupil and limbus (the boundary between the cornea and the white of the eye, i.e., the sclera), as well as reflections from one or more of the illumination components 314. In some embodiments, for example, the illumination component 314 may include a light-emitting diode (LED) ring positioned around the camera optics (e.g., coaxial illumination light around the imaging device) such that the center of the ring appears to be the center of the corneal curvature.

[0046] The system 300 includes control logic 318, which may include a logic device such as a processor that executes stored program instructions configured to perform the functions disclosed herein. In some embodiments, the control logic 318 performs a measurement sequence using multiple images captured by the first imaging device 312. The measurement sequence determines the position and orientation of the eye 302 by using the positions of detectable features of the eye 302 in the image data (e.g., eye tracking data 316), such as the pupil, limbus, and iris features. The measurement sequence may also determine the position of a reflection of the illumination system at the cornea (e.g., reflection 317, which includes a circular pattern of illuminated elements). In some embodiments, during the measurement sequence, the position and orientation of the eye 302 are continuously determined using the captured images. The control logic 318 may also perform an initialization and calibration component 318A (e.g., the sequence described with reference to FIG. 2 ), which includes calculating calibration offsets and gains for the patient's eye 302 from pairs of images captured from imaging devices 312 and 313, respectively. The calibration offsets and gains can be used to accurately calculate absolute eye position and orientation from the pixel locations of the eye characteristics identified in the captured image data.

[0047] The control logic 318 may further include a fixation tracking component 318B configured to track whether the eye 302 is properly fixating and / or an offset from the fixation position, and a tear film status component 318C configured to detect and track the status of the tear film on the eye 302. Eye measurements may be taken based on the fixation status and / or the tear film status. For example, the tear film status component 318C may include procedures to help maintain an intact tear film during measurements. In various embodiments, the patient may be instructed to blink or open and close the eye 302 to moisten the eye 302. The tear film status component 318C may detect eye closing and eye reopening and track the time to and during tear film stabilization. When the tear film is no longer stable for measurement (e.g., the eye is dry), the patient may be instructed to blink again to repeat the process.

[0048] The control logic 318 may be embodied in the eye tracking module 310, the retinal imaging system 330, and / or other system components. The control logic 318 is configured to detect relative eye movement during operation of the eye tracking module 310, which may include detecting and tracking ocular features (e.g., detecting the pupil) from captured images and knowledge of the illumination source position. For example, detecting and calculating the offset of the pupil center and the offset of the corneal curvature may provide information about the relative gaze of the eye.

[0049] Optional retinal imaging system 330 may include any device or system for imaging the retina of eye 302. Retinal imaging system 330 may be implemented as a retinal optical coherence tomography (OCT) system, a retinal optical system, or a similar system for imaging the retina. In some embodiments, retinal imaging system 330 and / or control logic 318 are configured to detect the patient's fovea at least once during the entire measurement sequence. As a result, retinal imaging system 330 does not need to be active during the entire diagnostic sequence (e.g., for technical or safety reasons) and can be shut down or paused as needed.

[0050] When a patient is fixating, the fovea will be present in the retinal imaging data. The fovea is often visible as a depression in the retina that can be detected by a retinal imaging system. In various embodiments, the retinal imaging system 330 generates retinal imaging data 332, such as a retinal OCT image 334 and / or a fundus image 336. The retinal imaging system 330 may include a retinal OCT scanning system, a fundus imaging system, or other similar devices. When a patient is fixating on a target object associated with the system 300, the fovea will be present at the center of the optical axis of the retinal imaging device. The retinal imaging device may only need to scan a central portion centered on the optical axis of the device. In some embodiments, the retinal imaging device is configured to image the back of the eye for fovea detection. When the system needs to image a different portion of the eye (e.g., a high-resolution scan of the cornea), the fovea will not be visible in the image, and the eye tracking module 310 is used to track the position and rotation of the eye.

[0051] System 300 coordinates the processing of information about eye orientation from imaging devices 312 and 313 of eye tracking module 310 (e.g., eye tracking data 316, including detected illumination source reflections 317 captured from each image capture component). System 300 may further enhance eye tracking data 316 with information from optional retinal imaging system 330 (retinal imaging data 332). During operation, if system 300 detects the fovea in a region of retinal imaging data 332 (e.g., via retinal imaging system 330 and / or control logic 318), the corresponding orientation of the eye is known to system 300. Using this information, system 300 can further determine whether the patient is fixating correctly, even during measurement phases when retinal imaging is not available. Fixation information can be used by eye tracking module 310 to identify images (e.g., images of the eye at fixation) to use in the initialization and calibration process. The calibrated eye tracking module 310 can then be used to accurately calculate absolute eye position and orientation from captured images.

[0052] The eye tracking module 310 may be configured to image and track eye position and eye rotation simultaneously with retinal imaging. In some embodiments, the captured images include associated temporal characteristics, such as a timestamp, a frame reference (e.g., 10 frames ago), or other information that enables synchronization of the retinal image with images captured from the first imaging device 312 and the second imaging device 313. After the fovea is detected, fovea detection information, which may include the corresponding temporal characteristics and an indication of whether the fovea is detected, may be provided to the control logic 318, the eye tracking module 310, and / or other system components.

[0053] In some embodiments, analyzing the position and orientation of the eye 302 includes comparing the eye orientation / position at the time the fovea was visible to the retinal imaging system with current eye tracking data. The system 300 may be used, for example, in a diagnostic procedure involving a measurement sequence. By tracking the eye position and orientation during the procedure using the eye tracking module 310, measurement data may be collected and analyzed along with the corresponding eye tracking data. In one embodiment, measurement data obtained when the eye 302 was fixating (e.g., when the eye position is within an acceptable offset from the fixation position) may be considered valid and used for further diagnosis / analysis, while measurement data obtained when the eye 302 was not fixating (e.g., when the eye position is outside the acceptable offset from the fixation position) may be ignored and / or discarded.

[0054] In various embodiments, the system 300 uses the foveal detection information to establish baseline fixation information, which may include a certain orientation of the pupil relative to the cornea. The eye tracking module 310 may receive the foveal detection information (e.g., a fixation determined at a specific time or other time reference), retrieve one or more corresponding images from the same time frame, and analyze the captured images to determine a specific relationship between the pupil and the center of the cornea during fixation. The system may be initialized and calibrated using the captured images to determine calibration offsets and gains for more accurate measurements. The eye position may then be tracked by comparing the eye position and orientation in the newly captured image with the eye position and orientation from the baseline image. This allows the eye tracking module 310 to confirm that the eye is fixating while the retinal imaging system 330 images another portion of the eye 302 (or operates other ophthalmic equipment, if necessary). The eye tracking module 310 may provide fixation information to the retinal imaging system 330 indicating whether the current scan occurred while the eye was fixating (within error relative to the reference data) or whether the current scan occurred while the eye was not fixating, such as when the offset between the current eye position and the reference eye position exceeds a threshold.

[0055] During operation of the system 300, the retinal imaging system 330 may be shut down during a diagnostic or other procedure so that retinal imaging data 332 is no longer generated. If the fovea has been previously detected by the retinal imaging system 330 at least once, the system 300 may continue to provide information about the patient's eye fixation to the device operator even during stages of the procedure when retinal imaging is not available. For example, the system 300 may compare the current eye position and orientation captured using the eye tracking module 310 with the eye position and orientation determined when the retinal imaging system 330 detected the fovea. The eye tracking module 310 may provide an indication to the device operator through one or more visual cues (e.g., indicator lights, status information on a display screen) or audible cues (e.g., beeps). The eye tracking module 310 may further provide fixation information to other components of the system 300, for example, to control actions requiring eye fixation and / or to enable / disable acquired data. It should be understood that the system and method described in FIG. 3 are example implementations of various embodiments, and that the teachings of the present disclosure may be used in other eye tracking systems, such as systems or devices that use an illumination system to generate a Purkinje reflex and a camera to capture digital images of the eye.

[0056] To assist in determining whether the eye is fixating, the control logic 318 may be configured to determine the current position and orientation of the eye and calculate an offset to determine whether the eye is sufficiently fixating on a desired target. In one embodiment, one or more thresholds may be determined, and any offset below a corresponding threshold results in a determination that the eye is fixating. In some embodiments, the fixation determination and thresholds are application dependent, and different offsets may be acceptable for different implementations.

[0057] In some embodiments, the retinal imaging system 330 identifies a time frame (e.g., a period of time, one or more images, sequential index values, etc.) in which the fovea was detected, allowing the eye tracker to identify corresponding eye tracking images taken at or near the same time. The eye tracking module 310 and / or control logic 318 may then perform initialization and calibration procedures to determine calibration offsets and gains that can be used to accurately calculate eye position and orientation from the captured images. The eye tracking module 310 may then determine a reference position of the eye associated with the fixation position, including the relative positions of the pupil and cornea. The eye fixation information may be used immediately by the system 300 to track eye position and orientation and / or may be stored and retrieved for later use by the system 100. For example, the eye fixation information may be determined and stored for the patient and retrieved for use by the system 300 (or a similar system) for subsequent procedures for the patient or for offline analysis of captured images.

[0058] While the retinal imaging system 330 is performing other scans and / or other ophthalmic components are operating, the eye tracking module 310 captures an image stream and analyzes the position and alignment of the eye relative to the position and orientation determined from the reference image. This analysis can be performed in real time during a procedure and / or offline (e.g., when analyzing previously captured data). The current image is compared to the reference image and an offset is calculated. If the offset is less than a threshold, the eye is fixating and the corresponding retinal image is accurate. If the offset is greater than the threshold, the eye is not fixating and the corresponding retinal image can be flagged, discarded, or other action can be taken. The images can be further synchronized with other information, including tear film status, which can be stored along with a time reference to allow for later synchronization and processing using the stored images.

[0059] In some embodiments, the eye tracking module 310 continuously images the eye throughout the procedure. For each frame, a pupil position may be detected in the image based at least in part on where the reflection is detected in the image stream. In various embodiments, the tracked and recorded information may include one or more of the image, image features extracted from the image, image properties, pupil position and / or reflection position within the image. The eye tracking system and retinal imaging system are synchronized so that one or more corresponding eye tracker images can be identified for each retinal scan image. In one embodiment, there is a one-to-one correspondence. In other embodiments, the images are synchronized through timestamps or other synchronization data associated with the captured images.

[0060] While the eye tracking module 310 and optional retinal imaging system 330 are described as separate components, it should be understood that system 300 may comprise a diagnostic device having various subcomponents, including the eye tracking module 310, the retinal imaging system 330, and other subcomponents. In some embodiments, a central processor may be provided to control the operation of system 300, synchronize and control communication between the two systems, and perform other system functions. Analysis of eye position and orientation may be performed by system 300 in real time or after the procedure is completed. Offline, system 300 may provide feedback to the patient and operator. Offline, system 300 and / or other systems may perform more complex analyses to achieve more accurate scans and results.

[0061] In some embodiments, the system 300 may include a larger diagnostic device that includes two or more cameras (e.g., for imaging the surface of the eye) and a second component for measuring the retina. The system 300 may include multiple sensors configured to image the eye to generate a 3D eye model. The first sensor system may include two or more cameras for recovering the corneal shape and for eye tracking. The second sensor system may include a wavefront sensor that measures the wavefront of the eye (optical parameters of the eye). The third sensor may include an OCT system that can measure the distance between different refractive surfaces of the eye. The OCT may include multiple modes and resolutions, including a full-eye mode, a half-eye mode (anterior surface of the eye), and a corneal mode (with higher resolution).

[0062] The sensor data can be provided to a processor (e.g., as shown in FIG. 5 ), which collects and stores the data in memory. The processor can use a fusion algorithm to derive a 3D model of the eye, including a parameterized model incorporating the various sensor data. The 3D model can be used, for example, for cataract and corneal refractive surgery planning. The data can be used for ray tracing in the eye, such as to assist in intraocular lens (IOL) implant placement. The fovea detection and eye tracking innovations described herein can be used with any diagnostic device or instrument, including devices that scan through the retina. Eye tracking can be implemented in keratometers, biometers, wavefront measurement devices, and other devices, including digital cameras and lighting.

[0063] In various embodiments, absolute eye orientation utilizes a device that scans through the retina, such as an OCT device, which may include a biometer and other devices that (i) provide retinal scanning and other diagnostic modes, and (ii) other sensors that perform other input functions. The systems disclosed herein may be used with more, different, and fewer components in various embodiments.

[0064] The benefits of the present application will be appreciated by those skilled in the art. The systems and methods disclosed herein provide automated initialization and calibration of eye tracking information that is calibrated to a patient's eyes. Eye tracking may be performed independently of the patient (e.g., without relying on the patient's cooperation) when the patient is fixating and not fixating, and may include tracking the tear film condition of the eye. The eye tracking information is collected and provided to a logic device for further analysis. Other sensor data may be obtained and validated by backtracking through the data to adjust for a known or projected orientation based on the eye tracking data. For example, eye position may be determined and provided to a retinal imaging system for use in analyzing scan data. The ability to flag whether a patient is fixating or not fixating is beneficial for many system operations, and the accuracy provided by the initialization and calibration of the present disclosure allows the system to more accurately determine fixation times / intervals and / or adjust for calculated offsets. The ability to flag whether a patient is fixating or not fixating is beneficial for many system operations. The ability to determine the degree of fixation allows the system to be adapted for use in a variety of implementations: storing the captured data for later retrieval and analysis allows for further offline calculations and more complex analyses and options, such as through the use of complex neural networks or other analytical processes.

[0065] In one embodiment, the control logic is configured with reference points and thresholds used to filter out unreliable sensor data. For example, the system may be configured so that small gaze changes (e.g., 0.03 degrees of offset) may be acceptable, but larger gaze changes indicate unreliable data that should be filtered out. In some embodiments, sensor data acquired during fixation may be averaged or combined together. In other embodiments, the acquired data may be analyzed with eye position and orientation information by calculating the eye position during acquisition using the calculated offset and known eye position and orientation at the reference points. In some embodiments, the various sensor and data inputs and calculations may be processed using a fusion engine to generate the desired output data.

[0066] In various embodiments, one or more neural networks may be used to analyze images and data, such as to determine whether the eye is fixating on a target object. FIG. 4 is a diagram of an example multi-layer neural network 400, according to some embodiments. Neural network 400 may represent a neural network used to implement at least some of the logic, image analysis, and / or eye fixation determination logic described herein. Neural network 400 processes input data 410 using an input layer 420. In some examples, input data 410 may correspond to the image capture data and captured retinal image data described previously herein. In some embodiments, the input data corresponds to input training data used to train neural network 400 to make fixation, orientation, and / or other determinations.

[0067] Input layer 420 includes a plurality of neurons used to condition input data 410, such as by scaling and / or range limiting. Each neuron in input layer 420 generates an output that is provided to an input of hidden layer 431. Hidden layer 431 includes a plurality of neurons that process the output from input layer 420. In some embodiments, each of the neurons in hidden layer 431 generates an output, which is then collectively propagated through one or more additional hidden layers, terminating in hidden layer 439, as shown. Hidden layer 439 includes a plurality of neurons that process the output from the previous hidden layer. The output of hidden layer 439 is provided to output layer 440. Output layer 440 includes one or more neurons that are used to condition the output from hidden layer 439, such as by scaling and / or range limiting. It should be understood that the architecture of neural network 400 is merely representative, and other architectures are possible, including neural networks with only one hidden layer, neural networks without input and / or output layers, and / or neural networks with recurrent layers.

[0068] In some embodiments, the input layer 420, the hidden layers 431-439, and / or the output layer 440 each include one or more neurons. In some embodiments, the input layer 420, the hidden layers 431-439, and / or the output layer 440 each may include the same or different numbers of neurons. In some embodiments, each of the neurons takes a combination (e.g., a weighted sum using a trainable weight matrix W) of its inputs x, adds an optional trainable bias b, and applies an activation function f to generate an output a, as shown in the equation a=f(Wx+b). In some embodiments, the activation function f may be a linear activation function, an activation function with upper and / or lower bounds, a log-sigmoid function, a hyperbolic tangent function, a rectified linear unit function, etc. In some embodiments, each of the neurons may have the same or different activation functions.

[0069] In some embodiments, the neural network 400 may be trained using supervised learning, which is a combination of training data including a combination of input data and ground truth (e.g., expected) output data. Differences between the generated output data 450 and the ground truth output data may be fed back to the neural network 400 to correct various trainable weights and biases. In some embodiments, the differences may be fed back using a backpropagation technique, such as a stochastic gradient descent algorithm. In some embodiments, multiple sets of training data combinations may be presented to the neural network 300 multiple times until an overall loss function (e.g., mean squared error based on the difference between each training combination) converges to an acceptable level. The trained neural network may be stored and implemented in an ophthalmic device (e.g., system 300 of FIG. 3) for real-time classification of captured images (e.g., as fixation or not fixation) and / or in an offline system for analysis of captured data.

[0070] 5 illustrates an example computing system that may include one or more components and / or devices of systems 100, 200, and 300, including an implementation of eye tracking module 310 and optional retinal imaging system 330. Computing system 500 may include one or more devices in electrical communication with each other, including computing device 510 including processor 512, memory 514, communication component 522, and user interface device 534.

[0071] The processor 512 may be coupled to various system components via a bus or other hardware configuration (e.g., one or more chipsets). The memory 514 may include read-only memory (ROM), random-access memory (RAM), and / or other types of memory (e.g., PROM, EPROM, FLASH-EPROM, and / or any other memory chip or cartridge). The memory 514 may further include a cache of high-speed memory directly connected to, nearby, or integrated as part of the processor 512. The computing device 510 may access data stored in the ROM, RAM, and / or one or more storage devices 524 through the cache for fast access by the processor 512.

[0072] In some embodiments, memory 514 and / or storage device 524 may store one or more software modules (e.g., software modules 516, 518, and / or 520), which may control and / or be configured to control processor 512 to perform various actions. While computing device 510 is shown with only one processor 512, it should be understood that processor 512 may represent one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and / or tensor processing units (TPUs), etc. In some embodiments, computing device 510 may be implemented as a standalone subsystem and / or as a board added to a computing device or as a virtual machine.

[0073] To enable a user to interact with system 500, computing device 510 includes one or more communications components 522 and / or one or more user interface devices 534 that facilitate user input / output (I / O). In some embodiments, one or more communications components 522 may include one or more network interfaces and / or network interface cards, etc., to provide communications according to one or more network and / or communications bus standards. In some embodiments, one or more communications components 522 may include an interface for communicating with computing device 510 over a network 580, such as a local area network, a wireless network, the Internet, or other network. In some embodiments, one or more user interface devices 534 may include one or more user interface devices, such as a keyboard, a pointing / selection device (e.g., a mouse, touchpad, scroll wheel, trackball, touchscreen), an audio device (e.g., a microphone and / or speaker), a sensor, an actuator, a display device, and / or other input / output devices.

[0074] According to some embodiments, the user interface device 534 may provide a graphical user interface (GUI) suitable for assisting a user (e.g., a surgeon and / or other medical personnel) in performing the processes disclosed herein. The GUI may include instructions regarding next actions to be performed, annotated and / or unannotated diagrams of anatomical structures, such as pre-operative and / or post-operative images of the eye, and / or input prompts. In some examples, the GUI may display true color and / or false color images of anatomical structures, etc.

[0075] Storage device 524 may include non-transitory and non-volatile storage such as that provided by a hard disk, optical media, and / or solid state drive, etc. In some embodiments, storage device 524 may be co-located with computing device 510 (e.g., a local storage device) and / or located remotely from system 500 (e.g., a cloud storage device).

[0076] The computing device 510 may be coupled to one or more diagnostic, imaging, surgical, and / or other devices for use by medical personnel. In the illustrated embodiment, the system 500 includes an ophthalmic device 550, an eye tracker 560, and an optional retinal imager 570, which may be embodied in one or more computing systems, including the computing device 510. The ophthalmic device 550 includes a user interface 554 for controlling and / or providing feedback to an operator performing a procedure on a patient's eye 552. The ophthalmic device 550 may include devices for imaging, measuring, diagnosing, tracking, and / or surgically correcting and / or repairing the patient's eye 552.

[0077] The ophthalmic device 550 is communicatively coupled to an eye tracker 560 (such as the eye tracking module 310 of FIG. 3 ), which receives eye imaging data from the ophthalmic device and provides status information on the position and alignment of the eye 552 during the procedure. The eye tracker 560 includes two or more imagers (e.g., imager A and imager B) positioned at known positions relative to the optical axis of the ophthalmic device 550. The eye tracker 560 is configured to perform an initialization and calibration procedure, which may be fully or partially automated. The calibration procedure includes instructing each of imager A and imager B to capture one or more images of the eye 552 while the eye is fixating and calculating calibration offsets and gains. The eye tracker 560 may then capture images of the eye 552, analyze the captured images for one or more eye characteristics, and calculate a gaze azimuth angle GA and a gaze elevation angle GE using the calibration offsets and gains. An optional retinal imager 570 is communicatively coupled to both the ophthalmic device 550 and the eye tracker 560 and configured to capture retinal images of the eye 552 for use in ophthalmic procedures and for foveal detection for use in fixation tracking.

[0078] In various embodiments, memory 514 includes an optional retinal imaging analysis module 516, an eye tracker module 518, a tear film condition module 519, and an ophthalmic procedure module 520. Retinal imaging analysis module 516 includes program instructions for instructing processor 512 to capture retinal images using retinal imager 570 and / or analyze the captured retinal images. Retinal image analysis module 516 may include a neural network trained to receive one or more captured retinal images (e.g., a captured image, a real-time stream of retinal images, stored retinal images, etc.), extract relevant image features, and detect the presence or absence of a fovea (e.g., outputting a classification indicative of foveal detection, outputting a likelihood of proper eye position and / or alignment, etc.).

[0079] The eye tracker module 518 includes program instructions for instructing the processor 512 to capture images of the eyes 552 using the eye tracker 560 and / or analyze the captured images. The eye tracker module 518 may include one or more neural networks trained to receive one or more captured images (e.g., a captured image, a real-time stream of eye images from the eye tracker 560, an image pair of image A and image B, stored eye images, etc.), extract relevant image features, and output eye tracking information (e.g., output eye alignment indicators, output likelihood of proper eye position and / or alignment, output eye offset from proper position and alignment, etc.).

[0080] In various embodiments, the eye tracker module 518 is configured to determine a reference eye position based on alignment data received while the eye 552 fixates on a known fixation point. For example, the eye tracker module 518 may receive foveal detection information from the retinal image analysis module 516, which is used to identify corresponding images from the eye tracker 560 that show the eye 552 in proper alignment. The eye tracker module 518 may receive fixation information from other sources, including operator feedback, statistical analysis, image analysis, and other sources available to the system 500. The eye tracker module 518 is further configured to automatically calibrate the patient's eye position calculations by a process that includes capturing an image from imager A during fixation, capturing an image from imager B during fixation, determining at least one characteristic in each image, comparing image coordinates of the eye characteristic in the two images, and calculating a calibration offset and gain for use in future patient's eye 552 position calculations. The eye tracker module 518 is further configured to analyze images captured by the eye tracker 560 and output eye tracking information related to the reference image and / or the calculated position.

[0081] The tear film status module 519 is configured to analyze images captured by the eye tracker 560, such as images from a camera (e.g., imager A) aligned with the device's optical axis. The tear film status module 519 receives an image sequence from the eye tracker and analyzes the images to determine one or more tear film status events, which may include blinks, eye-opening events, eye-closing events, etc. For example, it may be desirable to distinguish between an unintentional blink and an attempt by the patient to initiate a measurement sequence. In some embodiments, a blink sequence is defined as two or more such consecutive blinks, one or more intentional blinks, or a long blink in which the patient ensures that a period of eye closure can be detected by the tear film status module 519. In one approach, the tear film status module 519 detects one or more eye characteristics (e.g., pupil center, reflection of an illumination source) in an image sequence (e.g., an open eye), detects a disturbance of the one or more eye characteristics (e.g., a closed eye), and then detects the presence of the one or more eye characteristics (e.g., a re-open eye). In various embodiments, the tear film status module 519 may include one or more trained neural networks configured to receive the image stream and output tear film status events.

[0082] In some embodiments, the tear film status module 519 interfaces with one or more user interface devices 534 to aid in the process. For example, the film status module 519 may instruct the user interface device 534 (e.g., a loudspeaker) to generate a beep or other sound to indicate that a blink has been detected. When a blink and / or blink sequence has been detected and the measurement process is underway, the film status module 519 may further instruct the user interface device 534 to generate a second sound. In this manner, the patient is notified that the measurement process has begun, thereby reinforcing the need for the patient to fixate.

[0083] The ophthalmic procedure module 520 includes program instructions for instructing the processor 512 to perform an ophthalmic procedure, which may include user input and output during the procedure through the user interface 554 and analysis of captured data. In some embodiments, the ophthalmic procedure module 520 includes a trained neural network for analyzing data captured during the procedure. The ophthalmic procedure module 520 receives eye tracking information from the eye tracker module 518, which may include an alignment condition, among other information, an acceptable offset threshold, offset data, and / or other information. In some embodiments, the ophthalmic procedure module 520 is configured to operate when the patient's eye 552 is in an acceptable alignment position and tear film condition and to provide an indication to the patient through the user interface 554 that the procedure has begun (e.g., an audible signal such as a beep, a visual signal such as a flash of light, etc.). The ophthalmic procedure module 550 may further provide an indication to the operator when the patient is not aligned and / or when the tear film condition needs to be refreshed.

[0084] The system 500 may store captured retinal eye tracking data, tear film data, and ophthalmic procedure data for later processing, including online processing (e.g., during a subsequent procedure) and offline processing. The storage device 524 may store retinal image data 526 captured for a patient. The retinal image data 526 may include a patient identifier, a stream of captured images, time information (e.g., timestamp, sequential index, etc.), and / or information about whether the fovea was detected in the image. The storage device 524 may also store eye tracker data 528. The eye tracker data 528 may include a patient identifier, a stream of captured images, time information (e.g., timestamp, sequential index, etc.), whether the captured image corresponds to a period during which fixation was detected, and / or information providing a reference position of the eye during fixation, and / or calibration offset and gain information. The storage device 524 may also store procedure data 530 captured for a patient during a procedure. The procedure data 530 includes a patient identifier, a stream of data captured during the procedure (e.g., images, data readings, data calculations, etc.), time information (e.g., timestamps, sequential indexes, etc.), calculated offset information for the eye position at a point in the procedure and / or whether the eye was fixating at a point in the procedure.

[0085] The computing device 510 may communicate with one or more network servers 582 that provide one or more application services to the computing device. In some embodiments, the network server 582 includes a neural network training module 584 for training one or more of the neural networks using a training dataset 586, which may include labeled images. For example, the retinal image analysis module 516 may include a neural network trained using a set of retinal images labeled to identify the presence and / or absence of a fovea. The eye tracker module 518 may further include a neural network trained using a set of captured eye images labeled to identify the offset of the images relative to the reference data and the reference data. The ophthalmic procedure module 520 may include a neural network trained using a set of data representing data captured during a procedure, including alignment data and / or offset data from the eye tracker module 518.

[0086] Referring to FIG. 6A , an example embodiment of a process 600 for initializing and calibrating an ophthalmic device will now be described. In step 610, a patient is positioned in the ophthalmic system and oriented to focus on a target object so as to align the patient's gaze with the alignment axis of the ophthalmic device. In one embodiment, the patient's retina is analyzed to confirm that the patient is properly fixating. For example, the system may include a retinal imaging system configured to scan the retina, acquire scanned retinal data, and analyze the acquired data to detect the fovea. In some embodiments, a system operator may provide feedback to the system based on the operator's determination of whether the patient is fixating. In other approaches, image data may be acquired during a fixation procedure and analyzed to determine fixation (e.g., through analysis of histograms or other image data).

[0087] In step 620, multiple images of the surface of the eye are captured from corresponding imaging devices positioned to capture images from at least two known positions. For example, in the systems of Figures 2 and 3, two cameras are used to capture pairs of images of the eye, each from a different known position. In various embodiments, the images are captured simultaneously or sequentially over a short time interval to capture the current position and orientation of the eye.

[0088] In step 630, the captured images are analyzed to determine image coordinates of one or more eye characteristics. The eye characteristics may include the center of the pupil detected in the image, the center of the cornea detected in the image, the location of a reflection from an illumination source detected in the image, or other eye characteristics. In some embodiments, the image coordinates represent the (x,y) coordinates of pixel locations in each image, which may be mapped to real-world locations to determine the position and orientation of the eye.

[0089] In step 640, calibration offsets and gains are calculated from the known positions of the imaging device and image coordinates of the eye features. For example, the image coordinate difference between two eye features (e.g., the positions of the pupil center PC and the corneal reflection CR) may correspond to the gaze azimuth angle (Dx=CRx-PCx) and elevation angle (Dy=CRy-PCy). Calibration offset and gain values ​​may be used to more accurately derive the gaze azimuth angle GA and gaze elevation angle GE from the coordinate difference (e.g., Dx and Dy). GA=ax+bx*Dx GE=ay+by*Dy The simultaneous equations from the two imaging devices are used to solve for the values ​​of the calibration offset a and calibration gain b.

[0090] In step 650, the system performs eye tracking during the ophthalmic procedure. Eye tracking involves capturing a stream of images from one or more of the imaging devices, analyzing the images to determine image coordinates of detected eye characteristics, and calculating eye position and rotation using the calculated image coordinates and calibration offset and gain values. In some embodiments, the calibration offsets and gains for the patient's eyes and the patient identifier are stored in a lookup table or other storage device and can be accessed and used in subsequent ophthalmic procedures.

[0091] Referring to FIG. 6B , an example process 660 for operating a diagnostic system is now described according to one or more embodiments. In step 670, a patient is positioned in the ophthalmic system and oriented to focus on a target object so as to align the patient's gaze with the alignment axis of the ophthalmic device. In step 672, the system detects eye fixation, which may be performed by an operator, by a retinal imaging system to detect the fovea, through a histogram or other statistical analysis, or through another process. In one embodiment, the ophthalmic system includes a retinal imaging system (retinal imaging system 330 of FIG. 3 ) configured to scan the retina, acquire scanned retinal data, and analyze the acquired data to detect the fovea. In some embodiments, the fovea is visible at the center of the OCT image when the eye is fixating. In step 674, temporal characteristics associated with fixation detection are determined. In various embodiments, the temporal characteristics may include a timestamp, a sequential image index, or other criteria that enable synchronization of the detected eye fixation with the captured image stream captured by the eye tracking system.

[0092] Simultaneously, the eye tracking system captures a stream of eye images in step 680 and tracks eye movements using the captured image data in step 682. In step 684, captured images matching the temporal characteristics are identified and analyzed to determine the position and orientation of the eye when fixating the target object.

[0093] In step 686, the eye tracker analyzes the captured image stream relative to a fixation position (e.g., a reference position) to determine whether the eye is properly fixating within an error threshold. The image and fixation information may be stored (e.g., in storage 688) for later processing. Simultaneously, in step 690, a tear film state is determined. In one embodiment, images captured by the eye tracker are analyzed to detect blinks or other eye-open / eye-close / eye-open events. For example, the images may be analyzed to determine the presence of one or more eye characteristics indicative of an eye being open, the failure of one or more eye characteristics indicative of an eye being closed, and the reappearance of one or more eye characteristics in the image stream indicative of an eye being open. The tear film state may include detecting a blink event, waiting for a delay period, entering a stable tear film state for a time interval, and then exiting the stable film state. The tear film data may be stored (e.g., in storage 691) for later processing.

[0094] If the eye is properly fixating and the tear film condition is stable (step 692), a diagnosis is performed in step 694, which may include ocular measurements and other diagnostic data. In some embodiments, the analysis of the retina (step 672) and determination of temporal characteristics associated with the detected fovea (step 674) are performed by a retinal imaging system, which is disabled during the eye diagnosis of step 694. Thus, the retinal imaging system is unavailable for tracking the position of the eye during the diagnostic procedure.

[0095] During the measurement in step 694, the eye tracking system tracks the position and orientation of the eye in step 686 to determine whether the eye is properly positioned and aligned during the measurement. In some embodiments, the eye tracking system focuses on the anterior side of the cornea or the inside of the eye chamber. The eye tracking system may analyze captured images of the eye during the diagnosis (step 694) and determine the current position and rotation based on the captured images. The current position and rotation are compared to the fixation position and rotation to determine an offset. If the offset is less than an error threshold, the eye is determined to be properly positioned and aligned for the measurement. If the offset exceeds the error threshold, the diagnostic process and / or a system operator may be notified that the eye is not aligned, allowing the operator to pause the diagnostic procedure and instruct the patient to reposition the eye, determine whether associated measurement data is valid / invalid, or take other action. In some embodiments, data acquired during the eye diagnostic procedure is stored in a storage device 696 for subsequent processing and analysis.

[0096] Retinal imaging information and / or foveal detection information may not always be available for use in eye tracking. Some ophthalmic devices, for example, do not include an OCT retinal scanner. In some procedures, the fovea may not be reliably detected before the procedure begins (e.g., the patient did not fixate properly, the fovea was not detected in the image with a satisfactory degree of certainty, operator or system error, etc.). In these embodiments, the absolute fixation position may be determined based at least in part on analysis of images captured from the eye tracker (e.g., images of the eye surface). In other embodiments, the absolute fixation position may be determined through one or more of operator feedback, a detailed initialization procedure, image analysis, statistical analysis, and / or other methods.

[0097] In various embodiments, fixation analysis is performed by detecting eye positions in a stream of images captured from a camera and analyzing the results to estimate absolute fixation positions. The analysis may include statistical analysis using a histogram of eye positions determined from the captured images. If the histogram shows a clear peak according to the analysis, the method may estimate absolute fixation positions. If the histogram does not show a clear peak, the method may indicate that no fixation was detected. In some embodiments, analysis of the captured images may include comparison of the patient's eyes with other eyes in known positions (e.g., using a neural network trained using a set of labeled training images), historical fixation information for the patient, image analysis (including crossing / thresholding), and / or other analysis of available information. In some embodiments, the method may rely on the operator and patient to properly fixate the patient's eyes. In some embodiments, the method may address scenarios in which the images do not reflect fixation due to operator and / or patient error (e.g., when the patient intentionally fixates in the wrong place or when the operator does not properly instruct and / or monitor the patient).

[0098] Embodiments of systems and methods for eye tracking where a retinal OCT scan is not available and / or the fovea has not been reliably detected prior to the procedure are described herein with reference to Figures 7-10. As previously discussed, accurate measurement of the eye using an ophthalmic device may begin with the alignment of the patient's line of sight (the patient's visual axis) with the optical axis of the ophthalmic device. In this context, the line of sight may be the axis along which the patient views objects. Resulting diagnostic data and / or other results of the ophthalmic procedure may be unreliable during periods when the patient has not properly fixated.

[0099] The absolute eye fixation position may be used by the ophthalmic device to provide feedback to the device operator regarding whether the patient is fixating (or not fixating properly) along an optical axis of the diagnostic device during a procedure (e.g., a measurement procedure). The ophthalmic device may use the absolute eye fixation position during a procedure to identify periods when the patient is fixating properly. The system may also use the absolute eye fixation position to determine whether data acquired during a procedure is reliable and / or unreliable based at least in part on whether the patient is determined to be fixating during data acquisition.

[0100] Referring to FIG. 7 , an embodiment of a method 700 for estimating absolute eye fixation is now described. Method 700 is performed using a computing device and an imaging system, which may include a camera and an illumination system (e.g., imaging devices 312 and 313 and illumination component 314 of FIG. 3 ) for imaging the surface of a patient's eye. The method determines the position and orientation of the eye by using the location of detectable features of the eye (e.g., pupil, limbus, iris features, etc.) in the image and the location of the illumination system's reflection on the cornea. The eye position is determined during a procedure or other time when the patient is expected to be properly positioned and fixating relative to the optical axis of the ophthalmic device. The operator may initiate the process by providing feedback (e.g., by pressing one or more buttons) and / or the operator may initiate a sequence that the patient then follows. Optionally, the operator may provide confirmation of the patient's compliance with the procedure.

[0101] Method 700 illustrates an embodiment for implementation by a computing device of an ophthalmic device, which may include a retinal OCT imaging device. To determine the absolute fixation position, the computing system determines whether foveal detection information is available (step 702). For example, the foveal detection information may be available if the ophthalmic device includes a retinal imaging device that scanned the patient's eye while the patient properly fixated. If the foveal detection information is available, the method proceeds to step 704. In step 704, the computing system identifies eye tracking images corresponding to the detected foveal data (e.g., as described above with reference to FIG. 3). In step 706, the system calibrates the offset and gain and calculates absolute fixation parameters using the corresponding images. The patient's eye may then be tracked during the procedure using the eye tracking images, fixation parameters, and the calibrated equations.

[0102] Referring back to step 702, if foveal detection is not available, the method estimates absolute fixation parameters using captured images of the eye (e.g., images of the surface of the eye). In step 720, the computing device receives a stream of captured images from the camera, calibrates offset and gain values ​​using at least one pair of images captured from a different camera positioned at a known location, and determines the position and orientation of the eye in each of the multiple images. The computing device may process each received image or a subset of the received images (e.g., according to processing constraints). Images may be received before / during and / or after the procedure when analyzing the captured data.

[0103] After the eye position and orientation have been determined for the series of captured images, a histogram is generated from the determined positions and orientations in step 730. In some embodiments, the position and orientation information includes the pixel location of the pupil center in each of the images, which is used to construct a two-dimensional histogram of (x,y) coordinates. The position and orientation information may include the absolute position and orientation of the eye determined from each of the images, which is used to construct a two-dimensional histogram. Other representations of the position and orientation data may also be used in the method (e.g., heat maps). In some embodiments, operator feedback may be used to indicate the images the patient is instructed to fixate and / or to indicate if the patient is not fixating, and the corresponding images may be added to or discarded from the analysis. A procedure may be performed in which the system operator instructs the patient to fixate on the target during the measurement sequence.

[0104] Referring to FIG. 8 , a heat map 800 is shown illustrating an example distribution of fixation points where a patient is looking. The map may be color-coded, three-dimensional, or include indicia to track how often a patient fixates at a location. Other indicia (e.g., a color similar to the background color) may be used to indicate brief fixations at that location. In the illustrated embodiment, region 810 of the heat map indicates the most frequent coordinates and may indicate the position and orientation of the patient's eyes while properly fixating on a target object. Dashed circle 820 indicates positions and orientations that fall within a threshold offset that should be selected for fixation determination, depending on the level of precision required for the procedure or analysis.

[0105] 9 shows an example histogram 900 plotting eye coordinates detected from a captured image. The peak of this distribution 910 can be used to estimate the position and orientation of the fixating eye (e.g., by identifying the position and orientation at which the patient most often fixates). This estimated position and orientation can be used as a reference position for further eye fixation determinations. For example, analysis of medical data taken in a measurement sequence can use only data points acquired when the eye has an orientation and position within an acceptable offset (e.g., indicated by circle 920) from the reference position (e.g., based at least in part on the peak of the histogram).

[0106] As previously described, the histogram 900 may be constructed by plotting fixation points determined from captured images. For example, the histogram may track eye position as a series of pixel locations of detected pupils or identified eye centers (e.g., determined from reflexes or other measurements). As a sequence of images is received and analyzed, patterns may emerge that indicate where the patient most often fixates. In some embodiments, the values ​​in the histogram may include an average of neighboring pixels and / or incorporate other smoothing.

[0107] Referring back to method 700 of FIG. 7, in step 740, the histogram is analyzed to detect a fixation location. As previously mentioned, the fixation location may be related to the maximum value in the histogram that meets certain analytical criteria. For example, the peak may be selected based on a variety of factors, including the degree to which the peak exceeds the average value, the degree to which it exceeds a threshold for a given number of images, etc. In some embodiments, eye tracking continues during the procedure, and the peak / fixation location may be updated in real time as more images are analyzed.

[0108] Referring to step 750, if an acceptable peak point is not found (or other fixation criteria are met), eye fixation information is unavailable throughout this process. In some embodiments, eye tracking continues during the procedure, and peak / fixation locations may be identified and / or updated in real time as more images are analyzed.

[0109] Calibration offsets and gains and estimated fixation parameters (e.g., acceptable fixation positions and offset radii for the procedure) are determined based on the detected fixation information in step 760. The patient's eyes can then be tracked during the procedure in step 708 using the eye tracking images and estimated fixation parameters.

[0110] 10, an example system 1000 for implementing the methods of Figures 7-9 is now described. A computing device 1002 (such as computing device 510 of Figure 5) is communicatively coupled to ophthalmic equipment 1060 and configured to perform processes associated with eye tracker 1030, tear film analysis 1050, and ophthalmic procedure 1040. Computing device 1002 may be configured to perform retinal image analysis (through retinal image analysis module 1010) using a retinal imaging device (if available) of ophthalmic equipment 1060 and store retinal image data 1012. Computing device 1002 further includes a fixation analysis module 1020 for performing an embodiment of the method shown in Figure 7 or other methods for estimating absolute fixation parameters. In one embodiment, the fixation analysis module 1020 receives and analyzes a stream of eye images captured by one or more cameras (e.g., Imager A and Imager B) of the eye tracker 1030, constructs and analyzes a histogram of fixation positions, and determines reference positions and associated radii. The fixation data, including the histogram data, may be stored in storage 1022 (e.g., a memory or storage device).

[0111] In some embodiments, the computing device 1002 includes a logic device configured to execute program instructions stored in memory, which may include a fixation analysis module 1020, an optional retinal image analysis module 1010, an eye tracker 1030, a tear film analysis module 1050, and processes associated with an ophthalmic procedure 1040. The computing device 1002 may also be coupled to a storage device 1032 for storing eye tracker data, images, reference information, and other data.

[0112] The fixation analysis module 1020 may be configured to analyze the relative gaze of the patient's eyes using images captured by the eye tracker 1030. The fixation analysis module 1020 may calibrate measurements using pairs of images captured from different cameras to derive calibration offsets and gains, thereby enabling accurate determination of eye position and orientation from pixel coordinates of eye characteristics. The fixation analysis module 1020 may construct a histogram tracking gaze direction (e.g., eye pitch and yaw, relative up / down and left / right offsets, curvature / rotation, etc.) and analyze the histogram's peak value (e.g., number of data values ​​at each location) to obtain an estimate of the absolute reference. In some embodiments, the fixation analysis module 1020 estimates the eye's optical axis and its intersection with the eye tracker camera to track gaze direction.

[0113] The eye tracker 1030 may be configured to capture, store, and process images of the patient's eye. The eye tracker 1030 may be configured to determine the position and orientation of the patient's eye from one or more captured images for further analysis by the fixation analysis module 1020. In some embodiments, each analyzed image may include x, y positions representing the position and orientation of the eye (e.g., rotation about the x- and y-axes). The eye tracker may use information about the absolute fixation position (e.g., determined through retinal image analysis 1010) or the relative orientation change from one image to another in relation to an estimated absolute fixation position (e.g., determined through fixation analysis module 1020). In some embodiments, the fixation analysis module 1020 operates on the assumption that the patient was attempting to fixate most of the time, and that the estimated absolute fixation position can be determined by constructing a histogram of x and y rotations and determining the most prominent gaze direction. In various embodiments, the histogram may be constructed from pixel coordinates, rotations about x and / or y, offset values, or other data. Each image may provide a coordinate pair representing a calculated eye gaze direction that is added to the histogram.

[0114] In some embodiments, the fixation analysis module 1020 is configured to analyze the histogram by detecting one clear peak (e.g., a prominent peak surrounded by smaller peaks) and determining a level of confidence that a fixation position has been detected. If no clear peak is detected, the confidence level may be low. A radius around the detected peak may be used (e.g., humans may fixate + / - 0.5 degrees). The threshold of the peak to the mean and / or the size of the radius may vary depending on the system and procedural requirements.

[0115] The computing device 1002 may include one or more neural networks trained to make one or more determinations disclosed herein, including analyzing the histogram data to determine whether eye fixation location can be determined. In some embodiments, the fixation analysis may further include a comparison of known eye tracking images and / or eye fixation parameters for the patient and / or other patients. For example, one or more images may be input into a neural network trained using historical data to determine whether an eye in the image is fixating.

[0116] 11, an example method for tracking the state of a patient's tear film is now described. Method 1100 includes patient-induced trigger generation, which may be implemented in various systems as (i) the patient pressing a button on a user input device, (ii) the system detecting when the patient blinks, and (iii) the system detecting when there is a change in fixation state between a fixation state and a non-fixation state.

[0117] In some embodiments, the system is configured to detect when the patient blinks and when the patient is fixating so that measurements can be taken when tear film and fixation conditions are favorable. In one embodiment, the system is configured to detect when the patient blinks by tracking the eye using an eye tracker or other system components. This can be achieved using a video-based system or other imaging methods (e.g., optical coherence tomography), one or more trained neural networks, an expert system, and / or other systems and components. The patient is asked to blink in some manner, such as, for example, two blinks, a strong blink, etc. The system is configured to detect an event and trigger measurement acquisition after the tear film has stabilized.

[0118] The device may provide feedback to the user (e.g., beep each time a blink is detected). With an appropriate number of blinks, blink duration, and a defined delay in acquisition after the last blink, well-defined measurement conditions may be achieved to identify a stable tear film state. This method may be implemented in a variety of ophthalmic diagnostic devices, including imaging systems.

[0119] During operation, the patient is positioned relative to the device by the operator in step 1110. The patient is instructed to fixate a target object to align one of the patient's eyes with the optical axis of the device in step 1120, and the patient attempts to fixate the target object throughout the procedure. In step 1130, the patient is instructed to blink or perform another eye open / close sequence to restore the eye's tear film. In step 1132, the patient blinks as instructed. In some embodiments, the patient is instructed to blink in a manner (e.g., twice in succession) to restore the tear film and signal to the device that a blink is occurring. In step 1134, the device detects the blink performed in step 1132. In some embodiments, the patient's eye is imaged using an eye tracker to capture a visual representation of the eye's surface. A stream of images (e.g., a video stream) is analyzed to detect blink sequences. For example, a blink detection component may perform image analysis to detect the eye's pupil, the reflection of an illumination source from the eye, or other eye characteristics. The image sequence may be analyzed to determine, for example, open and closed eye states. Blinking patterns may be detected by searching for blink sequences (e.g., open eye state -> closed eye state -> open eye state) within a short interval (e.g., two blinks in three seconds). In some embodiments, a trained neural network may be used to detect open and closed eye states from the captured images.

[0120] If a blink sequence is detected in step 1134, a delay period is initiated to allow tear film stabilization in step 1140. In some embodiments, the patient is notified via an audible beep or other indicator. After t1 seconds, the device enters a stable tear film state for t2, during which measurements may be captured by the device. During the stable tear film state 1140, the patient continues to fixate on the target object (step 1120). In some embodiments, the device detects whether the patient is properly fixating using an eye tracker or other device component. The eye tracker may capture an image of the eye and compare the current position to a reference position to determine whether the eye is fixating within an acceptable offset range. One or more measurements may be captured during the stable tear film state t2 and during the fixation state in step 1142. After the stable tear film period (step 1140), the tear film is presumed to have deteriorated to a level that makes the captured measurements unreliable (step 1150). At this state, the patient may cease fixating and end the procedure. In some embodiments, the sequence is completed in 2-6 seconds and may be repeated to restore the tear film for additional measurement opportunities.

[0121] In various embodiments, feedback can be provided to the operator as to whether the patient is fixating or not fixating along this axis during data acquisition, even when retinal imaging data is not available (e.g., not part of the system and / or foveal detection is not available pre-procedure).The systems and methods disclosed herein provide a cost-effective solution suitable for use in ophthalmic diagnostic devices that use the image capture devices and illumination systems described herein.

[0122] As will be appreciated by those skilled in the art, the method of the illustrated embodiment provides an improved technique for independently verifying whether a patient's eyes are properly fixating on a target object during operation. By detecting the fovea at a particular time, the system can determine where the gaze / visual axis is located relative to the patient. This information allows the system to determine whether the patient is currently fixating during a measurement sequence or other diagnostic or corrective procedure. The method combines a system that images the retina with a system that tracks the eye using surface information. From the location of the fovea in the retinal image, the system can determine the eye tracking position and whether the eye is moving left or right / up or down. The system can track the user's gaze, calculate offsets, determine current eye position and orientation, make decisions regarding eye fixation, determine data validity, and provide other features according to the present disclosure.

[0123] Methods according to the above-described embodiments may be implemented as executable instructions stored on a non-transitory, tangible, machine-readable medium. The executable instructions, when executed by one or more processors (e.g., processor 512), may cause the one or more processors to perform one or more of the processes disclosed herein. Devices implementing methods according to these disclosures may include hardware, firmware, and / or software and may take any of a variety of form factors. Typical examples of such form factors include laptops, smartphones, small form factor personal computers and / or personal digital assistants, etc. Some of the functionality described herein may also be embodied in peripherals and / or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executed in a single device, as a further example.

[0124] While exemplary embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, variations, and substitutions, and in some cases, some features of the embodiments may be utilized without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention is to be limited only by the claims that follow, and it is appropriate that such claims be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A system comprising: an ophthalmic device configured to measure a characteristic of the eye; an eye tracker configured to capture a first stream of images of the eye; A logical device, analyzing the first stream of images to determine whether the eye is fixating on a target object; Detecting a predetermined blink sequence in the first stream of images; After a predetermined tear stabilization period, a stable tear film interval is initiated; the logic device configured to capture at least one measurement of the eye with the ophthalmic device while the eye fixates on the target object during the stable tear film interval; A system comprising:

2. the blink sequence includes a plurality of consecutive blinks; 2. The system of claim 1, wherein detecting the predetermined blink sequence in the first stream of images comprises processing the images through a neural network trained to detect open and / or closed eyes.

3. the eye tracker is configured to capture a first image of the eye from a first position and a second image of the eye from a second position; The logic device is detecting a first plurality of eye characteristics from the first image, the eye characteristics having first corresponding image coordinates; detecting a second plurality of eye characteristics from the second image, the eye characteristics having second corresponding image coordinates; further configured to determine a calibration offset and a calibration gain based at least in part on the first corresponding image coordinate, the second corresponding image coordinate, the first position, and the second position. The system of claim 1 .

4. The logic device is 4. The system of claim 3, further configured to determine an eye fixation position and orientation relative to an optical axis of the eye tracker based at least in part on the first corresponding image coordinates and / or the second corresponding image coordinates.

5. The logic device is estimating eye fixation parameters based at least in part on the determined eye fixation position and orientation; receiving a first stream of images from the eye tracker; further configured to track the current eye position and orientation by analyzing at least one image from the first stream of images to determine a current eye position and orientation relative to the eye fixation parameter; The system of claim 1 , wherein the eye fixation parameters include a reference position and orientation of the eye at the time of fixation.

6. the logic device is further configured to determine the gaze fixation position relative to an optical axis of the eye tracker by constructing and analyzing a histogram of detected eye positions and orientations; analyzing the histogram further includes determining whether a coordinate of a relative maximum value includes a fixation position and orientation; The system of claim 1 , wherein determining whether the coordinates of the relative maximum value include a fixation position and orientation further comprises comparing the relative maximum value with a threshold and / or a mean coordinate value of the histogram.

7. further comprising a retinal imaging system including an optical coherence tomography (OCT) scanner configured to perform a retinal scan; the eye tracker is further configured to capture a stream of images of the eye during the retinal scan; the retinal imaging system comprises: capturing a plurality of retinal images of the eye; Detecting whether a fovea is present in one or more of the plurality of retinal images of the eye; further configured to identify a first retinal image from the plurality of retinal images of the eye having the detected fovea; The logic device is determining a corresponding image from the stream of images that has temporal proximity to the first retinal image; and further configured to analyze the corresponding images to determine eye fixation parameters. The system of claim 1 .

8. the logic device is configured to track the position and orientation of the eye, calculate an offset from an eye fixation parameter, and determine whether the offset is less than a threshold; When the offset is less than the threshold, the eye is determined to be fixating and the logic device generates an indication of fixation; The system of claim 1 , wherein when the offset is greater than the threshold, the eyes are determined to be out of alignment and the control processor generates a no fixation indicator.

9. 10. The system of claim 1, wherein the logic device is further configured to perform an ophthalmic diagnostic procedure and to track eye position using the eye tracker during the ophthalmic diagnostic procedure.

10. 10. The system of claim 1, further comprising a diagnostic device configured to perform an ocular diagnostic procedure while tracking the position and orientation of the eye with the eye tracker, the diagnostic device configured to modify the ocular diagnostic procedure based at least in part on data representing ocular fixation parameters and tracked eye position.

11. 1. A method comprising: capturing a first stream of images of the eye with an eye tracker; analyzing the first stream of images to determine whether the eye is fixating on a target object; detecting a predetermined blink sequence in the first stream of images; tracking a stable tear film interval after a predetermined tear stabilization period; capturing at least one measurement of the eye with an ophthalmic device while the eye is fixating on the target object during the stable tear film interval; A method comprising:

12. the blink sequence includes a plurality of consecutive blinks; 12. The method of claim 11 , wherein detecting the predetermined blink sequence in the first stream of images comprises processing the images through a neural network trained to detect open and / or closed eyes.

13. capturing a first image of the eye from a first position; capturing a second image of the eye from a second position different from the first position; detecting a first plurality of eye characteristics from the first image, the eye characteristics having first corresponding image coordinates; Detecting a second plurality of eye characteristics from the second image, the eye characteristics having second corresponding image coordinates; and determining a calibration offset and a calibration gain based at least in part on the first corresponding image coordinate, the second corresponding image coordinate, the first position, and the second position; The method of claim 11 further comprising:

14. capturing a stream of images of the eye; detecting an eye position and orientation in the stream of images based at least in part on coordinates of the detected eye characteristic, the calibration offset, and the calibration gain; determining eye fixation position and orientation relative to an optical axis; 14. The method of claim 13, further comprising:

15. estimating eye fixation parameters based at least in part on the determined eye fixation position and orientation; tracking the position and orientation of the eye by analyzing one or more images from the stream of images to determine the eye position and orientation relative to the eye fixation parameter; further comprising The method of claim 14 , wherein the eye fixation parameters include a reference position and orientation of the eye at the time of fixation.

16. The method of claim 11 , further comprising training a neural network to receive the stream of images and output a judgment of eye position.

17. detecting the gaze fixation position relative to an optical axis of the device by constructing and analyzing a histogram of detected eye positions and orientations; The method of claim 11 , wherein analyzing the histogram further comprises determining a relative maximum value.

18. performing a retinal imaging scan of the eye using an optical coherence tomography (OCT) scanner; capturing a plurality of retinal images of the eye from the retinal imaging scan; capturing a stream of images with an imaging device configured to image the surface of the eye; detecting whether a fovea is present in one or more of the plurality of retinal images; identifying a first retinal image from the plurality of retinal images having the detected fovea; determining a corresponding image from the stream of images that has temporal proximity to the first retinal image; analyzing the corresponding images to determine eye fixation parameters; The method of claim 11 further comprising:

19. tracking eye position and orientation, calculating an offset from an eye fixation parameter, and determining whether the offset is less than a threshold; When the offset is less than the threshold, the eye is determined to be fixating and a fixation indication is generated; The method of claim 11 , wherein when the offset is greater than the threshold, the eyes are determined to be out of alignment and a no fixation indication is generated.

20. performing an ophthalmic diagnostic procedure while tracking the position and orientation of the eye with an image capture device; modifying the ocular diagnostic procedure based at least in part on data representative of ocular fixation parameters and tracked eye positions; The method of claim 11 further comprising:

Images