Eye tracking gaze monitoring system and method

Abstract

Systems and methods for tracking eye movement during a diagnostic procedure include an eye tracker that captures images of an eye, and a control processor configured to: detect an eye position and orientation in each of the images; determine an eye gaze position and orientation relative to an optical axis of the eye tracker; estimate eye gaze parameters based at least in part on the determined eye gaze position and orientation; and track eye position and orientation by analyzing the images to determine eye position and orientation relative to the eye gaze parameters. The eye gaze parameters can include a reference position and orientation of the eye when gazing. A histogram is constructed with the detected eye position and orientation and analyzed to determine the eye gaze position and orientation.

Description

background Technical Field

[0002] This disclosure relates generally to eye-tracking systems and methods, and more specifically to systems and methods for tracking the position and / or orientation of the eye in imaging, tracking, diagnostic, and / or surgical systems. Background Technology

[0004] A wide variety of ophthalmic devices are used for imaging, measuring, diagnosing, tracking, surgically correcting, and / or repairing a patient's eye. The operation of ophthalmic devices, such as tomography devices, corneal curvature measurement devices, wavefront analyzers, or other devices that measure various aspects of the eye (e.g., optical, geometric, etc.), is generally based on the assumption that the eye is maintained in a defined position and orientation relative to the diagnostic device. The human operator of the ophthalmic device allows the patient to position and be directed, for example, to look at a target object within the device (e.g., a gaze lamp), to align the patient's line of sight (e.g., the axis along which a person looks at an object) with the optical axis of the ophthalmic device. If the patient does not gaze properly, readings may be inaccurate and / or the system may not function properly.

[0005] To ensure accurate data acquisition, the human operator of the ophthalmic device is typically responsible for monitoring the patient and / or monitoring feedback from the device during data acquisition to determine whether the patient has properly gazed at the target object and aligned their eyeball. One known technique involves relying on the patient's cooperation to gaze at the target object as instructed by the device operator. However, existing approaches have several drawbacks, including human error when the patient attempts to gaze (e.g., elderly patients may not be able to maintain eye position, patients may lack sufficient attention to gaze, patients may not be able to look directly at the target object, etc.) and human error by the operator monitoring the patient during the procedure. In another approach, retinal scanning and imaging analysis can be used to track the patient's eye position and orientation, but the operation of the retinal imaging system can interfere with the diagnostic procedure. Therefore, during diagnostic procedures using ophthalmic devices, the retinal scanning and imaging system is often turned off or otherwise rendered unusable for eye tracking.

[0006] In view of the foregoing, there is a continued need in the art for improved techniques for determining and / or tracking the position and orientation of a patient’s eyeball during ophthalmic procedures. Summary of the Invention

[0007] Systems and methods for assessing and / or facilitating eye fixation are disclosed according to several different embodiments. The improvements disclosed herein can be used in various ocular imaging, tracking, diagnostic, and / or surgical systems to detect whether a patient is properly fixating during measurement sequences, surgical procedures, and / or other procedures. In some embodiments, the system assesses whether a patient is fixating on a particular optical axis of a diagnostic device. The system can be combined with a retinal imaging system to use eye-tracking techniques and statistical evaluation to allow for absolute fixation monitoring, even during diagnostic procedures where retinal imaging is not available.

[0008] In several different embodiments, a system includes an eye tracker configured to capture a first plurality of images of an eye, and a control processor configured to: detect eye position and orientation in each of the first plurality of images; determine eye fixation position and orientation relative to the optical axis of the eye tracker; estimate eye fixation parameters at least in part based on the determined eye fixation position and orientation; and track the eye position and orientation by analyzing one or more of the first plurality of images to determine the eye position and orientation relative to the eye fixation parameters. The eye fixation parameters may include a reference position and orientation of the eye during fixation.

[0009] In some embodiments, the control processor is further configured to detect the gaze position relative to the optical axis of the eye tracker by constructing and analyzing a histogram of the detected eye position and orientation. Analyzing the histogram further includes determining a relative maximum value and determining whether the coordinates of the relative maximum value contain the gaze position and orientation, and determining whether the coordinates of the relative maximum value might contain the gaze position and orientation further includes comparing the relative maximum value with a threshold and / or the average coordinate values ​​of the histogram.

[0010] The system may further include a retinal imaging system comprising an optical coherence tomography (OCT) scanner configured to perform retinal scanning, the retinal imaging system being configured to: capture a second plurality of images of the eye; detect the presence of a fovea in one or more of the second plurality of images; identify a first image having the detected fovea from the second plurality of images; determine a second image that is temporally close to the first image from the first plurality of images; and analyze the second image to determine eye fixation parameters.

[0011] The control processor can also be configured to track eye position and orientation, calculate the deviation from eye fixation parameters, and determine if the deviation is less than a threshold. When the deviation is less than the threshold, it is determined that the eye is fixating, and the control processor generates a fixation indication. When the deviation is greater than the threshold, it is determined that the eye is misaligned, and the control processor generates a non-fixation indication.

[0012] In some embodiments, the control processor is further configured to perform an eye diagnostic procedure and to use an eye tracker to track eye position during the eye diagnostic procedure. The system may further include a diagnostic device configured to perform the eye diagnostic procedure while using an image capture device to track the position and orientation of the eye; the diagnostic device is further configured to receive, during the eye diagnostic procedure, data representing gaze and eye position based at least in part on data representing gaze and eye position.

[0013] In several different embodiments, a method includes: capturing a first plurality of images of an eye; detecting the eye position and orientation in each of the first plurality of images; determining the eye gaze position and orientation relative to the optical axis of an eye tracker; estimating eye gaze parameters based at least in part on the determined eye gaze position and orientation; and tracking the eye position and orientation by analyzing one or more of the first plurality of images to determine the eye position and orientation relative to the eye gaze parameters. The method may further include: training a neural network to receive the first plurality of images and output a determination of the eye position. The eye gaze parameters include a reference position and orientation of the eye during gaze.

[0014] The method may further include detecting the gaze position relative to the optical axis of the eye tracker by constructing and analyzing a histogram of the detected eye position and orientation, and analyzing the histogram further includes determining the relative maximum value.

[0015] In some embodiments, the method includes: performing a retinal imaging scan of the eye using an optical coherence tomography (OCT) scanner; capturing a second plurality of images of the eye using the retinal imaging scan; detecting the presence of a fovea in one or more of the second plurality of images; identifying a first image having the detected fovea from the second plurality of images; determining a second image that is temporally close to the first image from the first plurality of images; and analyzing the second image to determine eye fixation parameters.

[0016] In some embodiments, the method further includes: tracking eye position and orientation and calculating a deviation from eye gaze parameters and determining whether the deviation is less than a threshold; wherein, when the deviation is less than the threshold, it is determined that the eye is gazing, and the control processor generates a gaze indication; and wherein, when the deviation is greater than the threshold, it is determined that the eye is not aligned, and the control processor generates a non-gazing indication.

[0017] The method may further include performing an eye diagnostic procedure and using an eye tracker to track the eye position during the eye diagnostic procedure. The method may perform the eye diagnostic procedure while using an image capture device to track the position and orientation of the eye; and modify the eye diagnostic procedure at least in part based on data representing eye fixation parameters and the tracked eye position.

[0018] In several different embodiments, a system includes: a retinal imaging system configured to capture a first plurality of images of an eye; an eye tracker configured to capture a second plurality of images of the eye and analyze the captured images to track eye position and orientation; and a control processor configured to detect a fovea in the first plurality of images; identify a first image having the detected fovea from the first plurality of images; determine a second image that is temporally close to the first image from the second plurality of images; and analyze the second image to determine eye fixation parameters.

[0019] The eye fixation parameters may include a reference position and orientation of the eye during fixation, and the control processor may be further configured to analyze the second plurality of images to determine the current eye position and orientation relative to the reference position and orientation during fixation. The control processor may be further configured to track the eye position and orientation and calculate the amount of deviation from the reference position and orientation. In some embodiments, when the deviation is less than a threshold, it may be determined that the eye is fixating, and the control processor generates a fixation indication. If the deviation is greater than the threshold, it is determined that the eye is misaligned, and the control processor generates an indication that the eye is not properly aligned.

[0020] In some embodiments, the retinal imaging system includes an optical coherence tomography (OCT) scanner configured to perform retinal scanning, and an ophthalmic device is provided to perform diagnostic procedures. During a portion of the ophthalmic diagnostic procedure, the retinal imaging system may not capture images of the eye and detect the fovea, and a control processor is configured to use an eye tracker to track the eye's position during the ophthalmic diagnostic procedure. The ophthalmic device can be configured to perform the ophthalmic diagnostic procedure while tracking the position and orientation of the eye using an image capture device. In some embodiments, the image capture device is configured to capture images of the surface of the eyeball without interfering with the diagnostic procedure. The ophthalmic device can be further configured to receive data representing gaze and eye position, at least in part, during the ophthalmic diagnostic procedure. Eye tracking can be analyzed and updated during the diagnostic procedure.

[0021] The system may further include one or more neural networks trained to analyze eye-tracking data from an eye tracker, retinal imaging data from a retinal imaging system, and / or data from diagnostic procedures. The neural network can be trained using a dataset of labeled data including images and information on eye fixation and / or deviation. In some embodiments, the patient's retinal imaging data and / or eye-tracking data may be captured, analyzed, and / or stored during one or more task periods and invoked by the system for use during diagnostic procedures.

[0022] In several different embodiments, a method includes: capturing a first plurality of images of the eye using a retinal imaging system; and capturing a second plurality of images of the eye using an eye tracker. The method may further include detecting a fovea in the first plurality of images and tracking the position and orientation of the eye based on analysis of the second plurality of images. The method may further include: identifying a first image with the detected fovea from the first plurality of images; determining a second image that is temporally close to the first image from the second plurality of images; and analyzing the second image to determine eye fixation parameters. In some embodiments, the first plurality of images may be captured by optical coherence tomography of the retina.

[0023] In some embodiments, the method can be performed using an ophthalmic diagnostic device including a retinal imaging system and an eye tracker. Eye fixation parameters may include a reference position and orientation of the eye during fixation, and the method may further include analyzing the second plurality of images to determine the current eye position and orientation relative to the reference position and orientation during fixation. The method may further include using the eye tracker to track the current eye position and orientation, calculating a deviation from the reference position and orientation, and generating an eye fixation indication when the deviation is less than a threshold.

[0024] The method may further include performing an ocular diagnostic procedure and using an eye tracker to track the eye position during the ocular diagnostic procedure, wherein the retinal imaging system does not capture images of the eye and detect the fovea during at least a portion of the ocular diagnostic procedure. The ocular diagnostic procedure may be performed while using an image capture device to track the position and orientation of the eye, and the method may include receiving data representing gaze and eye position, at least in part, during the ocular diagnostic procedure. Eye tracking may be analyzed and updated during the diagnostic procedure.

[0025] The scope of this disclosure is defined by the claims, which are incorporated herein by reference. A more complete understanding and additional advantages thereof will be achieved by those skilled in the art upon consideration of the following detailed description of one or more embodiments. A brief description of the accompanying drawings will be given first, with reference to the drawings. Attached Figure Description

[0026] A better understanding of the aspects and advantages of this disclosure can be achieved by referring to the following accompanying drawings and the subsequent detailed description. It should be understood that the same reference numerals are used to identify one or more of the same elements shown in the figures, and that the illustrations are for demonstrating embodiments of this disclosure and not for limiting it. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on clearly illustrating the principles of this disclosure.

[0027] Figure 1 Exemplary eye-tracking and imaging systems according to one or more embodiments of this disclosure are shown.

[0028] Figure 2 This illustrates one or more embodiments of the diagnostic procedure according to this disclosure. Figure 1 An exemplary eye-tracking and imaging system.

[0029] Figure 3 An exemplary neural network according to one or more embodiments of this disclosure is shown.

[0030] Figure 4 An exemplary computing system according to one or more embodiments of this disclosure is shown.

[0031] Figure 5 Exemplary operation of an ophthalmic system according to one or more embodiments of this disclosure is shown.

[0032] Figure 6 A method for estimating absolute eye position according to one or more embodiments of this disclosure is shown.

[0033] Figure 7 Exemplary heatmaps of eye position and orientation detected using an eye tracker according to one or more embodiments of this disclosure are shown.

[0034] Figure 8 Exemplary histograms constructed from eye position and orientation data detected using an eye tracker, according to one or more embodiments of this disclosure, are shown.

[0035] Figure 9 One or more embodiments of this disclosure are shown for implementation. Figure 6 An exemplary system for the method. Detailed Implementation

[0036] This article discloses systems and methods for evaluating and / or facilitating eye fixation in ocular imaging, tracking, diagnostic, and / or surgical systems. For example, the improvements disclosed herein can be used to help device operators align the patient's line of sight with the optical axis of the ocular device before activating it for measurement sequences or other diagnostic procedures.

[0037] According to various embodiments, a diagnostic system is used to facilitate accurate measurements of a patient's eyeballs, determining whether the patient's line of sight (also referred to herein as the patient's visual axis) is aligned with the optical axis of the diagnostic system. The patient's line of sight / visual axis can be the axis along which the patient's eyeballs are directed to look at an object. Diagnostic data acquired according to the systems and methods disclosed herein are more meaningful and accurate than data acquired through conventional methods. These systems and methods can also be configured to provide feedback to a human operator of the diagnostic device regarding whether the patient has gazed at the appropriate axis during data acquisition.

[0038] The systems and methods disclosed herein offer numerous advantages over conventional approaches. For example, data acquisition from a patient's eye may include instructing the patient to gaze at a target object to properly align their eye. However, this technique is error-prone because human operators of diagnostic devices often rely on patient cooperation, and the patient may not be properly gazing at the target object. In another approach, optical coherence tomography (OCT) can be used to image the patient's retina and provide the operator with an indication of whether the patient is properly gazing. However, analysis of OCT scans is only valid while the OCT is scanning the retina. If the diagnostic system uses different types of sensors during procedures that require switching the OCT scan to different parts of the eye or turning it off (e.g., for eye safety reasons), the analysis is unreliable for these measurement periods. Other optical retinal imaging techniques also have the same drawbacks as retinal OCT scans. In measurement sequences utilizing different sensors, gaze information is only valid during the period when retinal imaging is active. During periods when retinal imaging needs to be turned off, gaze information is unavailable.

[0039] The systems and methods disclosed herein overcome the aforementioned and other limitations of conventional systems and introduce numerous advantages. This disclosure provides a cost-effective, improved solution that can be implemented in a wide variety of systems, including conventional ophthalmic diagnostic devices using camera and illumination systems. Even with retinal OCT turned off, the combination of the ophthalmic diagnostic device and the retinal OCT system still allows for absolute gaze control. In some embodiments, the systems and methods provide accurate gaze information after the retinal OCT has detected the fovea at least once during the measurement sequence. In other embodiments, the systems and methods provide accurate gaze information in implementations where retinal OCT scans are unavailable and / or the fovea is not detected.

[0040] The embodiments disclosed herein will now be described in further detail with reference to the accompanying drawings. See also Figure 1 A system 100 according to one or more embodiments includes a communicatively coupled eye-tracking module 110 (also referred to herein as an "eye tracker") and a retinal imaging system 130. The eye-tracking module 110 is configured to track the orientation of an eye 102 and may include an imaging device 112 and one or more illumination components 114. In some embodiments, the imaging device 112 is a digital camera or other digital imaging device configured to image certain features of the eye, such as the pupil and limbus (the boundary between the cornea and the sclera, i.e., the sclera), and reflections from the illumination components 114. In some embodiments, for example, the illumination components 114 may include an LED ring positioned around camera optics (e.g., coaxial illumination around the imaging device) such that the center of the ring approximates the center of curvature of the cornea.

[0041] System 100 includes control logic 118, which may include a processor executing stored program instructions configured to perform the functions disclosed herein. In some embodiments, control logic 118 performs a measurement sequence on multiple images captured by imaging device 112. The measurement sequence determines the position and orientation of eyeball 102 by using the position of detectable features of eyeball 102 (e.g., pupil, limbus, and iris features) in image data (e.g., eye-tracking data 116). The measurement sequence may also determine the position of reflections of the illumination system at the cornea (e.g., reflections 117 including a circular pattern of illuminated elements). In some embodiments, the position and orientation of eyeball 102 are continuously determined using the captured images during the measurement sequence.

[0042] Control logic 118 may be implemented in eye tracker 110, retinal imaging system 130, and / or other system components. Control logic 118 is configured to detect relative eye movements during operation of system 110, which may include detecting and tracking eye features (e.g., detecting the pupil) based on captured images and knowledge of the location of the illumination source. For example, detecting and calculating the amount of deviation of the pupil center and the amount of deviation of the corneal curvature can provide information about the relative gaze of the eye.

[0043] The retinal imaging system 130 may include any means or system for imaging the retina of the eyeball 102. The retinal imaging system 130 may be implemented as a retinal optical coherence tomography (OCT) system, a retinal optics system, or a similar system for imaging the retina. In some embodiments, the retinal imaging system 130 and / or control logic 118 are configured to detect the patient's fovea at least once throughout the measurement sequence. Thus, the retinal imaging system 130 does not need to be operational throughout the diagnostic sequence (e.g., for technical or safety reasons) and can be shut down or paused when needed.

[0044] The fovea typically appears as a depression in the retina and can be detected in some retinal imaging systems. In several different embodiments, the retinal imaging system 130 generates retinal imaging data 132, such as retinal OCT images 134 and / or fundus images 136. The retinal imaging system 130 may include a retinal OCT scanning system, a fundus imaging system, or other similar devices. If a patient is gazing at a target object associated with system 100, the fovea will be present at the center of the optical axis of the retinal imaging device. The retinal imaging device may only need to scan the central portion around the optical axis of the device. If the patient is gazing, the fovea is present in the retinal imaging data. In some embodiments, the retinal imaging device is configured to image the posterior portion of the eyeball for fovea detection. If the system requires imaging different parts of the eyeball (e.g., a high-resolution scan of the cornea), the fovea will not be visible in the image, and an eye tracker 110 will be used to track eye position and rotation.

[0045] System 100 coordinates the processing of eye orientation-related information (e.g., eye tracking data 116, including detected illumination source reflections 117) from eye-tracking module 110 with information from retinal imaging system 130 (e.g., retinal imaging data 132). In operation, if system 100 (e.g., via retinal imaging system 130 and / or control logic 118) detects a fovea in a region of retinal imaging data 132, system 100 will know the corresponding eye orientation. Using this information, system 100 can further determine whether the patient is correctly fixating, even during measurement phases where retinal imaging is unavailable.

[0046] In some embodiments, the fovea appears at the center of the image if the patient is gazing. The eye-tracking module 110 is configured to image and track eye position and rotation simultaneously with retinal imaging. In some embodiments, the captured image includes associated temporal characteristics, such as timestamps, frame reference values ​​(e.g., 10 frames ago), or other information that allows synchronization of retinal images and eye-tracker information. After fovea detection, fovea detection information, which may include corresponding temporal characteristics and an indication of whether a fovea has been detected, can be provided to control logic 118, the eye-tracking module 110, and / or other system components.

[0047] In some embodiments, the analysis of the position and orientation of eye 102 includes comparing the orientation / position of the eye when the fovea is visible on a retinal imaging system with current eye-tracking data. For example, system 100 may be used in a diagnostic procedure that includes measurement sequences. By using eye tracker 110 to track the position and orientation of the eye during the procedure, measurement data can be collected and analyzed together with corresponding eye-tracking data. In one embodiment, measurement data acquired when eye 102 is fixating (e.g., when the eye position is within an acceptable deviation from the fixation position) is considered valid and used for further diagnosis / analysis, while measurement data acquired when eye 102 is not fixating (e.g., when the eye position exceeds an acceptable deviation from the fixation position) may be ignored and / or discarded.

[0048] In several different embodiments, system 100 uses foveal detection information to establish reference gaze information, which may include a certain orientation of the pupil relative to the cornea. Eye tracker 110 may receive foveal detection information (e.g., gaze determined at a specific time or other time reference value), recall one or more corresponding images from the same time frame, and analyze the captured images(s)(s) to determine a specific relationship between the pupil and the corneal center during gaze. Eye position can then be tracked by comparing the eye position and orientation in a newly captured image with the eye position and orientation in the reference image. This allows retinal imaging system 130 to image another portion of eye 102 (or operate other ophthalmic devices as needed) when eye tracker 110 confirms that the eye is gazed. Eye tracking module 110 may provide gaze information to retinal imaging system 130, indicating whether the current scan was performed while the eye was gazed (within error relative to reference data), or whether the current scan was performed while not gazed, such as when the deviation between the current eye position and the reference eye position exceeds a threshold.

[0049] See Figure 2During the operation of system 100, retinal imaging system 130 may be shut down during diagnostic or other procedures, thereby ceasing the generation of retinal imaging data 132. This is possible if retinal imaging system 130 has previously detected the fovea at least once, as seen in [reference needed]. Figure 1 As described, even during procedural phases where retinal imaging is unavailable, system 100 can continue to provide the device operator with information about the patient's eye gaze. For example, system 100 can compare the current eye position and orientation captured using eye tracker 110 with the eye position and orientation determined when retinal imaging system 130 detects the fovea. Eye tracker 110 can provide instructions to the device operator via one or more visual cues (e.g., indicator lights, status information on a display screen) or auditory cues (e.g., beeps). Eye tracker 110 can also provide gaze information to other components of system 100, for example, to control operations requiring eye gaze and / or verify the validity / invalidity of acquired data.

[0050] It should be understood that Figure 1 and Figure 2 The systems and methods described herein are exemplary implementations of various embodiments, and the teachings of this disclosure can be used in other eye-tracking systems, such as systems or apparatus that use an illumination system that generates Purkinje reflections and a camera that captures digital images of the eye.

[0051] To help determine whether the eye is fixating, control logic 118 can be configured to determine the current position and orientation of the eye and calculate a deviation amount to determine whether the eye is sufficiently fixating on the desired object. In one embodiment, a threshold can be determined, and any deviation amount below that threshold will determine that the eye is fixating. In some embodiments, the determination of fixation and the threshold are application-dependent, and different implementations may accept different deviation amounts.

[0052] In exemplary operation, the retinal imaging system 130 can be configured to perform multiple scans and retinal imaging analyses, which can focus on multiple different portions of the eyeball 102. In one configuration, the retinal imaging system 130 is configured to image the posterior portion of the eyeball and identify the fovea. The fovea may appear in the center of the image, indicating that the patient's gaze is aligned with the optical axis of the system 100. The retinal imaging system 130 can be configured to subsequently perform additional scans of different portions of the eyeball from which fovea detection is not possible. During these scans, the patient is expected to gaze at a target object, but the retinal imaging system may not be able to detect the fovea to confirm proper eye position and alignment.

[0053] In some embodiments, the retinal imaging system 130 identifies time frames (e.g., a time segment, one or more images, sequential index values, etc.) at which the fovea is detected, allowing the eye tracker to identify corresponding eye-tracking images acquired at or approximately the same time. The eye-tracking module 110 can then determine an eye reference position associated with the gaze position, including the relative positions of the pupil and cornea. The system 100 can immediately use the eye gaze information to track eye position and orientation and / or store and recall it for later use by the system 100. For example, eye gaze information can be determined and stored for a patient and recalled by the system 100 (or a similar system) for subsequent patient procedures or for offline analysis of captured images.

[0054] While the retinal imaging device 130 is performing other scans and / or other ophthalmic components are in operation, the eye tracker 110 captures an image stream and analyzes eye position and alignment by referencing positions and orientations determined from multiple reference images. This analysis can be performed in real time during the procedure and / or offline (e.g., when analyzing previously captured data). The current image is compared to the multiple reference images and the deviation is calculated. If the deviation is less than a threshold, the eye is fixating and the corresponding retinal image is accurate. If the deviation is greater than the threshold, the eye is not fixating and the corresponding retinal image can be flagged, discarded, or otherwise acted upon.

[0055] In some embodiments, the eye tracker 110 continuously images the eye throughout the procedure. For each frame, the pupil position in the image can be detected at least in part based on the location where a reflection is detected in the image stream. In several different embodiments, the tracked and recorded information may include one or more of the following: images, image features extracted from images, image properties, pupil position and / or reflection position in the image. The eye tracking system and the retinal imaging system are synchronized such that for each scanned retinal image, a corresponding one or more eye tracker images can be identified. In one embodiment, a one-to-one correspondence exists. In other embodiments, these images are synchronized via timestamps or other synchronization data associated with the captured images.

[0056] It should be understood that although the eye-tracking module 110 and the retinal imaging system 130 are described as separate components, the system 100 may include a diagnostic device with various sub-components, including the eye-tracking module 110, the retinal imaging system 130, and other sub-components. In some embodiments, a central processing unit may be provided to control the operation of the system 100, synchronize and control communication between the two systems, and perform other system functions. Analysis of eye position and orientation may be performed by the system 100 in real time or after the procedure is completed. When online, the system 100 may provide feedback to the patient and operator. When offline, the system 100 and / or other systems may perform more complex analyses to obtain more accurate scans and results.

[0057] In some embodiments, system 100 may include a larger diagnostic device including a camera (e.g., for imaging the surface of the eyeball) and a second component for measuring the retina. System 100 may include multiple sensors configured to image the eyeball to create a 3-D model of the eyeball. The first sensor may include multiple cameras for reconstructing corneal shape and performing eye tracking. The second sensor may include a wavefront sensor for measuring the wavefront (optical parameters of the eyeball). The third sensor may include an OCT system capable of measuring the distance between different refractive surfaces of the eyeball. OCT may include multiple modes and resolutions, including whole-eye mode, half-eye mode (anterior part of the eyeball), and corneal mode (with higher resolution).

[0058] Sensor data can be provided to a processor (e.g., such as...) Figure 4 As shown, the processor collects data and stores it in memory. The processor can use a fusion algorithm to derive a 3D model of the eye, which includes a parametric model incorporating data from various sensors. The 3D model can be used, for example, for planning cataract and corneal refractive surgeries. This data can be used for ray tracking to assist in the placement of intraocular lens (IOL) implants in the eye, etc. The innovative foveal detection and eye-tracking techniques described herein can be used with any diagnostic device or instrument, including those for scanning the retina. Eye tracking can be implemented in keratometers, biometers, wavefront measurement devices, and other devices including digital cameras and illumination components.

[0059] In several different embodiments, absolute eye orientation utilizes a device for scanning the retina, such as an OCT device, which may include a biometric meter and (i) other devices providing retinal scans and other diagnostic modalities, and (ii) other sensors performing other input functions. In several different embodiments, the system disclosed herein can be used with more components, different components, and fewer diagnostic devices.

[0060] Those skilled in the art will appreciate the advantages of this application. The systems and methods disclosed herein provide information about the time a patient is fixating and not fixating, independent of the patient (e.g., without relying on the patient's cooperation). Eye-tracking information is collected and provided to a processor for further analysis. Other sensor data can be acquired and validated through backtracking data to adjust known or projected orientations based on eye-tracking data. For example, eye position can be determined and provided to a retinal imaging system for analysis of scan data. The ability to identify whether a patient is fixating or not is valuable for many system operations. The ability to determine the degree of fixation allows the system to be adapted for use in a variety of implementations. Storing the captured data for subsequent retrieval and analysis will allow for further offline computation and more complex analysis and selection, for example, through the use of sophisticated neural networks or other analytical processes.

[0061] In one embodiment, the processor is configured with a reference point and a threshold to filter out unreliable sensor data. For example, the system may be configured such that small gaze changes (e.g., a deviation of 0.03 degrees) may be acceptable, but larger gaze changes will indicate unreliable data that should be filtered out. In some embodiments, sensor data acquired during fixation may be averaged together or otherwise combined. In other embodiments, the acquired data, along with eye position and orientation information, can be analyzed by calculating eye position during acquisition using a calculated deviation and known eye position and orientation relative to a reference point. In some embodiments, a fusion engine may be used to process various sensor and data inputs and perform calculations to generate desired output data.

[0062] In several different embodiments, one or more neural networks may be used to perform image and data analysis to determine whether the eye is looking at a target object. Figure 3 This is a diagram of an exemplary multilayer neural network 300 according to some embodiments. The neural network 300 may represent a neural network used to implement at least some of the logic, image analysis, and / or eye gaze determination logic described herein. The neural network 300 uses an input layer 320 to process input data 310. In some examples, the input data 310 may correspond to image capture data and captured retinal image data, as previously described herein. In some embodiments, the input data corresponds to input training data used to train the neural network 300 to make gaze, orientation, and / or other determinations.

[0063] Input layer 320 includes multiple neurons for regulating input data 310 by scaling, range limiting, and / or similar methods. The output generated by each neuron in input layer 320 is fed to the input of hidden layer 331. Hidden layer 331 includes multiple neurons that process the output from input layer 320. In some examples, each neuron in hidden layer 331 generates an output, which is then uniformly propagated through one or more additional hidden layers (ending in hidden layer 339), as shown. Hidden layer 339 includes multiple neurons that process the output from the previous hidden layer. The output of hidden layer 339 is fed to output layer 340. Output layer 340 includes one or more neurons for regulating the output from hidden layer 339 by scaling, range limiting, and / or similar methods. It should be understood that the architecture of neural network 300 is only representative, and other architectures are possible, including neural networks with only one hidden layer, neural networks without input and / or output layers, neural networks with recurrent layers, etc.

[0064] In some examples, each of the input layer 320, hidden layers 331-339, and / or output layer 340 includes one or more neurons. In some examples, each of the input layer 320, hidden layers 331-339, and / or output layer 340 may include the same or different numbers of neurons. In some examples, each neuron combines its input x (e.g., a weighted sum obtained using a trainable weighted matrix W), adds an optional trainable bias b, and applies an activation function f to generate an output a, as shown in the equation a = f(Wx + b). In some examples, the activation function f may be a linear activation function, an activation function with an upper and / or lower bound, a log-sigmoid function, a hyperbolic tangent function, a modified linear unit function, etc. In some examples, each neuron may have the same or different activation functions.

[0065] In some examples, supervised learning can be used to train neural network 300, where the combination of training data includes a combination of input data and standard ground truth (e.g., expected) output data. The differences between the generated output data 350 and the standard ground truth output data can be fed back into neural network 300 to correct for individual trainable weights and biases. In some examples, these differences can be fed back using backpropagation techniques such as stochastic gradient descent. In some examples, a large set of training data combinations can be presented to neural network 300 multiple times until the total loss function (e.g., the mean squared error based on the differences for each training combination) converges to an acceptable level. The trained neural network can be stored in an ophthalmic device (e.g., Figure 1The system 100) is used to classify captured images in real time (e.g., gaze or non-gaze) and / or stored in an offline system and used to analyze the captured data.

[0066] Figure 4 An exemplary computing system is illustrated, which may include one or more components and / or devices of system 100, including implementations of eye-tracking module 110 and retinal imaging system 130. Computing system 400 may include one or more devices that are electrically in communication with each other, including computing device 410, which includes processor 412, memory 414, multiple communication components 422, and user interface 434.

[0067] Processor 412 may be connected to various system components via a bus or other hardware arrangement (e.g., one or more chipsets). Memory 414 may include read-only memory (ROM), random access memory (RAM), and / or other types of memory (e.g., PROM, EPROM, FLASH-EPROM, and / or any other memory chip or cartridge). Memory 414 may further include a cache of high-speed memory directly connected to, adjacent to, or integrated into processor 412. Computing device 410 can access data stored in ROM, RAM, and / or one or more storage devices 424 via the cache for high-speed access by processor 412.

[0068] In some examples, memory 414 and / or storage device 424 may store one or more software modules (e.g., software modules 416, 418, and / or 420) that can control and / or be configured to control processor 412 to perform various actions. Although computing device 410 is shown as having only one processor 412, it should be understood that processor 412 may represent one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), etc. In some examples, computing device 410 may be implemented as a stand-alone subsystem, and / or as a board added to a computing device, or as a virtual machine.

[0069] To enable a user to interact with system 400, computing device 410 includes one or more communication components 422 and / or one or more user interface devices 434 facilitating user input / output (I / O). In some examples, the one or more communication components 422 may include one or more network interfaces, network interface cards, etc., to provide communication according to one or more network and / or communication bus standards. In some examples, the one or more communication components 422 may include an interface for communicating with computing device 410 via network 480 (such as a local area network, wireless network, Internet, or other network). In some examples, the one or more user interface devices 434 may include one or more user interface devices (e.g., keyboard, pointing / selection device (e.g., mouse, touchpad, scroll wheel, trackball, touch screen), audio device (e.g., microphone and / or speaker), sensor, actuator, display device, and / or other input / output device).

[0070] According to some embodiments, the user interface device 434 may provide a graphical user interface (GUI) suitable for assisting users (e.g., surgeons and / or other medical personnel) in performing the procedures disclosed herein. The GUI may include instructions regarding the next action to be performed, annotated and / or unannotated anatomical diagrams (such as preoperative and / or postoperative images of the eye), input requests, etc. In some examples, the GUI may display true-color and / or false-color images of anatomical structures, etc.

[0071] Storage device 424 may include nontransitory, non-volatile storage devices such as hard disks, optical media, solid-state drives, etc. In some examples, storage device 424 may be located in the same location as computing device 410 (e.g., local storage device) and / or remote from system 400 (e.g., cloud storage device).

[0072] The computing device 410 can be coupled to one or more diagnostic devices, imaging devices, surgical devices, and / or other devices for use by medical personnel. In the illustrated embodiment, system 400 includes an ophthalmic device 450, an eye tracker 460, and a retinal imager 470, which can be implemented in one or more computing systems (including the computing device 410). The ophthalmic device 450 includes a user interface 454 for controlling and / or providing feedback to an operator performing procedures on a patient's eyeball 452. The ophthalmic device 450 may include means for imaging, measuring, diagnosing, tracking, and / or surgically correcting and / or repairing a patient's eyeball 424.

[0073] The ophthalmic device 450 can be communicatively connected to the eye tracker 460 (e.g. Figure 1An eye tracker 110 receives eye imaging data from an ophthalmic device and provides position and alignment status information of the eye 452 during procedures. A retinal imager 470 is communicatively coupled to both the ophthalmic device 450 and the eye tracker 460 and is configured to capture retinal images of the eye 452 for use in ophthalmic procedures and for detecting the fovea for use in gaze tracking.

[0074] In several different embodiments, memory 414 includes a retinal image analysis module 416, an eye tracker module 418, and an ophthalmic program module 420. The retinal image analysis module 416 includes program instructions for instructing processor 412 to capture retinal images using retinal imager 470 and / or analyze the captured retinal images. The retinal image analysis module 416 may include a neural network trained to receive one or more captured retinal images (e.g., captured images, real-time retinal image streams, stored retinal images, etc.), extract relevant image features, and detect the presence or absence of the fovea (e.g., outputting a classification indicating foveal detection, outputting the probability of proper eye position and / or alignment, etc.).

[0075] The eye tracker module 418 includes program instructions for instructing the processor 412 to use the eye tracker 460 to capture images of the eye 452 and / or analyze the captured images. The eye tracker module 418 may include a neural network trained to receive one or more captured images (e.g., captured images, a real-time stream of eye images from the eye tracker 460, stored eye images, etc.), extract relevant image features, and output eye tracking information (e.g., outputting an indication of eye alignment, outputting the probability of proper eye position and / or alignment, outputting the amount of deviation of the eye from the proper position and alignment, etc.).

[0076] In several different embodiments, the eye tracker 418 is configured to determine the position of a reference eye based on alignment data received from the retinal image analysis module 416. For example, the eye tracker 418 may receive foveal detection information from the retinal image analysis module 416, which is used to identify a corresponding image from the eye tracker 460 that shows the eye 452 properly aligned. The eye tracker module 418 is further configured to analyze the image captured by the eye tracker 460 and output eye-tracking information with reference to the reference image.

[0077] The ophthalmology program 420 includes program instructions for instructing the processor 412 to perform ophthalmology procedures, and may include user input and output via a user interface 454 during the program, as well as analysis of captured data. In some embodiments, the ophthalmology program 420 includes a trained neural network for analyzing data captured during the program. The ophthalmology program 420 receives eye-tracking information from the eye tracker module 418, which may include alignment status within an acceptable deviation threshold, deviation data, and / or other information. In some embodiments, the ophthalmology program 420 is configured to operate when the patient's eye 452 is in an acceptable alignment position and to provide instructions (e.g., audible sounds such as beeps, visual cues such as flashes, etc.) to the operator of the ophthalmology device 450 via the user interface 454 when the patient's eye is misaligned.

[0078] System 400 can store captured retinal data, eye-tracking data, and ophthalmic procedure data for subsequent processing, including online processing (e.g., during subsequent procedures) and offline processing. Storage device 424 can store retinal image data 426 captured for the patient, which may include a patient identifier, captured image stream, time information (e.g., timestamps, sequence indexes, etc.), and / or information regarding whether a fovea was detected in the image. Storage device 424 can also store eye-tracker data 428, which may include a patient identifier, captured image stream, time information (e.g., timestamps, sequence indexes, etc.), whether the captured image corresponds to a time period in which a fovea was detected, and / or gaze information providing a reference position of the eye during gaze. The storage device 424 may also store program data 430 captured for the patient during the program, including patient identifier, data streams captured during the program (e.g., images, data readings, data calculation results, etc.), time information (e.g., timestamps, sequence indexes, etc.), deviation information calculated for the eye position at a certain point in the program, and / or whether the eye was fixating at a certain moment during the program.

[0079] Computing device 410 may communicate with one or more network servers 482 that provide one or more application services to the computing device. In some embodiments, network server 482 includes a neural network training module 484 for training one or more neural networks using a training dataset 486, which may include labeled images. For example, retinal image analysis module 416 may include a neural network trained using a set of labeled retinal images to identify the presence and / or absence of the fovea. Eye tracker module 418 may include a neural network trained using a set of captured labeled eye images to identify the presence and / or absence of the fovea. Eye tracker module 418 may further include a neural network trained using a set of captured eye images and reference data, labeled to identify the amount of deviation of the image relative to the reference data. Ophthalmological program 420 may include a neural network trained using a set of data representing data captured during the program, including alignment and / or deviation data from eye tracker module 418.

[0080] See Figure 5 The operation will now be described. Figure 1 An embodiment of the method 500 of the system 100. In step 510, the patient is positioned at the ophthalmic system and guided to focus on a target object while aligning the patient's line of sight with the alignment axis of the ophthalmic device. In step 512, the retina is analyzed to detect the fovea. In one embodiment, the ophthalmic system includes a retinal imaging system (e.g., Figure 1 A retinal imaging system 130 is configured to scan the retina, acquire scanned retinal data, and analyze the acquired data to detect the fovea. In some embodiments, if the eye is fixating, the fovea is visible at the center of the OCT image. In step 514, the temporal characteristics of the retinal image data associated with the detected fovea are determined. In several different embodiments, the temporal characteristics may include timestamps, sequential image indexes, or other metrics that allow the retinal imaging data to be synchronized with other components of the system.

[0081] Simultaneously, the eye-tracking system captures an image stream of the eye in step 520 and uses the captured image data to track eye movements in step 522. In some embodiments, when the retina is detected in steps 512 and 514, the eye-tracking system tracks eye movements and determines the gaze position and applies an acceptable deviation amount for a given procedure. In step 530, one or more captured images matching this temporal characteristic are identified and analyzed to determine the position and orientation of the eye when gazing at a target object. In step 540, system diagnostics are performed, which may include ocular measurements and other acquired data. In some embodiments, the analysis of the retina (step 512) and the determination of the temporal characteristics associated with the detected fovea (step 514) are performed by a retinal imaging system, which is disabled during the eye diagnostics in step 540. Therefore, the retinal imaging system is not used to track eye position during the diagnostic procedure.

[0082] During the measurement in step 540, the eye-tracking system tracks the position and orientation of the eye in step 550 to determine whether the eye is properly positioned and aligned during the measurement. In some embodiments, the eye-tracking system focuses on the anterior or lateral side of the cornea. The eye-tracking system may analyze the captured eye images during diagnosis (step 540) and determine the current position and rotation based on the captured images. The current position and rotation are compared with the gaze position and rotation to determine the amount of deviation. If the deviation is below an error threshold, the eye is determined to be in the appropriate position and alignment for the measurement. If the deviation is above the error threshold, the diagnostic process and / or system operator may be informed that the eye is misaligned, allowing the operator to pause the diagnostic procedure and instruct the patient to reposition the eye, allow determination of whether the associated measurement data is valid / invalid, or allow other actions to be taken. In some embodiments, data acquired during the eye diagnostic procedure (step 540) is stored in storage device 560, and data acquired during the eye-tracking procedure (step 550) is stored in storage device 570 for subsequent processing and analysis. In one embodiment, in addition to foveal information, patient data is tracked, and the foveal information can be used to verify the patient data when available. In this way, the range of values ​​and the average fixation position can be determined.

[0083] Retinal imaging information and / or foveal detection information may not always be available for eye tracking. For example, some ophthalmic devices do not include an OCT retinal scanner. In some procedures, the fovea may not be reliably detected before the procedure begins (e.g., the patient is not properly fixating, the fovea is not detected in the image with satisfactory certainty, operator or systematic errors, etc.). In these embodiments, the absolute fixation position can be determined at least in part based on the analysis of images captured by the eye tracker (e.g., images of the surface of the eye).

[0084] In several different embodiments, gaze analysis is performed as follows: eye positions are detected in an image stream captured by a camera, and the results are analyzed to estimate the absolute gaze position. This analysis may include statistical analysis using histograms of eye positions determined from these captured images. If, according to the analysis, the histogram shows a significant maximum value, the method can estimate the absolute gaze position. If the histogram does not show a significant maximum value, the method can indicate that no gaze was detected. In some embodiments, the analysis of the captured images may include comparisons between the patient's eyes and other eyes at known locations (e.g., using a neural network trained with a set of labeled training images), the patient's historical gaze information, image analysis (including tolerances / thresholds), and / or other analyses of available information. In some embodiments, the method may rely on the operator and the patient to ensure proper gaze of the patient's eyes. In some embodiments, the method may address situations where operator and / or patient errors (e.g., if the patient intentionally gazes at the wrong spot, or the operator does not properly instruct and / or monitor the patient) cause the image to not reflect a gaze.

[0085] Now refer to Figures 6 to 9 This describes embodiments of systems and methods for eye tracking, wherein a retinal OCT scan is not available prior to the procedure and / or the fovea is not reliably detected. As previously discussed, accurate measurements of the eye using an ophthalmic device typically begin by aligning the patient's line of sight (the patient's visual axis) with a certain optical axis of the ophthalmic device. In this context, the line of sight can be the axis along which the patient looks at something. During periods when the patient does not have proper fixation, the resulting diagnostic data and / or other results of the ophthalmic procedure may be unreliable. This document discloses methods that enable ophthalmic devices to track the eye via an eye tracker module (e.g., Figure 1 A system and method for estimating absolute gaze position by analyzing images captured by an eye tracker 110 and / or another image capture device.

[0086] Ophthalmic devices can use estimated eye fixation information to provide feedback to the device operator regarding whether the patient was fixating (or not properly fixating) a particular optical axis of the diagnostic device during a procedure (e.g., a measurement procedure). The ophthalmic device can use the estimated eye fixation information during the procedure to identify time periods when the patient was properly fixating. The system can also use the estimated eye fixation information, at least in part based on determining whether the patient was fixating during data acquisition, to determine whether the data acquired during the procedure is reliable and / or unreliable.

[0087] See Figure 6Embodiments of a method 600 for estimating absolute eye fixation will now be described. Method 600 is performed using a computing device and an imaging system that may include a camera for imaging the surface of a patient's eyeball and an illumination system (e.g., Figure 1 The method utilizes the position of detectable features of the eyeball in an image (e.g., pupil, limbus, iris features, etc.) and the position of the reflection of the illumination system at the cornea to determine the position and orientation of the eyeball. The eyeball position is determined during the procedure, or at other times when the patient is expected to properly position and gaze with reference to the optical axis of the ophthalmic device. The operator may initiate the procedure and / or the sequence may be started by providing feedback (e.g., by pressing one or more buttons), which the patient then follows. Optionally, the operator may provide confirmation of the patient's compliance with the procedure.

[0088] Method 600 illustrates an embodiment implemented by a computing device of an ophthalmic apparatus that may include a retinal OCT imaging device. To determine the absolute gaze position, the computing system determines whether foveal detection information is available (step 602). For example, if the ophthalmic apparatus includes a retinal imaging device that scans the patient's eyeball when the patient is properly gazed, the foveal detection information may be available. If the foveal detection information is available, the method proceeds to step 604, where the computing system identifies an eye-tracking image corresponding to the detected foveal data (e.g., see above). Figures 1 to 5 (As described). In step 606, these corresponding images are used to calculate absolute fixation parameters. The eye-tracking images and fixation parameters can then be used during the procedure to track the patient's eyes.

[0089] Returning to step 602, if foveal detection is unavailable, the method uses captured eye images (e.g., images of the eye surface) to estimate absolute fixation parameters. In step 620, the computing device receives the captured image stream from the camera and determines the eye position and orientation in each of the multiple images. The computing device can process each received image or a subset of the received images (e.g., according to processing constraints). Images can be received before / during and / or after the procedure when analyzing the captured data.

[0090] After determining the position and orientation of the eyeballs for a series of captured images, a histogram of the determined position and orientation is generated in step 630. In some embodiments, the position and orientation information includes the pixel position of the pupil center in each image, which is used to construct a two-dimensional histogram of (x, y) coordinates. The position and orientation information may include the absolute position and orientation of the eyeballs determined from each image, which is used to construct the two-dimensional histogram. Other representations of the position and orientation data (e.g., heatmaps) may also be used in this method. In some embodiments, operator feedback may be used to indicate images in which the patient has been eyeing, and / or to indicate whether the patient has not yet eyeed, and the corresponding images may be added to or discarded from the analysis. A procedure may be performed whereby the operator of the system instructs the patient to eyeballs during the measurement sequence.

[0091] See Figure 7 A heatmap 700 is shown, illustrating an exemplary distribution of the points of fixation being looked at by the patient. This graph may be color-coded, three-dimensional, or otherwise include markers for tracking the frequency with which the patient fixates on certain spots. Other indicators (e.g., colors close to the background color) may be used to indicate short-term fixation on the spot. In the illustrated embodiment, region 710 of the heatmap shows the most common coordinates and may indicate the position and orientation of the patient's eyeball when properly fixating on the target object. The position and orientation indicated by the dashed circles 720 will deviate from a threshold amount selected for fixation determination, depending on the level of precision required by the procedure or analysis.

[0092] Figure 8 An exemplary histogram 800 is shown plotting the eye coordinates detected from a captured image. The maximum value of this distribution 810 can be used to estimate the position and orientation of the eye during fixation (e.g., by identifying the position and orientation when the patient most frequently fixates). This estimated position and orientation can be used as a reference position for further eye fixation determination. For example, analysis of medical data acquired in a measurement sequence can use only data points acquired when the orientation and position of the eye are within an acceptable deviation from a reference position (e.g., based at least in part on the maximum value of the histogram) (e.g., as shown in circle 820).

[0093] As previously discussed, histogram 800 can be constructed by plotting fixation points determined from the captured images. For example, the histogram can track eye position as a series of pixel locations of the detected pupil or otherwise identified center of the eye (e.g., determined based on reflectance or other measurements). As the image sequence is received and analyzed, a pattern indicating the location where the patient most frequently fixates can be displayed. In some embodiments, the values ​​in the histogram may include the average of neighboring pixels and / or incorporate other smoothing.

[0094] See back Figure 6 In method 600, in step 640, a histogram is analyzed to detect gaze location. As previously discussed, gaze location can be related to the maximum value of the histogram that satisfies a specific analytical metric. For example, the maximum value can be selected based on a variety of factors, including the extent to which the maximum value exceeds the average, the extent to which it exceeds a threshold for a given number of images, etc. In some embodiments, eye tracking continues during the procedure and the maximum value / gaze location can be updated in real time as more images are analyzed.

[0095] Referring to step 650, if no acceptable maximum value (or other fixation metric) is found, eye fixation information cannot be obtained through this process. In some embodiments, eye tracking continues during the procedure and the maximum value / fixation location can be identified and / or updated in real time as more images are analyzed.

[0096] In step 660, estimated gaze parameters (e.g., procedurally acceptable gaze position and deviation radius) are determined based on the detected gaze information. The patient's eyes can then be tracked during the procedure using the eye-tracking images and the estimated gaze parameters in step 608.

[0097] See Figure 9 Now we will discuss the implementation Figures 6 to 8 An exemplary system 900 of the method. Computing device 902 (e.g., Figure 4 The computing device 410 is communicatively connected to the ophthalmic device 960 and configured to perform processes associated with the eye tracker 930 and the ophthalmic program 940. The computing device 902 can be configured to perform retinal image analysis 910 and store retinal image data 912 using the retinal imaging device of the ophthalmic device 960 (if available). The computing device 902 further includes a gaze analysis module 920 for performing... Figure 6 The method is implemented as follows. In one embodiment, the gaze analysis module 920 receives and analyzes an eye image stream captured and stored (e.g., in storage device 932) by the eye tracker 930, constructs and analyzes a histogram of the gaze position, and determines a reference position and associated radius. The gaze data (including histogram data) may be stored in storage device 922 (e.g., a memory or storage device).

[0098] In some embodiments, the computing device 902 includes a processor configured to execute program instructions stored in a memory, the processor including a gaze analysis module 920, an optional retinal image analysis module 910, an eye tracker 930, and a process 940 associated with an ophthalmological program.

[0099] The gaze analysis module 920 can be configured to analyze the relative gaze of a patient's eyes using images captured by the eye tracker 930. The gaze analysis module 920 can construct a histogram tracking gaze orientation (e.g., vertical and horizontal eye movements and yaws, relative upward / downward and left / right deviations, curvature / rotation, etc.) and analyze the peak values ​​of the histogram (e.g., the number of data values ​​at each location) to obtain an estimate of an absolute reference value. In some embodiments, the gaze analysis module 920 estimates the optical axis of the eye and its intersection with the eye tracker camera to track gaze orientation.

[0100] Eye tracker module 930 can be configured to capture, store, and process images of a patient's eyes. Eye tracker module 930 can be configured to determine the position and orientation of the patient's eyes based on the captured images for further analysis by gaze analysis module 920. In some embodiments, each image analyzed may include x, y positions representing the eye's position and orientation (e.g., rotation about the x and y axes). The eye tracker may use information about relative orientation changes from one image to another by combining absolute gaze position (e.g., determined by retinal image analysis 910) or estimated absolute gaze position (e.g., determined by gaze analysis module 920). In some embodiments, gaze analysis module 920 operates based on the assumption that the patient attempts to gaze most of the time and that the estimated absolute gaze position can be determined by constructing a histogram of x and y rotations and identifying the most dominant gaze orientation. In several different embodiments, the histogram may be constructed from pixel coordinates, rotation about x and / or y, deviation values, or other data. Each image may provide a coordinate pair representing the calculated eye gaze orientation, which is added to the histogram.

[0101] In some embodiments, the gaze analysis module 920 is configured to analyze a histogram by detecting a distinct peak (e.g., a main peak surrounded by multiple smaller peaks) and determining a confidence level for a gaze location that has been detected. If no significant peak is detected, the confidence level may be low. The radius around the detected peak can be used (e.g., humans can gaze at ±0.5 degrees). The threshold for peak-to-mean ratio and / or the size of the radius can be varied depending on system and program requirements.

[0102] The computing device 902 may include one or more neural networks trained to make one or more determinations disclosed herein, including analyzing histogram data to determine whether an eye gaze position can be determined. In some embodiments, gaze analysis may further include comparing known eye-tracking images and / or eye gaze parameters for the patient and / or other patients. For example, one or more images may be fed into a neural network trained using historical data to determine whether the eyes in the images are gazing.

[0103] In several different embodiments, feedback on whether the patient is gazing at this axis can be provided to the operator during data acquisition, even when retinal imaging data is unavailable (e.g., not part of the system and / or foveal detection is unavailable before the procedure). The systems and methods disclosed herein provide a cost-effective solution suitable for use with ophthalmic diagnostic devices that use image capture devices and illumination systems as described herein.

[0104] Those skilled in the art will understand that the methods of the illustrated embodiments provide an improved technique for independently verifying whether a patient's eye is properly fixating on a target object during operation. By detecting the fovea at specific points in time, the system can determine where the patient's gaze / visual axis is located. This information allows the system to determine whether the patient is currently fixating during a measurement sequence or other diagnostic or corrective procedure. This method combines a system for imaging the retina with a system for tracking the eye using surface information. Based on the foveal position in the retinal image, the system can determine the eye-tracking position and whether the eye is moving left or right / up or down. According to this disclosure, the system can track the user's gaze, calculate the amount of deviation, determine the current eye position and orientation, make determinations about eye fixation, determine data validity, and provide other features.

[0105] The methods according to the above embodiments can be implemented as executable instructions stored on a non-transitory tangible machine-readable medium. These executable instructions, when run by one or more processors (e.g., processor 412), can cause the one or more processors to perform the processes of methods 500, 600, or one or more other processes disclosed herein. Apparatus for implementing the methods according to these disclosures can include hardware, firmware, and / or software, and can take any of a variety of form factors. Typical examples of such form factors include laptop computers, smartphones, minicomputers, personal digital assistants, etc. A portion of the functionality described herein can also be embodied in peripheral devices and / or add-on cards. By further example, such functionality can also be implemented on a circuit board between different chips in a single device or between different processes executed therein.

[0106] While illustrative embodiments have been shown and described, various modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without correspondingly using other features. Many variations, alternatives, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be interpreted broadly in accordance with the scope of the embodiments disclosed herein.

Claims

1. An eye-tracking system, comprising: An eye tracker configured to capture a first plurality of images of the eye; as well as Control processor, the control processor being configured to: Detect the eye position and orientation in each of the first plurality of images; Determine the eye gaze position and orientation relative to the optical axis of the eye tracker; Eye fixation parameters are estimated at least in part based on the determined eye fixation location and orientation; as well as Eye position and orientation are tracked by analyzing one or more of the first plurality of images to determine eye position and orientation relative to eye gaze parameters. The control processor is further configured to detect the gaze position relative to the optical axis of the eye tracker by constructing and analyzing a histogram of the detected eye position and orientation, wherein the histogram is a two-dimensional histogram constructed using the pixel position of the pupil center in each of the first plurality of images.

2. The system as claimed in claim 1, wherein, The eye fixation parameters include the reference position and orientation of the eyeball during fixation.

3. The system as described in claim 1, wherein, Analyzing the histogram further includes determining the relative maximum value and determining whether the coordinates of the relative maximum value contain the gaze position and orientation.

4. The system as described in claim 3, wherein, Determining whether the coordinates of the relative maximum value include gaze position and orientation further includes comparing the relative maximum value with a threshold and / or the average coordinate values ​​of the histogram.

5. The system of claim 1, further comprising: A retinal imaging system comprising an optical coherence tomography (OCT) scanner and / or a fundus camera configured to perform retinal scans.

6. The system of claim 5, wherein, The retinal imaging system is configured to capture a second plurality of images of the eye; Detect whether the central fovea exists in one or more of the second plurality of images; Identify a first image with the detected central fovea from the second plurality of images; Determine a second image from the plurality of images that is temporally close to the first image; as well as The second image is analyzed to determine eye fixation parameters.

7. The system as claimed in claim 1, wherein, The control processor is configured to track the eye position and orientation and calculate the deviation from the eye gaze parameters and determine whether the deviation is less than a threshold. Wherein, when the deviation is less than the threshold, it is determined that the eye is fixating, and the control processor generates a fixation indication; and Specifically, when the deviation exceeds a threshold, it is determined that the eye is not aligned, and the control processor generates a non-gazing indication.

8. The system as claimed in claim 1, wherein, The control processor is further configured to perform an eye diagnostic procedure and use the eye tracker to track the position of the eye during the eye diagnostic procedure.

9. The system of claim 1, further comprising a diagnostic device configured to perform an eye diagnostic procedure while using an image capture device to track the position and orientation of the eyeball; in, The diagnostic device is further configured to receive the data representing gaze and eye position based at least in part on the data representing gaze and eye position during the eye diagnostic procedure.

10. An eye-tracking method, the method comprising: Capture the first multiple images of the eye using an eye tracker; Detect the eye position and orientation in each of the first plurality of images; Determine the eye gaze position and orientation relative to the optical axis of the eye tracker; Eye fixation parameters are estimated at least in part based on the determined eye fixation location and orientation; as well as Eye position and orientation are tracked by analyzing one or more of the first plurality of images to determine the eye position and orientation relative to eye gaze parameters. The method further includes detecting the gaze position relative to the optical axis of the eye tracker by constructing and analyzing a histogram of the detected eye position and orientation, wherein the histogram is a two-dimensional histogram constructed using the pixel position of the pupil center in each of the first plurality of images.

11. The method of claim 10, further comprising training a neural network to receive the first plurality of images and output a determination of eye position.

12. The method of claim 10, wherein the eye fixation parameters include the reference position and orientation of the eye during fixation.

13. The method of claim 10, wherein analyzing the histogram further includes determining a relative maximum value.

14. The method of claim 10, further comprising: The retinal imaging of the eyeball was performed using an optical coherence tomography (OCT) scanner.

15. The method of claim 14, further comprising: A second plurality of images of the eyeball are captured by the retinal imaging scan; Detect whether the central fovea exists in one or more of the second plurality of images; Identify a first image with the detected central fovea from the second plurality of images; Determine a second image from the plurality of images that is temporally close to the first image; as well as The second image is analyzed to determine eye fixation parameters.

16. The method of claim 10, further comprising tracking the eye position and orientation and calculating a deviation from the eye fixation parameter and determining whether the deviation is less than a threshold; in, When the deviation is less than the threshold, it is determined that the eye is gazing, and a gaze indication is generated; as well as When the deviation exceeds a threshold, it is determined that the eye is not aligned and a non-focusing indication is generated.

17. The method of claim 10, further comprising performing an eye diagnostic procedure and using an eye tracker to track the position of the eye during the eye diagnostic procedure.

18. The method of claim 10, further comprising performing an eye diagnostic procedure while using an image capture device to track the position and orientation of the eyeball; and The eye diagnostic procedure is modified, at least in part, based on data representing eye fixation parameters and the tracked eye position.

Images