Limbus segment tracking to quantify full range of ocular movements

Abstract

A computer-based information processing system, and a computer-implemented method, and a non-transitory computer-readable storage medium, for automatically tracking eye rotation visible in captured images. The method, with a processing system, memory, a camera system, and a neural network, captures images of a patient's eye in various gazes, including a straight ahead gaze and at least one "as far as possible" gaze. The neural network segments the captured images into areas corresponding to the iris and the sclera, and performs feature extraction to identify the shape and location of visible limbus boundaries in each of the captured images. The shape and location of visible limbus boundaries are mapped to elliptical segments. The direction and magnitude of how far the eye has rotated is estimated based on a comparison of the reference ellipse and the at least one observed elliptical segment.

Description

[0001] LIMBUS SEGMENT TRACKING TO QUANTIFY FULL RANGE OF OCULAR MOVEMENTS

[0002] BACKGROUND

[0003] The present application generally relates to methods of determining a patient's ocular range of motion and, more specifically, to a system and method for objectively and automatically quantifying measurements of the ocular range of motion.

[0004] The human eye sits in a protective bony socket called the orbit. Six extra-ocular muscles in the orbit are attached to the eye. These muscles rotate the eye up and down, side to side, and clockwise and counterclockwise. The extraocular muscles are attached to the white part of the eye called the sclera. This is a strong layer of tissue that covers nearly the entire surface of the eyeball.

[0005] Light is focused into the eye through the clear, dome-shaped front portion of the eye called the cornea. Behind the cornea is the eye’s iris (the colored part of the eye) and the dark hole in the middle called the pupil. Muscles in the iris dilate (widen) or constrict (narrow) the pupil to control the light reaching the back of the eye.

[0006] The ocular range of motion is ty pically assessed by an eye care professional watching how far a patient’s eye can move. Movements that "‘stop too soon” or "go too far” are taken to indicate abnormal eye muscles. This approach is valuable but does not document the exact amount of ocular movement. While valuable, direct observation by an eye care professional is a subjective approach and does not allow ocular rotations to be quantified.

[0007] BRIEF SUMMARY

[0008] To address the problems of previous methods of determining the ocular range of motion of a patient, the inventor developed a system and method to automatically measure the ocular range of motion so that the eye care professional has a tool capable of evaluating eye muscle function before, during, and after both pathologic events and / or therapeutic interventions.

[0009] Specifically, according to various embodiments, the inventor created a novel imageprocessing method, limbus segment tracking (LST), to address the failings of the previous methods.

[0010] The limbus of the eye is a ring-shaped (elliptical) area around 1-2 mm wide that marks the transition between the cornea and sclera. The limbus provides a structural barrier between the cornea and sclera. The limbus starts off transparent by the cornea and becomes opaque near the sclera. Behind the cornea is the eye’s iris (the colored part of the eye). The inventor has discovered that the limbus ring-shaped (elliptical) area between the iris (the colored part of the eye) and the sclera (the white part of the eye), or the segment(s) thereof that is / are visibly exposed during various gazes of the eye, can be optically tracked real-time by a machine vision system or can be analyzed in captured images of the eye, to reliably estimate the location of the visible limbus segment as the eye moves.

[0011] Limbus segment tracking (LST) is built on the premise that the location and shape of the corneoscleral limbus changes predictably with ocular rotations. Thus, there is an infinite number of ellipses, and related elliptical segments, each linked to specific magnitudes and directions of ocular rotation. This makes it possible to know where and how far the eye has moved by comparing any two limbal ellipses or limbal elliptical segments. Using the limbus as a proxy for ocular rotation expands the recording range beyond that of conventional trackers of eye rotation because a portion (i.e., segment) of the limbus is typically visible no matter how far the eye moves. The use of either limbal ellipses or limbal elliptical segments as proxies for ocular rotation imparts flexibility to this approach because the entire limbus need not be visible during tracking.

[0012] Features and advantages of the various embodiments will become readily apparent from the following description and accompanying drawings. Certain preferred embodiments of the invention and their benefits will also become more apparent to a person of ordinary skill in the art through the description and selected examples given herein below, as well as through the appended claims.

[0013] All references, publications, patents, and patent applications, cited herein and / or cited in any accompanying Information Disclosure Statement (IDS), are hereby incorporated herein by reference in their entirety for all purposes.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 is a front planar view of a human eye highlighting certain features thereof.

[0016] FIG. 2 illustrates several eye rotations that can be analyzed and tracked by identifying and tracking visible limbus segments according to various embodiments.

[0017] FIG. 3 is an illustration of example steps in limbus segment tracking (LST) algorithms that can be followed in real-time by a machine vision system or by a computer-based information processing system analyzing captured images of the eye, relying on tracking the visible limbus segments of the eye during various eye rotations, according to various embodiments.

[0018] FIG. 4 is a pair of charts illustrating a comparison of estimated and actual rotation magnitudes and directions across all possible gaze locations using simulations of limbus segments representative of a full range of ocular movements, in which limbus segment tracking (LST) algorithms were used to estimate movement parameters from the simulated limbus segments. FIG. 5 illustrates the test results of an example LST ocular motility assessment comparing estimated ocular ranges of motion from human eyes with and without movement limitations.

[0019] FIGs. 6 and 7 are flow diagrams illustrating example methods of quantifying the magnitude and direction of ocular rotation, wherein a first example method is based on LST algorithms operating in a computer-based information processing system, and wherein a second example is based on mapping locations and shapes of observed segments followed by calibrations to predict magnitude and direction given the observed locations and shapes, according to various embodiments.

[0020] FIG. 8 is a block diagram illustrating an example of a computer-based information processing system suitable for use with systems and methods of disclosure as shown in the previous figures, according to various embodiments of the invention

[0021] DETAILED DESCRIPTION

[0022] To promote an understanding of the principles of the invention, reference will now be made to the embodiments illustrated herein, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Any alterations and further modifications in the described systems and methods for using limbus segment tracking (LST) to estimate eye movement parameters from eye images are contemplated as would usually occur to one of ordinary skill in the art to which the invention relates.

[0023] Introduction

[0024] As illustrated in FIG. 1, the limbus 104 of the eye 102 is a ring-shaped (elliptical) area around 1-2 mm wide that marks the transition between the cornea 108 and sclera 106. The limbus 104 provides a structural barrier between the cornea 108 and sclera 106. The limbus 104 starts off transparent by the cornea 108 and becomes opaque near the sclera 106. Behind the cornea 108 is the eye’s iris (the colored part of the eye). For the present discussion the iris and cornea are both referred to as the iris 108. The inventor has discovered that the limbus 104 ring-shaped (elliptical) area between the iris 108 (the colored part of the eye) and the sclera 106 (the white part of the eye), or the segment(s) of the limbus 104 that is / are visibly exposed during various gazes of the eye 102, can be optically tracked real-time by a machine vision system or can be analyzed in captured images of the eye 102, to reliably estimate the location of the visible limbus segment 104 as the eye 102 moves (e.g.. rotates). Limbus segment tracking (LST) is built on the premise that the apparent location and shape of the corneoscleral limbus 104 change predictably with ocular rotations (i.e., rotations of the eye 102) when imaged with a stationary frontoparallel camera (i.e., in planar view). Eye movements (rotations) are produced by the coordinated action of six extraocular muscles in each eye 102. Optometrists and ophthalmologists use ocular motility examinations to evaluate the function of each muscle. During examination, to assess the ocular range of motion, clinicians ask patients to “follow a light’’ in several directions while the extent of ocular movement is physically observed by the clinician. Clinicians move the light until ocular movement ceases. This indicates that maximum ocular rotation has been achieved. Movements that “stop too soon” or “continue too far” are taken to indicate abnormal muscle function, which itself can signal the presence of a compressive, ischemic, or vascular disease. This practice leaves the actual magnitude and direction of ocular rotations unknown because the extent of ocular movement is not routinely quantified. Instead, evaluating the ocular range of motion relies entirely upon qualitative scales that summarize results at a gross level (i.e., smooth and full movements). This kind of assessment stands in contrast to the plethora of micron-level measurements utilized to evaluate the health of the cornea, retina, and optic nen e.

[0025] There is a need to quantify the ocular range of motion because the results of motility examinations are used to gauge both pathological progression and therapeutic improvement of eye 102 movement disorders. Better tools and techniques in oculomotor assessment will improve the care and study of patients with a multitude of clinical conditions because changes in ocular motility often accompany systemic disorders (e.g., Grave’s disease, multiple sclerosis, myasthenia gravis, and Parkinson’s disease).

[0026] Quantifying the ocular range of motion is difficult because maximum ocular rotations are large. The normal limits of ocular movement are approximately 30 degrees for upward movements and 45 degrees for horizontal and downward movements. These movement magnitudes meet or exceed the recording range of most eye trackers (i.e., ~ 30°) which compare the location of the pupil and comeal reflection (P-CR) to estimate e e position. A technique called the scleral search coil can provide high resolution measurements for eye movements of any size; however, it is typically relegated to laboratory' use because it is expensive and requires the patient to wear an uncomfortable rigid lens. The open methodological space has gradually been filled by “chair-side” techniques in which clinicians use the relative location of the limbus, the boundary' between the “colored” and “white” parts of the eyes (i.e., the iris and sclera), to quantify ocular rotations one movement at a time. For example, in one limbus test, movement of the limbus is used to estimate the amount of ocular rotation. This is done by first placing a ruler against the eye and then comparing the location of the limbus when a patient is looking “straight ahead” versus when they' are looking “as far as possible” in one direction. The lateral version light reflex test created uses a similar principle: It judges the location of the “straight ahead” and “as far as possible” limbal edges to the position of a stationary’ light to quantify the amount of ocular rotation. For both tests, the limbus is an ideal structure for movement estimation because it is easily visualized and only moves when the eye moves. Each “chair-side” technique’s step towards quantification represents an improvement over qualitative assessment. Still, these techniques can be dexterous and time consuming, ultimately limiting the number of movements which can be examined, as measuring movements in different directions requires manually and repeatedly measuring the relative location of the limbus.

[0027] Limbus segment tracking (LST)

[0028] To address the unmet needs posed by currently available approaches, the inventor proposes to use limbus segment tracking (LST) as a method to automatically quantify the magnitude and direction of any ocular rotation. LST is built upon the premise that both the location and shape of the limbus (previous methods omit consideration of the limbal shape) change predictably with ocular rotations. This occurs because, when viewed from a camera, movement of the eye progressively decenters the limbus and makes it appear more elliptical. This makes it possible to know where and how far the eye has moved by comparing any two limbal ellipses. The elliptical comparisons relieve the recording range restriction imposed by conventional eye trackers because a portion of the limbus is typically visible no matter how far the eye moves. The use of either limbal ellipses or limbal elliptical segments as proxies for ocular rotation imparts flexibility7to this approach because the entire limbus need not be visible during tracking. This, in turn, allows LST to be used for ocular motility assessments in patients with palpebral apertures of varying sizes.

[0029] The rows 202, 204, 206, shown in FIG. 2 illustrate certain steps in an example LST method used to quantify7ocular rotations. It should be noted that the ophthalmic speculum visible in the three rows 202, 204, 206, is intended for use in basic scientific investigations. Clinicians can manually lift the eyelids instead.

[0030] The columns 208, 210, 212, 214, 216, illustrate several different gazes of a right eye in captured images. The first (left-most) column 208 shows the straight ahead “reference’’ gaze of a patient's eye. The second column 210 shows an “as far as possible” gaze to the right. The third column 212 shows an “as far as possible” gaze straight up. The fourth column 214 shows an “as far as possible” gaze up and left. Lastly, the fifth column 216 shows an “as far as possible” gaze to the left.

[0031] The example LST method begins by capturing images of the eye, as shown in the first (upper-most) row 202 in FIG. 2. Next, as illustrated in the second row7204, a machine learning process is used to partition the captured images into areas corresponding to the iris 108 and sclera 106. This segmentation leads to further processing by computational algorithms to extract from the captured images the visible limbus boundaries 222, 224, 226, 228, 230, as shown in the third row 206 in FIG. 2. Limbal segments 222, 224, 226, 228, 230, are extracted by finding pixels adjacent to both the iris 108 and the sclera 106.

[0032] The limbus boundaries obtained when looking “straight ahead” and “as far as possible” are then used to define a “reference” ellipse 222 mapped to the visible limbal segments (in the first column 208) and “observed” elliptical segments mapped to the visible limbal segments (in the remaining columns 224, 226, 228, 230) whose locations and shapes represent minimal and maximal ocular movement, respectively. Lastly, as illustrated in FIG. 3 and further discussed below, the “observed” limbus segments 324, 326, 328, 330, are compared to the reference ellipse 322 to estimate the magnitude and direction 325, 327, 329, 331, of eye rotation.

[0033] The mathematical comparison is performed through an iterative process called optimization. The computer processor repeatedly rotates the reference ellipse 322 by different magnitudes and directions. This rotation produces a new “candidate” ellipse each time the computer tries a different rotation. Movement estimation is performed by the computer “launching” the reference ellipse 322 towards each observed limbus segment 324, 326, 328, 330. The computer processor compares 342, 344, 346. 348, the “candidate” ellipses to the respective “observed” elliptical segments 324. 326, 328, 330. to find a best match between the shape and location of the “observed” elliptical segments 324, 326, 328, 330 and the shape and location of the “candidate” ellipses 342, 344, 346, 348. The shape and location of arc-length matched segments (shaded regions) 343, 345, 347, 349, from the “candidate” ellipses 342, 344, 346, 348 and the “observ ed” elliptical segments 324, 326, 328. 330, are compared to find this best match.

[0034] The process concludes by the computer processor determining the “best matching candidate ellipses” 352, 354, 356, 358, shown in the third row in FIG. 3, whose locations and shapes are most similar to the “observed” elliptical segments 324, 326, 328, 330. Note that portions of the “best matching candidate ellipses” 352, 354, 356, 358 are obscured by the “observed” elliptical segments in FIG. 3.

[0035] Estimated ocular rotation magnitudes and directions, to be used for ocular motility assessment, are derived from the rotation parameters which created the best matching candidate ellipses 352, 354, 356, 358, and thus equivalently the rotation parameters which “launched” the reference ellipse 322 closest to the “observed” elliptical segments 324. 326, 328, 330. The above-described processing pipeline allows recordings to be performed in individuals of any age or level of vision because movements are estimated directly from images of the e e without a preceding calibration sequence.

[0036] Calibration 615 (see FIG. 7) is compatible with, but not necessary for, LST’s estimation of rotation parameters from limbus segments. Utilization of a calibration sequence involves mapping the location and shapes of a limited number of “observed” elliptical segments 324, 326, 328, 330 to a limited number of known fixation locations in physical space and / or on a computer display prior to ey e tracking. This process allows “live” extraction of elliptical locations and shapes and “live” mapping of said segments to known locations in physical space and / or on a computer. This means of estimating rotation parameters is most ideal for reducing processing time but requires accurate fixation be exerted by the patient.

[0037] FIG. 4 shows a comparison of estimated and actual rotation magnitudes and directions across all possible gaze locations using simulations of limbus segments representative of a full range of ocular movements, and limbus segment tracking (LST) algorithms used to estimate movement parameters from images. LST segment comparison achieved near perfect accuracy when comparing estimated and actual rotation magnitudes (A) 402, 406 and directions (B) 404, 408 across all possible gaze locations [quantified as gaze magnitudes (x-axis) and directions (y-axis)]. The LST algorithms directly estimate movement parameters from images, which obviates the need for device calibration and thus can quantify ocular motility in individuals of any age and with any level of vision. As stated above, calibration is compatible with, but not necessary’ for, LST’s estimation of the ocular range of motion.

[0038] FIG. 5 shows an example set of LST estimated ranges of motion derived from right eyes with clinically apparent movement limitations 502 and without clinically apparent movement limitations 504. The range of motion (shaded grey area) is derived by estimating how far each eye can move in different directions. Each movement is indicated by a vector arrow’, with the exact movement magnitudes and directions indicted in the format magnitude @ direction. The boundaries of the iris 108, sclera 106, and limbus 104, used to estimate the eye movements are shown adjacent to their respective movements. The LST estimated pattern of movement of the eye 102 with a known movement disorder 502 show s a smaller (limited) range of motion (magnitude and direction of eye rotation) than the LST estimated pattern of movement of the normal eye 504, with the normal eye’s range of motion (magnitude and direction of eye rotation).

[0039] FIG. 6 is a flow diagram illustrating an example method of quantify ing magnitude and direction of ocular rotation based on LST algorithms operating in a computer-based information processing system, according to various embodiments.

[0040] Optional Preparation Before Performing the Method of FIG. 6

[0041] As a first step toward preparing for performing the method of FIG. 6, a computer-based information processing system can be operated under guidance from an expert reviewer (e.g., an eye care professional) to review images of different eyes in various different gazes and label (annotate) the images to manually delineate the iris area and the sclera area in a collection of, for example, 1,000+ gaze-sorted (i.e., “straight”, “up”, “down”, “left”, etc.) images. The collection of annotated images becomes a training set for a machine learning system. For example, and not for limitation, a neural network set of software tools and libraries can be created using commercially available MATLAB’s (Matrix Laboratory) Computer Vision and Deep Learning toolboxes. Next, the annotated images are “fed” into a “naive” neural network to help it learn the relevant portions of each image (e.g., the iris 108 and the sclera 106) and optionally also the respective gaze of the eye in each image.

[0042] Training sessions, which involve the neural network “looking through” all the annotated images, enable learning by letting the neural network continually adjust its connections to best “highlight” the iris area and the sclera area on a pixel-by -pixel level.

[0043] Next, an independent test set of gaze-sorted and annotated images (e.g., 250+ images) can be “fed” into the neural netw ork to assess the post-training accuracy and precision of the neural netw ork. The independent test set contains images the neural network has not yet been exposed to. Gaze positions containing images with less than 90 percent accuracy or precision will undergo targeted refinement through further annotation, training, and testing. The network exhibiting the highest levels of accuracy and precision for all gaze positions will be selected to perform image processing in the next step, based on the method shown in FIG. 6. It should be noted that once a neural network, or other machine learning system, is trained and ready for use. there will be no need for the preparation discussed above.

[0044] Referring now to FIG. 6, the example method is entered, at step 602, by a computer-based information processing system which can be coupled to a machine vision system or a camerabased image capture system. The computer-based information processing system, according to the example, comprises a machine learning artificial intelligence system (e.g.. aneural network as discussed above). In the following discussion of this method it is assumed that the machine learning artificial intelligence system (e.g., the neural network) has been trained for reliably identifying the iris and the sclera in an image of an eye.

[0045] The method then immediately proceeds to capture, at step 604, one or more images of a patient’s eye in various gazes. Then, the neural network, using a set of trained classifiers, partitions the images of the patient’s eye into areas corresponding to the iris and the sclera, and identifies the iris area and the sclera area in each image.

[0046] The neural network performs, at step 608. feature extraction to identify the limbus boundaries in each of the images; the limbus boundaries define a “reference” ellipse for the “straight ahead” gaze, and “observed elliptical segments” for the “as far as possible” gazes whose locations and shapes represent minimal and maximal ocular movement, respectively.

[0047] If an optional calibration is not done, at step 609, according to certain embodiments the computer-based information processing system, at step 610, performs mathematical comparison of the “reference” ellipse and “observed elliptical segments” through an iterative process of optimization to estimate where and how far the eye moved (rotated) relative to the “reference"’ ellipse. As has been discussed above, the computer-based information processing system repeatedly rotates the “reference” ellipse by different magnitudes and directions. Each different rotation produces a new “candidate” ellipse. The process concludes, and magnitude and direction of ocular rotation of the eye is quantified, when a “candidate” ellipse containing a piece very similar to the observed limbus segment is found.

[0048] Optionally, if calibration is to be done, at step 609, then rotation estimation via calibration, at steps 611, 615,613, is performed.

[0049] The method then exits at step 612.

[0050] The above method can be quickly and accurately repeated for a very' large number of images of an eye which are processed by the computer-based information processing system. As has been discussed above, this novel method can be quickly and accurately performed by an automated machine vision system or by a computer-based information processing system operating to automatically process images of eyes. However, this method cannot be performed manually by clinicians due to at least the very complicated multi-step image processing analysis of a large number of images of eye rotations to identify the limbus segment of the eye while in various gazes.

[0051] FIG. 8 is a block diagram illustrating an example of a computer-based information processing system suitable for use with systems and methods of the disclosure as shown in the previous figures, such as with the methods shown in FIGs. 6 and 7, according to various embodiments of the invention.

[0052] Regarding FIG. 8. an example of a computer-based information processing system 700 includes various components. A processor 702 executes instructions 707 that cause the information processing system 700 to perform operations according to various embodiments of the invention. The processor 702, in this example, is communicatively coupled with various other components of the information processing system 700 via a system bus 708. Main memory 704 contains instructions 707, which can include computer instructions, configuration parameters, and data used by processor 702. Persistent memory 706 can store the instructions 707 in persistent storage for the processor 702. Additionally, configuration data 752 can be used by processor 702 to configure the computer-based information processing system 700, and various interoperating components of the information processing system 700.

[0053] A user interface 710 includes a user output interface 712 and a user input interface 714 for communicating with the user (e.g., an operator or other technical personnel such an eye care professional) using the information processing system 700. The user output interface 712 includes various output devices, such as a computer display device, indicator lights, a speaker that generates sound output to a user, or a data output interface device that can provide data and control signals to a user that comprises a computer system.

[0054] The user input interface 714 can include various input devices such as a computer keyboard, mouse device, touch screen display, a microphone that receives sound input signals from a user. The received sound signals, for example, can be converted to an electronic digital representation and stored in memory, and optionally can be used with voice recognition software executed by the processor 702 to receive user input data and commands. The user input interface 714 can include a data input interface device that can receive data and control signals from a user that comprises a computer system, such as a remote user device 770 or a remote server 772.

[0055] According to the example, a models database 716 and a captured images database 718 can be communicatively coupled with the processor 702. The processor 702 interoperates with a models and an eye inspection process controller 740 to perform the novel method according to various embodiments of the invention. See, for example, the method shown in FIG. 6 and the associated discussion above.

[0056] A machine-learning artificial intelligence (ML / Al) system is communicatively coupled with the information processing system 700 and can be used by the eye image analyzer 742, for example, to analyze the captured images of eyes and thereby identify the iris and sclera portions of the eye, and accordingly extract and identify' the limbus segment visible in each image of an eye.

[0057] Such an ML / Al system can include, according to various embodiments, a set of classifiers 720, 748. that are used to extract certain features (e.g., using the feature set values extractor 744 in the instruction 707) of eyes from the captured images 718, 726. Classifiers 720 can also include functions and operations stored as a collection of classifiers 748 in instructions 707. The ML / Al system trains the classifiers 720 with a classifier builder 746 and training data set 722 that includes annotated images of certain eyes that identify the iris and the sclera in the eye. In this example ML / Al system, each training data set image is labeled (annotated) by an expert such as an eye care professional. The ML / Al system receives captured images from a camera system (e.g., a machine visions system) and stores the information in a repository' as session data 726. The ML / Al system uses the trained classifiers 720, the decision rules 724, and the decision rules processor 750 to identify' individual areas, e.g., the iris, the sclera, and the limbus segment, in the captured images. The ML / Al system uses machine learning based artificial intelligence image recognition and identification of images of eyes, as has been discussed above.

[0058] The eye image analyzer 742 can compare identified features, such as iris area, sclera area, and limbus segment, in the captured image to expected (classified) features of eyes from the models database 716, which correspond to the identified features of the actual eye in the captured image.

[0059] Optionally, according to the example, the eye image analyzer 742 also uses time information from the captured images database 718 and the session data 726 which is associated with the expected individual limbus segment feature of eye in the models database 716. Each individual limbus segment feature of eyes, in this example, is associated with time information in the session data 726. According to the example, the time information defines points in time (time points) within spans of time (time spans) in an overall timeline associated with the rotation of the eye in the captured images. Each individual limbus segment feature captured from an image of an eye, therefore, can be associated with one or more time points, which indicate expected time(s) for eye motion (rotation) corresponding to a model in the models database 716. That is, besides the measurement of maximum and minimum rotation direction and magnitude of an eye rotation, according to the example, the velocity (direction and speed) of the eye rotation over time can be captured and analyzed by the eye image analyzer 742. This time information, in certain embodiments, is available to complement the eye rotation information measured by the novel method of FIGs. 6 and 7.

[0060] As shown in FIG. 8, the processor 702 can be communicatively coupled with a computer- readable medium 732. The computer-readable medium 732, according to the present example, is communicatively coupled to a reader / writer device 730, which is communicatively coupled via the system bus 708 to the processor 702.

[0061] The instructions 707, which can include computer instructions, configuration parameters, and data, can be stored in the computer-readable medium 732, the main memory 704, the persistent memory' 706, and the processor's internal memory' such as cache memory' and registers.

[0062] A network interface device 760 is communicatively coupled with the processor 702 and provides a communication interface for the information processing system 700 to communicate via one or more networks 762. The networks can include wired or wireless networks or a combination of both, and can be any of local area networks, wide area networks, or a combination of such networks. For example, wide area networks, including the Internet and the web, can intercommunicate the information processing system 700 with other information processing systems 770, 772, that may be locally or remotely located relative to the information processing system 700. It should be noted that mobile communications devices, such as mobile phones. Smartphones, tablet computers, lap top computers, and the like, which are capable of at least one of wired or wireless communication, are also examples of information processing systems within the scope of the present disclosure. As illustrated in FIG. 8, an eye inspection process controller 740 instructs the information processing system 700 how to inspect and track the eye rotation from the captured images. The eye inspection process controller 740 also interoperates with the eye image analyzer 742 to coordinate and perform aspects of the various embodiments of the invention.

[0063] All patents and publications mentioned in this specification are indicative of the levels of those of ordinary skill in the art to which the invention pertains. All patents and publications are herein incorporated by reference to the same extent as if each individual publication was specifically and individually indicated to be incorporated by reference.

[0064] Non-Limiting Examples

[0065] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The present invention may be implemented as a system and / or a method, at any possible technical detail level of integration.

[0066] Although the present specification may describe components and functions implemented in various examples with reference to particular standards and protocols, the invention is not limited to such standards and protocols. Each of the standards represents examples of the state of the art. Such standards are from time-to-time superseded by faster or more efficient equivalents having essentially the same functions.

[0067] The illustrations of examples described herein are intended to provide a general understanding of the structure of various embodiments, and they are not intended to serve as a complete description of all the elements and features of apparatus and systems that might make use of the structures described herein. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description and the following claims. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this invention. Figures are also merely representational and may not be drawn to scale. Certain proportions thereof may be exaggerated, while others may be minimized. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. For the sake of clarity and succinctness, the components and details which are not essential in order to explain the scope of the invention have been omitted in the drawings.

[0068] The Abstract is provided with the understanding that it is not intended be used to interpret or limit the scope or meaning of the claims. Headings are provided for the convenience of the reader, and are not intended to be limiting in any way. In addition, in the foregoing Detailed Description, various features are grouped together in a single example embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0069] As will be appreciated by one of ordinary skill in the art, in view of the discussions herein, aspects of the present invention may be embodied as a sy stem, method, or computer program product.

[0070] Accordingly, one or more aspects of the present invention may take the form of an entire hardware embodiment, an entire software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module", or "system". Furthermore, parts of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having the computer readable program code embodied thereon.

[0071] An information processing system may utilize any combination of computer-readable medium(s). The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the preceding.

[0072] More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory). an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the preceding. In the context of this document, a computer- readable storage medium is any tangible non-transitory medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0073] A computer-readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier w ave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is any computer-readable medium that is not a computer-readable storage medium. A computer-readable signal medium can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0074] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc.. or any suitable combination of the preceding. Computer program code for carry ing out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. According to various embodiments of the invention, the program code may execute entirely on a user's computer, partly on a user's computer, as a stand-alone software package, partly on a user's computer and partly on a remote computer or entirely on a remote computer or a server. In the latter scenario, the remote computer or the server may be connected to the user's computer through any type of network, including one or more of a local area network (LAN), a wireless communication network, a wide area network (WAN), or a connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0075] Aspects of the present invention have been discussed above with reference to flow diagram illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to various embodiments of the invention. It will be understood that each block of the flow diagram illustrations and / or block diagrams, and combinations of blocks in the flow diagram illustrations and in the block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general- purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flow diagram and / or block diagram block or blocks.

[0076] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices, to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0077] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, to cause operational steps to be performed on the computer, other programmable apparatus, or other devices, to produce a computer-implemented process (or method) such that the computer instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0078] The terminology used herein is to describe particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0079] As used herein, the term "about" or "approximately" applies to all numeric values, whether or not explicitly indicated. These terms generally refer to a range of numbers that one of skill in the art w ould consider equivalent to the recited values (i.e., having the same function or result). In many instances these terms may include numbers that are rounded to the nearest significant figure. As used herein, the terms "substantial" and "substantially" means, when comparing various parts to one another, that the parts being compared are equal to or are so close enough in dimension that one skill in the art would consider the same. Substantial and substantially, as used herein, are not limited to a single dimension and specifically include a range of values for those parts being compared. The range of values, both above and below (e g., "+ / -" or greater / lesser or larger / smaller), includes a variance that one of ordinary skill in the art would know to be a reasonable tolerance for the parts mentioned.

[0080] The description in the present application should not be read as implying that any particular element, step, or function is an essential or critical element that must be included in the claim scope. The scope of patented subject matter is defined only by the allowed claims. Moreover, none of the claims invokes 35 U.S.C. § 112(f) with respect to any of the appended claims or claim elements unless the exact words "means for" or "step for" are explicitly used in the particular claim, followed by a participle phrase identify ing a function.

[0081] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term “another”, as used herein, is defined as at least a second or more. The terms “including” and “having,” as used herein, are defined as comprising (i.e., open language). The term “coupled,” as used herein, is defined as “connected,” although not necessarily directly, and not necessarily mechanically. “Communicatively coupled” refers to coupling of components such that these components are able to communicate with one another through, for example, optical, wired, wireless, or other communications media. The terms “communicatively coupled” or “communicatively coupling” include, but are not limited to, communicating light signals and / or electronic control signals, by which one element may direct or control another. The term "configured to" describes one or more structures, or a combination of structures, that is set up. arranged, built, composed, constructed, designed or that has any combination of these characteristics to carry out a given function. The term "adapted to" describes one or more structures or a combination of structures that is capable of, able to accommodate, to make, or that is suitable to carry out a given function.

[0082] It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0083] The phrases “at least one of . . . . and <N>” or “at least one of , , . . . <N>, or combinations thereof’ or “, , . . . and / or <N>” are defined by the Applicant in the broadest sense, superseding any other implied definitions hereinbefore or hereinafter unless expressly asserted herein by the Applicant to the contrary7, to mean one or more elements selected from the group comprising A, B, . . . and N, that is to say. any combination of one or more of the elements A, B. . . . or N including any one element alone or in combination with one or more of the other elements which may also include, in combination, additional elements not listed.

[0084] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed.

[0085] Reference in the specification to “one embodiment” or “an embodiment” of the present principles, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment”, as well any other variations, appeanng in various places throughout the specification are not necessarily all referring to the same embodiment.

[0086] The description of the various embodiments of the present invention has been presented by various examples for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. The examples were chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CLAIMS1. A computer-implemented method of automatically tracking eye rotation visible in captured images, comprising: providing a processing system; providing memory, a camera system, and a neural network, coupled with the processing system; capturing, with the camera system, a plurality of images of a patient’s eye in various gazes, including a straight ahead gaze and at least one “as far as possible” gaze selected from a set of gazes including left, up and left, straight up, up and right, right, down and right, straight down, and down and left; segmenting, with the neural netw ork, the captured images into areas corresponding to the patient eye’s iris and sclera; performing feature extraction, with the neural network, and identify a shape and location of visible limbus boundaries in each of the captured images that have been segmented, the straight ahead gaze being associated with a reference ellipse representing a shape and location of a visible limbus and the at least one “as far as possible” gaze being associated with a respective at least one observed elliptical segment representing a shape and location of at least one visible limbus segment, in the captured images; estimating a direction and magnitude of how far the eye has rotated based on a comparison of the reference ellipse and the at least one observed elliptical segment; and outputting the estimated direction and magnitude.

2. The method of claim 1, further comprising: performing, with the processing system, a mathematical comparison of a shape and location of the reference ellipse and a shape and location of the at least one observed elliptical segment to estimate the direction and magnitude of how far the eye has rotated.

3. The method of claim 2, further comprising: repeatedly rotating, with the processing system, the reference ellipse by different directions and magnitudes producing a different candidate ellipse by each rotation; and comparing each different candidate ellipse and the at least one observed elliptical segment to find a best match therebetw een: andquantifying the estimated direction and magnitude of rotation of the eye from the reference ellipse to the at least one observed elliptical segment, based on the best match found.

4. The method of claim 1, further comprising: performing an ocular rotation estimation via calibration by mapping locations and shapes of observed elliptical segments to known locations in physical space; and predicting a magnitude and direction of ocular rotation based on the known locations in physical space of the mapped observed elliptical segments.

5. A computer-based information processing system, comprising: a processor; memory, a camera system, and a neural network, coupled with the information processing system; and the processor, in response to executing computer instructions, performs the following operations: capturing, with the camera system, a plurality of images of a patient’s eye in various gazes, including a straight ahead gaze and at least one “as far as possible” gaze selected from a set of gazes including left, up and left, straight up, up and right, right, down and right, straight down, and down and left; segmenting, with the neural network, the captured images into areas corresponding to the patient eye’s iris and sclera; performing feature extraction, with the neural netw ork, and identify a shape and location of visible limbus boundaries in each of the captured images that have been segmented, the straight ahead gaze being associated with a reference ellipse representing a shape and location of a visible limbus and the at least one “as far as possible” gaze being associated with a respective at least one observed elliptical segment representing a shape and location of at least one visible limbus segment, in the captured images; estimating a direction and magnitude of how far the eye has rotated based on a comparison of the reference ellipse and the at least one observed elliptical segment; and outputting the estimated direction and magnitude.

6. The computer-based information processing system of claim 5, wherein the processor, in response to executing computer instructions, performs the following operations:mathematically comparing a shape and location of the reference ellipse and a shape and location of the at least one observed elliptical segment to estimate the direction and magnitude of how far the eye has rotated.

7. The computer-based information processing system of claim 6, wherein the processor, in response to executing computer instructions, performs the following operations: repeatedly rotating, by the processing system, the reference ellipse by different directions and magnitudes producing a different candidate ellipse by each rotation; and comparing each different candidate ellipse and the at least one observed elliptical segment to find a best match therebetween; and quantifying the estimated direction and magnitude of rotation of the eye from the reference ellipse to the at least one observed elliptical segment, based on the best match found.

8. The computer-based information processing system of claim 5, wherein the processor, in response to executing computer instructions, performs the following operations: performing an ocular rotation estimation via calibration by mapping locations and shapes of observed elliptical segments to known locations in physical space; and predicting a magnitude and diretion of ocular rotation based on the known locations in physical space of the mapped observed elliptical segments.

9. A non-transitory computer-readable storage medium having stored therein instructions which, when executed by at least one processor coupled with memory', a camera system, and a neural network, cause the processor to perform a computer-implemented method of automatically tracking eye rotation visible in captured images, comprising: capturing, with the camera system, a plurality of images of a patient’s eye in various gazes, including a straight ahead gaze and at least one “as far as possible” gaze selected from a set of gazes including left, up and left, straight up. up and right, right, down and right, straight down, and down and left; segmenting, with the neural network, the captured images into areas corresponding to the patient eye’s iris and sclera; performing feature extraction, with the neural network, and identify a shape and location of visible limbus boundaries in each of the captured images that have been segmented, the straight ahead gaze being associated with a reference ellipse representing a shape and location of a visible limbus and the at least one “as far as possible” gaze being associated with a respectiveat least one observed elliptical segment representing a shape and location of at least one visible limbus segment, in the captured images; estimating a direction and magnitude of how far the eye has rotated based on a comparison of the reference ellipse and the at least one observed elliptical segment; and outputting the estimated direction and magnitude.

10. The non-transitory computer-readable storage medium of claim 9, wherein the processor performs the computer-implemented method further comprising: performing a mathematical comparison of a shape and location of the reference ellipse and a shape and location of the at least one observed elliptical segment to estimate the direction and magnitude of how far the eye has rotated.1 1. The non-transitory computer-readable storage medium of claim 10, wherein the processor performs the computer-implemented method further comprising: repeatedly rotating the reference ellipse by different directions and magnitudes producing a different candidate ellipse by each rotation; and comparing each different candidate ellipse and the at least one observed elliptical segment to find a best match therebetween; and quantifying the estimated direction and magnitude of rotation of the eye from the reference ellipse to the at least one observed elliptical segment, based on the best match found.

12. The non-transitory computer-readable storage medium of claim 9, wherein the processor performs the computer-implemented method further comprising: performing an ocular rotation estimation via calibration by mapping locations and shapes of observed elliptical segments to known locations in physical space; and predicting a magnitude and diretion of ocular rotation based on the knowm locations in physical space of the mapped observed elliptical segments.

Images