Using infrared to detect proper eye alignment before capturing retinal images

Abstract

To provide using infrared to detect proper eye alignment before capturing retinal images.SOLUTION: Systems and methods are disclosed herein for detecting eye alignment during retinal imaging. In an embodiment, the system receives an infrared stream from an imaging device, the infrared stream showing characteristics of an eye of a patient. The system determines, based on the infrared stream, that the eye is improperly aligned at a first time, and outputs sensory feedback indicative of the improper alignment. The system detects, based on the infrared stream at a second time later than the first time, that the eye is properly aligned, and receives an image of a retina of the properly aligned eye from the imaging device.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Utility Application No. 16 / 997,843, filed August 19, 2020, which is incorporated by reference in its entirety.

[0002] The present invention relates generally to the autonomous diagnosis of retinal abnormalities, and more particularly to the use of spectral waves, such as infrared, to detect proper eye alignment when capturing retinal images. [Background technology]

[0003] (background) Historically, for abnormality examination of a patient's eye, a medical professional personally operated an imaging device to capture retinal images. The medical professional was trained to guide the camera and ensure it was properly aligned with the patient's pupil when capturing the retinal image. Autonomous systems have been developed to diagnose retinal abnormalities and are designed to capture images of a patient's retina (the term "fundus" is used interchangeably herein) and analyze those images for abnormalities without requiring a medical professional to analyze the images. However, without a medical professional operating the imaging device, the imaging device is incapable of ensuring that the patient's eye is properly aligned, and therefore these autonomous systems are prone to error. For example, a guide light may be illuminated within the imaging device and the patient may be instructed (e.g., by an operator rather than a medical professional) to focus on the guide light to align the patient's pupil with the camera lens; however, the patient may be unable to do so, and an image would nevertheless be captured that is insufficient for abnormality analysis. Summary of the Invention [Means for solving the problem]

[0004] (summary) Systems and methods for detecting eye alignment during retinal imaging are provided herein. For example, during an eye exam, a patient may place their eye within the field of view of an imaging device configured to capture a retinal image. Rather than requiring a medical professional to operate the imaging device to instruct the patient to properly align the patient's eye with the camera, the imaging device may be configured to determine proper alignment without any action by the medical professional. For example, infrared imaging of the patient's eye may be performed to determine eye alignment, and instructions (e.g., voice commands) may be output to assist the patient in adjusting the positioning of the patient's eye to properly align the eye. In this manner, fewer flashes of light need to be cast to capture an image of the patient's eye for retinal disease diagnosis, thus preventing the need to unnecessarily expose the patient's eye to multiple flashes of light due to unsuccessful imaging.

[0005] To achieve these and other objects, in one embodiment, a processor (e.g., in a server) receives infrared light from an imaging device, the infrared light indicative of a characteristic of a patient's eye. The processor determines that the eye is improperly aligned at a first time based on the infrared light and outputs sensor feedback indicative of the improper alignment. The processor then detects that the eye is properly aligned based on the infrared light at a second time after the first time and receives an image of the retina of the properly aligned eye from the imaging device. The present invention provides, for example, the following items. (Item 1) 1. A method for detecting eye alignment during retinal imaging, the method comprising: receiving an infrared beam from an imaging device, the infrared beam indicative of a characteristic of the patient's eye; inputting the frames of infrared light into a machine learning model; receiving information as output from the machine learning model indicating that the eye is improperly aligned at a first time; outputting sensor feedback indicative of said improper match; detecting that the eye is properly aligned based on the infrared light beam at a second time after the first time; receiving an image of the properly aligned retina of the eye from the imaging device; A method comprising: (Item 2) The information is the pupil of the eye is not shown in the infrared light beam; The imaging device is out of focus. Item 10. The method of item 1, wherein the eye is indicated to be improperly aligned at the first time due to at least one of the following: (Item 3) 10. The method of claim 1, wherein the imaging device is operated autonomously and the sensor feedback includes a visual or audio output to the patient with instructions on how to correct the improper alignment. (Item 4) Item 10. The method of item 1, wherein the imaging device is operated by an operator other than the patient, and the sensor feedback is provided to the operator. (Item 5) Item 10. The method of item 1, further comprising, in response to detecting that the eye is properly aligned, commanding the imaging device to capture the image. (Item 6) Item 10. The method of item 1, wherein detecting that the eye is properly aligned comprises detecting a twitch of the eye. (Item 7) Detecting that the eye is properly aligned includes: capturing a candidate image in response to detecting said twitch; determining whether the candidate image satisfies a quality parameter; determining that the eye is properly aligned in response to determining that the candidate image satisfies the quality parameter; and Item 7. The method of item 6, further comprising: (Item 8) 8. The method of claim 7, wherein the quality parameter comprises at least one of image quality of the candidate image and detection of artifacts in the candidate image. (Item 9) 2. The method of claim 1, further comprising autonomously diagnosing a retinal disease of the retina based on characteristics of the image of the retina. (Item 10) 1. A computer program product for detecting ocular alignment during retinal imaging, the computer program product comprising: a non-transitory computer-readable storage medium; receiving an infrared beam from an imaging device, the infrared beam indicative of a characteristic of the patient's eye; inputting the frames of infrared light into a machine learning model; receiving information as output from the machine learning model indicating that the eye is improperly aligned at a first time; outputting sensor feedback indicative of said improper match; detecting that the eye is properly aligned based on the infrared light beam at a second time after the first time; receiving an image of the properly aligned retina of the eye from the imaging device; A computer program product containing computer program code for performing the steps of: (Item 11) The information is the pupil of the eye is not shown in the infrared light beam; The imaging device is out of focus. Item 11. The non-transitory computer-readable medium of item 10, wherein the eye is improperly aligned at the first time due to at least one of: (Item 12) Item 11. The non-transitory computer-readable medium of item 10, wherein the imaging device is operated autonomously and the sensor feedback includes a visual or audio output to the patient with instructions on how to correct the improper alignment. (Item 13) Item 11. The non-transitory computer-readable medium of item 10, wherein the imaging device is operated by an operator other than the patient, and the sensor feedback is provided to the operator. (Item 14) Item 11. The non-transitory computer-readable medium of item 10, wherein the computer program code is further for commanding the imaging device to capture the image in response to detecting that the eye is properly aligned. (Item 15) Item 11. The non-transitory computer-readable medium of item 10, wherein detecting that the eye is properly aligned comprises detecting a twitch of the eye. (Item 16) Detecting that the eye is properly aligned includes: capturing a candidate image in response to detecting said twitch; determining whether the candidate image satisfies a quality parameter; determining that the eye is properly aligned in response to determining that the candidate image satisfies the quality parameter; and Item 16. The non-transitory computer-readable medium of item 15, further comprising: (Item 17) Item 17. The non-transitory computer-readable medium of item 16, wherein the quality parameters comprise at least one of image quality of the candidate image and detection of artifacts in the candidate image. (Item 18) 11. The non-transitory computer-readable medium of claim 10, wherein the computer program code is further for autonomously diagnosing a retinal disease of the retina based on characteristics of the image of the retina. (Item 19) 1. A computer program product for detecting ocular alignment during retinal imaging, the computer program product comprising: a computer readable storage medium containing computer program code; the computer program product comprising: a first module for receiving an infrared beam from an imaging device, the infrared beam indicative of a characteristic of the patient's eye; a second module for inputting the frames of infrared light into a machine learning model; a third module for receiving as output from the machine learning model information indicating that the eye is improperly aligned at a first time; and a fourth module for outputting sensor feedback indicative of the improper match; a fifth module for detecting that the eye is properly aligned based on the infrared light beam at a second time after the first time; and a sixth module for receiving an image of the properly aligned retina of the eye from the imaging device; and A computer program product comprising: (Item 20) The information is the pupil of the eye is not shown in the infrared light beam; The imaging device is out of focus. 20. The computer program product of claim 19, wherein the eye is improperly aligned at the first time due to at least one of: [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is an exemplary block diagram of system components in an environment for utilizing a matching tool, according to one embodiment.

[0007] [Figure 2]FIG. 2 is an exemplary block diagram of modules and components of an imaging device, according to one embodiment.

[0008] [Figure 3] FIG. 3 is an exemplary block diagram of modules and components of a matching tool, according to one embodiment.

[0009] [Figure 4] FIG. 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).

[0010] [Figure 5] FIG. 5 is an exemplary flowchart for detecting eye alignment during retinal imaging, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be developed without departing from the principles of the present invention as described herein.

[0012] (a) Environmental Overview 1 is an exemplary block diagram of system components in an environment for utilizing an alignment tool, according to one embodiment. Environment 100 includes imaging device 110, alignment tool 115, network 120, and retinal disease diagnosis tool 130. Imaging device 110 is a device configured to capture one or more images of the retina of a patient's eye. Imaging device 110 may capture such images through manual operation, autonomously when commanded by computer program instructions or an external signal (e.g., received from alignment tool 115), or a combination thereof. Examples of what these images may look like and how they are derived are described in commonly owned U.S. patent application Ser. No. 15 / 466,636, filed March 22, 2017, the disclosure of which is hereby incorporated by reference in its entirety.

[0013] Alignment tool 115 is operably coupled to imaging device 110 and may verify that the patient's eye is properly aligned. In response to verifying that the patient's eye is properly aligned, alignment tool 115 may instruct imaging device 110 to capture an image. Alignment tool 115 may use an image or stream of images from various spectrums, such as infrared images, to determine whether the patient's eye is properly aligned. While depicted as a standalone entity, this is for illustrative purposes only; alignment tool 115 may alternatively be included with imaging device 110 (e.g., installed as a module) or may be implemented as part of retinal disease diagnosis tool 130, acting based on information received via network 120. Further details regarding the functionality of alignment tool 115 are described in more detail below with respect to FIGS. 3-5 .

[0014] After capturing the image (based on instructions from alignment tool 115), imaging device 110 (or alternatively, alignment tool 115) transmits the image to retinal disease diagnosis tool 130 for processing. Retinal disease diagnosis tool 130 autonomously analyzes the received retinal image and uses machine learning analysis of biomarkers therein to determine a diagnosis. The diagnosis may specifically be a determination that the user has a particular disease, such as diabetic retinopathy, or that the user likely has a disease and should therefore see a physician for confirmation and treatment. The manner in which retinal disease diagnosis tool 140 performs the analysis and determines a diagnosis is further discussed in commonly owned U.S. Pat. No. 10,115,194, the disclosure of which is hereby incorporated by reference in its entirety. Although depicted as a separate entity from realignment tool 115, retinal disease diagnosis tool 130 may be instantiated on the same device or set of devices as alignment tool 130 and may be installed as a module on imaging device 110, in part or in whole, similar to the manner in which alignment tool 130 may be installed as a module within imaging device 110. Although not depicted in FIG. 1 , one or more additional imaging devices may be used to capture external images that include part or all of the patient (e.g., the patient's skin or hair). Any retinal diagnosis performed by retinal diagnostic tool 130 may utilize such external images.

[0015] (b) Exemplary Imaging Device Components 2 is an exemplary block diagram of modules and components of an imaging device, according to one embodiment. The imaging device 110 includes an image capture component 211, a flashlight component 212, an infrared component 213, an alignment tool application protocol interface (API) 214, a visual guidance light 215, and a user interface 216. While not depicted, the imaging device 110 may include other components, such as built-in instances of either or both of the alignment tool 115 and the retinal disease diagnosis tool 130, as well as any components thereof. The imaging device 110 may also include any databases or memory for performing any functionality described herein. The imaging device 110 may also exclude some depicted components. For example, the imaging device may exclude the visual guidance light 215.

[0016] The image capture component 211 may be any sensor configured to capture an image of the patient's retina. For example, a specialized lens may be used to capture an image of the patient's retina. The flash component 212 may be any component capable of illuminating the patient's retina during image capture by the image capture component 211 and may be configured to emit light in coordination with the image capture operation of the image capture component 211. The image capture component 211 may also be configured with an external image capture component 211 to capture images including the patient's skin and / or hair.

[0017] The infrared component 213 is an infrared sensor configured to transmit infrared radiation to the patient's retina and determine its absorptance. The infrared component 213 may generate a heat map indicating the absorptance of the infrared transmission across the patient's retina. The infrared component 213 transmits the absorptance determination and / or heat map to a processor (e.g., of the alignment tool 130) for processing toward determining alignment of the patient's eye. The infrared component 213 may stream the images it captures and / or absorptance maps or other post-processed renderings of the infrared images (hereinafter collectively referred to as infrared images) to the alignment tool 115. As used herein, the term "stream" may refer to some or all of the transmission of the infrared images as they are captured by the infrared component 213 or shortly thereafter.

[0018] The alignment tool API 214 interfaces with the alignment tool 130 to translate commands from the alignment tool 130 to the imaging device 110. Exemplary commands may include commands for capturing an image, for adjusting the intensity of light emitted by the flash component 212, for activating the gaze guidance light 215, and the like. These commands and how they are generated are discussed in further detail with reference to FIG. 3 below. The alignment tool API 214 is also used by the imaging device 110 to transmit infrared images to the alignment tool 130.

[0019] The visual guidance lights 215 may include one or more lights in the patient's vicinity who is using the imaging device 110 that can be selectively activated. The activated guidance lights are designed to capture the patient's attention, who is instructed to gaze at the guidance lights (e.g., to align the pupil with the lens for retinal image capture). The visual guidance lights may include arrows in any direction (e.g., left, right, up, down, or any diagonal direction), illuminated by illuminating certain pixels or lights (e.g., LEDs) within a grid of pixels or lights. The arrows may indicate the direction the patient's gaze should be moved to improve alignment. Further details regarding the visual guidance lights 215 and their functionality are described with reference to the visual feedback module 340, which may generate instructions to selectively activate one or more of the visual guidance lights 215.

[0020] User interface 216 is an interface through which a user of imaging device 110 (e.g., an operator or a patient) may command imaging device 110 to perform any function possible, such as capturing an image, adjusting flash intensity, capturing infrared information, and the like. User interface 216 may also output information (e.g., auditory or visual information) to the user. User interface 216 may be any hardware or software interface and may include physical components (e.g., buttons) and / or graphical components (e.g., on a display such as a touchscreen display). User interface 216 may be located on imaging device 110, may be a peripheral device of imaging device 110, or may be located on a device separated from imaging device 110 by network 120, thus enabling remote operation of imaging device 110.

[0021] (c) Exemplary Alignment Tool Components 3 is an exemplary block diagram of modules and components of a matching tool, according to one embodiment. Matching tool 115 includes an infrared image processing module 310, a matching determination module 320, an auditory feedback module 330, a visual feedback module 340, and an imaging device API 350. Although not depicted, matching tool 115 may include other components, such as additional modules, and any databases or memories for performing any functionality described herein. Matching tool 115 may also exclude some depicted components. For example, matching tool 130 may exclude visual feedback module 340.

[0022] The infrared image processing module 310 receives infrared light from the imaging device 110. Infrared refers to an exemplary spectrum that may be used in accordance with embodiments disclosed herein, however, this is non-limiting. Whenever infrared is described herein, other spectrums, such as ultraviolet, X-ray, etc., may also be used. The infrared image processing module 310 may receive the infrared light via the imaging device API 350, which interfaces the alignment tool 115 with the imaging device 110. The infrared image processing module 310 may command the imaging device 110 to start streaming infrared data (e.g., when a new patient is detected using the imaging device 110). Alternatively, the infrared image processing module 310 may command the imaging device 110 to start streaming infrared data in response to an occurring condition (e.g., the imaging device 110 being initialized to capture retinal images).

[0023] The infrared image processing module 310 may perform processing on the received infrared light (e.g., to make the data contained therein usable by the match determination module 320). For example, the infrared image processing module 310 may take raw infrared data as input and generate an infrared heat map or contour map showing portions of one or both of the patient's eyes. As another example, the infrared image processing module 310 may generate a vector map showing changes in the position of the eye between one or more frames, and thus showing the direction and distance the eye has moved or rotated. This vector map may be used in addition to, or separate from, the raw or otherwise processed infrared information in determining eye match.

[0024] The alignment determination module 320 determines whether the patient's eye is properly aligned for the capture of a retinal image. As used herein, the term "properly aligned" may refer to an image that meets the criteria for performing an autonomous diagnosis of whether the patient has a retinal disease. An improper alignment generally occurs because the patient's gaze is directed toward a direction that obstructs an object of interest, such as the patient's pupil or retina. In an embodiment in which the patient is instructed to look toward a fixation light, a proper alignment occurs when the image is aligned with a retinal landmark (e.g., the center of the retina). A high-intensity flash of light is used when the image is captured, and guidelines suggest that limits on the number of times a patient is flashed during a single appointment should be limited for patient health and safety reasons. An improper alignment results in the capture of an invalid image for the autonomous diagnosis, thus requiring additional images to be captured, which in turn requires additional flashes of light to be cast on the patient's eye. The alignment determination module 320 may be used to determine proper alignment before flashing the light into the patient's eye, thus improving the patient's health and safety.

[0025] In some embodiments, the match determination module 320 may determine whether a match is adequate based on whether and where an object of interest (i.e., either the pupil or the retina of the eye) is present in the infrared light. The match determination module 320 may determine that the presence of the object of interest in the infrared light alone is sufficient to determine that a match is adequate. Alternatively, or in addition, the match determination module 320 may require that other factors be true. For example, the match determination module 320 may require that the imaging device 110 be in focus to determine that the images are adequately aligned. The imaging device 110 may be programmed to autofocus in response to detecting an object of interest, such as the pupil of the eye, or that the pupil is at the center of or within a threshold distance of the infrared image. The match determination module 320 may determine whether the imaging device 110 is in focus based on the infrared light or based on auxiliary information indicating whether the imaging device 110 is in focus. In response to determining that imaging device 110 is in focus, alignment determination module 320 may determine that alignment is adequate. If the object of interest and / or the focal point of imaging device 110 is not in the infrared light beam, alignment determination module 320 may determine that the patient's eye is improperly aligned.

[0026] The alignment determination module 320 may use other means to determine that alignment is appropriate based on the infrared light. For example, the patient may twitch their eye in a given direction based on a stimulus, such as the illumination of a guidance light or an auditory command from an operator, or as an output from a speaker on the alignment tool 115 and / or imaging device 110. The alignment determination module 320 may determine that alignment is appropriate in response to detecting the twitch. That is, it is likely that the patient moved their eye into the appropriate alignment based on the stimulus. The alignment determination module 320 may detect the twitch based on a threshold movement of the patient's eye over a certain time interval or across a certain number of frames. The alignment determination module 320 may detect the twitch using the vector map described above and determine whether a movement vector on the vector map indicates that a twitch has occurred.

[0027] Because twitching may occur due to reasons other than a stimulus directly designed to cause the patient to twitch their eye to a particular position, alignment determination module 320 may perform further processing before determining that the eye is properly aligned. In an embodiment, in response to detecting a twitch, alignment determination module 320 may capture an image. The captured image may be a frame of infrared light followed by a twitch, or it may be a visual image captured by imaging device 110 and commanded to be transmitted to alignment determination module 320 for processing.

[0028] The match determination module 320 may then determine whether the image satisfies quality parameters. The quality parameters may include determining whether the image quality is sufficient. For example, if the frame is blurred due to continuous eye movement during twitching, the image quality may be poor. The match determination module 320 may determine the image quality based on one or more metrics, such as blur, focus, etc. The quality parameters may also include other characteristics, such as whether biomarkers are detected in the candidate image. For example, the match determination module 320 may determine whether the optic disc and / or any other biomarkers are detectable in the image based on either visual or infrared frames. Another example quality parameter may include determining whether artifacts are blocking part or all of the image (e.g., the patient's eyelashes obscure the image). In response to determining that the candidate image satisfies the quality parameters, the match determination module 320 may determine that the patient's eyes are properly aligned. In response to determining that the candidate image does not satisfy the quality parameters, the match determination module 320 may determine that the patient's eye is improperly matched.

[0029] In an embodiment, quality may be determined by inputting a candidate image into a machine learning model and receiving as output from the machine learning model an indication of whether the candidate image is of sufficient quality. In some embodiments, the machine learning model may be a supervised learning model. In such embodiments, the machine learning model may be trained using a labeled training set of images, where each image in the training set is labeled as to whether the image is of sufficient quality. In some embodiments, the machine learning model may be trained using unsupervised techniques.

[0030] If the alignment is determined to be improper, feedback may be provided to the patient to prompt them to adjust the position of their eyes so that the eyes are properly aligned. In one embodiment, the alignment determination module 320 determines the specific adjustments that must be made to properly align the eyes. For example, the alignment determination module 320 uses an infrared light to determine where the pupils are focused and determines that the pupils must move in a specific direction to properly align. In another embodiment, the alignment determination module 320 is unable to identify the specific adjustments that must be made to properly align the eyes (e.g., because the pupils are not detectable within the infrared light). Depending on this scenario, the auditory feedback module 330 and the visual feedback module 340 may provide feedback to the patient to make the appropriate adjustments.

[0031] In one embodiment, the match determination module 320 determines whether the eyes are matched based on the output from the machine learning model. For example, the match determination module 320 may input one or more frames of infrared light into the machine learning model and receive as output from the machine learning model a probability that the eyes are likely matched and / or an output indicating whether the eyes are properly matched. If the output indicates a high probability, the match determination module 320 may compare the probability to a threshold (e.g., a likelihood of the eyes being matched greater than 92%) and determine that the eyes are matched based on the probability exceeding the threshold.

[0032] The machine learning model may be trained using a labeled dataset of infrared frames, where the labels indicate whether the eyes are aligned or not. In some embodiments, the dataset may include additional labels, such as labels indicating an offset from proper alignment (e.g., if rotated three degrees to the right, the eyes would be aligned). In such cases, the alignment determination module 320 may additionally or alternatively output the offset, which may be used by the auditory feedback module 330 and / or the visual feedback module 340 in determining the command. The machine learning model may be any machine learning model. For example, the machine learning model may be a neural network, such as a convolutional neural network.

[0033] In one embodiment, a model, such as a machine learning model and / or a deterministic model, may be trained to take infrared frames as input. For example, frames may be input into the model from the beginning of the alignment process until a correct alignment is obtained. The model may also take as input a representation of instructions provided to the patient (e.g., a speech-to-text transcription if a human operator provides the instructions, or a reading of automated instructions provided to the patient). Based on these inputs, the model may output an indication of whether the provided instructions were correctly administered by the operator and / or whether the provided instructions were accurately followed by the patient. The output may be binary (e.g., incorrect or correct administration). The output may indicate a specific error in the instructions (e.g., a specific portion of the instructions was incorrect, so some other action should be performed instead). The output may also be a specific new instruction that will correct the alignment (e.g., move eye gaze 3 degrees to the right). The output may be a set of probabilities corresponding to different corrective activities and / or instructions and / or a set of probabilities of whether the administration is correct or not, whereby the alignment determination module 320 may determine what to output to the patient and / or operator by comparing the probabilities to a representative threshold. The machine learning model may be a trained region-based convolutional neural network (RCNN) or any other type of machine learning model.

[0034] The auditory feedback module 330 may output auditory instructions to the patient (e.g., using a speaker on the imaging device 110 or a peripheral speaker operably coupled to the alignment tool 115). The auditory instructions may move the eyes in any particular direction, such as up, down, left, right, or some combination thereof. In embodiments in which the exact needed adjustment is known to properly align the patient's eyes, the instructions may coincide with the needed eye movement. In embodiments in which the exact needed adjustment is not known, the patient may be instructed to move their eyes until their pupils are detected within the infrared light (e.g., "Slowly move your eyes up"), at which point the alignment determination module 320 may determine the needed adjustment using the process described above. The auditory feedback module 330 may then provide further instructions based on the needed adjustment.

[0035] Visual feedback module 340 operates in a similar manner to auditory feedback module 330 and may operate in combination with or separately from auditory feedback module 330 (i.e., auditory and visual feedback may occur in parallel or separately). As mentioned above, imaging device 110 may include imaging components (e.g., pixels, light-emitting diodes, etc.) that appear in the patient's field of view when imaging device 110 is used. In some embodiments, visual feedback module 340 may activate a guidance light. The guidance light is activated at a location in the patient's field of view that, when gazed at, will properly align the patient's eyes. Alignment tool 115 may have auditory feedback module 330 and may provide instructions to gaze at the guidance light when activated.

[0036] In another embodiment, or if the patient does not respond to the guiding light, the visual feedback module 340 may activate the imaging components to form an indicator, such as an arrow, in the direction in which the patient's gaze should be adjusted. In an embodiment, the visual feedback module 340 may activate the imaging components, and the auditory feedback module 330 may output auditory instructions to follow the imaging components with the patient's gaze as they move. The visual feedback module 340 may selectively activate and deactivate the imaging components as the light continues to move within the patient's field of view. As the patient moves their gaze to follow the movement of the light, the alignment determination module 320 may determine whether the alignment is inadequate and / or whether a pupil can be detected, and may feed back to the visual feedback module 340 any further adjustments needed to obtain proper alignment (as determined using the process discussed above) for further movement of the light.

[0037] (d) Exemplary Computer Architecture 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller). Specifically, FIG. 4 shows a diagrammatic representation of a machine, in the exemplary form of a system 400, within which program code (e.g., software) for causing the machine to perform any one or more of the methods discussed herein may be implemented. The program code may consist of instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0038] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions 424 that specify actions to be taken by the machine. Further, while only a single machine is illustrated, the term "machine" should also be taken to include any collection of machines that individually or together execute the instructions 124 to perform any one or more of the methodologies discussed herein.

[0039] Exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. Computer system 400 may further include a visual display interface 410. The visual interface may include software drivers that enable a user interface to be displayed on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, or the like. For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with a touch-enabled screen. The computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touchscreen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a storage unit 416, a signal generating device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate via the bus 408.

[0040] The storage unit 416 includes a machine-readable medium 422 having stored thereon instructions 424 (e.g., software) that embody any one or more of the methods or functions described herein. The instructions 424 (e.g., software) may also reside, completely or partially, within the main memory 404 or within the processor 402 (e.g., within a processor's cache memory) during execution thereof by the computer system 400, and the main memory 404 and the processor 402 may also constitute machine-readable media. The instructions 424 (e.g., software) may be transmitted or received via a network 426 via the network interface device 420.

[0041] Although machine-readable medium 422 is shown in the exemplary embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" should also be taken to include any medium capable of storing instructions (e.g., instructions 424) for execution by a machine, causing the machine to perform any one or more of the methodologies disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0042] (e) Exemplary Data Flow for Determining Eye Alignment FIG. 5 depicts an exemplary flowchart for detecting eye alignment during retinal imaging, according to one embodiment. Process 500 begins with one or more processors (e.g., processor 402) of a device used to activate alignment tool 130 receiving 502 infrared light from an imaging device (e.g., using infrared image processing module 310), the infrared light indicative of characteristics of the patient's eye. Based on the infrared light, alignment tool 130 determines 504 (e.g., using alignment determination module 320) that the eye is improperly aligned the first time. The determination of improper alignment may be based on any number of factors, such as the pupil of the patient's eye not being shown in the infrared light or the imaging device 110 being out of focus. In one embodiment, alignment determination 504 is made by inputting frames of infrared light into a machine learning model and receiving information as output from the machine learning model indicating that the eye is improperly aligned the first time.

[0043] The alignment tool 130 outputs 506 sensory feedback indicating an improper alignment (e.g., using the auditory feedback module 330 and / or the visual feedback module 340). The sensory feedback may be output to an operator (e.g., a non-specialist operating the imaging device 110), to the patient, or to both the patient and the operator. The alignment tool 130 detects 508 that the eyes are properly aligned based on the infrared light beam at a second, subsequent time (e.g., using the alignment determination module 320). The proper alignment determination may be based on detecting twitches of the patient's eye. If twitches form the basis for a proper alignment determination, an image (e.g., an infrared frame) may be captured and verified against quality parameters (e.g., image quality, artifacts, biomarker detection, etc.) before confirming the determination that the eyes are indeed properly aligned. The alignment tool 130 receives 510 an image of the retina of the properly aligned eye from the imaging device 110. In response to detecting a proper alignment, alignment tool 130 may command imaging device 110 to automatically capture an image of the retina and transmit it to alignment tool 130. Alignment tool 130 may transmit the image to retinal disease diagnosis tool 130 for performing an autonomous diagnosis of the patient's eye.

[0044] (f) Overview The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will recognize that many modifications and variations are possible in light of the above disclosure.

[0045] Some portions of this description will describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they may be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Further, it has proven convenient at times to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0046] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0047] Embodiments of the present invention may also relate to apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referenced in the specification may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0048] Embodiments of the present invention may also relate to products produced by the computational processes described herein. Such products may comprise information resulting from the computational processes, and when the information is stored on a non-transitory tangible computer-readable storage medium, may include any embodiment of a computer program product or other data combination described herein.

[0049] Finally, the terminology used within this specification has been selected primarily for readability and guidance purposes, and not to precisely describe or delineate the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but by any claims that may be published upon filing based on this specification. Thus, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

[Claim 1] The invention described in this specification.

Images