Image Holding and Stitching for Minimal Flash Eye Disease Diagnosis

Abstract

Systems and methods for minimizing retinal exposure to flash light during diagnostic image acquisition are provided herein. In one embodiment, the system captures multiple retinal images of different retinal regions. The system determines that a first portion of the first image does not meet a criterion, while a second portion of the first image meets the criterion, identifies a portion of the retina depicted in the first portion that does not meet the criterion, and determines whether the portion of the retina is depicted in a third portion of the second image and whether the third portion meets the criterion. In response to determining that the third portion meets the criterion, the system performs a diagnosis. In response to determining that the portion of the retina is not depicted in the second image, the system captures additional images of the retinal region.

Description

Technical Field

[0001] (Background) The present invention generally relates to autonomous diagnosis of retinal abnormalities, and more specifically, to image retention for reducing the need to flash the patient's retina.

Background Art

[0002] An autonomous system for diagnosing retinal abnormalities captures images of a patient's retina (used herein with the same meaning as the word "fundus") and analyzes those images for abnormalities. Typically, images of different parts of the retina (e.g., an image with the fovea at the center and an image with the optic nerve head at the center) are captured. When the captured images are insufficient for performing a thorough diagnosis, additional images are captured until sufficient images of each part of the retina are obtained. Repeated exposure to the flash by the imaging device can further cause pupil constriction, which lowers the prospects of normal imaging after each flash. Thus, reducing the need for additional images to be captured until sufficient images are obtained reduces the number of flashes and the need to take numerous photographs, or the need to keep the patient waiting to be diagnosed until the pupil has recovered to its normal dilation. Additionally, reducing the need for additional images, in terms of preventing the need to delay the patient's diagnosis until such time that the number of flashes that significantly increases pupil constriction is unlikely to be reached while seated, and thus the likelihood of obtaining an executable image to flash the patient's eye again, can enable a more rapid diagnosis.

Summary of the Invention

Means for Solving the Problems

[0003] (Summary) Systems and methods are provided herein for reducing the need to re-flash a patient's eye to capture an image of the retinal region when a previously captured image is inadequate. As an example, for performing a diagnosis of a patient's right eye, two images, namely an image with the fovea centered and an image with the optic nerve head centered, may be captured. There are some overlapping portions between each part of the retina depicted in these two images. If the region of the retina depicted in one of the images is unusable for some reason (e.g., overexposure or underexposure), instead of recapturing the image, the system may determine whether, in the other image, the same region is depicted without the same deficiency. If the region is usable from the other image, the system may avoid the need to re-flash the patient's right eye to capture another image of that region and instead may stitch portions of the two images to perform the diagnosis.

[0004] In some embodiments, to minimize retinal exposure to the flash during image acquisition for diagnosis, a retinal image preprocessing tool captures a plurality of retinal images (e.g., by instructing an imaging device to take images and send them to the retinal image preprocessing tool). Each retinal image may correspond to a different retinal region of a plurality of retinal regions (e.g., a region with the fovea centered and a region with the optic nerve head centered). The retinal image preprocessing tool may determine that a first portion of a first image (e.g., the portion with the fovea centered) does not meet a criterion (e.g., the image is overexposed or underexposed, the image has a shadow or other artifact, etc.), while a second portion of the first image (e.g., the portion with the optic nerve head centered) meets the criterion (e.g., the second image is properly exposed).

[0005] The retinal image preprocessing tool can identify a part of the retina depicted within a first portion that does not meet the criteria, and can determine that the same part of the retina is depicted within a third portion of a second image. The retinal image preprocessing tool can determine whether the third portion meets the criteria. In response to determining that the third portion meets the criteria, a diagnosis can be performed using a plurality of retinal images. In response to determining that that part of the retina is not depicted within the second image, the retinal image preprocessing tool can capture additional images of the retinal region depicted within the first image. The present invention provides, for example, the following items. (Item 1) A method for minimizing retinal exposure to a flash during image acquisition for diagnosis, capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among a plurality of retinal regions, the plurality of retinal images including a first image and a second image, determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion, identifying a portion of the retina depicted within the first portion that does not meet the criterion, determining whether the portion of the retina is depicted within a third portion of the second image, determining whether the third portion meets the criterion, performing the diagnosis using the plurality of retinal images in response to determining that the third portion meets the criterion, capturing an additional image of the retinal region depicted in the first image in response to determining that the portion of the retina is not depicted within the second image and a method including the same. (Item 2) Performing the diagnosis includes passing the images through a fully autonomous machine learning model that outputs a likelihood of a disease based on the plurality of retinal images, according to the method of Item 1. (Item 3) Each retinal image of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at a different level of flash exposure, according to the method of Item 1. (Item 4) Determining that the first portion of the first image does not meet the criterion includes determining that the first portion of the first image is either overexposed or underexposed, according to the method of Item 1. (Item 5) The second image is an image of the retina of the same eyeball as depicted by the first image, according to the method of Item 1. (Item 6) Performing the diagnosis using the plurality of retinal images includes generating a composite image including the first portion of the image together with the third portion of the second image stitched within the first image, performing the diagnosis using the composite image and a method including the same, according to the method of Item 1. (Item 7) Performing the diagnosis using the plurality of retinal images includes analyzing the first image without considering the first portion and generating a first analysis, Analyzing the third portion of the second image to generate a second analysis; Performing the diagnosis using the first analysis and the second analysis; The method according to item 1, comprising the above. (Item 8) Capturing the additional image comprises: Transmitting, to the imaging device, a command to capture an additional image using the perspective used to capture the first image, using an Application Programming Interface (API); Receiving the additional image from the imaging device; The method according to item 1, comprising the above. (Item 9) Performing the diagnosis using the plurality of retinal images comprises using the third portion in the diagnosis. The method according to item 1, comprising the above. (Item 10) A computer program product for minimizing retinal exposure to a flash during image collection for diagnosis, comprising: Capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among a plurality of retinal regions, the plurality of retinal images including a first image and a second image; Determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion; Identifying a part of the retina depicted within the first portion that does not meet the criterion; Determining whether the part of the retina is depicted within a third portion of the second image; Determining whether the third portion meets the criterion; In response to determining that the third portion meets the criterion, performing the diagnosis using the plurality of retinal images; In response to determining that the part of the retina is not depicted within the second image, capturing an additional image of the retinal region depicted within the first image; A computer program product comprising a non-transitory computer-readable storage medium containing computer program code for performing the above. (Item 11) The computer program code for performing the diagnosis comprises computer program code for passing the images through a fully autonomous machine learning model that outputs the likelihood of a disease based on the plurality of retinal images. The computer program product according to item 10, comprising the above. (Item 12) Each of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at a different level of flash exposure. The computer program product according to item 10. (Item 13) The computer program code for determining that the first portion of the first image does not meet the criteria includes computer program code for determining that the first portion of the first image is either overexposed or underexposed. The computer program product according to item 10. (Item 14) The second image is an image of the retina of the same eyeball as depicted by the first image. The computer program product according to item 10. (Item 15) The computer program code for performing the diagnosis using the plurality of retinal images is generating a composite image including the first portion of the image, together with the third portion of the second image stitched within the first image, and performing the diagnosis using the composite image The computer program product according to item 10 includes computer program code for performing the above. (Item 16) The computer program code for performing the diagnosis using the plurality of retinal images is analyzing the first image without considering the first portion to generate a first analysis, analyzing the third portion of the second image to generate a second analysis, and performing the diagnosis using the first analysis and the second analysis The computer program product according to item 10 includes computer program code for performing the above. (Item 17) The computer program code for capturing the additional image is transmitting, using an application protocol interface (API), a command for capturing an additional image to an imaging device using the viewpoint used for capturing the first image, and receiving the additional image from the imaging device The computer program product according to item 10 includes computer program code for performing the above. (Item 18) Performing the diagnosis using the plurality of retinal images includes using the third portion in the diagnosis. The computer program product according to item 10. (Item 19) A computer program product for minimizing retinal exposure to a flash during image acquisition for diagnosis, a first module for capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among a plurality of retinal regions, the plurality of retinal images including a first image and a second image, a second module, determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion; identifying a portion of the retina depicted within the first portion that does not meet the criterion; determining whether the portion of the retina is depicted within a third portion of the second image; determining whether the third portion meets the criterion and a second module for performing; a third module for performing the diagnosis using the plurality of retinal images in response to determining that the third portion meets the criterion; and a fourth module for capturing additional images of the retinal region depicted within the first image in response to determining that the portion of the retina is not depicted within the second image A computer program product comprising a computer-readable storage medium including computer program code. (Item 20) The computer program product according to item 19, wherein performing the diagnosis includes passing the images through a fully autonomous machine learning model that outputs a likelihood of a disease based on the plurality of retinal images.

Brief Description of the Drawings

[0006] (Brief Description of the Drawings)

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

[0012] The figures depict various embodiments of the invention for illustrative purposes only. Those skilled in the art will immediately recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

DETAILED DESCRIPTION

[0013] (DETAILED DESCRIPTION) (a) ENVIRONMENT OVERVIEW FIG. 1 is an exemplary block diagram of system components in an environment for using a retinal image preprocessing tool according to one embodiment. Environment 100 includes an imaging device 110, a network 120, a retinal image preprocessing tool 130, and a retinal disease diagnosis tool 140. The imaging device 110 is a device configured to capture one or more images of a patient's eye retina. The imaging device 110 can be commanded to autonomously capture such images through manual operation, by computer program instructions, or by external signals (e.g., received from the retinal pigmentation determination tool 130), or a combination thereof. Examples of how these images may look and how they are derived are described in its U.S. Patent Application No. 15 / 466,636, filed on March 22, 2017, the disclosure of which is hereby incorporated by reference in its entirety. Other examples are described with respect to FIG. 5 of the present disclosure.

[0014] After capturing the images, the imaging device 110 transmits the images to the retinal disease diagnosis tool 140. The retinal disease diagnosis tool 140 can take one or more images as input and use a machine learning model to output a diagnosis completely autonomously based on the input. The retinal disease diagnosis tool 140 autonomously analyzes the retinal images and uses machine learning analysis of biomarkers therein to determine the diagnosis. The diagnosis can specifically be a determination that the user has a specific disease such as diabetic retinopathy, or a determination that the user is likely to have a disease and thus should receive a doctor's examination for confirmation and treatment. The manner in which the retinal disease diagnosis tool 140 performs the analysis and determines the diagnosis is further discussed in its U.S. Patent No. 10,115,194, published on October 30, 2019, the disclosure of which is hereby incorporated by reference in its entirety.

[0015] Prior to performing the diagnosis, the retinal disease diagnosis tool 140 may include a retinal image pre - processing tool 130 that determines whether the image(s) captured by the imaging device 110 is / are sufficient for performing the diagnosis. The manner in which the retinal image pre - processing tool 130 performs this analysis is described in further detail below with respect to FIG. 3. While depicted as a component of the retinal disease diagnosis tool 140, the retinal image pre - processing tool 130 may be a stand - alone entity that receives an image, examines the image with respect to sufficiency, and then transmits the image to the retinal disease diagnosis tool 140 if the image is sufficient for diagnosis. The retinal image pre - processing tool 130 and / or the retinal disease diagnosis tool 140 may be instantiated on one or more servers. Further, the retinal image pre - processing tool 130 and / or the retinal disease diagnosis tool 140 may be instantiated, in whole or in part, at the imaging device 110 and thus may eliminate some or all of the need for communication traffic via the network 120.

[0016] (b) Exemplary Imaging Device Components FIG. 2 is an exemplary block diagram of modules and components of an imaging device according to one embodiment. The imaging device 110 includes an image capture component 211, a flash component 212, a retinal disease diagnosis tool application protocol interface (API) 214, and a user interface 215. Although not depicted, the imaging device 110 may include other components such as built - in instances of either or both of the retinal image pre - processing tool 130 and the retinal disease diagnosis tool 140, as well as any of their components. The imaging device 110 may include any database or memory for performing any of the functions described herein. The imaging device 110 may similarly exclude some of the depicted components.

[0017] The image capture component 211 may be any sensor configured to capture an image of a patient's retina. For example, a special lens may be used to capture an image of the patient's retina. The flash component 212 may be any component capable of illuminating the patient's retina during image capture by the image capture component 211, and may be configured to emit light in cooperation with the image capture operation of the image capture component 211.

[0018] The retinal disease diagnosis tool API 214 interfaces with the retinal disease diagnosis tool 130 to convert commands from the retinal image preprocessing tool 130 to the imaging device 110. Exemplary commands may include commands to capture or recapture an image, commands to adjust the intensity of the light emitted by the flash component 212, and the like. These commands, and how they are generated, are considered in more detail below with reference to FIG. 3.

[0019] The user interface 215 is an interface through which an operator of the imaging device 110 can command the imaging device 110 to perform any function that enables the operator to capture an image, adjust the flash intensity, and the like. The user interface 215 may be any hardware or software interface, and may include physical components (e.g., buttons) and / or graphic components (e.g., on a display such as a touch screen display). The user interface 215 may be located on the imaging device 110, may be a peripheral device of the imaging device 110, or may be located on a device separated from the imaging device 110 by the network 120, thereby enabling remote operation of the imaging device 110. An exemplary user interface is shown and considered in more detail with respect to FIG. 5.

[0020] (c) Exemplary retinal image preprocessing tool components 3 is an exemplary block diagram of modules and components of a retinal image preprocessing tool, according to one embodiment. The retinal image preprocessing tool 130 includes an initial image capture module 331, a preprocessing module 232, a preliminary image evaluation module 233, an image recapture module 234, and an image stitching module 235. Although not depicted, the retinal image preprocessing tool 110 may include other components, such as additional modules, and any databases or memories for performing any functionality described herein. Additionally, the retinal image preprocessing tool 130 may exclude some depicted components.

[0021] The initial image capture module 331 captures a retinal image of one or both of the patient's eyes. The term "capture," as used herein, may refer to causing an image to be taken, i.e., "capturing" may include instructing the imaging device 110 from a remote device or server to take an image and transmit the image over the network 120 to the initial image capture module 331. Alternatively, if the initial image capture module 331 is present on the imaging device 110, the initial image capture module 331 may capture the image by instructing the imaging device to take an image and route it internally to the pre-processing module 332.

[0022] The initial images captured by the initial image capture module 331 may include images of different portions of the patient's retina. The images of the different portions, when used together as inputs to one or more machine learning models of the retinal disease diagnosis tool 140, result in an output of a diagnosis of a retinal disease. The retinal regions may be predefined. For example, an administrator of the retinal disease diagnosis tool 140 may indicate that certain retinal regions should be captured for diagnosis. The regions may be defined as images with the fovea at the center and as images with the optic disc at the center. Thus, the initial image capture module 331 may capture images for one or both of the patient's eyes with those defined retinal regions at the center.

[0023] In certain embodiments, the captured image may be a multi-frame video over a period of time. For example, similar to "Live Photos", each frame may be captured from the moment the flash is first emitted until the moment the flash is completely dissipated, and when each frame is displayed continuously, it forms a video. Capturing multiple frames improves the likelihood that one of the frames includes a portion of the retina depicted by an image that meets a preprocessing criterion (described below), even if that portion of another frame does not meet the criterion. For example, following a flash example, if an underexposure, overexposure, or shadow caused by a certain flash intensity is corrected when the flash is adjusted to another intensity (e.g., as the flash is blocked, it gets dimmer after 0.05 seconds), the portion of the retina depicted within the frame at the other intensity can be used to meet the criterion.

[0024] The preprocessing module 332 is used to determine whether the captured image is sufficient for input into the machine learning model. The sufficiency may be defined by an administrator and may be based on parameters such as underexposure (e.g., the image was captured using a very low flash intensity), overexposure (e.g., the image was captured using a very high flash intensity), blurriness (e.g., the patient moved when the image was captured, obscuring the image), shadows, and / or any other defined parameters. The criteria may be established based on these parameters to determine whether an image is sufficient. For example, "the image must be within a certain distance of exposure" and / or "a landmark (e.g., the boundary of the optic nerve head) must be of a sufficiently narrow width" (in which case a wide boundary indicates blurriness), etc. The preprocessing module 332 compares the parameters of the image to the criteria to determine whether each image is sufficient for input into the machine learning model.

[0025] In one embodiment, if the preprocessing module 332 determines that the image is insufficient for input into the machine learning model, the retinal image preprocessing tool 130 may instruct that the insufficient image be recaptured. However, this embodiment has the disadvantage that the patient's health may be affected in that an extra flash of the patient's eyeball is required. There is also a technical disadvantage because extra bandwidth and processing power are required to capture the image and retransmit it to the retinal image preprocessing tool 130. In one embodiment, the preliminary image evaluation module 333 is used to determine whether a diagnosis can be performed without recapturing the image, despite the inadequacy of a given image. The preliminary image evaluation model 333 identifies one or more depicted portions of the retina in an image that do not meet one or more criteria. For example, the image may be partially overexposed or underexposed because the retinal pigmentation in the patient's retina is inconsistent and a consistent level of flash causes appropriate exposure in one part of the patient's retina while causing inappropriate exposure in another part of the patient's retina. As another example, the image may have shadows that can obscure one or more biomarkers depicted in portions of the retina while leaving the rest of the image in a state of sufficient quality. Thus, the portion(s) that meet one or more criteria and / or the portion(s) that do not meet the criteria may be separated by the preliminary image evaluation model 333. The preliminary image evaluation model 333 may identify these portions by examining them at any basis (e.g., pixel unit, quadrant unit, region unit, etc.) for the portions of the image that meet and do not meet the criteria.

[0026] If it is determined that the portion of the retina depicted by the image is insufficient, the preliminary image evaluation model 333 examines one or more other images captured by the initial image capture module 331 to determine whether that portion can be recovered without recapturing the image, and removes the insufficiency. For example, if a portion of the retina depicted within an image where the fovea of the patient's retina is centered is insufficient, the preliminary image evaluation model 333 may determine whether the same portion of the retina is depicted within an image where the optic disc of the patient's retina, which was also captured during the initial image capture, is centered. Another example may include evaluating whether frames of a multi-frame video of which the insufficient image is a part can be used. Since the preliminary image evaluation model 333 evaluates whether each captured image is sufficient for diagnosis, if the same portion of the retina is depicted within another image, the preliminary image evaluation model 333 can immediately determine whether that same portion is of sufficient quality. If the same portion is of sufficient quality, the preliminary image evaluation module 333 determines that image recapture is not necessary when a preliminary image is available. If the preliminary image evaluation module 333 determines that the same image of the retina is not depicted within another image of sufficient quality, the preliminary image evaluation module 333 determines that the image should be recaptured. In one embodiment, if different insufficient portions of the image are repaired by portions from two or more preliminary images, the two or more images may be used together as a preliminary for a single image.

[0027] The image recapture module 334 recaptures the image if the image is insufficient and, in some embodiments, if there is no appropriate preliminary image to remove the insufficiency. Except for being used to capture an image of an already captured area of the retina, the image recapture module 334 functions in the same manner as the image capture module 331.

[0028] The image stitching module 335 stitches sufficient portions of an image with replacement content for insufficient portions of the image, where the replacement content is from a preliminary image. The term "stitching", as used herein, refers to either the physical or logical integration of portions from two images. Physical stitching refers to generating a composite image that includes portions from at least two images. Logical stitching refers to obtaining different portions from different images for use in a diagnosis of those separate portions. For example, each portion is separately input into a machine learning model without generating a composite image, and a diagnosis is output therefrom. As another example, sufficient portions of an image may be used to generate a first analysis without considering the insufficient portions of that image. The preliminary image may have portions corresponding to the insufficient portion(s) of the first image that are used to generate a second analysis. The two analyses may be used to perform a diagnosis (e.g., by inputting the analyses into a machine learning model, whether or not the images themselves are present). In one embodiment, the image stitching module 335 weights areas of insufficient image quality based on the distribution rate of biomarkers in those areas with respect to the distance from anatomical markers such as the fovea and optic nerve head, enabling a higher confidence in processing those investigations even if there are areas with insufficient image quality in the fovea of the stitched image. Such weighting can be based on the training of random forests, ANNs, and equivalents on a representative dataset to learn and weight the importance of various sufficient image quality areas in the fovea. The image stitching module 335 is depicted as a module of the retinal image preprocessing tool 130, while the image stitching module 335 may instead be, in whole or in part, a module of the retinal disease diagnosis tool 140.

[0029] (d) Exemplary computer architecture FIG. 4 is a block diagram illustrating exemplary components of a machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller). Specifically, FIG. 4 shows a schematic representation of a machine in an exemplary form of system 400, in which program code (e.g., software) for causing a machine to perform any one or more of the methods discussed herein may be implemented. The program code may comprise instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., network-connected) to other machines. In a network-connected deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0030] The machine may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions 424 (sequentially or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute instructions 124 to perform any one or more of the methods discussed herein.

[0031] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include a software driver that enables the display of a user interface on a screen (or display). The visual interface may display the user interface directly (e.g., on the screen) or indirectly, on a surface, window, or the like (e.g., via a visual projection unit). For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with an interface with a touch screen. The computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or a touch screen keyboard), a cursor control device 414 (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate via the bus 408.

[0032] The memory unit 416 may include a machine-readable medium 422, on which instructions 424 (e.g., software) are stored that embody any one or more of the methods or functions described herein. The instructions 424 (e.g., software) may be present, in whole or in part, within the main memory 404 or within the processor 402 (e.g., within the processor's cache memory) during execution thereof by the computer system 400, and the main memory 404 and the processor 402 may constitute a machine-readable medium. The instructions 424 (e.g., software) may be transmitted or received via the network interface device 420 over the network 426.

[0033] In one exemplary embodiment, the machine-readable medium 422 is shown as a single medium, but the term "machine-readable medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" should also be construed to include any medium capable of storing instructions (e.g., instructions 424) for machine execution and causing a machine to perform any one or more of the methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid state memory, optical media, and magnetic media.

[0034] (e) Exemplary Images and User Interfaces FIG. 5 depicts an exemplary image of a patient's eye according to one embodiment. The user interface 500 may be displayed to the operator of the imaging device 110 as part of the user interface 215. The user interface 500 includes images 510, 520, 530, and 540. As depicted, the patient has two captured images of each eye, namely, an image with the fovea centralis in the center and an image with the optic nerve head in the center. As depicted, these images are merely exemplary, and any parameters may be used as input by the operator of the imaging device 110 and / or an administrator who determines what images are necessary for performing a diagnosis to indicate where the images should be centered. Further, in the depicted embodiment, two images are used, while any number of images may be used. The images are centered around the optic nerve head and the fovea centralis, but portions of the images overlap, and thus, if any of the overlapping portions of one image do not meet the defined criteria, each image may be used as a backup for the other images.

[0035] (f) Exemplary data flow for minimizing retinal exposure to the flash during image collection for diagnosis FIG. 6 is an exemplary flowchart for minimizing retinal exposure to the flash during image collection for diagnosis according to one embodiment. Process 600 begins with one or more processors (e.g., processor 402) of the device used to activate the retinal image preprocessing tool 130 capturing a plurality of retinal images (602) (e.g., using the initial image capture module 331), where each retinal image corresponds to a different retinal region of a plurality of retinal regions, and the plurality of retinal images includes a first image and a second image. The plurality of retinal images may include, for example, an image with the fovea centralis in the center and an image with the optic nerve head in the center for each eye. The first image may be the image 510 with the fovea centralis of the patient's left eye in the center, the second image may be the image 530 with the optic nerve head of the patient's left eye in the center, and the different retinal regions may correspond to the retinal regions in which each image is centered.

[0036] The retinal image preprocessing tool 130 determines (604) that a first portion of the first image does not meet the criteria while a second portion of the first image meets the criteria. For example, due to retinal striations, overexposure, underexposure, shadows, or other problems, it is determined that a part of the left-eye image 510 with the fovea at the center is insufficient, while the remaining part of the left-eye image 510 with the fovea at the center is sufficient. The determination of sufficiency may be performed by the preprocessing module 332 using any manner described herein.

[0037] The retinal image preprocessing tool 130 identifies (606) a part of the retina depicted within the first portion that does not meet the criteria. That is, a part of the retina itself may be depicted at different coordinates of the preliminary image, and thus, it is the part of the retina that is identified (not the part of the image). The preliminary image evaluation module 333 may perform this determination in any manner described herein. The retinal image preprocessing tool 130 then determines (608) whether the part of the retina is depicted within a third portion of the second image. That is, the preliminary image evaluation module 333 determines whether another captured image includes a depiction of the same part of the retina at a certain part of the other captured image. For example, the retinal image preprocessing tool 130 determines whether the left-eye image 530 with the optic disc at the center depicts the same part of the retina that was insufficient in the left-eye image 510 with the fovea at the center.

[0038] The retinal image preprocessing tool 130 determines (610) whether a third portion of the second image (e.g., a portion of the left-eye image 530 with the optic nerve head at the center that captures the same portion of the retina as that which is insufficient in the left-eye image 510 with the fovea at the center) meets a criterion. In response to the third portion meeting the criterion, the retinal image preprocessing tool 130 performs a diagnosis (612) using a plurality of retinal images (e.g., using the image stitching module 335). In response to the third portion not meeting the criterion, the retinal image preprocessing tool 130 captures (614) additional images of the retinal region depicted within the first image (e.g., using the image recapture module 334).

[0039] (g) Overview The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0040] While the present disclosure specifically focuses on retinal diseases, the present disclosure is generally applicable to disease diagnosis in general, as well as the capture (and potentially recapture) and preprocessing of images of other parts of a patient's body. For example, the disclosed preprocessing may be applied to the diagnosis of other organs such as the kidneys, liver, brain, or heart, or portions thereof, and the images considered herein may be of the relevant organs.

[0041] Some portions of this description describe embodiments of the present invention from the perspective of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the gist of their research to other skilled artisans. While these operations are functionally, computationally, or logically described, it is understood that they may be implemented by a computer program, equivalent electrical circuitry, microcode, or the like. Further, it has also been proven convenient at times to refer to these arrangements of operations as modules without loss of generality. The operations described and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0042] Any of the steps, operations, or processes described herein may be performed or implemented alone or in combination with other devices using one or more hardware or software modules. In one embodiment, a software module is implemented using a computer program product that includes a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0043] Embodiments of the present invention may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored within a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Further, any computing system referred to herein may include a single processor or may be an architecture that employs multiple processor designs to increase computing power.

[0044] Embodiments of the present invention may also relate to a product produced by the computing processes described herein. Such a product may comprise information resulting from the computing process, and where the information is stored on a non-transitory tangible computer-readable storage medium, may include any embodiment of a computer program product or other combination of data described herein.

[0045] Finally, the terms used herein are primarily selected for readability and instructional purposes and are not selected to precisely describe or circumscribe the subject matter of the invention. Accordingly, the scope of the present invention is intended to be limited not by this detailed description but by any claims that issue upon an application based on this specification. Thus, the disclosure of embodiments of the present invention is intended to be illustrative but not to limit the scope of the invention as set forth in the following claims.

Claims

1. A method of operating a system to minimize retinal exposure to a flash during image acquisition for diagnosis, the system comprising one or more processors, the method comprising: the one or more processors capturing a plurality of retinal images, each retinal image corresponding to a different retinal region of a plurality of retinal regions, the plurality of retinal images including a first image and a second image; the one or more processors determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion; the one or more processors identifying a portion of the retinal region corresponding to the first portion that does not meet the criterion; the one or more processors determining whether the portion of the retinal region corresponds to a third portion of the second image; the one or more processors determining whether the third portion meets the criterion; in response to the one or more processors determining that the third portion meets the criterion, the one or more processors performing the diagnosis using the second portion of the first image and the third portion of the second image; in response to the one or more processors determining that the portion of the retinal region does not correspond to any portion of the second image, the one or more processors capturing additional images of the retinal region corresponding to the first image A method, comprising.

2. Performing the diagnosis includes transmitting the plurality of retinal images to a retinal disease diagnosis tool, the retinal disease diagnosis tool taking as input the second portion of the first image and the third portion of the second image and using a machine learning model to fully autonomously output the diagnosis based on the input, the diagnosis being a determination that there is a high likelihood that the user of the system has a disease. The method according to claim 1.

3. Each retinal image of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at a different level of flash exposure. The method according to claim 1.

4. Determining that the first part of the first image does not meet the criterion includes determining that the first part of the first image is either overexposed or underexposed, the method according to claim 1.

5. The method according to claim 1, wherein the second image is an image of the retina of the same eyeball as depicted by the first image.

6. Performing the diagnosis using the second part of the first image and the third part of the second image comprises generating a composite image including the second part of the first image and the third part of the second image, and performing the diagnosis using the composite image, the method according to claim 1.

7. Performing the diagnosis using the second part of the first image and the third part of the second image comprises analyzing the first image without considering the first part to generate a first analysis, analyzing the third part of the second image to generate a second analysis, and performing the diagnosis using the first analysis and the second analysis, the method according to claim 1.

8. Capturing the additional image comprises transmitting, using an application protocol interface (API), a command for capturing an additional image to an imaging device using the viewpoint used for capturing the first image, and receiving the additional image from the imaging device, the method according to claim 1.

9. A computer program product for minimizing retinal exposure to a flash during image collection for diagnosis, comprising capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among a plurality of retinal regions, the plurality of retinal images including a first image and a second image, determining that a first part of the first image does not meet a criterion while a second part of the first image meets the criterion, identifying a part of the retinal region corresponding to the first part that does not meet the criterion, determining whether the part of the retinal region corresponds to a third part of the second image, and determining whether the third part meets the criterion. ​ In response to determining that the third portion meets the criteria, performing the diagnosis using the second portion of the first image and the third portion of the second image; In response to determining that the portion of the retinal region does not correspond to any portion of the second image, capturing an additional image of the retinal region corresponding to the first image; A computer program product including a non-transitory computer-readable storage medium including computer program code for performing the above. **Claim 10** The computer program code for performing the diagnosis includes computer program code for transmitting the plurality of retinal images to a retinal disease diagnosis tool, the retinal disease diagnosis tool taking as input the second portion of the first image and the third portion of the second image, and using a machine learning model to completely autonomously output the diagnosis based on the input, the diagnosis being a determination that the user is likely to have a disease, and the plurality of retinal images having been captured from the user. The computer program product according to claim 9. **Claim 11** Each retinal image of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at a different level of flash exposure. The computer program product according to claim 9. **Claim 12** The computer program code for determining that the first portion of the first image does not meet the criteria includes computer program code for determining that the first portion of the first image is either overexposed or underexposed. The computer program product according to claim 9. **Claim 13** The second image is an image of the retina of the same eyeball as depicted by the first image. The computer program product according to claim 9. **Claim 14** The computer program code for performing the diagnosis using the second portion of the first image and the third portion of the second image includes generating a composite image including the second portion of the first image and the third portion of the second image; performing the diagnosis using the composite image; and computer program code for performing the above. The computer program product according to claim 9. The computer program code for performing the diagnosis using the second portion of the first image and the third portion of the second image is analyzing the first image without considering the first portion and generating a first analysis; analyzing the third portion of the second image and generating a second analysis; performing the diagnosis using the first analysis and the second analysis The computer program product according to claim 9, comprising computer program code for performing the above.

16. The computer program code for capturing the additional image is transmitting, using an application protocol interface (API), a command for capturing an additional image to an imaging device using the viewpoint used for capturing the first image; receiving the additional image from the imaging device The computer program product according to claim 9, comprising computer program code for performing the above.

17. A computer program product for minimizing retinal exposure to a flash during image acquisition for diagnosis, the computer program product comprising a computer-readable storage medium including computer program code, the computer program code, when executed by one or more processors, a first software module for capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among a plurality of retinal regions, the plurality of retinal images including a first image and a second image; a second software module, determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion; identifying a part of the retinal region corresponding to the first portion that does not meet the criterion; determining whether the part of the retinal region corresponds to a third portion of the second image; determining whether the third portion meets the criterion a second software module for performing the above; a third software module for performing the diagnosis using the second portion of the first image and the third portion of the second image in response to determining that the third portion meets the criterion; In response to determining that the portion of the retinal region does not correspond to any part of the second image, a fourth software module for capturing an additional image of the retinal region corresponding to the first image and A computer program product implemented on the one or more processors. **Claim 18** Performing the diagnosis includes transmitting the plurality of retinal images to a retinal disease diagnosis tool, the retinal disease diagnosis tool taking as input the second portion of the first image and the third portion of the second image, and using a machine learning model to fully autonomously output the diagnosis based on the input, the diagnosis being a determination that the user is likely to have a disease, the computer program product according to claim 17, wherein the plurality of retinal images have been captured from the user.

Images