Image retention and stitching for minimal flash ocular disease diagnosis

Abstract

To provide image retention and stitching for minimal-flash eye disease diagnosis.SOLUTION: There are provided systems and methods for minimizing retinal exposure to flash during image gathering for diagnosis. In an embodiment, a system captures a plurality of retinal images of different retinal regions. The system determines that a first portion of a first image does not meet a criterion while a second portion of the first image does meet the criterion, identifies a portion of the retina depicted in the first portion that does not meet the criterion, and determines whether the portion of the retina is depicted in a third portion of a second image and whether the third portion meets the criterion. Responsive to determining that the third portion meets the criterion, the system performs the diagnosis. Responsive to determining that the portion of the retina is not depicted in the second image, the system captures an additional image of the retinal region.SELECTED DRAWING: Figure 6

Description

Technical Field

[0002] , ,

[0003] , , , , , , ,

[0001] (Background) The present invention generally relates to autonomous diagnosis of retinal abnormalities, and more specifically to image retention for reducing the need to flash the patient's retina.

Background Art

[0002] An autonomous system for diagnosing retinal abnormalities captures images of a patient's retina (used herein with the same meaning as the word "fundus") and analyzes those images for abnormalities. Typically, images of different parts of the retina (e.g., an image with the fovea at the center and an image with the optic nerve head at the center) are captured. When the captured images are insufficient for performing a sufficient diagnosis, additional images are captured until sufficient images of each part of the retina are obtained. Repeated exposure to the flash by the imaging device can further cause pupil constriction that reduces the likelihood of normal imaging after each flash. Thus, reducing the need for additional images to be captured until sufficient images are captured reduces the number of flashes and the need to take numerous photographs or the need to make the patient wait to be diagnosed until the pupil has recovered to normal dilation. In addition, reducing the need for additional images in terms of preventing the need to delay the patient's diagnosis until such time that the number of flashes that significantly increases pupil constriction is unlikely to be reached in a seated position and thus the likelihood of obtaining an executable image to flash the patient's eye again can enable a more rapid diagnosis.

Summary of the Invention

Means for Solving the Problems

[0003] (Summary) A system and method are provided herein for reducing the need to flash the patient's eye again to capture an image of the retinal region when the previously captured image is insufficient. In one embodiment, two images may be captured to perform a diagnosis of a patient's right eye, namely an image with the fovea in the center and an image with the optic nerve head in the center. There is some overlap between the different parts of the retina depicted in these two images. If a region of the retina depicted in one of the images is unusable for any reason (e.g., overexposure or underexposure), instead of recapturing the image, the system may determine whether the same region is depicted in the other image without the same deficiencies. If the region is usable from the other image, the system can avoid the need to flash the patient's right eye again to capture another image of that region, and instead, the parts of the two images may be stitched together to perform the diagnosis.

[0004] In one embodiment, to minimize retinal exposure to flash during image acquisition for diagnostic purposes, a retinal image preprocessing tool captures multiple retinal images (for example, by instructing the imaging device to take images and send them to the retinal image preprocessing tool). Each retinal image may correspond to a different retinal region among multiple retinal regions (e.g., a region with the fovea in the center and a region with the optic nerve head in the center). The retinal image preprocessing tool may determine that a first portion of the first image (e.g., the portion with the fovea in the center) does not meet the criteria (e.g., the image is overexposed or underexposed, the image has shadows or other artifacts, etc.), while a second portion of the first image (e.g., the portion with the optic nerve head in the center) does meet the criteria (e.g., the second image is properly exposed).

[0005] A retinal image preprocessing tool may identify a portion of the retina depicted within a first portion that does not meet the criteria and may determine that the same portion of the retina is depicted within a third portion of a second image. The retinal image preprocessing tool may determine whether the third portion meets the criteria. In response to determining that the third portion meets the criteria, a diagnosis may be performed using multiple retinal images. In response to determining that the portion of the retina is not depicted within the second image, the retinal image preprocessing tool may capture additional images of the retinal region depicted within the first image. The present invention provides, for example, the following items: (Item 1) A method for minimizing retinal exposure to flash during image acquisition for diagnostic purposes, The method involves capturing multiple retinal images, each retinal image corresponding to a different retinal region among multiple retinal regions, and the multiple retinal images include a first image and a second image. It is determined that the first portion of the first image does not meet the criteria, while the second portion of the first image does meet the criteria. Identifying the portion of the retina depicted within the first portion that does not meet the aforementioned criteria, To determine whether the portion of the retina is depicted within the third portion of the second image, The third part above is to determine whether it satisfies the criteria, In response to determining that the third part satisfies the criteria, the diagnosis is performed using the plurality of retinal images, In response to determining that the portion of the retina is not depicted in the second image, to capture an additional image of the retinal region depicted in the first image. A method that includes this. (Item 2) The method according to item 1, wherein performing the diagnosis includes passing the images through a fully autonomous machine learning model that outputs the likelihood of disease based on the multiple retinal images. (Item 3) The method according to item 1, wherein each of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at different levels of flash exposure. (Item 4) The method according to item 1, wherein determining that the first portion of the first image does not meet the criteria includes determining that the first portion of the first image is either overexposed or underexposed. (Item 5) The method according to item 1, wherein the second image is an image of the retina of the same eyeball as the first image depicts. (Item 6) Performing the diagnosis using the aforementioned multiple retinal images means To generate a composite image that includes the first part of the image together with the third part of the second image stitched into the first image, The diagnosis is performed using the aforementioned composite image. The method described in item 1, including the method described in item 1. (Item 7) Performing the diagnosis using the aforementioned multiple retinal images means Analyzing the first image without taking the first part into consideration and generating the first analysis, Analyzing the third portion of the second image and generating the second analysis, The diagnosis is performed using the first and second analyses described above. The method described in item 1, including the method described in item 1. (Item 8) Capturing the aforementioned additional images is Using an Application Protocol Interface (API), a command is transmitted to the imaging device to capture an additional image using the viewpoint used to capture the first image, Receiving the additional image from the aforementioned imaging device The method described in item 1, including the method described in item 1. (Item 9) The method of item 1, wherein performing the diagnosis using the aforementioned plurality of retinal images includes using the third portion in the diagnosis. (Item 10) A computer program product for minimizing retinal exposure to flash during image acquisition for diagnostic purposes, The method involves capturing multiple retinal images, each retinal image corresponding to a different retinal region among multiple retinal regions, and the multiple retinal images include a first image and a second image. It is determined that the first portion of the first image does not meet the criteria, while the second portion of the first image does meet the criteria. Identifying the portion of the retina depicted within the first portion that does not meet the aforementioned criteria, To determine whether the portion of the retina is depicted within the third portion of the second image, The third part above is to determine whether it satisfies the criteria, In response to determining that the third part satisfies the criteria, the diagnosis is performed using the plurality of retinal images, In response to determining that the portion of the retina is not depicted in the second image, to capture an additional image of the retinal region depicted in the first image. A computer program product including a non-temporary computer-readable storage medium containing computer program code for performing the following actions. (Item 11) The computer program product according to item 10, wherein the computer program code for performing the diagnosis includes computer program code for passing the images through a fully autonomous machine learning model that outputs the likelihood of disease based on the plurality of retinal images. (Item 12) The computer program product described in item 10, wherein each of the plurality of retinal images includes a multi-frame video, and each frame of the multi-frame video captures an image at different levels of flash exposure. (Item 13) The computer program code for determining that the first portion of the first image does not meet the criteria includes computer program code for determining that the first portion of the first image is either overexposed or underexposed, the computer program product according to item 10. (Item 14) The computer program product according to item 10, wherein the second image is an image of the retina of the same eyeball as depicted in the first image. (Item 15) The computer program code for performing the diagnosis using the plurality of retinal images includes generating a composite image including the first portion of the image, together with the third portion of the second image stitched within the first image, and performing the diagnosis using the composite image, the computer program product according to item 10. (Item 16) The computer program code for performing the diagnosis using the plurality of retinal images includes analyzing the first image without considering the first portion and generating a first analysis, analyzing the third portion of the second image and generating a second analysis, and performing the diagnosis using the first analysis and the second analysis, the computer program product according to item 10. (Item 17) The computer program code for capturing the additional image includes transmitting, to an imaging device, a command for capturing an additional image using a perspective used for capturing the first image using an application protocol interface (API), and receiving the additional image from the imaging device, The computer program product according to item 10, comprising computer program code for performing (Item 18) The computer program product according to item 10, wherein performing the diagnosis using the plurality of retinal images includes using the third portion in the diagnosis (Item 19) A computer program product for minimizing retinal exposure to a flash during image acquisition for diagnosis, A first module for capturing a plurality of retinal images, each retinal image corresponding to a different retinal region among the plurality of retinal regions, the plurality of retinal images including a first image and a second image, a first module; A second module, Determining that a first portion of the first image does not meet a criterion while a second portion of the first image meets the criterion; Identifying a part of the retina depicted within the first portion that does not meet the criterion; Determining whether the part of the retina is depicted within a third portion of the second image; Determining whether the third portion meets the criterion A second module for performing; A third module for performing the diagnosis using the plurality of retinal images in response to determining that the third portion meets the criterion; A fourth module for capturing additional images of the retinal region depicted within the first image in response to determining that the part of the retina is not depicted within the second image A computer program product including a computer-readable storage medium including computer program code including (Item 20) The computer program product according to item 19, wherein performing the diagnosis includes passing the images through a fully autonomous machine learning model that outputs the likelihood of a disease based on the plurality of retinal images [Brief explanation of the drawing]

[0006] (Brief explanation of the drawing) [Figure 1] Figure 1 is an exemplary block diagram of system components in an environment for using a retinal image preprocessing tool according to one embodiment.

[0007] [Figure 2] Figure 2 is an illustrative block diagram of an imaging device module and components according to one embodiment.

[0008] [Figure 3] Figure 3 is an illustrative block diagram of modules and components of a retinal image preprocessing tool according to one embodiment.

[0009] [Figure 4] Figure 4 is a block diagram illustrating the components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).

[0010] [Figure 5] Figure 5 illustrates an exemplary image of a patient's eyeball according to one embodiment.

[0011] [Figure 6] Figure 6 is an illustrative flowchart of one embodiment for minimizing retinal exposure to flash during image acquisition for diagnostic purposes.

[0012] The figures illustrate various embodiments of the present invention for illustrative purposes only. Those skilled in the art will immediately recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles of the present invention described herein. [Modes for carrying out the invention]

[0013] (Detailed explanation) (a) Overview of the environment Figure 1 is an exemplary block diagram of system components in an environment for utilizing a retinal image preprocessing tool according to one embodiment. The environment 100 includes an imaging device 110, a network 120, a retinal image preprocessing tool 130, and a retinal disease diagnostic tool 140. The imaging device 110 is a device configured to capture one or more images of the retina of a patient's eyeball. The imaging device 110 can be made to capture such images autonomously when commanded through manual operation, by computer program instructions or external signals (e.g., received from the retinal pigment deposition determination tool 130), or a combination thereof. Embodiments of how these images may appear and how they are derived are described in the user's own U.S. Patent Application No. 15 / 466,636, filed March 22, 2017, the disclosure of which is incorporated herein by reference in its entirety. Other embodiments are described with respect to Figure 5 of this disclosure.

[0014] After capturing an image, the imaging device 110 transmits the image to the retinal disease diagnostic tool 140. The retinal disease diagnostic tool 140 may take one or more images as input and may, using a machine learning model, fully autonomously output a diagnosis based on the input. The retinal disease diagnostic tool 140 autonomously analyzes the retinal images and determines the diagnosis using machine learning analysis of biomarkers therein. The diagnosis may specifically be a determination that the user has a specific disease such as diabetic retinopathy, or a determination that the user is highly likely to have a disease and should therefore see a doctor for confirmation and treatment. The manner in which the retinal disease diagnostic tool 140 performs the analysis and determines the diagnosis is further discussed in its own U.S. Patent No. 10,115,194, published October 30, 2019, the disclosure of which is incorporated herein by reference in its entirety.

[0015] Prior to performing a diagnosis, the retinal disease diagnostic tool 140 may include a retinal image preprocessing tool 130 that determines whether the images(s) captured by the imaging device 110 are sufficient for performing a diagnosis. The manner in which the retinal image preprocessing tool 130 performs this analysis is described in more detail below with reference to Figure 3. While depicted as a component of the retinal disease diagnostic tool 140, the retinal image preprocessing tool 130 may also be a standalone entity that receives images, examines them for sufficiency, and then, if the images are sufficient for diagnosis, transmits them to the retinal disease diagnostic tool 140. The retinal image preprocessing tool 130 and / or the retinal disease diagnostic tool 140 may be instantiated on one or more servers. Furthermore, the retinal image preprocessing tool 130 and / or the retinal disease diagnostic tool 140 may be instantiated in whole or in part on the imaging device 110, thus eliminating the need for some or all of the communication traffic over the network 120.

[0016] (b) Components of an exemplary imaging device Figure 2 is an illustrative block diagram of modules and components of an imaging device according to one embodiment. The imaging device 110 includes an image acquisition component 211, a flash component 212, a retinal disease diagnostic tool application protocol interface (API) 214, and a user interface 215. Although not depicted, the imaging device 110 may include other components, such as built-in instances of either or both of the retinal image preprocessing tool 130 and the retinal disease diagnostic tool 140, as well as any components thereof. The imaging device 110 may include any database or memory for performing any functions described herein. The imaging device 110 may also exclude some components that are depicted.

[0017] The image capture component 211 may be any sensor configured to capture an image of the patient's retina. For example, a special lens may be used to capture an image of the patient's retina. The flash component 212 may be any component capable of illuminating the patient's retina during image capture by the image capture component 211, and may be configured to emit light in coordination with the image capture operation of the image capture component 211.

[0018] The retinal disease diagnostic tool API 214 interfaces with the retinal disease diagnostic tool 130 to translate commands from the retinal image preprocessing tool 130 to the imaging device 110. Exemplary commands may include commands for capturing or recapturing images, commands for adjusting the intensity of light emitted by the flash component 212, and their equivalents. These commands, and how they are generated, are discussed in further detail below with reference to Figure 3.

[0019] The user interface 215 is an interface through which an operator of the imaging device 110 can instruct the imaging device 110 to perform any function, such as capturing an image, adjusting the flash intensity, and so on. The user interface 215 may be any hardware or software interface and may include physical components (e.g., buttons) and / or graphic components (e.g., on a display such as a touchscreen display). The user interface 215 may be located on the imaging device 110, on a peripheral device of the imaging device 110, or on a device isolated from the imaging device 110 by the network 120, thereby enabling remote operation of the imaging device 110. An exemplary user interface is shown with reference to Figure 5 and will be discussed in more detail.

[0020] (c) Exemplary retinal image preprocessing tool component Figure 3 is an illustrative block diagram of modules and components of a retinal image preprocessing tool according to one embodiment. The retinal image preprocessing tool 130 includes an initial image acquisition module 331, a preprocessing module 232, a preliminary image evaluation module 233, an image recapture module 234, and an image stitching module 235. Although not depicted, the retinal image preprocessing tool 110 may include other components such as additional modules, and any database or memory for performing any functions described herein. The retinal image preprocessing tool 130 may also omit some of the depicted components.

[0021] The initial image acquisition module 331 captures images of the retina of one or both of the patient's eyes. The term “capture,” as used herein, may mean to cause an image to be taken; i.e., “capturing” may include instructing the imaging device 110 from a remote device or server to take an image and transmit the image to the initial image acquisition module 331 via the network 120. Alternatively, if the initial image acquisition module 331 resides on the imaging device 110, the initial image acquisition module 331 may capture an image by instructing the imaging device to take an image and route it internally to the preprocessing module 332.

[0022] The initial images captured by the initial image capture module 331 may include images of different parts of the patient's retina. These images of different parts, when used together as input to one or more machine learning models of the retinal disease diagnostic tool 140, yield an output of a retinal disease diagnosis. The retinal regions may be predefined. For example, the administrator of the retinal disease diagnostic tool 140 may indicate that certain retinal regions should be captured for diagnosis. Regions may be defined as images with the fovea centrally located, and images with the optic nerve head centrally located. Thus, the initial image capture module 331 may capture images relating to one or both of the patient's eyes, where those defined retinal regions are centrally located.

[0023] In one embodiment, the captured image may be a multi-frame video over a period of time. For example, similar to “Live Photo,” each frame may be captured from the moment the flash is first emitted until the moment the flash has completely dissipated, and each frame, when displayed consecutively, forms an image. Capturing multiple frames increases the likelihood that one of the frames will contain a portion of the retina depicted by an image that satisfies the pre-processing criteria (described below) (even if that portion in another frame does not satisfy the criteria). For example, if, following a flash example, underexposure, overexposure, or shadowing caused by a certain flash intensity is corrected when the flash is adjusted to a different intensity (e.g., the flash dims after 0.05 seconds as it is blocked), then a portion of the retina depicted in the frame at the other intensity may be used to satisfy the criteria.

[0024] The preprocessing module 332 is used to determine whether the captured images are sufficient for input into the machine learning model. Sufficiency may be defined by the administrator and may be based on parameters such as underexposure (e.g., the image was captured using a very low flash intensity), overexposure (e.g., the image was captured using a very high flash intensity), blurriness (e.g., the patient moved when the image was captured, blurring the image), shadows, and / or any other specified parameters. Criteria may be established based on these parameters to determine whether an image is sufficient, for example, "the image must be within a certain exposure distance" and / or "landmarks (e.g., the boundary of the optic nerve head) must be sufficiently narrow" (in this case, a wide boundary indicates blurriness), etc. The preprocessing module 332 compares the image parameters to the criteria and determines whether each image is sufficient for input into the machine learning model.

[0025] In one embodiment, if the preprocessing module 332 determines that the image is insufficient for input into the machine learning model, the retinal image preprocessing tool 130 may be instructed to recapture the insufficient image. However, this embodiment has the disadvantage that the patient's health may be affected, as it requires an extra flash of the patient's eyeball. There is also a technical disadvantage, as extra bandwidth and processing power are required to capture the image and retransmit it to the retinal image preprocessing tool 130. In one embodiment, a preliminary image evaluation module 333 is used to determine whether a diagnosis can be performed without recapturing the image despite the inadequacy of a given image. The preliminary image evaluation model 333 identifies one or more depicted portions of the retina in an image that do not meet one or more criteria. For example, the image may be partially overexposed or underexposed because retinal pigment deposition in the patient's retina is inconsistent, leading to a consistent level of flash causing proper exposure in some parts of the patient's retina while causing inappropriate exposure in other parts. In another embodiment, the image may have shading, which may obscure one or more biomarkers depicted in parts of the retina while leaving the rest of the image in a good quality state. Thus, portions (one or more) that meet one or more criteria and / or portions (one or more) that do not meet the criteria may be isolated by the preliminary image evaluation model 333. The preliminary image evaluation model 333 may identify these portions by examining the portions of the image that meet and do not meet the criteria on any basis (e.g., pixel-level, quadrant-level, region-level, etc.).

[0026] If it is determined that the portion of the retina depicted in an image is insufficient, the preliminary image evaluation model 333 examines one or more other images captured by the initial image acquisition module 331 to determine whether the portion is recoverable without recapturing the image and to eliminate the insufficiency. For example, if a portion of the retina depicted in an image where the fovea of ​​the patient's retina is central is insufficient, the preliminary image evaluation model 333 may determine whether the same portion of the retina depicted in an image where the optic nerve head of the patient's retina, which was also captured during initial image acquisition, is central. Another embodiment may include evaluating whether a frame of a multi-frame video in which the insufficient image is part can be used. Since the preliminary image evaluation model 333 evaluates whether each captured image is sufficient for diagnosis, if the same portion of the retina is depicted in another image, the preliminary image evaluation model 333 can immediately determine whether that identical portion is of sufficient quality. If the identical portion is of sufficient quality, the preliminary image evaluation module 333 determines that recapture of the image is not necessary when a preliminary image is available. If the backup image evaluation module 333 determines that the same image of the retina is not depicted in another image with sufficient quality, the backup image evaluation module 333 determines that the image should be recaptured. In one embodiment, if different inadequate parts of an image are restored by parts from two or more backup images, the two or more images may be used together as backups for a single image.

[0027] The image recapture module 334 recaptures an image if the image is insufficient, and in some embodiments, if a suitable backup image to eliminate the insufficiency is not available. Aside from being used to capture images of already captured areas of the retina, the image recapture module 334 functions in the same manner as the image capture module 331.

[0028] The image stitching module 335 stitches together a sufficient portion of an image with replacement content for insufficient portions of the image, the replacement content being from a backup image. The term “stitching,” as used herein, refers to either the actual or logical aggregation of portions from two images. Actual stitching refers to generating a composite image containing portions from at least two images. Logical stitching refers to obtaining different portions from different images using those distinct portions for diagnosis. For example, each portion is individually input into a machine learning model without generating a composite image, from which a diagnosis is output. In another embodiment, a sufficient portion of an image may be used to generate a first analysis without considering the insufficient portions of that image. The backup image may have portions corresponding to the insufficient portions (one or more) of the first image, which are used to generate a second analysis. The two analyses may be used to perform a diagnosis (for example, by inputting the analyses into a machine learning model, with or without the images themselves). In one embodiment, the image stitching module 335 weights areas of insufficient image quality based on the distribution rate of biomarkers in those areas with respect to distance from anatomical markers such as the fovea and optic disc, enabling greater confidence in processing their investigation even when areas with insufficient image quality exist in the fovea of ​​the image being stitched. Such weighting can be based on training a random forest, ANN, and equivalent on a representative dataset to learn and weight the importance of various areas of sufficient image quality in the fovea. While the image stitching module 335 is described as a module of the retinal image preprocessing tool 130, the image stitching module 335 may instead be a module of the retinal disease diagnostic tool 140, in whole or in part.

[0029] (d) Exemplary computer architecture Figure 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller). Specifically, Figure 4 shows a schematic representation of the machine in an exemplary form of system 400, in which program code (e.g., software) may be implemented to cause the machine to perform any one or more of the methods discussed herein. The program code may consist of instructions 424 that can be executed by one or more processors 402. In alternative embodiments, the machine may operate as a standalone device or be connected to other machines (e.g., networked). In a networked deployment, the machine may operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0030] A machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions 424 that define the actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term “machine” should also be understood to include any collection of machines that individually or together execute instructions 124 to perform any one or more of the methods considered herein.

[0031] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), main memory 404, and static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include a software driver that enables the display of a user interface on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly on a surface, window, or equivalent (e.g., via a visual projection unit). For ease of consideration, the visual interface may be described as a screen. The visual interface 410 may include an interface with a touchable screen, or may interface with such a screen. The computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touchscreen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing device), a memory unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate via a bus 408.

[0032] The storage unit 416 may include a machine-readable medium 422 on which instructions 424 (e.g., software) embodying any one or more of the methods or functions described herein are stored. The instructions 424 (e.g., software) may reside, all or in part, in the main memory 404 or in the processor 402 (e.g., in the processor's cache memory) during their execution by the computer system 400, and the main memory 404 and the processor 402 may constitute a machine-readable medium. The instructions 424 (e.g., software) may be transmitted or received via the network interface device 420 and the network 426.

[0033] In one exemplary embodiment, the machine-readable medium 422 is shown as a single medium, but the term “machine-readable medium” should be understood to include a single or multiple mediums (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term “machine-readable medium” should also be understood to include any medium capable of storing instructions for machine execution (e.g., instructions 424) and causing a machine to perform any one or more of the methods disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0034] (e) Exemplary images and user interfaces Figure 5 depicts exemplary images of a patient's eyeball according to one embodiment. User interface 500 may be displayed to the operator of the imaging device 110 as part of user interface 215. User interface 500 includes images 510, 520, 530, and 540. As depicted, the patient has two captured images of each eyeball, namely an image with the fovea centered and an image with the optic disc centered. As depicted, these images are merely illustrative and may be used to indicate where the images should be centered, with any parameters being input by the operator of the imaging device 110 and / or the administrator to determine what images are needed to perform the diagnosis. Furthermore, while two images are used in the embodiment depicted, any number of images may be used. Images are centered around the optic disc and fovea, but some of the images overlap, and therefore, if any of the overlapping portions of one image does not meet the specified criteria, each image may be used as a backup for the other image.

[0035] (f) Exemplary data flow to minimize retinal exposure to flash during image acquisition for diagnosis Figure 6 is an illustrative flowchart of one embodiment for minimizing retinal exposure to flash during image acquisition for diagnostic purposes. Process 600 begins with one or more processors (e.g., processor 402) of the device used to launch the retinal image preprocessing tool 130 capturing multiple retinal images (602) (e.g., using the initial image acquisition module 331), where each retinal image corresponds to a different retinal region of multiple retinal regions, and the multiple retinal images include a first image and a second image. The multiple retinal images may include, for example, an image with the fovea centered and an image with the optic disc centered for each eyeball. The first image may be an image 510 with the fovea centered for the patient's left eye, and the second image may be an image 530 with the optic disc centered for the patient's left eye, and the different retinal regions may correspond to the retinal region in which each image is centered.

[0036] The retinal image preprocessing tool 130 determines that a first portion of the first image does not meet the criteria, while a second portion of the first image does (604). For example, a portion of the left eye image 510 with the fovea in the center is determined to be insufficient due to retinal striates, overexposure, underexposure, shadows, or other issues, while the remaining portion of the left eye image 510 with the fovea in the center is determined to be sufficient. The determination of sufficiency may be carried out by the preprocessing module 332 using any of the methods described herein.

[0037] The retinal image preprocessing tool 130 identifies a portion of the retina that is depicted within a first portion that does not meet the criteria (606). That is, a portion of the retina itself may be depicted at different coordinates in the preliminary image, and therefore it is the identified portion of the retina (not a portion of the image). The preliminary image evaluation module 333 may perform this determination in any manner described herein. The retinal image preprocessing tool 130 then determines whether the portion of the retina is depicted within a third portion of the second image (608). That is, the preliminary image evaluation module 333 determines whether another captured image includes a depiction of the same portion of the retina in a portion of the other captured image. For example, the retinal image preprocessing tool 130 determines whether the left eye image 530, with the optic disc in the center, depicts the same portion of the retina that was insufficient in the left eye image 510, with the fovea in the center.

[0038] The retinal image preprocessing tool 130 determines whether a third portion of the second image (for example, a portion of the left eye image 530 with the optic nerve head in the center that captures the same portion of the retina as the insufficient portion in the left eye image 510 with the fovea in the center) meets the criteria (610). In response that the third portion meets the criteria, the retinal image preprocessing tool 130 performs a diagnosis using multiple retinal images (for example, using the image stitching module 335) (612). In response that the third portion does not meet the criteria, the retinal image preprocessing tool 130 captures additional images of the retinal region depicted in the first image (for example, using the image recapture module 334) (614).

[0039] (g) Overview The foregoing description of embodiments of the present invention is presented for illustrative purposes only and is not intended to be exhaustive or to limit the invention to any specific form disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.

[0040] While this disclosure specifically focuses on retinal diseases, it generally applies to disease diagnosis in general, as well as the acquisition (and possible recapture) and preprocessing of images of other parts of a patient's body. For example, the preprocessing disclosed may be applied to the diagnosis of other organs such as the kidneys, liver, brain, or heart, or parts thereof, and the images considered herein may be of the relevant organs.

[0041] Several parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of information-related operations. These algorithmic descriptions and representations are commonly used by those skilled in the art in data processing techniques to convey the gist of the research to others skilled in the art. While these operations are described functionally, computationally, or logically, it is understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or equivalents. Furthermore, it has been demonstrated that it is sometimes convenient to refer to these sequences of operations as modules without loss of generality. The operations described and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0042] Any step, operation, or process described herein can be performed or implemented, either alone or in combination with other devices, using one or more hardware or software modules. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described herein.

[0043] Embodiments of the present invention may also relate to apparatus for carrying out the operations described herein. Such apparatus may comprise a general-purpose computing device that is specifically constructed for a required purpose and / or selectively activated or reconfigured by a computer program stored within the computer. Such computer programs may be stored in non-temporary tangible computer-readable storage media or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referenced herein may comprise a single processor or may be an architecture employing multiple processor designs to increase computing power.

[0044] Embodiments of the present invention may also relate to products produced by the computational processes described herein. Such products may comprise information arising from the computational processes, and may include any embodiment of a computer program product or any combination of other data described herein, provided that the information is stored on a non-temporary, tangible, computer-readable storage medium.

[0045] Finally, the terminology used herein has been selected primarily for readability and guidance purposes, and not to precisely describe or define the subject matter of the invention. Therefore, the scope of the invention is intended to be limited not by this detailed description, but by any claims published at the time of filing a claim herein. Thus, the disclosure of embodiments of the invention, while illustrative, is not intended to limit the scope of the invention as set forth in the following claims.

Claims

1. A method for operating a system to minimize retinal exposure to flash during image acquisition for diagnostic purposes, wherein the system comprises one or more processors, and the method One or more processors, in response to a capture device deciding to focus on a first retinal region, capture a first retinal image of the first retinal region by flashing the first retinal region, The one or more processors capture a second retinal image of the second retinal region by flashing the second retinal region, wherein the first retinal image and the second retinal image together form a plurality of retinal images. The one or more processors determine whether to flash again on the first retinal region, and the determination is The one or more processors determine that the first portion of the first retinal image corresponding to the first portion of the first retinal region does not meet the quality standards, In response to one or more processors determining that the first portion of the first retinal image does not meet the quality criteria, one or more processors determine whether the first portion of the first retinal region corresponds to the second portion of the second retinal image. In response to one or more processors determining that the first portion of the first retinal region corresponds to the second portion of the second retinal image, one or more processors determine whether the second portion of the second retinal image meets the quality criteria, In response to one or more processors determining that the second portion of the second retinal image meets the quality criteria, one or more processors determine not to flash the first retinal region again. The act of being done by, The one or more processors transmit the plurality of retinal images to a retinal disease diagnostic tool, the retinal disease diagnostic tool takes a portion of the plurality of retinal images that meet the quality criteria as input, and uses a machine learning model to output a fully autonomous decision that the user of the system is highly likely to have the disease based on the input. Methods that include...

2. The method according to claim 1, further comprising the one or more processors deciding to flash the first retinal region again and recapture the first retinal image in response to the one or more processors determining that the second portion of the second retinal image does not meet the quality criteria.

3. The method according to claim 1, wherein capturing the first retinal image comprises capturing a multi-frame image while the first retinal region is illuminated by emitting the flash, the emitting of the flash being caused by a single flash, and each frame of the multi-frame image captures an image at different levels of flash exposure.

4. The method according to claim 1, wherein determining that the first portion of the first retinal image does not meet the quality criteria includes determining that the first portion of the first retinal image is either overexposed or underexposed.

5. The method according to claim 1, wherein the second retinal image is an image of the retina of the same eyeball as the first retinal image.

6. The aforementioned method, The one or more processors generate a composite image that includes a second portion of the first retinal image and a second portion of the second retinal image that meet the quality criteria, The one or more processors perform the diagnosis using the synthesized image. The method according to claim 1, further comprising:

7. The aforementioned method, The one or more processors analyze the first retinal image without taking into account the first portion of the first retinal image and generate the first analysis. The one or more processors analyze the second portion of the second retinal image and generate the second analysis, The one or more processors perform the diagnosis using the first analysis and the second analysis. The method according to claim 1, further comprising:

8. A computer program product for minimizing retinal exposure to flash during image acquisition for diagnostic purposes, wherein the computer program product is In response to the capture device deciding to focus on a first retinal region, a first retinal image of the first retinal region is captured by flashing onto the first retinal region, The method involves capturing a second retinal image of a second retinal region by emitting a flash onto the second retinal region, wherein the first retinal image and the second retinal image together form a plurality of retinal images. The determination of whether to flash again in the first retinal region, the determination of which Determining that the first portion of the first retinal image corresponding to the first portion of the first retinal region does not meet the quality standards, In response to determining that the first portion of the first retinal image does not meet the quality criteria, it is determined whether the first portion of the first retinal region corresponds to the second portion of the second retinal image, In response to determining that the first retinal region, the first portion, corresponds to the second portion of the second retinal image, it is determined whether the second portion of the second retinal image meets the quality criteria, In response to determining that the second portion of the second retinal image meets the quality criteria, it is decided not to flash the first retinal region again. The act of being done by, The process involves transmitting the aforementioned multiple retinal images to a retinal disease diagnostic tool, wherein the retinal disease diagnostic tool takes a portion of the aforementioned multiple retinal images that meet the aforementioned quality standards as input, and uses a machine learning model to output a fully autonomous decision that the user is highly likely to have a disease based on the input. A computer program product including a non-temporary computer-readable storage medium containing computer program code for performing the following.

9. The computer program product according to claim 8, wherein the computer program code further determines, in response to determining that the second portion of the second retinal image does not meet the quality criteria, to flash the first retinal region again and recapture the first retinal image.

10. The computer program product according to claim 8, wherein capturing the first retinal image includes capturing a multi-frame image while the first retinal region is illuminated by emitting the flash, the emitting of the flash being caused by a single flash, and each frame of the multi-frame image captures an image at different levels of flash exposure.

11. The computer program product according to claim 8, wherein determining that the first portion of the first retinal image does not meet the quality criteria includes determining that the first portion of the first retinal image is either overexposed or underexposed.

12. The computer program product according to claim 8, wherein the second retinal image is an image of the retina of the same eyeball as the first retinal image.

13. The aforementioned computer program code is: To generate a composite image that includes a second portion of the first retinal image and a second portion of the second retinal image that meet the aforementioned quality standards, The diagnosis is performed using the aforementioned composite image. A computer program product according to claim 8, which further performs the above.

14. The aforementioned computer program code is: Analyzing the first retinal image without taking into account the first portion of the first retinal image, and generating the first analysis, Analyzing the second portion of the second retinal image and generating a second analysis, The diagnosis is performed using the first analysis and the second analysis. A computer program product according to claim 8, which further performs the above.

15. A system for minimizing retinal exposure to flash during image acquisition for diagnostic purposes, wherein the system is: Memory with encoded instructions, One or more processors and Equipped with, When one or more processors execute the instruction, In response to the capture device deciding to focus on a first retinal region, a first retinal image of the first retinal region is captured by flashing onto the first retinal region, The method involves capturing a second retinal image of a second retinal region by emitting a flash onto the second retinal region, wherein the first retinal image and the second retinal image together form a plurality of retinal images. The determination of whether to flash again in the first retinal region, the determination of which Determining that the first portion of the first retinal image corresponding to the first portion of the first retinal region does not meet the quality standards, In response to determining that the first portion of the first retinal image does not meet the quality criteria, it is determined whether the first portion of the first retinal region corresponds to the second portion of the second retinal image, In response to determining that the first retinal region, the first portion, corresponds to the second portion of the second retinal image, it is determined whether the second portion of the second retinal image meets the quality criteria, In response to determining that the second portion of the second retinal image meets the quality criteria, it is decided not to flash the first retinal region again. The act of being done by, The process involves transmitting the aforementioned plurality of retinal images to a retinal disease diagnostic tool, wherein the retinal disease diagnostic tool takes a portion of the plurality of retinal images that meet the aforementioned quality criteria as input, and uses a machine learning model to output a fully autonomous decision that, based on the input, the user of the system is highly likely to have a disease. A system that can perform actions including those mentioned above.

16. The system according to claim 15, further comprising determining, in response to determining that the second portion of the second retinal image does not meet the quality criteria, to flash again over the first retinal region and recapture the first retinal image.

17. The system according to claim 15, wherein capturing the first retinal image includes capturing a multi-frame image while the first retinal region is illuminated by emitting the flash, the emitting of the flash being caused by a single flash, and each frame of the multi-frame image captures an image at different levels of flash exposure.

18. The system according to claim 15, wherein determining that the first portion of the first retinal image does not meet the quality criteria includes determining that the first portion of the first retinal image is either overexposed or underexposed.

19. The system according to claim 15, wherein the second retinal image is an image of the retina of the same eyeball as the first retinal image depicts.

20. The aforementioned operation is, To generate a composite image that includes a second portion of the first retinal image and a second portion of the second retinal image that meet the aforementioned quality standards, The diagnosis is performed using the aforementioned composite image. The system according to claim 15, further comprising:

Images