Dynamic regulation of flash intensity based on retinal pigment deposition

Abstract

To provide dynamic adjustment of flash intensity based on retinal pigmentation.SOLUTION: Systems and methods are disclosed herein for adjusting flash intensity based on retinal pigmentation. In an embodiment, a processor determines a retinal pigmentation of a retina of an eye positioned at an imaging device. The processor commands the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation. The processor commands the imaging device to capture an image that is lit by the flash component at the second intensity, and receives the image from the imaging device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] (Background) The present invention generally relates to autonomous diagnosis of retinal abnormalities, and more specifically, to adjusting the flash intensity of an imaging device based on retinal pigmentation.

Background Art

[0002] An autonomous system for diagnosing retinal abnormalities captures images of a patient's retina (used herein with the same meaning as the word "fundus") and analyzes those images for abnormalities. The images are captured using a preset flash intensity. However, like the outer skin, a user's fundus can be pigmented, and pigmentation can result in under-exposed or over-exposed images when using a preset flash intensity. Under-exposure or over-exposure can obscure the biomarkers used to diagnose abnormalities, making the images sub-optimal, and thus preventing the system from diagnosing abnormalities that should appear on the image if it were properly exposed.

[0003] Images that are under-exposed or over-exposed can also be insufficient for diagnosis, thus requiring the patient to hold still through multiple images and each image to be flashed again. Repeated exposure to the flash by the imaging device can damage the patient's eye, and thus preventing or reducing the probability of under-exposure or over-exposure of the image improves the patient's health in that fewer flashes are required to capture a sufficient image of the patient's eye.

Summary of the Invention

Means for Solving the Problems

[0004] (Summary) Systems and methods for determining a patient's retinal pigmentation and adjusting flash intensity based on the determined pigmentation are provided herein. For example, retinal pigmentation may be determined based on an initial image that is overexposed or underexposed, and the flash intensity used to capture a subsequent image may be adjusted based on that overexposure or underexposure. In another embodiment, infrared light may be used to determine retinal pigmentation, thus eliminating the need for an initial image to be captured from the patient using a flash. In yet another embodiment, an image of the patient's skin or hair may be captured, from which pigmentation may be determined. The systems and methods disclosed herein, once pigmentation is learned, advantageously prevent overexposure and underexposure of the captured image and prevent the need to unnecessarily expose the patient's eyeball to multiple flashes due to unsuccessful imaging.

[0005] To achieve these and other objectives, in one embodiment, a processor (e.g., in a server) determines the retinal pigment deposition of the retina of an eyeball positioned in an imaging device (e.g., a camera located away from the server). Based on the retinal pigment deposition, the processor instructs the imaging device to adjust the intensity of the flash component from a first intensity to a second intensity. The processor then instructs the imaging device to capture an image illuminated by the flash component at the second intensity, and receives the image from the imaging device. The image may be used to diagnose whether retinal abnormalities are observed within the patient's retina. The present invention provides, for example, the following items: (Item 1) A method for adjusting flash intensity based on retinal pigmentation, To determine the retinal pigment deposition of the retina of the eyeball positioned in the imaging device, Based on the retinal pigment deposition, the imaging device is instructed to adjust the intensity of the flash component from a first intensity to a second intensity, Commanding the imaging device to capture an image illuminated by the flash component at the second intensity, Receiving the image from the aforementioned imaging device and A method that includes this. (Item 2) Determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving a first image of the eyeball from the imaging device, To determine whether the first image is underexposed, In response to determining that the first image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. The method described in item 1, including the method described in item 1. (Item 3) Determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving a first image of the eyeball from the imaging device, To determine whether the first image is overexposed, In response to determining that the first image is overexposed, it is determined that the second intensity is an intensity reduced relative to the first intensity. The method described in item 1, including the method described in item 1. (Item 4) Determining the retinal pigment deposition of the eyeball positioned in the imaging device is: The imaging device is instructed to obtain feedback by emitting an infrared signal in the retina, Receiving the feedback from the imaging device, Based on the aforementioned feedback, the retinal pigment deposition of the eyeball is determined. The method described in item 1, including the method described in item 1. (Item 5) The method according to item 1, wherein instructing the imaging device to adjust the intensity of the flash component includes transmitting the instruction using an Application Protocol Interface (API). (Item 6) The method according to item 1, wherein the first intensity is at least one of the default intensity or the last used intensity. (Item 7) To determine whether the received image is properly exposed, In response to determining that the received image is not properly exposed, the imaging device is instructed to capture an additional image using a third intensity. The method described in item 1, further including the method described in item 1. (Item 8) Determining the retinal pigment deposits of the retina includes identifying a plurality of pigment deposits within the retina, wherein the second intensity is determined based on the first retinal pigment deposit among the plurality of pigment deposits, and the method is The imaging device is instructed to adjust the intensity of the flash component from the second intensity to the third intensity based on the second retinal pigment deposit among the plurality of pigment deposits. Commanding the imaging device to capture an additional image illuminated by the flash component at the third intensity, Receiving the additional image from the aforementioned imaging device The method described in item 1, further including the method described in item 1. (Item 9) The method according to item 1, further comprising diagnosing a retinal disease of the retina based on the characteristics of the aforementioned image. (Item 10) A computer program product for adjusting flash intensity based on retinal pigmentation, To determine the retinal pigment deposition of the retina of the eyeball positioned in the imaging device, Based on the retinal pigment deposition, the imaging device is instructed to adjust the intensity of the flash component from a first intensity to a second intensity, Commanding the imaging device to capture an image illuminated by the flash component at the second intensity, Receiving the image from the aforementioned imaging device and A computer program product comprising a non-temporary computer-readable storage medium containing computer program code for performing the following actions. (Item 11) The computer program code for determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving a first image of the eyeball from the imaging device, To determine whether the first image is underexposed, In response to determining that the first image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. A computer program product as described in item 10, including computer program code for performing the following actions. (Item 12) The computer program code for determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving a first image of the eyeball from the imaging device, To determine whether the first image is overexposed, In response to determining that the first image is overexposed, it is determined that the second intensity is an intensity reduced relative to the first intensity. A computer program product as described in item 10, including computer program code for performing the following actions. (Item 13) The computer program code for determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Instructing the imaging device to obtain feedback by emitting an infrared signal in the retina, Receiving the feedback from the imaging device, Determining the retinal pigmentation of the eye based on the feedback, The computer program product according to item 10, comprising computer program code for performing the above. (Item 14) The computer program code for instructing the imaging device to adjust the intensity of the flash component includes computer program code for transmitting commands using an application protocol interface (API). The computer program product according to item 10. (Item 15) The computer program product according to item 10, wherein the first intensity is at least one of a default intensity or the last used intensity. (Item 16) The computer program code, Determining whether the received image is properly exposed, In response to determining that the received image is not properly exposed, instructing the imaging device to capture an additional image using a third intensity, The computer program product according to item 10, further comprising computer program code for performing the above. (Item 17) The computer program code for determining the retinal pigmentation of the retina includes computer program code for identifying a plurality of pigmentations in the retina. The second intensity is determined based on a first retinal pigmentation among the plurality of pigmentations. The computer program code, Based on a second retinal pigmentation among the plurality of pigmentations, instructing the imaging device to adjust the intensity of the flash component from the second intensity to a third intensity, Commanding the imaging device to capture an additional image illuminated by the flash component at the third intensity, Receiving the additional image from the aforementioned imaging device The computer program products described in item 10, further including computer program code for performing the following actions. (Item 18) The computer program product according to item 10, further comprising computer program code for diagnosing retinal diseases of the retina based on the characteristics of the image. (Item 19) A computer program product for adjusting flash intensity based on retinal pigmentation, A first module for determining retinal pigment deposition in the retina of an eyeball positioned in an imaging device, A second module for instructing the imaging device to adjust the intensity of the flash component from a first intensity to a second intensity based on the retinal pigment deposition, A third module for instructing the imaging device to capture an image illuminated by the flash component at the second intensity, A fourth module for receiving the image from the imaging device and A computer program product comprising a computer-readable storage medium containing computer program code. (Item 20) The aforementioned first module is, Receiving a first image of the eyeball from the imaging device, To determine whether the first image is underexposed, In response to determining that the first image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. A computer program product as described in item 19, comprising submodules for performing the following. [Brief explanation of the drawing]

[0006] (Brief explanation of the drawing) [Figure 1] Figure 1 is an exemplary block diagram of system components in an environment for using a retinal pigmentation determination tool according to one embodiment.

[0007] [Figure 2] Figure 2 is an illustrative block diagram of an imaging device module and components according to one embodiment.

[0008] [Figure 3] Figure 3 is an illustrative block diagram of modules and components of a retinal pigmentation determination tool according to one embodiment.

[0009] [Figure 4] Figure 4 is a block diagram illustrating the components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).

[0010] [Figure 5] Figure 5 illustrates exemplary images of various levels of exposure according to one embodiment.

[0011] [Figure 6] Figure 6 illustrates an exemplary image of a retina with striated retinal pigmentation according to one embodiment.

[0012] [Figure 7] Figure 7 illustrates an exemplary flowchart for adjusting flash intensity based on retinal pigmentation according to one embodiment.

[0013] The figures illustrate various embodiments of the present invention for illustrative purposes only. Those skilled in the art will immediately recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles of the present invention described herein. [Modes for carrying out the invention]

[0014] (Detailed explanation) (a) Overview of the environment Figure 1 is an exemplary block diagram of system components in an environment for utilizing a retinal pigment deposition determination tool according to one embodiment. The environment 100 includes an imaging device 110, a network 120, a retinal pigment deposition determination tool 130, and a retinal disease diagnostic tool 140. The imaging device 110 is a device configured to capture one or more images of the retina of a patient's eyeball. The imaging device 110 can be made to capture such images autonomously when commanded through manual operation, by computer program instructions or external signals (e.g., received from the retinal pigment deposition determination tool 130), or a combination thereof. Examples of how these images may appear and how they are derived are described in the user's own U.S. Patent Application No. 15 / 466,636, filed March 22, 2017, the disclosure of which is incorporated herein by reference in its entirety.

[0015] After capturing an image, the imaging device 110 transmits the image to the retinal pigmentation determination tool 130 for processing. In one embodiment, though not depicted, the retinal pigmentation determination tool 130 is installed on the imaging device 110 as a module, and therefore the transmission is internal to the imaging device 110. In the embodiment depicted, the retinal pigmentation determination tool 130 is instantiated on a server separate from the imaging device 110, and the image is transmitted via the network 120. The network 120 may be any communication network such as a local area network, wide area network, the Internet, and equivalents.

[0016] Although not depicted in Figure 1, one or more additional imaging devices may be used to capture external images, including part or all of the patient (e.g., the patient's skin or hair). Retinal pigmentation can then be determined.

[0017] The retinal pigmentation determination tool 130 receives an image and determines whether the image is suitable for processing. In one embodiment, suitability for processing means that the image is exposed using or within a specified range of light intensity and that biomarkers in the patient's retina can be detected by the retinal disease diagnostic tool 140. In another embodiment, suitability for processing means that a certain marker in the patient's retina is identifiable in the image. Further consideration of methods for determining whether an image is suitable for processing is discussed with reference to Figures 2-7 below.

[0018] A factor contributing to an image being unsuitable for processing is retinal pigmentation; if the patient's retina is pigmented (whether light or dark), the retinal image may be overexposed or underexposed if a consistent flash intensity is used when acquiring the image. The retinal pigmentation determination tool determines, based on the retinal pigmentation, the necessary adjustments to the retinal pigmentation and / or flash intensity, and instructs the imaging device 110 (or its human operator) to adjust the flash intensity prior to acquiring the image or further images. The manner in which the retinal pigmentation determination tool 130 makes its determination and commands the adjustments is described in more detail with respect to Figure 3 below.

[0019] The retinal disease diagnostic tool 140 autonomously analyzes retinal images and uses machine learning analysis of biomarkers therein to determine a diagnosis. The diagnosis may specifically be a determination that the user has a specific disease, such as diabetic retinopathy, or a determination that the user is highly likely to have a disease and therefore should consult a physician for confirmation and treatment. The manner in which the retinal disease diagnostic tool 140 performs the analysis and determines the diagnosis is further described in its own U.S. Patent No. 10,115,194, the disclosure of which is incorporated herein by reference in its entirety. While described as a separate entity from the retinal pigment deposition determination tool 130, the retinal disease diagnostic tool 140 may be instantiated on the same server or set of servers as the retinal pigment deposition determination tool 130, and the retinal pigment deposition determination tool 130 may be installed in part or in whole on the imaging device 110, similar to how the retinal pigment deposition determination tool 130 may be installed in part or in whole as a module within the imaging device 110.

[0020] (b) Components of an exemplary imaging device Figure 2 is an illustrative block diagram of modules and components of an imaging device according to one embodiment. The imaging device 110 includes an image acquisition component 211, a flash component 212, an infrared component 213, a retinal pigment deposition determination tool application protocol interface (API) 214, and a user interface 215. Not depicted, the imaging device 110 may include other components, such as built-in instances of either or both of the retinal pigment deposition determination tool 130 and the retinal disease diagnostic tool 140, as well as any components thereof. The imaging device 110 may include any database or memory for performing any functions described herein. The imaging device 110 may also exclude some components that are depicted. For example, the imaging device may exclude the infrared component 213.

[0021] The image capture component 211 may be any sensor configured to capture an image of the patient's retina. For example, a special lens may be used to capture an image of the patient's retina. The flash component 212 may be any component capable of illuminating the patient's retina during image capture by the image capture component 211, and may be configured to emit light in coordination with the image capture operation of the image capture component 211. The image capture component 211 may also be configured with an external image capture component 211 for capturing images including the patient's skin and / or hair.

[0022] The infrared component 213 is an infrared sensor configured to transmit infrared radiation to the patient's retina and determine its absorption. The infrared component 213 may generate a heatmap showing the absorption of infrared transmission across the patient's retina. The infrared component 213 transmits the absorption determination and / or heatmap to a processor (e.g., of the retinal pigmentation determination tool 130) for processing with respect to the determination of the patient's retinal pigmentation.

[0023] The retinal pigment deposition determination tool API 214 interfaces with the retinal pigment deposition determination tool 130 to translate commands from the retinal pigment deposition determination tool 130 to the imaging device 110. Exemplary commands may include commands for capturing an image, commands for adjusting the intensity of light emitted by the flash component 212, and their equivalents. These commands, and how they are generated, are discussed in more detail below with reference to Figure 3.

[0024] The user interface 215 is an interface through which an operator of the imaging device 110 can instruct the imaging device 110 to perform any function that it is capable of, such as capturing an image, adjusting the flash intensity, capturing infrared information, and the same. The user interface 215 may be any hardware or software interface and may include physical components (e.g., buttons) and / or graphic components (e.g., on a display such as a touchscreen display). The user interface 215 may be located on the imaging device 110, on a peripheral device of the imaging device 110, or on a device that is isolated from the imaging device 110 by the network 120, thereby enabling remote operation of the imaging device 110.

[0025] (c) Exemplary retinal pigmentation determination tool component Figure 3 is an illustrative block diagram of modules and components of a retinal pigmentation determination tool according to one embodiment. The retinal pigmentation determination tool 130 includes a pigment determination module 331, a flash intensity adjustment module 334, and a striated retinal pigment imaging module 335. Not depicted, the retinal pigmentation determination tool 110 may include other components such as additional modules and any database or memory for performing any functions described herein. The retinal pigmentation determination tool 130 may also omit some of the depicted components. For example, the retinal pigmentation determination tool 130 may omit the striated retinal pigment imaging module 355.

[0026] The pigment determination module 331 determines the patient's retinal pigment deposition based on information received from the imaging device 110. The information may include an image or a series of images of the patient's retina, infrared absorption information, and / or a combination thereof. The pigment determination module 331 may perform submodules such as the exposure determination module 332 and / or the infrared pigment determination module 333. Alternatively, the exposure determination module 332 and / or the infrared pigment determination module 333 may be standalone modules rather than submodules of the pigment determination module 331.

[0027] Herein, we refer to an embodiment in which retinal pigmentation is determined based on a patient's retinal image received from the imaging device 110. The exposure determination module 332 determines whether the image is underexposed or overexposed. As used herein, the term “underexposed” may refer to an image being captured while the patient’s retina is exposed to insufficient light intensity from the flash component 212, which thus reduces the likelihood that biomarkers appearing in a properly exposed image due to insufficient illumination will be detected in the underexposed image. Similarly, the term “overexposed” may refer to an image being captured while the patient’s retina is exposed to excessive light intensity from the flash component 212, which thus reduces the likelihood that biomarkers appearing in a properly exposed image due to excessive illumination will be detected in the overexposed image. The term “biomarker” as used herein refers to an object in the image that is part of the patient’s retina corresponding to a retinal disease.

[0028] The exposure determination module 332 can determine whether an image is overexposed or underexposed in part or in whole by analyzing the appearance of the captured image (or a portion thereof) and determining the exposure level from it. For example, the brightness level, the grayscale level (e.g., at individual points or averages), the color intensity level, or any other unit of measurement of brightness, intensity, color scheme, and equivalents can be determined. The exposure level can be determined on a pixel-by-pixel basis, and the exposure determination module 332 generates a heatmap, where each part of the heatmap reflects the exposure level of the corresponding pixel in the captured image. The exposure determination module 332 can also determine the exposure level on an aggregated basis for the image. The exposure level for the entire image can be captured by performing statistical calculations (e.g., mean, median, mode) on the exposure level on a pixel-by-pixel or pixel group basis to identify the exposure level that reflects the entire image.

[0029] The exposure determination module 332 can determine whether the exposure level for each pixel, each pixel group, or the image as a whole falls within a predetermined exposure range. For example, the ranges of overexposure, underexposure, and proper exposure may be predefined by the administrator of the retinal pigmentation determination tool 130. The exposure determination module 332 can determine which range the exposure level falls within and, based on that range, determine whether the image is overexposed, underexposed, or properly exposed. The exposure determination module 332 can perform this overexposure, underexposure, or proper exposure determination for the entire image or for different parts of the image based on the exposure levels of pixel groups.

[0030] The exposure determination module 332 may use additional information to determine the exposure level for a pixel, a group of pixels, or an entire pixel. For example, the exposure determination module 332 may calculate the similarity of an image (or its pixels or groups of pixels) by referring to a reference image with typical exposure levels. A database showing similarity levels mapped to exposure levels may be referenced, and the exposure levels may be determined from there. In another embodiment, the exposure determination module 332 may filter the image and determine the difference in tonality in the color space (e.g., red-green-blue, cyan-magenta-yellow tones, etc.) with respect to the brightness of the image. The difference in tonality may be compared to entries in a database that maps the difference in tonality of exposure levels.

[0031] The exposure determination module 332 can determine retinal pigmentation based on the exposure level. To determine pigmentation, the exposure determination module 332 can access a data structure that maps exposure levels to pigmentation. Any aforementioned exposure level, i.e., exposure levels calculated on a pixel-by-pixel basis, for pixel groups, or for the entire image, can be mapped to pigmentation. A map showing pigmentation can be generated on a pixel-by-pixel basis based on the exposure level of each pixel.

[0032] The exposure determination module 332 may determine adjustment values ​​based on pigmentation and the flash intensity used to capture the image. For example, a data structure may be referenced that shows the exposure level corresponding to each flash intensity for each possible type of pigmentation. The data structure shows the adjustment value for that exposure level. Alternatively, the exposure determination module 332 may calculate adjustment values ​​based on the exposure level without referencing pigmentation. To perform this adjustment when considering the exposure level of the entire image, the exposure determination module 332 may determine the difference between the exposure level and a predetermined appropriate exposure value and assign that difference as an adjustment value.

[0033] In one embodiment, the exposure determination module 332 may calculate an adjustment value based on the exposure level for each pixel in the image, or based on the exposure levels for different groups of pixels in the image (for example, each quadrant of the image may have a different calculated exposure level). The exposure determination module 332 may calculate an adjustment value by maximizing the amount of pixels or groups of pixels that have an adjusted exposure level within the range of appropriate exposure. For example, if three of the four quadrants are overexposed, and an adjustment value is calculated that, when applied to the exposure levels of all four quadrants, would properly expose three of the four quadrants, and if there is no adjustment value that properly exposes all four quadrants, the adjustment value is applied to maximize the amount of properly exposed image.

[0034] In one embodiment, the exposure determination module 332 may not take into account the exposure levels of properly exposed pixels or groups of pixels when determining the adjustment value. This is because properly exposed pixels can be stitched together in the image, and poorly exposed images are compensated for by adjusting the flash intensity or otherwise considered by analyzing two separate images when the image is used for an intended purpose (e.g., detecting biomarkers). Therefore, the exposure determination module 332 may determine the intensity level for the image to be used when calculating the adjustment value by ignoring the intensity levels of properly exposed portions of the image while considering pixels or groups of pixels that are overexposed or underexposed.

[0035] The exposure determination module 332 may store the adjustment value and / or the adjusted value after the adjustment value has been applied in the user information database 336. As will be described in more detail below, the flash intensity adjustment module 334 may instruct the imaging device 110 to adjust its flash intensity based on the adjustment value. During a future imaging session, the retinal pigmentation determination tool 130 may identify the patient, read the flash intensity adjustment value from the user information 336, and determine the flash adjustment prior to image acquisition, thus eliminating the need to expose the patient to a flash that would result in an improperly exposed image. The adjustment value may be assigned to each eye of the patient (if the image is labeled to indicate which eyeball (left or right eye) it corresponds to) and may be used depending on which eyeball is being imaged.

[0036] In one embodiment, the exposure determination module 332 may determine retinal pigmentation and / or exposure values ​​by inputting an image into a machine learning model and receiving retinal pigmentation as output from the machine learning model. The machine learning model may be trained using images that are labeled as having certain retinal pigmentation and / or exposure values.

[0037] Herein, we refer to an embodiment in which, once infrared absorption information of the patient's eyeball is received from the imaging device 110, retinal pigmentation is determined based thereon. The infrared pigment determination module 333 receives the absorption information and / or heatmap from the imaging device 110 and calculates the patient's retinal pigmentation therefrom. In one embodiment, the infrared retinal pigmentation determination module 333 calculates the pigmentation by comparing the absorption information with information in a data structure that maps absorption to pigmentation. In another embodiment, the infrared retinal pigmentation determination module 333 inputs the absorption information into a machine learning model and receives the pigmentation as output from the machine learning model. When absorption information such as any patient's vital signs, history, or demographic information is read from the user information database 336, or when received from the imaging device 110 along with the absorption information, other inputs along with them are also sent to the machine learning model. After determining the pigmentation, the pigment determination module 331 may, with reference to the exposure determination module 332, determine the accommodation values ​​using the techniques described above.

[0038] In addition, adjustment values ​​may be communicated using other parameters. For example, flash intensity and / or gain may be compared separately from or in combination with other factors (e.g., gamma setting of imaging device 110) to statistical parameters of the infrared level (e.g., mean, median, etc.). The pigment determination module 331 may refer to a database using the comparison results and determine the corresponding pigment deposition in the database mapped to the comparison results.

[0039] In one embodiment, the exposure determination module 332 may determine retinal pigmentation and / or exposure values ​​by inputting an infrared image (or a heatmap as described above) into a machine learning model and receiving retinal pigmentation (or information that can be used to determine retinal pigmentation) as output from the machine learning model. The machine learning model may be a trained image-level classifier that outputs flash exposure time and intensity from an infrared image. The classifier may use predetermined features extracted from the infrared image to output flash exposure time and intensity. The flash exposure time and intensity may be used to determine retinal pigmentation by either referring to a database or inputting the flash exposure time and intensity into a trained classifier, which is trained to convert flash exposure time and intensity into pigmentation. The resulting exposure time and intensity received from the model may be derived from a representative normative database of various pigmentations.

[0040] The flash intensity adjustment module 334 transmits a command to the imaging device 110 to have a flash intensity emitted by the adjusted flash component 212. The command may include an amount to adjust the flash intensity, or it may include a new flash intensity value. In some embodiments, the adjustment to the flash intensity described above, such as being performed by the dye determination module 331 and / or its submodules, may instead be performed by the flash intensity adjustment module 334.

[0041] In one embodiment, the retinal pigmentation determination tool 130 may categorize the retina into a set of known retinal types or pigmentations, each known type or pigmentation in the set having a corresponding exposure time and / or flash intensity mapped to it in an electronic data structure. For example, the retinal pigmentation determination tool 130 may perform statistical calculations on the entire infrared image (e.g., or an infrared heatmap) or on specific features of the infrared image (e.g., pixels or groups of pixels, or features such as the optic disc, fovea, or blood vessels) to determine the mapping of the statistical calculations to known retinal types. The retinal pigmentation determination tool 130 may then determine retinal pigmentations from these. In another embodiment, the retinal pigmentation determination tool 130 may use an encoder in combination with a clustering algorithm (e.g., k-means clustering) to map the encoder output to a matching known retinal type or pigmentation. In yet another embodiment, the retinal pigmentation determination tool 130 may input the retinal image into a classification neural network and receive a known retinal type as the output. In yet another embodiment, one or more images of the iris may be used as input for determining a known retinal type, either separately from or in addition to the embodiments described above. The retinal pigmentation determination tool 130 may determine the exposure time and / or flash intensity for capturing an image of the retina based on the mapping in the electronic data structure. The electronic data structure may be continuously refined, for example, by storing essential metrics or by embedding the resulting image quality scores of infrared and fundus color images and feeding them into the classifier data as updated training data.

[0042] The striated retinal pigment imaging module 335 identifies striates within the retinal pigment deposits and determines remedial actions to ensure that a properly exposed retinal image can be acquired or constructed. The striated retinal pigment imaging module 335 may use exposure levels, pixel-by-pixel or pixel-group exposure maps, infrared absorption information, or any other information as described above, to determine whether the retinal image indicates that the patient has striated pigment deposits within the patient's retina.

[0043] In one embodiment, the striated pigment imaging module 335 may determine that a contiguous group of pixels exceeding a predetermined amount in the group has a predetermined amount of different intensity or exposure levels (received using, for example, infrared or intensity information) from neighboring pixel groups. For example, a pixel group with a high intensity adjacent to a pixel group with a low intensity may show different pigmentation at each point in the patient's retina where those pixels correspond. The striated retina pigment imaging module 335 may determine from these that there are striates in the patient's retina. To prevent or minimize misdetermination, the striated retina pigment imaging module 335 considers pixel groups that are at least a threshold size (e.g., at least 30 pixels wide) to ensure that what is causing the variation in intensity or exposure levels is pigmentation and not a biomarker or other artifact. While a common situation is vertical or horizontal stripes or streaks traversing the retina with different pigmentation, non-striated patterns of different pigmentation are also within the scope of this disclosure whenever the term “striate” is used herein. For example, whenever the term “linear” is used herein, tessellation may be identified. Tessellation may be detected using grayscale in the image, edge detectors, orientation edge detectors, Bag-of-Words, lexicon learning based on patch histograms, template matching, and equivalents.

[0044] Alternatively, or in addition, the streaking pigment imaging module 335 may use computer vision to determine if the retina is streaked. In one embodiment, the streaking pigment imaging module 335 may use computer vision to determine whether an edge (e.g., a boundary between two pigment deposits) is present. If an edge is present, the streaking pigment imaging module may determine the streaks using static or adaptive thresholding techniques, such as determining whether the difference in pigment deposits on each side of the edge differs by a threshold amount (e.g., threshold intensity or infrared information). The streaking pigment imaging module 335 performs a Hough transform at one or more angles in the computer vision model to identify semilinear objects that may represent an edge. In addition, or alternatively, the streaking pigment imaging module 335 compares contours to various color channels to identify streaks, and the contour comparison assists streaking detection when the color channel indicates that the streaks are part of pigment deposits that are less visible than typical. Statistical analysis may be performed on the infrared and intensity distributions, and the standard deviation is applied to determine the likelihood that streaks are detected. Furthermore, neural networks such as classification neural networks and segmentation neural networks can be trained and used to identify lines.

[0045] The striated retinal pigment imaging module 335 can instruct the imaging device 110 to capture two or more retinal images at different flash intensities, resulting in different exposure levels corresponding to the intensity levels required to be properly exposed at two or more intensities corresponding to the striates. The striated retinal pigment imaging module 335 can then use the properly exposed portions from each image to stitch together two or more images to create a composite image that is properly exposed throughout. Alternatively, the striated retinal pigment imaging module 335 can maintain each image separately for further analysis. For example, a module performing a diagnosis of retinal disease can analyze the properly exposed portions of each image to detect biomarkers and output a diagnosis based on the biomarkers identified across two or more images.

[0046] The user information database 336 is a database that can maintain a profile for each patient. The profile may include any collected information, including brief history information (e.g., name, height, weight), demographic information (e.g., ethnicity, geographical location), and any other information (e.g., health records).

[0047] Although not described, in one embodiment, an image including the patient's hair and / or skin is received by the retinal pigmentation determination tool 130. The retinal pigmentation tool may then determine the patient's retinal pigmentation. In one embodiment, the retinal pigmentation tool 130 may refer to a data structure that maps the patient's hair and / or skin color to retinal pigmentation and takes a mapped retinal pigmentation value as the assumed patient's retinal pigmentation. In another embodiment, the retinal pigmentation tool 130 may input the image or data derived therefrom into a machine learning model that is trained to output retinal pigmentation. This may be done before an image of the patient's retina is captured and may be used to adjust the flash intensity prior to capturing the image in any manner described herein.

[0048] (d) Exemplary computer architecture Figure 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller). Specifically, Figure 4 shows a schematic representation of the machine in an exemplary form of system 400, in which program code (e.g., software) may be implemented to cause the machine to perform any one or more of the methods discussed herein. The program code may consist of instructions 424 that can be executed by one or more processors 402. In alternative embodiments, the machine may operate as a standalone device or be connected to other machines (e.g., networked). In a networked deployment, the machine may operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0049] A machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions 424 that define the actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term “machine” should also be understood to include any collection of machines that individually or together execute instructions 124 to perform any one or more of the methods considered herein.

[0050] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), main memory 404, and static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include a software driver that enables the display of a user interface on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly on a surface, window, or equivalent (e.g., via a visual projection unit). For ease of consideration, the visual interface may be described as a screen. The visual interface 410 may include an interface with a touchable screen, or may interface with such a screen. The computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touchscreen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing device), a memory unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate via a bus 408.

[0051] The storage unit 416 may include a machine-readable medium 422 on which instructions 424 (e.g., software) embodying any one or more of the methods or functions described herein are stored. The instructions 424 (e.g., software) may reside, all or in part, in the main memory 404 or in the processor 402 (e.g., in the processor's cache memory) during their execution by the computer system 400, and the main memory 404 and the processor 402 may constitute a machine-readable medium. The instructions 424 (e.g., software) may be transmitted or received via the network interface device 420 and the network 426.

[0052] In one exemplary embodiment, the machine-readable medium 422 is shown as a single medium, but the term “machine-readable medium” should be understood to include a single or multiple mediums (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term “machine-readable medium” should also be understood to include any medium capable of storing instructions for machine execution (e.g., instructions 424) and causing a machine to perform any one or more of the methods disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0053] (e) Exemplary image exposure Figure 5 illustrates exemplary images at various exposure levels according to one embodiment. Image 510 is an underexposed image and includes a biomarker 501. The biomarker 501 is difficult to distinguish due to shading caused by the underexposure of the image. Image 520 is a properly exposed image and similarly includes the biomarker 501, which is easily distinguishable. Image 530 is an overexposed image and similarly includes the biomarker 501, which is difficult to distinguish because the overexposure obscures the biomarker by the intensity of light, causing its features to appear blurred or invisible. Machine learning models or pattern matching tools or their components may not be able to detect the biomarker 501 from overexposed or underexposed images because the biomarker may blend in with the background, whereas the biomarker in the properly exposed image 520 has features that are easily distinguishable from its background.

[0054] Figure 6 depicts an exemplary image of a retina with striated retinal pigmentation according to one embodiment. Image 600 includes sections 621, 622, 623, and 624. As depicted, each section is a group of pixels having shared pigmentation that deviates from each adjacent section. Sections 621 and 623 are depicted as having identical pigmentation to each other, and similarly sections 622 and 624, while within each section there may be more than two different types of pigmentation, some with some commonalities and some with no commonalities. Biomarkers 601 are depicted within each section, but biomarkers 601 may or may not be present in a given section, and more than one biomarker 601 may be present in a given section. As in the case of Figure 5, biomarker 601 may be obscured by pigmentation in the image, and therefore adjustments to the flash intensity may be required for two or more additional images to properly expose each streak.

[0055] (f) Exemplary data flow for adjusting retinal imaging flash intensity Figure 7 illustrates an exemplary flowchart for adjusting flash intensity based on retinal pigmentation according to one embodiment. Process 700 begins with one or more processors (e.g., processor 202) of the devices used to invoke the retinal pigmentation determination tool 130 determining (702) the retinal pigmentation of the retina of the eyeball positioned in the imaging device 110. Determining (702) may be performed by a pigment determination module 331, which may execute an exposure determination module 332 and / or an infrared pigment determination module 333 to obtain the information necessary to determine the retinal pigmentation. The retinal pigmentation determination tool 130 may then instruct the imaging device to adjust the intensity of a flash component (e.g., flash component 212) from a first intensity to a second intensity based on the retinal pigmentation (704). For example, a flash intensity adjustment module 334 may instruct the imaging device 110 to adjust the intensity of a flash component from a default intensity to a lower intensity to avoid overexposure of lightly pigmented retinas. Commands may be issued by transmitting commands using an API configured to facilitate communication between the imaging device 110 and the retinal pigment deposition determination tool 130.

[0056] The retinal pigmentation determination tool 130 may then instruct the imaging device 110 to capture an image illuminated by the flash component at a second intensity (706), and may receive the image from the imaging device (708). Further processing may be performed, such as determining whether the image is properly exposed, and capturing additional images if the image is improperly exposed overall or partially (for example, in the case of striated retina). Once all parts of the patient's retina have been captured by one or more properly exposed images, the retinal pigmentation determination tool 130 may pass one or more images to another tool, such as a tool that takes one or more images as input and outputs a diagnosis of retinal disease.

[0057] (g) Overview The foregoing description of embodiments of the present invention is presented for illustrative purposes only and is not intended to be exhaustive or to limit the invention to any specific form disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.

[0058] Several parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of information-related operations. These algorithmic descriptions and representations are commonly used by those skilled in the art in data processing techniques to convey the gist of the research to others skilled in the art. While these operations are described functionally, computationally, or logically, it is understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or equivalents. Furthermore, it has been demonstrated that it is sometimes convenient to refer to these sequences of operations as modules without loss of generality. The operations described and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0059] Any step, operation, or process described herein can be performed or implemented, either alone or in combination with other devices, using one or more hardware or software modules. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described herein.

[0060] Embodiments of the present invention may also relate to apparatus for carrying out the operations described herein. Such apparatus may comprise a general-purpose computing device that is specifically constructed for a required purpose and / or selectively activated or reconfigured by a computer program stored within the computer. Such computer programs may be stored in non-temporary tangible computer-readable storage media or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referenced herein may comprise a single processor or may be an architecture employing multiple processor designs to increase computing power.

[0061] Embodiments of the present invention may also relate to products produced by the computational processes described herein. Such products may comprise information arising from the computational processes, and may include any embodiment of a computer program product or any combination of other data described herein, provided that the information is stored on a non-temporary, tangible, computer-readable storage medium.

[0062] Finally, the terminology used herein has been selected primarily for readability and guidance purposes, and not to precisely describe or define the subject matter of the invention. Therefore, the scope of the invention is intended to be limited not by this detailed description, but by any claims published at the time of filing a claim herein. Thus, the disclosure of embodiments of the invention, while illustrative, is not intended to limit the scope of the invention as set forth in the following claims.

Claims

1. A method for operating a system for adjusting flash intensity based on retinal pigment deposition, wherein the system comprises one or more processors, and the method The one or more processors determine the retinal pigment deposition of the retina positioned in the imaging device based on the level of exposure present in the initial image of the retina of the eyeball, The one or more processors instruct the imaging device to adjust the intensity of the flash component from a first intensity to a second intensity based on the retinal pigment deposition, The one or more processors instruct the imaging device to capture an image illuminated by the flash component at the second intensity, The one or more processors receive the image from the imaging device. Methods that include...

2. The one or more processors determine the retinal pigment deposition of the eyeball positioned in the imaging device, Receiving the initial image of the eyeball from the imaging device, To determine whether the initial image is underexposed, In response to determining that the initial image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. The method according to claim 1, including the method described in claim 1.

3. The one or more processors determine the retinal pigment deposition of the eyeball positioned in the imaging device, Receiving the initial image of the eyeball from the imaging device, To determine whether the initial image is overexposed, In response to determining that the initial image is overexposed, it is determined that the second intensity is an intensity reduced from the first intensity. The method according to claim 1, including the method described in claim 1.

4. The method according to claim 1, wherein the one or more processors instruct the imaging device to adjust the intensity of the flash component by transmitting the instruction using an Application Protocol Interface (API).

5. The method according to claim 1, wherein the first intensity is at least one of the default intensity or the last used intensity.

6. The method described above is: The one or more processors determine whether the received image is properly exposed, In response to one or more processors determining that the received image is not properly exposed, one or more processors instruct the imaging device to capture an additional image using a third intensity. The method according to claim 1, further comprising:

7. The determination of the retinal pigment deposits of the retina by one or more processors includes identifying a plurality of pigment deposits within the retina, wherein the second intensity is determined based on a first retinal pigment deposit among the plurality of pigment deposits, and the method is The one or more processors instruct the imaging device to adjust the intensity of the flash component from the second intensity to the third intensity based on the second retinal pigment deposit among the plurality of pigment deposits, The one or more processors instruct the imaging device to capture additional images illuminated by the flash component at the third intensity, The one or more processors receive the additional images from the imaging device. The method according to claim 1, further comprising:

8. A computer program product for adjusting flash intensity based on retinal pigment deposition, wherein the computer program product is Based on the level of exposure present in the initial image of the retina of the eyeball, the retinal pigment deposition of the retina positioned in the imaging device is determined, Based on the retinal pigment deposition, the imaging device is instructed to adjust the intensity of the flash component from a first intensity to a second intensity, Commanding the imaging device to capture an image illuminated by the flash component at the second intensity, Receiving the image from the aforementioned imaging device and A computer program product comprising a non-temporary computer-readable storage medium containing computer program code for performing the following actions.

9. The computer program code for determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving the initial image of the eyeball from the imaging device, To determine whether the initial image is underexposed, In response to determining that the initial image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. The computer program product according to claim 8, comprising computer program code for performing the following.

10. The computer program code for determining the retinal pigment deposition of the eyeball positioned in the imaging device is: Receiving the initial image of the eyeball from the imaging device, To determine whether the initial image is overexposed, In response to determining that the initial image is overexposed, it is determined that the second intensity is an intensity reduced from the first intensity. The computer program product according to claim 8, comprising computer program code for performing the following.

11. The computer program code for determining the retinal pigment deposits of the retina includes a computer program code for identifying a plurality of pigment deposits within the retina, wherein the second intensity is determined based on a first retinal pigment deposit among the plurality of pigment deposits, and the computer program code The imaging device is instructed to adjust the intensity of the flash component from the second intensity to the third intensity based on the second retinal pigment deposit among the plurality of pigment deposits. Commanding the imaging device to capture an additional image illuminated by the flash component at the third intensity, Receiving the additional image from the aforementioned imaging device The computer program product according to claim 8, further comprising computer program code for performing the following actions.

12. The computer program product according to claim 8, wherein the computer program code further comprises a computer program code for diagnosing a retinal disease of the retina based on the characteristics of the image.

13. The computer program product according to claim 8, wherein the computer program code for instructing the imaging device to adjust the intensity of the flash component includes computer program code for transmitting commands using an Application Protocol Interface (API).

14. The computer program product according to claim 8, wherein the first intensity is at least one of the default intensity or the last used intensity.

15. The computer program code is: To determine whether the received image is properly exposed, In response to determining that the received image is not properly exposed, the imaging device is instructed to capture additional images using a third intensity. The computer program product according to claim 8, further comprising computer program code for performing the following actions.

16. A computer program product for adjusting flash intensity based on retinal pigment deposition, wherein the computer program product comprises a computer-readable storage medium containing computer program code, and the computer program code is A first module for determining retinal pigment deposition of the retina positioned in an imaging device based on the level of exposure present in the initial image of the retina of the eyeball, A second module for instructing the imaging device to adjust the intensity of the flash component from a first intensity to a second intensity based on the retinal pigment deposition, A third module for instructing the imaging device to capture an image illuminated by the flash component at the second intensity, A fourth module for receiving the image from the imaging device and A computer program product that includes the following features.

17. The first module is Receiving the initial image of the eyeball from the imaging device, To determine whether the initial image is underexposed, In response to determining that the initial image is underexposed, it is determined that the second intensity is an increased intensity relative to the first intensity. The computer program product according to claim 16, comprising a submodule for performing the following.

18. The computer program product according to claim 16, wherein the computer program code further comprises a fifth module for diagnosing retinal diseases of the retina based on the characteristics of the image.

19. The computer program product according to claim 16, wherein instructing the imaging device to adjust the intensity of the flash component includes transmitting the instruction using an Application Protocol Interface (API).

20. The computer program product according to claim 16, wherein the first intensity is at least one of the default intensity or the last used intensity.

Images