Dynamic adjustment of flash intensity based on retinal pigmentation
By dynamically adjusting flash intensity based on retinal pigmentation, the system ensures optimal image exposure for diagnosing retinal abnormalities, enhancing diagnostic accuracy and reducing flash-related harm.
Patent Information
- Application Number
- JP2024157258
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-03-19
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-03-15
AI Technical Summary
Existing autonomous systems for diagnosing retinal abnormalities face challenges with underexposed or overexposed images due to retinal pigmentation, leading to suboptimal image quality and potential damage from repeated flash exposure.
Systems and methods for determining retinal pigmentation and adjusting flash intensity based on pigmentation to optimize image exposure, using infrared light and machine learning algorithms to adjust flash intensity dynamically.
Prevents underexposure and overexposure, reducing the need for repeated flashes and improving image quality for accurate diagnosis while minimizing patient exposure to harmful light.
Smart Images

Figure 2024169447000001_ABST
Abstract
Description
[Technical field]
[0001] (background) The present invention relates generally to autonomous diagnosis of retinal abnormalities, and more particularly to adjusting flash intensity of an imaging device based on retinal pigmentation. [Background technology]
[0002] An autonomous system for diagnosing retinal abnormalities captures images of a patient's retina (used interchangeably herein with the word "fundus") and analyzes those images for abnormalities. The images are captured using a preset flash intensity. However, like the outer skin, the user's fundus may be pigmented, which may cause the image to be under- or over-exposed using a preset flash intensity. If the biomarkers used to diagnose the abnormality may be obscured, the under- or over-exposure may cause the image to be suboptimal, thus preventing the system from diagnosing an abnormality that would have appeared on the image if the image had been properly exposed.
[0003] An image that is underexposed or overexposed may be insufficient for diagnosis, thus requiring the patient to hold still through multiple images, each requiring the use of another flash. Repeated exposure to the flash by the imaging device may be damaging to the patient's eye, therefore preventing or reducing the probability that images are underexposed or overexposed improves patient health in that fewer flashes are required to capture an adequate image of the patient's eye. Summary of the Invention [Means for solving the problem]
[0004] (overview) Provided herein are systems and methods for determining a patient's retinal pigmentation and adjusting flash intensity based on the determined pigmentation. For example, retinal pigmentation can be determined based on an initial image that is overexposed or underexposed, and the flash intensity used to capture subsequent images can be adjusted based on the overexposure or underexposure. As another example, infrared light can be used to determine retinal pigmentation, thus preventing the need for an initial image to be captured from the patient using a flash. As yet another example, an image of the patient's skin or hair can be captured and the pigmentation determined from this. The systems and methods disclosed herein advantageously prevent overexposure and underexposure of captured images once the pigmentation has been learned, preventing the need to unnecessarily expose the patient's eye to multiple flashes due to unsuccessful imaging.
[0005] To achieve these and other objectives, in an embodiment, a processor (e.g., of a server) determines retinal pigmentation of a retina of an eye located at an imaging device (e.g., a camera remote from the server). The processor directs the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation. The processor then directs the imaging device to capture an image illuminated by the flash component at the second intensity and receives the image from the imaging device. The image may be used to diagnose whether a retinal abnormality is observed in the patient's retina. The present invention provides, for example, the following items. (Item 1) 1. A method for adjusting flash intensity based on retinal pigmentation, comprising: determining retinal pigmentation of a retina of an eye positioned at an imaging device; instructing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; directing the imaging device to capture an image illuminated by the flash component at the second intensity; receiving the image from the imaging device; The method includes: (Item 2) Determining the retinal pigmentation of the eye positioned at the imaging device includes: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; The method according to item 1, comprising: (Item 3) Determining the retinal pigmentation of the eye positioned at the imaging device includes: receiving a first image of the eye from the imaging device; determining whether the first image is overexposed; determining, in response to determining that the first image is overexposed, that the second intensity is a reduced intensity relative to the first intensity; The method according to item 1, comprising: (Item 4) Determining the retinal pigmentation of the eye positioned at the imaging device includes: instructing the imaging device to obtain feedback by emitting an infrared signal at the retina; receiving the feedback from the imaging device; and determining the retinal pigmentation of the eye based on the feedback; and The method according to item 1, comprising: (Item 5) 2. The method of claim 1, wherein instructing the imaging device to adjust the intensity of the flash component includes transmitting an instruction using an application protocol interface (API). (Item 6) 2. The method of claim 1, wherein the first intensity is at least one of a default intensity or a last used intensity. (Item 7) determining whether the received image is properly exposed; in response to determining that the received image is not properly exposed, directing the imaging device to capture an additional image using a third intensity; The method of claim 1, further comprising: (Item 8) Determining the retinal pigmentation of the retina includes identifying a plurality of pigmentations within the retina, and the second intensity is determined based on a first retinal pigmentation of the plurality of pigmentations, and the method further comprises: instructing the imaging device to adjust an intensity of the flash component from the second intensity to a third intensity based on a second retinal pigmentation of the plurality of pigmentations; directing the imaging device to capture an additional image illuminated by the flash component at the third intensity; receiving the additional image from the imaging device; The method of claim 1, further comprising: (Item 9) 2. The method of claim 1, further comprising diagnosing a retinal disease of the retina based on features of the image. (Item 10) 1. A computer program product for adjusting flash intensity based on retinal pigmentation, comprising: determining retinal pigmentation of a retina of an eye positioned at an imaging device; instructing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; directing the imaging device to capture an image illuminated by the flash component at the second intensity; receiving the image from the imaging device; A computer program product comprising a non-transitory computer readable storage medium containing computer program code for performing the steps of: (Item 11) The computer program code for determining the retinal pigmentation of the eye positioned at the imaging device further comprises: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; Item 11. A computer program product according to item 10, comprising computer program code for carrying out the steps of: (Item 12) The computer program code for determining the retinal pigmentation of the eye positioned at the imaging device further comprises: receiving a first image of the eye from the imaging device; determining whether the first image is overexposed; determining, in response to determining that the first image is overexposed, that the second intensity is a reduced intensity relative to the first intensity; Item 11. A computer program product according to item 10, comprising computer program code for carrying out the steps of: (Item 13) The computer program code for determining the retinal pigmentation of the eye positioned at the imaging device further comprises: instructing the imaging device to obtain feedback by emitting an infrared signal at the retina; receiving the feedback from the imaging device; and determining the retinal pigmentation of the eye based on the feedback; and Item 11. A computer program product according to item 10, comprising computer program code for carrying out the steps of: (Item 14) Item 11. The computer program product of item 10, wherein the computer program code for instructing the imaging device to adjust the intensity of the flash component comprises computer program code for transmitting instructions using an application protocol interface (API). (Item 15) Item 11. The computer program product of item 10, wherein the first intensity is at least one of a default intensity or a last used intensity. (Item 16) The computer program code comprises: determining whether the received image is properly exposed; in response to determining that the received image is not properly exposed, directing the imaging device to capture an additional image using a third intensity; Item 11. The computer program product of item 10, further comprising computer program code for performing the steps: (Item 17) The computer program code for determining the retinal pigmentation of the retina includes computer program code for identifying a plurality of pigmentations within the retina, the second intensity being determined based on a first retinal pigmentation of the plurality of pigmentations, the computer program code comprising: instructing the imaging device to adjust an intensity of the flash component from the second intensity to a third intensity based on a second retinal pigmentation of the plurality of pigmentations; directing the imaging device to capture an additional image illuminated by the flash component at the third intensity; receiving the additional image from the imaging device; Item 11. The computer program product of item 10, further comprising computer program code for performing the steps: (Item 18) Item 11. The computer program product of item 10, wherein the computer program code further comprises computer program code for diagnosing a retinal disease of the retina based on features of the image. (Item 19) 1. A computer program product for adjusting flash intensity based on retinal pigmentation, comprising: a first module for determining retinal pigmentation of a retina of the eye positioned at the imaging device; a second module for directing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; a third module for directing the imaging device to capture an image illuminated by the flash component at the second intensity; a fourth module for receiving the image from the imaging device; and 23. A computer program product comprising a computer readable storage medium containing computer program code comprising: (Item 20) The first module includes: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; 20. The computer program product of item 19, comprising a sub-module for performing the steps of: [Brief description of the drawings]
[0006] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] FIG. 1 is an exemplary block diagram of system components in an environment for utilizing a retinal pigmentation determination tool, according to one embodiment.
[0007] [Diagram 2] FIG. 2 is an exemplary block diagram of modules and components of an imaging device, according to one embodiment.
[0008] [Diagram 3] FIG. 3 is an exemplary block diagram of modules and components of a retinal pigmentation determination tool, according to one embodiment.
[0009] [Figure 4] FIG. 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).
[0010] [Diagram 5] FIG. 5 depicts example images at various levels of exposure, according to one embodiment.
[0011] [Figure 6] FIG. 6 depicts an exemplary image of a retina with striated retinal pigmentation, according to one embodiment.
[0012] [Figure 7] FIG. 7 depicts an exemplary flow chart for adjusting flash intensity based on retinal pigmentation, according to one embodiment.
[0013] The figures depict various embodiments of the invention for illustrative purposes only. Those skilled in the art will readily recognize from discussion below that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the inventive principles described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] (Detailed Description) (a) Environmental Overview 1 is an exemplary block diagram of system components in an environment for utilizing a retinal pigmentation determination tool, according to one embodiment. The environment 100 includes an imaging device 110, a network 120, a retinal pigmentation determination tool 130, and a retinal disease diagnosis tool 140. The imaging device 110 is a device configured to capture one or more images of a retina of a patient's eye. The imaging device 110 may be caused to capture such images autonomously, through manual operation, when commanded by computer program instructions or an external signal (e.g., received from the retinal pigmentation determination tool 130), or by a combination thereof. Examples of what these images may look like and how they are derived are described in co-pending U.S. patent application Ser. No. 15 / 466,636, filed Mar. 22, 2017, the disclosure of which is hereby incorporated by reference in its entirety.
[0015] After capturing the image, the imaging device 110 transmits the image to the retinal pigmentation determination tool 130 for processing. In an embodiment not depicted, the retinal pigmentation determination tool 130 is installed on the imaging device 110 as a module, and thus the transmission is internal to the imaging device 110. In the depicted embodiment, the retinal pigmentation determination tool 130 is instantiated on a server separate from the imaging device 110, and the image is transmitted over the network 120. The network 120 may be any communications network, such as a local area network, a wide area network, the Internet, and the like.
[0016] Although not depicted in Figure 1, one or more additional imaging devices may be used to capture external images that include some or all of the patient (e.g., the patient's skin or hair). Retinal pigmentation may then be determined.
[0017] The retinal pigmentation determination tool 130 receives the image and determines whether the image is suitable for processing. In one embodiment, suitability for processing means that the image is exposed with a specified light intensity or within a specified range of light intensities such that biomarkers within the patient's retina can be detected by the retinal disease diagnosis tool 140. In another embodiment, suitability for processing means that certain landmarks within the patient's retina are identifiable within the image. Further discussion regarding methods for determining whether an image is suitable for processing is discussed with reference to Figures 2-7 below.
[0018] A contributing factor to an image not being suitable for processing is retinal pigmentation; if a patient's retina is pigmented (either lightly or darkly pigmented), the retinal image may be overexposed or underexposed if a consistent flash intensity is used when capturing the image. The retinal pigmentation determination tool determines necessary adjustments to the retinal pigmentation and / or flash intensity based on the retinal pigmentation and instructs the imaging device 110 (or its human operator) to adjust the flash intensity prior to capturing the image or further images. The manner in which the retinal pigmentation determination tool 130 makes the determinations and instructs the adjustments is described in further detail with respect to FIG. 3 below.
[0019] The retinal disease diagnostic tool 140 autonomously analyzes the retinal images and determines a diagnosis using machine learning analysis of biomarkers therein. The diagnosis may specifically be a determination that the user has a particular disease, such as diabetic retinopathy, or may be a determination that the user is likely to have a disease and should therefore see a physician for confirmation and treatment. The manner in which the retinal disease diagnostic tool 140 performs the analysis and determines a diagnosis is further discussed in commonly owned U.S. Pat. No. 10,115,194, the disclosure of which is hereby incorporated by reference in its entirety. While depicted as a separate entity from the retinal pigmentation determination tool 130, the retinal disease diagnostic tool 140 may be instantiated on the same server or set of servers as the retinal pigmentation determination tool 130, similar to the manner in which the retinal pigmentation determination tool 130 may be installed as a module within the imaging device 110, in part or in whole.
[0020] (b) Exemplary Imaging Device Components 2 is an exemplary block diagram of modules and components of an imaging device, according to one embodiment. The imaging device 110 includes an image capture component 211, a flash component 212, an infrared component 213, a retinal pigmentation determination tool application protocol interface (API) 214, and a user interface 215. Although not depicted, the imaging device 110 may include other components, such as built-in instances of either or both of the retinal pigmentation determination tool 130 and the retinal disease diagnosis tool 140, as well as any components thereof. The imaging device 110 may include any database or memory for performing any functionality described herein. The imaging device 110 may also exclude some depicted components. For example, the imaging device may exclude the infrared component 213.
[0021] The image capture component 211 may be any sensor configured to capture an image of the patient's retina. For example, a specialized lens may be used to capture an image of the patient's retina. The flash component 212 may be any component capable of illuminating the patient's retina during image capture by the image capture component 211 and may be configured to emit light in coordination with the image capture operation of the image capture component 211. The image capture component 211 may also be configured with an external image capture component 211 for capturing images including the patient's skin and / or hair.
[0022] The infrared component 213 is an infrared sensor configured to transmit infrared radiation to the patient's retina and determine its absorption. The infrared component 213 may generate a heat map indicative of the absorption of the infrared transmission across the patient's retina. The infrared component 213 transmits the absorption determination and / or the heat map to a processor (e.g., of the retinal pigmentation determination tool 130) for processing with respect to determining the patient's retinal pigmentation.
[0023] The retinal pigmentation determination tool API 214 interfaces with the retinal pigmentation determination tool 130 to translate instructions from the retinal pigmentation determination tool 130 to the imaging device 110. Exemplary instructions may include instructions for capturing an image, instructions for adjusting the intensity of light emitted by the flash component 212, and the like. These instructions and how they are generated are discussed in further detail below with reference to FIG.
[0024] User interface 215 is an interface by which an operator of imaging device 110 may instruct imaging device 110 to perform any function capable of performing such as capturing images, adjusting flash intensity, capturing infrared information, and the like. User interface 215 may be any hardware or software interface and may include physical components (e.g., buttons) and / or graphical components (e.g., on a display such as a touch screen display). User interface 215 may be located on imaging device 110, may be a peripheral device of imaging device 110, or may be located on a device separated from imaging device 110 by network 120, thereby enabling remote operation of imaging device 110.
[0025] (c) Exemplary retinal pigmentation determination tool components 3 is an exemplary block diagram of modules and components of a retinal pigmentation determination tool, according to one embodiment. The retinal pigmentation determination tool 130 includes a pigment determination module 331, a flash intensity adjustment module 334, and a striated retinal pigment imaging module 335. Although not depicted, the retinal pigmentation determination tool 110 may include other components, such as additional modules, and any databases or memories for performing any of the functions described herein. Also, the retinal pigmentation determination tool 130 may exclude some of the depicted components. For example, the retinal pigmentation determination tool 130 may exclude the striated retinal pigment imaging module 355.
[0026] The pigment determination module 331 determines the patient's retinal pigmentation based on information received from the imaging device 110. The information may include an image or a series of images of the patient's retina, infrared absorption information, and / or a combination thereof. The pigment determination module 331 may execute sub-modules such as an exposure determination module 332 and / or an infrared pigment determination module 333. Alternatively, the exposure determination module 332 and / or the infrared pigment determination module 333 may be stand-alone modules rather than sub-modules of the pigment determination module 331.
[0027] Reference is now made to an embodiment in which retinal pigmentation is determined based on a patient's retinal image received from the imaging device 110. The exposure determination module 332 determines whether the image is underexposed or overexposed. As used herein, the term "underexposed" may refer to an image being captured while the patient's retina is exposed to insufficient light intensity from the flash component 212, thus reducing the likelihood that a biomarker that appears in a properly exposed image due to insufficient illumination is detected in an image that is underexposed. Similarly, the term "overexposed" may refer to an image being captured while the patient's retina is exposed to excessive light intensity from the flash component 212, thus reducing the likelihood that a biomarker that appears in a properly exposed image due to excessive illumination is detected in an image that is overexposed. As used herein, the term "biomarker" is an object in an image that is part of the patient's retina that corresponds to a retinal disease.
[0028] The exposure determination module 332 may determine that an image is partially or completely overexposed or underexposed by analyzing the appearance of the captured image (or portions thereof) and determining an exposure level therefrom. For example, brightness levels, grayscale levels (e.g., at individual points or averages), color intensity levels, or any other measure of brightness, intensity, coloration, and the like, may be determined. The exposure level may be determined on a pixel-by-pixel basis, and the exposure determination module 332 generates a heat map, with each portion of the heat map reflecting the exposure level of the corresponding pixel in the captured image. The exposure determination module 332 may determine the exposure level on a collective basis for the image. The exposure level for the entire image may be captured by performing statistical operations (e.g., average, median, mode) of the exposure levels on a pixel-by-pixel or pixel group basis to identify an exposure level that reflects the entire image.
[0029] The exposure determination module 332 may determine whether the exposure level for each pixel, each group of pixels, or the entire image is within a predefined exposure range. For example, the ranges that are overexposed, underexposed, and properly exposed may be predefined by an administrator of the retinal pigmentation determination tool 130. The exposure determination module 332 may determine which range the exposure level falls into and, based on that range, may determine whether the image is overexposed, underexposed, or properly exposed. The exposure determination module 332 may perform this determination of overexposed, underexposed, or properly exposed for the entire image, or for different portions of the image based on the exposure levels of groups of pixels.
[0030] The exposure determination module 332 may use additional information to determine exposure levels for either pixels, groups of pixels, or the entire pixel set. For example, the exposure determination module 332 may calculate the similarity of an image (or its pixels or groups of pixels) with reference to a reference image with typical exposure levels. A database may be consulted indicating similarity levels that map to exposure levels, and the exposure level may then be determined. As another example, the exposure determination module 332 may filter the image to determine gradients in a color space related to the brightness of the image (e.g., red-green-blue, cyan-magenta-yellow tones, etc.). The gradient differences may be compared to entries in a database that maps gradient differences to exposure levels.
[0031] The exposure determination module 332 may determine retinal pigmentation based on the exposure level. To determine pigmentation, the exposure determination module 332 may access a data structure that maps exposure level to pigmentation. Any of the aforementioned exposure levels, i.e., exposure levels calculated on a pixel-by-pixel basis, for groups of pixels, for the entire image, may be mapped to pigmentation. A map indicative of pigmentation may be generated on a pixel-by-pixel basis based on the exposure level of each pixel.
[0032] The exposure determination module 332 may determine the adjustment value based on the pigmentation and the flash intensity used to capture the image. For example, a data structure may be referenced that indicates the exposure level corresponding to each flash intensity for each possible type of pigmentation. The data structure indicates the adjustment value for that exposure level. Alternatively, the exposure determination module 332 may calculate the adjustment value based on the exposure level without reference to the pigmentation. To implement this adjustment when considering the exposure level of the entire image, the exposure determination module 332 may determine the difference between the exposure level and a predetermined appropriate exposure value and assign the difference to be the adjustment value.
[0033] In an embodiment, the exposure determination module 332 may calculate the adjustment value based on the exposure level for each pixel in the image or based on the exposure level for different groups of pixels in the image (e.g., each quadrant of the image may have a different calculated exposure level). The exposure determination module 332 may calculate the adjustment value by maximizing the amount of pixels or groups of pixels that have adjusted exposure levels that are within a range that are properly exposed. For example, if three of four quadrants are overexposed and an adjustment value is calculated that, when applied to the exposure levels of all four quadrants, properly exposes the three of the four quadrants, and no adjustment value exists that properly exposes all four quadrants, then the adjustment value is applied to maximize the amount of the image that is properly exposed.
[0034] In an embodiment, the exposure determination module 332 may not take into account the exposure levels of properly exposed pixels or groups of pixels when determining the adjustment value, since the properly exposed pixels may be stitched into the image, and the improperly exposed images may be corrected by adjusting the flash intensity or otherwise accounted for by analyzing two separate images when using the image for its intended purpose (e.g., detecting biomarkers). Thus, the exposure determination module 332 may determine intensity levels for an image to use in calculating the adjustment value by ignoring the intensity levels of properly exposed portions of the image, while taking into account pixels or groups of pixels that are over- or under-exposed.
[0035] The exposure determination module 332 may store the adjustment value and / or the adjusted value after the adjustment value has been applied in the user information database 336. As described in more detail below, the flash intensity adjustment module 334 may instruct the imaging device 110 to adjust its flash intensity based on the adjustment value. During future imaging sessions, the retinal pigmentation determination tool 130 may identify the patient, retrieve the flash intensity adjustment value from the user information 336, and determine the flash adjustment prior to taking an image, thus eliminating the need to expose the patient to a flash that would result in an improperly exposed image. Adjustment values may be assigned to each eye of the patient (if the image is labeled as to which eye (left or right) corresponds) and may be used depending on which eye is being imaged.
[0036] In an embodiment, the exposure determination module 332 may determine the retinal pigmentation and / or exposure value by inputting the image into a machine learning model and receiving the retinal pigmentation as output from the machine learning model. The machine learning model may be trained using images that are labeled as having a certain retinal pigmentation and / or exposure value.
[0037] Reference is now made to an embodiment in which infrared absorption information of the patient's eye is received from the imaging device 110 and retinal pigmentation is determined thereon. The infrared pigment determination module 333 receives the absorption information and / or heat map from the imaging device 110 and calculates the patient's retinal pigmentation therefrom. In an embodiment, the infrared retinal pigmentation determination module 333 calculates the pigmentation by comparing the absorption information to information in a data structure that maps absorption to pigmentation. In another embodiment, the infrared retinal pigmentation determination module 333 inputs the absorption information into a machine learning model and receives the pigmentation as an output from the machine learning model. When absorption information such as any patient vitals, biographical, or demographic information is retrieved from the user information database 336 or received from the imaging device 110 along with the absorption information, other inputs along with them are also made to the machine learning model. After determining the pigmentation, the pigment determination module 331 may determine an adjustment value using the techniques described above with reference to the exposure determination module 332.
[0038] Additionally, adjustments may be informed using other parameters. For example, flash intensity and / or gain may be compared separately or in combination with other factors (e.g., gamma settings of imaging device 110) against statistical parameters of infrared levels (e.g., average, median, etc.). Pigment determination module 331 may use the results of the comparison to reference a database and determine the corresponding pigmentation in the database that is mapped to the results of the comparison.
[0039] In an embodiment, the exposure determination module 332 may determine retinal pigmentation and / or exposure values by inputting an infrared image (or a heat map as described above) into a machine learning model and receiving retinal pigmentation (or information that can be used to determine retinal pigmentation) as output from the machine learning model. The machine learning model may be a trained image-level classifier that outputs flash exposure time and intensity from the infrared image. The classifier may use predefined features extracted from the infrared image to output the flash exposure time and intensity. The flash exposure time and intensity may be used to determine retinal pigmentation by either referencing a database or inputting the flash exposure time and intensity into a trained classifier, which is trained to convert the flash exposure time and intensity to pigmentation. The resulting exposure times and intensities received from the model may be derived from a representative normative database for various pigmentations.
[0040] The flash intensity adjustment module 334 transmits instructions to the imaging device 110 to have the flash intensity emitted by the flash component 212 adjusted. The instructions may include an amount by which to adjust the flash intensity or may include a new flash intensity value. In an embodiment, the adjustments to the flash intensity described above, as performed by the pigment determination module 331 and / or its sub-modules, may instead be performed by the flash intensity adjustment module 334.
[0041] In an embodiment, the retinal pigmentation determination tool 130 may categorize the retina into a set of known retinal types or pigmentations, with each known type or pigmentation in the set having a corresponding exposure time and / or flash intensity that is mapped to it in an electronic data structure. For example, the retinal pigmentation determination tool 130 may perform statistical operations on the entire infrared image (e.g., or an infrared heat map) or on specific features of the infrared image (e.g., pixels or groups of pixels, or features such as the optic disc, fovea, or blood vessels) and determine the mapping of the statistical operations to known retinal types. The retinal pigmentation determination tool 130 may determine the retinal pigmentation therefrom. As another example, the retinal pigmentation determination tool 130 may use an encoder in combination with a clustering algorithm (e.g., k-means clustering) and map the output of the encoder to a matching known retinal type or pigmentation. As another example, the retinal pigmentation determination tool 130 may input the retinal image into a classification neural network and receive the known retinal types as an output. As yet another example, one or more images of the iris may be used as input for determining a known retina type, either separately or in addition to the above-mentioned examples. The retinal pigmentation determination tool 130 may determine the exposure time and / or flash intensity for capturing the retinal image based on the mapping in the electronic data structure. The electronic data structure may be continually refined, for example, by storing the required metrics or by embedding the resulting image quality scores of the infrared image and the color image of the fundus and feeding it into the classifier data as updated training data.
[0042] The striated retinal pigment imaging module 335 identifies striated lines in the retinal pigmentation and determines remedial action to ensure that a properly exposed retinal image can be acquired or constructed. The striated retinal pigment imaging module 335 may use exposure levels as described above, pixel-by-pixel or pixel group exposure maps, infrared absorption information, or any other information to determine whether a retinal image indicates that the patient has striated pigmentation in the patient's retina.
[0043] In an embodiment, the striated retinal pigment imaging module 335 may determine that a contiguous pixel group exceeding a predetermined amount of pixels in the group has a level of intensity or exposure (e.g., received using infrared or intensity information) that differs from a neighboring pixel group by a predetermined amount. For example, a pixel group with a high level of intensity adjacent to a pixel group with a low level of intensity may indicate different pigmentation at each point in the patient's retina to which the pixels correspond. The striated retinal pigment imaging module 335 may determine from them that the patient's retina has a striated line. To prevent or minimize false determinations, the striated retinal pigment imaging module 335 considers pixel groups that are at least a threshold size (e.g., at least 30 pixels wide) to ensure that what is changing the level of intensity or exposure is pigmentation and not a biomarker or other artifact. While a common situation is vertical or horizontal stripes or stripes across the retina with different pigmentation, whenever the term "striations" is used herein, non-striped patterns of different pigmentation are also within the scope of this disclosure. For example, tessellations may be identified whenever the term "striations" is used herein. Tessellations may be detected using gradients in images, edge detectors, oriented edge detectors, Bag-of-Words, dictionary learning based on histograms of patches, template matching, and the like.
[0044] Alternatively or additionally, the striate pigment imaging module 335 may use computer vision to determine that the retina is a striate. In an embodiment, the striate pigment imaging module 335 may use computer vision to determine whether an edge (e.g., a boundary between two pigmentations) exists. The striate pigment imaging module may use static or adaptive thresholding techniques to determine striations, such as determining if the difference in pigmentation on each side of the edge differs by a threshold amount (e.g., threshold intensity or infrared information) if an edge exists. The striate pigment imaging module 335 performs a Hough transform at one or more angles in a computer vision model to identify semi-linear objects that may represent edges. Additionally or alternatively, the striate pigment imaging module 335 compares contours for various color channels to identify striations, where the comparison of contours aids in striation detection when the color channels are part of a pigmentation where the striations are less visible than typical. Statistical analysis may be performed on the infrared and intensity distributions, and standard deviations are applied to ascertain the likelihood that a striate has been detected. Additionally, neural networks such as classification neural networks and segmentation neural networks can be trained and used to identify striatum.
[0045] The striated retinal pigment imaging module 335 may direct the imaging device 110 to capture two or more retinal images, each at a different flash intensity, causing different exposure levels corresponding to the intensity levels required to be properly exposed at the two or more intensities corresponding to the striated lines. The striated retinal pigment imaging module 335 may then stitch the two or more images together using the properly exposed portions from each image to create a composite image that is properly exposed throughout. Alternatively, the striated retinal pigment imaging module 335 may keep each image separate for further analysis. For example, a module that performs a diagnosis of a retinal disease may analyze the properly exposed portions of each image to detect biomarkers and output a diagnosis based on the biomarkers identified across the two or more images.
[0046] The user information database 336 is a database that may maintain a profile for each patient. The profile may include any collected information, including biographical information (e.g., name, height, weight), demographic information (e.g., ethnicity, geographic location), and any other information (e.g., health records).
[0047] Although not depicted, in one embodiment, an image including the patient's hair and / or skin is received by the retinal pigmentation determination tool 130. The retinal pigmentation tool may then determine the patient's retinal pigmentation. As an example, the retinal pigmentation tool 130 may reference a data structure that maps the patient's hair and / or skin color to retinal pigmentation and takes the mapped retinal pigmentation value as the patient's hypothesized retinal pigmentation. As another example, the retinal pigmentation tool 130 may input the image or data derived therefrom into a machine learning model that has been trained to output retinal pigmentation. This may be performed before an image of the patient's retina is captured and may be used to adjust flash intensity prior to capturing the image in any manner described herein.
[0048] (d) Exemplary Computer Architecture FIG. 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them in a processor (or controller). In particular, FIG. 4 shows a diagrammatic representation of a machine in the exemplary form of a system 400 in which program code (e.g., software) for causing the machine to perform any one or more of the methods discussed herein may be implemented. The program code may consist of instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or may operate as a peer machine in a peer-to-peer (or distributed) network environment.
[0049] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions 424 that specify actions to be taken by the machine. Additionally, although only a single machine is illustrated, the term "machine" should also be taken to include any collection of machines that individually or together execute instructions 124 to perform any one or more of the methodologies discussed herein.
[0050] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include software drivers that allow for displaying a user interface on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, or the like. For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with an interface with a touch-enabled screen. Computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touch screen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a memory unit 416, a signal generating device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate over bus 408.
[0051] The storage unit 416 may include a machine-readable medium 422 on which are stored instructions 424 (e.g., software) embodying any one or more of the methods or functions described herein. The instructions 424 (e.g., software) may reside, completely or partially, within the main memory 404 or within the processor 402 (e.g., within a processor's cache memory) during its execution by the computer system 400, and the main memory 404 and the processor 402 may constitute machine-readable media. The instructions 424 (e.g., software) may be transmitted or received over a network 426 via the network interface device 420.
[0052] While in an exemplary embodiment, the machine-readable medium 422 is shown to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" should also be taken to include any medium capable of storing instructions (e.g., instructions 424) for execution by a machine and causing a machine to perform any one or more of the methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.
[0053] (e) Exemplary Image Exposure FIG. 5 depicts example images at various exposure levels, according to one embodiment. Image 510 is an underexposed image and includes biomarkers 501. Biomarkers 501 are difficult to see due to shadows caused from underexposure of the image. Image 520 is a properly exposed image and also includes biomarkers 501, which are easy to see. Image 530 is an overexposed image and also includes biomarkers 501, which are difficult to see because overexposure has obscured the biomarkers 501 due to light intensity that makes their features appear blurred or invisible. Machine learning models or pattern matching tools or components thereof may not be able to detect biomarkers 501 from overexposed or underexposed images because the biomarkers may blend in with the background, whereas the biomarkers in properly exposed image 520 have features that are easily distinguished from their background.
[0054] FIG. 6 depicts an example image of a retina with striated retinal pigmentation, according to one embodiment. Image 600 includes sections 621, 622, 623, and 624. As depicted, each section is a pixel group that has shared pigmentation that deviates from each of its neighboring sections. While sections 621 and 623 are depicted as having identical pigmentation relative to each other, as are sections 622 and 624, there may be more than two different types of pigmentation with some commonality or no commonality in the pigmentation in each section. Although biomarkers 601 are depicted in each section, biomarkers 601 may or may not be present in a given section, and more than one biomarker 601 may be present in a given section. As in FIG. 5, the biomarkers 601 may be obscured by pigmentation in the image and therefore may require adjustments to the flash intensity for two or more additional images to properly expose each striae.
[0055] (f) Exemplary Data Flow for Adjusting Retinal Imaging Flash Intensity FIG. 7 depicts an exemplary flowchart for adjusting flash intensity based on retinal pigmentation, according to one embodiment. The process 700 begins with one or more processors (e.g., processor 202) of a device used to launch the retinal pigmentation determination tool 130 determining (702) retinal pigmentation of a retina of an eye positioned at the imaging device 110. The determining (702) may be performed by the pigment determination module 331, which may execute the exposure determination module 332 and / or the infrared pigment determination module 333 to obtain the information necessary to determine the retinal pigmentation. The retinal pigmentation determination tool 130 may then instruct the imaging device to adjust the intensity of a flash component (e.g., flash component 212) from a first intensity to a second intensity based on the retinal pigmentation (704). For example, the flash intensity adjustment module 334 instructs the imaging device 110 to adjust the intensity of the flash component from a default intensity to a lower intensity to avoid overexposure of a lightly pigmented retina. The commanding may be performed by transmitting instructions using an API configured to facilitate communication between the imaging device 110 and the retinal pigmentation determination tool 130 .
[0056] The retinal pigmentation determination tool 130 may then instruct the imaging device 110 to capture an image illuminated by the flash component at a second intensity (706) and may receive the image from the imaging device (708). Further processing may be performed, such as determining whether the image is properly exposed and capturing additional images if the image is improperly exposed in whole or in part (e.g., in the case of a striated retina). Once all portions of the patient's retina are captured with one or more properly exposed images, the retinal pigmentation determination tool 130 may pass one or more images to another tool, such as a tool that takes an image(s) as input and outputs a diagnosis of retinal disease.
[0057] (g) Overview The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will recognize that many modifications and variations are possible in light of the above disclosure.
[0058] Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or the like. Moreover, it has proven convenient at times to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.
[0059] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product that includes a computer readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described.
[0060] An embodiment of the present invention may relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.
[0061] Embodiments of the invention may relate to products produced by the computational processes described herein. Such products may comprise information resulting from the computational processes, and when the information is stored on a non-transitory tangible computer-readable storage medium, may include any embodiment of a computer program product or other data combination described herein.
[0062] Finally, the terms used herein are primarily selected for readability and instructional purposes, and not for precisely describing or delineating the subject matter of the invention. Thus, the scope of the invention is intended to be limited not by this detailed description, but by any claims that may be issued upon filing based on this specification. Thus, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
Claims
1. A method of operating a system for adjusting flash intensity based on retinal pigmentation, the system comprising one or more processors, the method comprising: determining, by the one or more processors, retinal pigmentation of a retina located at an imaging device based on feedback indicative of absorption of infrared signals by a retina of the eye; the one or more processors directing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; the one or more processors directing the imaging device to capture an image illuminated by the flash component at the second intensity; the one or more processors receiving the image from the imaging device; A method comprising:
2. The one or more processors determining the retinal pigmentation of the eye positioned at the imaging device comprises: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; The method of claim 1 , comprising:
3. The one or more processors determining the retinal pigmentation of the eye positioned at the imaging device comprises: receiving a first image of the eye from the imaging device; determining whether the first image is overexposed; determining, in response to determining that the first image is overexposed, that the second intensity is a reduced intensity relative to the first intensity; The method of claim 1 , comprising:
4. The method comprises: the one or more processors inputting the image into a machine learning model; receiving an output from the machine learning model, the output representing a diagnosis of a retinal disease of the retina based on features of the image; The method of claim 1 further comprising:
5. The method of claim 1, wherein the one or more processors instructing the imaging device to adjust the intensity of the flash component includes transmitting instructions using an application protocol interface (API).
6. The method of claim 1, wherein the first intensity is at least one of a default intensity or a last used intensity.
7. The method comprises: the one or more processors determining whether the received image is properly exposed; in response to the one or more processors determining that the received image is not properly exposed, the one or more processors directing the imaging device to capture an additional image using a third intensity; The method of claim 1 further comprising:
8. The method of claim 7, wherein the one or more processors determining the retinal pigmentation of the retina includes identifying a plurality of pigmentations within the retina, and the second intensity is determined based on a first retinal pigmentation of the plurality of pigmentations, the method further comprising: the one or more processors directing the imaging device to adjust an intensity of the flash component from the second intensity to a third intensity based on a second retinal pigmentation of the plurality of pigmentations; directing the one or more processors to capture an additional image illuminated by the flash component at the third intensity; the one or more processors receiving the additional image from the imaging device; and The method of claim 1 further comprising:
9. A computer program product for adjusting flash intensity based on retinal pigmentation, the computer program product comprising: determining retinal pigmentation of a retina located at the imaging device based on feedback indicative of absorption of the infrared signal by the retina of the eye; instructing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; directing the imaging device to capture an image illuminated by the flash component at the second intensity; receiving the image from the imaging device; A computer program product comprising a non-transitory computer readable storage medium containing computer program code for performing the steps of:
10. The computer program code for determining the retinal pigmentation of the eye positioned at the imaging device, comprising: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; 10. A computer program product as claimed in claim 9, comprising computer program code for performing the following:
11. The computer program code for determining the retinal pigmentation of the eye positioned at the imaging device, comprising: receiving a first image of the eye from the imaging device; determining whether the first image is overexposed; determining, in response to determining that the first image is overexposed, that the second intensity is a reduced intensity relative to the first intensity; 10. A computer program product as claimed in claim 9, comprising computer program code for performing the following:
12. The computer program code for determining the retinal pigmentation of the retina includes computer program code for identifying a plurality of pigmentations in the retina, and the second intensity is determined based on a first retinal pigmentation of the plurality of pigmentations, the computer program code comprising: instructing the imaging device to adjust an intensity of the flash component from the second intensity to a third intensity based on a second retinal pigmentation of the plurality of pigmentations; directing the imaging device to capture an additional image illuminated by the flash component at the third intensity; receiving the additional image from the imaging device; 10. The computer program product of claim 9, further comprising computer program code for:
13. The computer program product of claim 9, wherein the computer program code further comprises computer program code for diagnosing retinal disease of the retina based on features of the image.
14. The computer program product of claim 9, wherein the computer program code for instructing the imaging device to adjust the intensity of the flash component includes computer program code for transmitting instructions using an application protocol interface (API).
15. The computer program product of claim 9, wherein the first intensity is at least one of a default intensity or a last used intensity.
16. The computer program code comprising: determining whether the received image is properly exposed; in response to determining that the received image is not properly exposed, directing the imaging device to capture an additional image using a third intensity; 10. The computer program product of claim 9, further comprising computer program code for:
17. A computer program product for adjusting flash intensity based on retinal pigmentation, the computer program product comprising: a first module for determining retinal pigmentation of a retina located at an imaging device based on feedback indicative of absorption of infrared signals by the retina of the eye; a second module for directing the imaging device to adjust an intensity of a flash component from a first intensity to a second intensity based on the retinal pigmentation; a third module for directing the imaging device to capture an image illuminated by the flash component at the second intensity; a fourth module for receiving the image from the imaging device; and 23. A computer program product comprising a computer readable storage medium containing computer program code comprising:
18. The first module comprises: receiving a first image of the eye from the imaging device; determining whether the first image is underexposed; determining, in response to determining that the first image is underexposed, that the second intensity is an increased intensity relative to the first intensity; 20. The computer program product of claim 17, comprising a sub-module for performing:
19. The computer program product of claim 17, further comprising a fifth module for diagnosing retinal disease of the retina based on features of the image.
20. The computer program product of claim 17, wherein instructing the imaging device to adjust the intensity of the flash component includes transmitting instructions using an application protocol interface (API).
Citation Information
Patent Citations
Measurement of macular pigment distribution
JP2006525054A
Grayscale redistribution system to improve retinal imaging by reducing the effect of a highly reflective optic nerve
US20070076169A1
Methods for measuring therapeutic effects of retinal disease therapies
WO2018170494A1
Systems and methods for improved ophthalmic imaging
WO2019030375A2
