Color correction in ophthalmic imaging

Abstract

Certain embodiments provide a method (20) of generating (240) a color corrected ophthalmic image. The method includes obtaining (210) an ophthalmic image of an eye (106) having a nuclear sclerotic cataract or having a colored intraocular lens ("IOL"). The method further comprises determining (220) color shift information associated with the ophthalmic image based on the user input and at least one of by processing the obtained image, the color shift information indicating a degree to which a color in the obtained image is to be corrected, to at least partially remedy the effect of nuclear sclerotic cataract or colored IOL on the color of the obtained image. The method further includes color correcting (230) the ophthalmic image based on the color shift information. The method further includes generating (240) a color-corrected ophthalmic image based on the color correction.

Description

Background Art

[0001] During ophthalmic screening (such as retinal screening), the practitioner must be able to clearly observe or capture a clear and accurate image of the eye in order to make a correct diagnosis, preferably with minimal discomfort or invasiveness to the patient. To this end, the practitioner can use various visualization devices (e.g., indirect ophthalmoscopes, 28-30 diopter lenses, or slit lamps with 78-90 diopter lenses) or various types of specialized imaging devices that produce digital retinal imaging, multispectral imaging, or hyperspectral imaging to observe the eye. Retinal screening is usually performed to diagnose retinal diseases such as age-related macular degeneration (AMD), nuclear sclerosis cataracts, diabetic retinopathy, hypertensive retinopathy, etc.

[0002] However, whether viewed through a visualization device or as a digital image, the color appearance of the vitreous and retina may be altered due to the presence of a nuclear sclerotic cataract, a yellow intraocular lens ("IOL"), or for some other reason. For example, a white lesion may appear yellow due to the yellowish color of a nuclear sclerotic cataract or a yellow IOL. This yellowing may make it more difficult to determine, for example, geographic atrophy used to diagnose dry macular degeneration in a patient. Further, the yellowing may make it more difficult to accurately determine the grade, extent, or severity of a dry macular degeneration condition. Discoloration of the vitreous and retina may also potentially confuse an artificial intelligence or digital imaging system, leading to errors. Summary of the invention

[0003] The present disclosure generally relates to systems and methods for color correction of ophthalmic images.

[0004] Certain embodiments provide a method for generating a color-corrected ophthalmic image. The method includes obtaining an ophthalmic image of an eye having a nuclear sclerotic cataract or having a colored intraocular lens ("IOL"). The method further includes determining color shift information associated with the ophthalmic image based on at least one of a user input and by processing the obtained image, the color shift information indicating the degree to which the color in the obtained image is to be corrected to at least partially remedy the effect of the nuclear sclerotic cataract or the colored IOL on the color of the obtained image. The method further includes color correcting the ophthalmic image based on the color shift information. The method further includes generating a color-corrected ophthalmic image based on the color correction.

[0005] Certain embodiments provide an ophthalmic imaging system comprising an imager configured to obtain an ophthalmic image of an eye having a nuclear sclerotic cataract or having a colored intraocular lens ("IOL"). The ophthalmic imaging system further comprises one or more memories configured to store executable instructions, and one or more processors in data communication with the one or more memories. The one or more processors are configured to execute instructions to determine color shift information associated with the ophthalmic image based on at least one of a user input and by processing the obtained image, the color shift information indicating the degree to which the color in the obtained image is to be corrected to at least partially remedy the effect of the nuclear sclerotic cataract or the colored IOL on the color of the obtained image. The one or more processors are further configured to execute instructions to color correct the ophthalmic image based on the color shift information. The one or more processors are configured to execute instructions to generate a color-corrected ophthalmic image based on the color correction.

[0006] The following description and the associated drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The drawings described herein are for illustration purposes only, are schematic in nature, and are intended to be exemplary and not to limit the scope of the present disclosure.

[0008] Figure 1 A diagram of an example ophthalmic environment including an imaging device and a patient's eye is shown in accordance with some embodiments.

[0009] Figure 2 is a flow chart of a method of providing a color-corrected ophthalmic image in accordance with certain embodiments.

[0010] The above summary is not intended to present every possible embodiment or every aspect of the subject disclosure. Rather, the above summary is intended to illustrate some of the novel aspects and features disclosed herein. The above features and advantages and other features and advantages of the subject disclosure will become clear through the following detailed description of representative embodiments and implementation modes of the subject disclosure in conjunction with the accompanying drawings and the attached claims. DETAILED DESCRIPTION

[0011] Embodiments of color correction in ophthalmic imaging disclosed herein can overcome some or all of the above challenges by generating color-corrected digital images that reduce or eliminate color shifts compared to raw images of the vitreous and retina captured, for example, during ophthalmic screening. In certain embodiments, the color-corrected digital images are generated by applying a filter to the raw image to reduce color shifts. Color shift in an image refers to the color of the image or various structures (e.g., anatomical objects, tissues, proteins, cells, etc.) or parts therein being different from the expected (e.g., true or natural) color of such structures or parts in the image.

[0012] In some embodiments, a filter is applied to the original image based on or in response to color shift information, which can include information input via a user interface (UI) and / or information automatically determined from the original image. For example, in some embodiments, a practitioner can provide input via a UI indicating that the patient's eye suffers from a nuclear sclerotic cataract or has a yellow IOL. In another example, an image including a structure having a known color can have associated color shift information, which is determined by measuring the difference between the known color of the structure and the corresponding color of the structure as it appears in the original image (which may have been color shifted). For example, in some embodiments, the optic nerve head present in the original image is used as a reference. The optic nerve head has a natural color of white, but may appear yellowed due to the presence of a nuclear sclerotic cataract or a yellow IOL. In such an example, the color shift information can be automatically determined by measuring the difference between the color of the optic nerve head as shown in the original image and white (or the natural color of the optic nerve head, which may be white). In some embodiments, instead of or in addition to using the optic nerve head as a reference, oxyhemoglobin and / or deoxyhemoglobin within the retinal arteries and veins may be used as a reference. Oxyhemoglobin is generally red or bright red, while deoxyhemoglobin is bluish purple. However, due to the presence of nuclear sclerosis cataracts or yellow IOLs, the colors of oxyhemoglobin and deoxyhemoglobin may differ in the image. Therefore, color shift information may be automatically determined by measuring the difference between the color of oxyhemoglobin and / or deoxyhemoglobin within the retinal arteries and veins as shown in the original image and the natural color of oxyhemoglobin and / or deoxyhemoglobin.

[0013] In various embodiments, reference may be made to xanthochromia, yellow shift, etc., due to yellowing of the original multispectral image caused by cataracts or certain IOLs. Although xanthochromia is illustrated in various embodiments, it should be understood that the disclosed aspects are not so limited and may be applicable to reducing any kind of color shift in ophthalmic imaging.

[0014] Figure 1 A diagram of an example ophthalmic environment 105 is shown showing an ophthalmic imaging device 104 for capturing images of a patient's eye 106. For example, the ophthalmic environment 105 may correspond to an ophthalmic screening environment in a screening room where a screening procedure is being performed to screen for macular degeneration. In such an example, a screener or practitioner may use the imaging device 104 to capture images of the vitreous and / or retina of a patient.

[0015] According to various embodiments, the imaging device 104 may be a digital imaging device, a multispectral imaging device, or a hyperspectral imaging device, or any other ophthalmic imaging device capable of performing ophthalmic imaging, etc. As shown, the imaging device 104 includes an imager 102, a controller 120, a user interface 125, an interconnect 130, and an I / O device interface 126 that may allow various I / O devices to be connected to the viewing device 104.

[0016] Imager 102 may be any imager, imaging component, or imaging device known to one of ordinary skill in the art for capturing ophthalmic images. For example, imager 102 may be a digital imager (such as a digital camera) for capturing images of various optical components of the eye (such as the vitreous body and the retina). Figure 1 In some embodiments, the imager 102 may be integrally formed with the imaging device 104 and controlled by the controller 120. However, in other embodiments, the imager 102 and the imaging device 104 may be separate devices, in which case the imager 102 may instead be controlled by the controller 120 via a wired or wireless connection.

[0017] The controller 120 includes a central processing unit (CPU) 123, a memory 121, and a storage device 124. The CPU 123 can retrieve and execute programming instructions stored in the memory 121. Similarly, the CPU 123 can retrieve and store application data present in the memory 121. The interconnect 130 transmits programming instructions and application data between the CPU 123, the I / O device interface 126, the user interface 125, the memory 121, the storage device 124, etc. The CPU 123 can represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc. Additionally, in some embodiments, the memory 121 represents a volatile memory, such as a random access memory (RAM). In addition, in some embodiments, the storage device 124 can be a non-volatile memory, such as a disk drive, a solid-state drive, or a collection of storage devices distributed on multiple storage systems.

[0018] As shown, the memory 121 includes a color shift detection module 122 and a color shift filtering module 134. The color shift detection module 122 refers to an application or set of software instructions for detecting color shifts present in a raw image captured by the imager 102. In certain embodiments, the color shift detection module 122 may detect color shifts based on color shift information provided via user input. For example, as further described below, a practitioner may provide input indicating that a patient has a nuclear sclerotic cataract or has a yellow IOL. Using the color shift information provided via such user input, the color shift detection module 122 is configured to detect the presence of a color shift in a raw image captured from the patient's eye.

[0019] Alternatively or additionally, the color shift detection module 122 can detect color shift by processing and analyzing the original image. For example, in some embodiments, the color shift detection module 122 can detect color shift by processing the original image, identifying structures in the original image, and determining color shift information associated with the original image based on the difference between the color of the structure as captured in the image and the actual color of the structure. The actual color of the structure refers to the natural color of the structure, which is consistent in all patients and is not related to any ophthalmic condition or disease. As described in further detail below, an example of a structure that can be used as a reference in the above embodiments is the optic nerve head, which is always white. As described above, other reference structures include oxyhemoglobin and deoxyhemoglobin in the retinal arteries and veins.

[0020] The color shift filter module 134 may be configured to perform color correction on the raw image captured by the imager 102. The raw image may be automatically color corrected based on color shift information input by a user and / or automatically derived from the raw image using various techniques known to those of ordinary skill in the art. As described below, the color shift information input by the user may indicate one or more of the following: the color of the patient's nuclear sclerotic cataract, the type and / or color of the IOL, etc. The color shift information automatically derived by processing the raw image may indicate an intensity value (e.g., red, green, blue (RGB) value) associated with one or more pixels in the raw image. Based on the color shift information, the color shift filter module 134 is configured to adjust the RGB values ​​of one or more pixels in the raw image so as to achieve a desired RGB value for each of the one or more pixels. For example, in some embodiments, the color shift filter module 134 may generate a color compensation or correction filter and apply it to the raw image, wherein the filter represents an offset of the RGB value of each pixel in the raw image, the offset representing the difference between the intensity value of each pixel in the raw image and the desired intensity value of the pixel. The desired intensity values ​​may be predefined colors or RGB intensity values ​​associated with natural or desired colors of various optical components or structures (e.g., retina, vitreous body, lesions, etc.) The predefined intensity values ​​may be stored in memory 121 as part of instructions associated with color shift filter module 134.

[0021] As an illustrative example, the user input may provide color shift information indicating that the patient has a yellow IOL, which corresponds to a hue of yellow. In such an example, the color shift filter module 134 may then apply a filter to the original image, thereby adjusting the RGB value of each pixel in the image by a shift corresponding to the hue of yellow. After adjusting the RGB value of each pixel, a corrected image may be generated as if the patient's eye had not had a yellow IOL inserted.

[0022] In various embodiments, the user interface 125 may include a display showing one or more menu items for receiving color shift information from a user. The user interface 125 may include a series of UI features to allow a practitioner to indicate the presence, color intensity, and other aspects of color shift in the original image. For example, the user interface 125 may provide the practitioner with options for indicating whether the patient has a nuclear sclerotic cataract or has a yellow IOL.

[0023] In another example, the user interface 125 may provide a plurality of UI features (e.g., radio buttons) and an associated rating scale (e.g., Lens Opacity Classification (LOCS)) and / or a visual color representation (e.g., an image) to prompt the practitioner to input color shift (e.g., yellow shift) information. For example, LOCS II and LOCS III rating scales may be used, which are image-based rating scales. A degree of yellowing or yellow shift may be associated with each of the various levels of the LOCS grading scale. As an example, a practitioner may observe an original image and determine that a degree of yellowing in the image corresponds to a certain color or level of nuclear sclerosis cataract in the LOCS II system.

[0024] Alternatively or additionally, images may be provided so that the practitioner can make selections using visual matching of these images to the original image or view of the patient's eye. As an example, multiple radio buttons with multiple corresponding sample images may be provided so that, for example, the practitioner can view the degree of yellowing in the original image, visually match it with a sample image reflecting a similar degree of yellowing, and select the corresponding radio button. By providing a sample image and a radio button selector for each level of the LOCS grading scale, the practitioner can quickly indicate the color and / or color intensity of the color shift.

[0025] The UI feature may also include radio buttons and associated images indicating one plus (1+) nuclear sclerosis, two plus (2+) nuclear sclerosis, three plus (3+) nuclear sclerosis, and four plus (4+) nuclear sclerosis. In another example, a UI feature (e.g., a check box) for tinted intraocular lenses (IOLs) or IOLs with chromophores may also be included, as well as a drop-down menu for individually named IOLs for which it is desirable to have a filter to eliminate color shift caused by tinting.

[0026] The information provided through one or more menu selections of the user interface as described above is processed by the color shift detection module 122 to determine how to color correct the raw image. For example, once the intensity, color, and / or hue of the yellow color associated with a nuclear sclerotic cataract or IOL is determined, the color shift detection module 122 may share such information as part of the color shift information with the color shift filtering module 134 to generate and apply a filter to correct the color shift in the raw image.

[0027] In addition to or in lieu of receiving user input indicating color shift information, color shift detection module 122 may also be configured to automatically determine color shift information by processing a raw image received by imager 102. For example, as discussed, color shift detection module 122 may include an image detection model configured to detect the presence of structures of known natural colors in the raw image. By determining a difference between the color of a structure present in the raw image and the known natural color of the structure, a color shift of the image may be determined.

[0028] For example, the structure present in the original image may be an optic nerve head that is known to have a natural white color. In such an example, the color shift detection module 122 may use an image detection model to determine the location of the optic nerve head in the original image and the color of the optic nerve head. The color shift detection module 122 may then determine the difference between the color of the optic nerve head present in the original image and the natural white color of the optic nerve head. Information indicating the difference may then be shared as part of the color shift information to the color shift filtering module 134 to generate a filter and apply the filter to correct the color shift in the original image. Similar techniques may be applied when using oxyhemoglobin and deoxyhemoglobin in the retinal arteries and veins as reference structures.

[0029] The image detection model of the color shift detection module 122 may need to be trained using a training data set. The training data set may include information associated with a plurality of images (typically hundreds, thousands, or more) that are manually annotated to reflect attributes, metadata, or other information about the images. For example, a set of hundreds or thousands of captured images including the optic nerve head may be manually annotated to indicate the location of the optic nerve head (e.g., pixel information associated with the optic nerve head). The model is then "trained" by using a machine learning algorithm to reduce the prediction error about the location of the optic nerve head until the error is reduced to a negligible amount.

[0030] Once the location of the optic nerve head is determined, the color shift detection module can measure a set of color intensities (e.g., RGB values) for each pixel associated with the optic nerve head. The color shift detection module can then determine the difference between the measured color intensities and a set of known color intensities for the optic nerve head to determine the presence of a light source causing the color shift. Therefore, in response to receiving color shift information from the color shift detection module 122 (whether as a result of user input or through image detection), the color shift filtering module can generate a color correction filter using the color shift information. When the color correction filter is applied to the original image, the color correction filter can be used to generate a corrected image in which some or all of the color shifts exhibited in the original image are eliminated. Similar techniques for training image detection models can be applied when using oxyhemoglobin and deoxyhemoglobin in the retinal arteries and veins as reference structures.

[0031] Figure 2 is a flow chart of a method 200 performed by an ophthalmic system ("system"). In certain embodiments, the system is Figure 1 The system may include or include an imaging device 104 in which the imager 102 is incorporated. In some embodiments, the system includes a computing device in communication with an imager, such as the imager 102.

[0032] At box 210, the system obtains an ophthalmic image (also referred to as a "raw image"), such as a retinal image. The ophthalmic image can be a digital retinal image, a multispectral image, or a hyperspectral image, or a similar ophthalmic image. The image may have a color shift, such that the color appearance of various optical components (e.g., vitreous and retina) may be altered due to the presence of a nuclear sclerotic cataract, a yellow intraocular lens ("IOL"), or for some other reason. The image can be captured by an imager that is part of the system or an imager that communicates with the system via a wired or wireless network.

[0033] At block 220, the system determines color shift information associated with the obtained image. As described above, the system may determine the color shift information based on user input and / or by processing and analyzing the obtained image. The color shift information indicates how, or to what extent, the color in the obtained image may be corrected to at least partially remove or remedy the effect of, for example, a nuclear sclerosis cataract or a yellow IOL on the color of the obtained image. As described above, the color shift information input by the user may indicate one or more of the following: the color of the patient's nuclear sclerosis cataract, the type and / or color of the IOL in the lens capsule, the observed color of the obtained image, etc. The color shift information automatically derived by processing the obtained image (e.g., without user input) may indicate the color of the obtained image, such as intensity values ​​(e.g., red, green, blue (RGB) values) associated with one or more pixels in the obtained image. In some embodiments, the system may determine the color shift information based on user input and automatic processing of the obtained image as described above.

[0034] At block 230, the system automatically color corrects the obtained image based on the color shift information. Various techniques may be used to color correct the obtained image based on the determined color shift information. As an example, a filter may be generated based on the color shift information and then applied to the obtained image to correct the color shift in the obtained image, thereby producing a corrected image having a desired color intensity or a target color intensity. Additional information about how filters may be generated and applied to the obtained image is provided above and is omitted here for brevity.

[0035] At box 240, the system generates a color-corrected image. For example, the system can generate and display the color-corrected image using a display that is part of the system, further process the color-corrected image, and / or transmit the color-corrected image to another display or system via a wired or wireless network for display to a clinician and / or further processing. Examples of further processing may include additional image detection techniques and / or other types of processing that are performed to derive information that can be used to provide actionable information or guidance to the clinician. As discussed, it is advantageous to correct the color of the obtained image prior to additional processing because the additional processing can be performed by artificial intelligence models that are trained using images without color shifts, and therefore, when the input to such models is images with color shifts, these models may make inaccurate predictions.

[0036] It should be understood that Figure 2The elements and configurations shown in are merely exemplary in nature, and various elements may be added or omitted, or otherwise configured or arranged for various applications without departing from the scope of the present disclosure.

[0037] It will be appreciated by those skilled in the art that the embodiments encompassed by the present disclosure are not limited to the specific exemplary embodiments described above. In this regard, although illustrative embodiments have been shown and described, a wide variety of modifications, variations, and substitutions are contemplated in the foregoing disclosure. It should be understood that such changes may be made to the foregoing without departing from the scope of the present disclosure. Therefore, it should be understood that the appended claims should be interpreted broadly and in a manner consistent with the present disclosure.

Claims

1. An ophthalmic imaging device, comprising: an imager configured to obtain an ophthalmic image of an eye having a nuclear sclerotic cataract or implanted with a colored intraocular lens (IOL); one or more memories configured to store executable instructions; One or more processors in data communication with the one or more memories and configured to execute the instructions to perform the following operations: determining color shift information associated with the ophthalmic image based on at least one of a user input and by processing the acquired image, the color shift information indicating a degree to which color in the acquired image is to be corrected to at least partially remedy an effect of the nuclear sclerotic cataract or the colored IOL on the color of the acquired image; performing color correction on the ophthalmic image based on the color shift information; and A color-corrected ophthalmic image is generated based on the color correction.

2. The ophthalmic imaging device of claim 1, wherein: The one or more processors being configured to determine the color shift information includes the one or more processors being configured to determine the color shift information based on a user input.

3. The ophthalmic imaging device of claim 2, wherein: The user input indicates one or more of: the color of the nuclear sclerotic cataract, the type of the IOL, the color of the IOL, the observed color of the acquired image.

4. The ophthalmic imaging device of claim 2, wherein: The user input includes a selection of a sample image corresponding to a color or grade of the nuclear sclerotic cataract, and The selection is based on a user visually matching the color of the nuclear sclerotic cataract in the acquired image with the sample image.

5. The ophthalmic imaging device of claim 2, wherein: The user input includes a selection of a rating associated with the grade of the nuclear sclerotic cataract.

6. The ophthalmic imaging device of claim 5, wherein: The rating is based on the Lens Opacities Classification (LOCS) system.

7. The ophthalmic imaging device of claim 2, wherein: The user input comprises a selection of a sample image corresponding to a type and / or color of the IOL, and The selection is based on a user visually matching the color of the IOL in the acquired image with the sample image.

8. The ophthalmic imaging device of claim 1, wherein: The one or more processors being configured to determine the color shift information comprises the one or more processors being configured to: automatically determining the locations of structures of the eye in the acquired images; and The difference between the color of the structure as it appears in the acquired image and a defined color associated with the structure is measured.

Images