Method and system for enhanced ophthalmic visualization

Abstract

In certain embodiments, a system, computer-implemented method, and computer-readable medium for enhanced ophthalmic visualization are disclosed. Multiple images corresponding to different portions of the electromagnetic spectrum are acquired and combined, such as by pixel-wise subtraction, to obtain a combined image. The images may be weighted with weights selected to enhance visualization of features, such as features corresponding to retinal layers or pathological conditions. The combined image may be processed, such as by a machine learning model, to extract features.

Description

[Technical Field]

[0001] The diagnosis and treatment of many eye diseases requires imaging of a patient's eye. The retina has many complex features that are imaged to diagnose eye diseases and other conditions that cause physiological changes in the retina. Ophthalmoscopes can be used to image the retina for the diagnosis of eye diseases. When performing eye surgery, surgeons typically use an ophthalmic microscope, such as a digital surgical microscope. [Background technology]

[0002] A digital ophthalmic microscope or ophthalmoscope can image the retina using a color (i.e., red, green, and blue) digital camera that captures an image of the retina illuminated by a broadband light source (e.g., visible white light).

[0003] Multispectral imaging (MSI) is another technique that can be used in ophthalmic microscopes or ophthalmoscopes. MSI involves measuring (or capturing) light reflected from the retina at different wavelengths or spectral bands across the electromagnetic spectrum, such as from infrared to ultraviolet wavelengths. MSI can capture more information from the retina that may not be visible with traditional imaging.

[0004] Improved imaging of the retina to better diagnose ocular diseases and provide a more accurate representation of a patient's eye during ophthalmic surgery would be an advancement in the art. Summary of the Invention [Means for solving the problem]

[0005] In certain embodiments, a system is provided that includes one or more processing devices and one or more memory devices coupled to the one or more processing devices. The one or more memory devices store executable code that, when executed by the one or more processing devices, causes the one or more processing devices to receive multiple images of a patient's eye, each image corresponding to a different portion of the electromagnetic spectrum. A combined image is obtained by performing a pixel-by-pixel combination of two or more images of the multiple images. A representation of the combined image is output to a display device.

[0006] So that the above-mentioned features of the present disclosure can be understood in detail, a more particular description of the present disclosure briefly summarized above can be had by reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only exemplary embodiments and therefore should not be considered as limiting the scope of the invention, as other equally effective embodiments are possible. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary operating environment in which enhanced ophthalmic visualization may be used, according to certain embodiments. [Figure 2] FIG. 2 is a diagram illustrating a system for enhanced ophthalmic visualization, in accordance with certain embodiments. [Figure 3A-3C] 3A-3C are diagrams illustrating the reflection of various wavelengths of light from layers of the retina, according to certain embodiments. [Figure 4A-4B] FIG. 4A includes images of a sample retina for various wavelengths, and FIG. 4B includes a difference image obtained from the images of the retina according to certain embodiments. [Figure 5] FIG. 5 is a process flow diagram of a method for enhanced ophthalmic visualization, according to certain embodiments. [Figure 6]FIG. 6 illustrates an exemplary computing device that at least partially implements one or more functions for performing enhanced ophthalmic visualization, according to certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] For ease of understanding, where possible, identical elements common to the figures are designated with the same reference numerals. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further reference.

[0009] FIG. 1 illustrates an exemplary ophthalmic observation system 100 in which the enhanced ophthalmic visualization methods disclosed herein may be used. The system 100 includes an ophthalmic microscope 102. A surgeon 104 uses the ophthalmic microscope 102 to visualize structures on and within the eye 106 of a patient 108 undergoing surgery or examination. The microscope 102, in this illustration, is supported on an adjustable overhead arm 110 of a microscope support stage 112. The patient 108 may be supported on an operating table 114. The ophthalmic microscope 102 is movable in three dimensions by the support arm 110 so that the surgeon 104 can position the ophthalmic microscope 102 as desired relative to the eye 106 of the patient 108.

[0010] In certain embodiments, the ophthalmic microscope 102 comprises a high-resolution, high-contrast stereoscopic surgical microscope. The ophthalmic microscope 102 often includes a monocular or binocular eyepiece 116 through which the surgeon 104 has an optically magnified view of the relevant ocular structures that the surgeon 104 needs to see to complete a given surgery or to diagnose an ocular condition of the patient 108.

[0011] The ophthalmic microscope 102 includes a digital camera and a broadband light source for capturing RGB images, an MSI imaging device, and / or other types of imaging devices. Digital images captured using the camera can be displayed on a display device within the ophthalmic microscope 102.

[0012] The ophthalmic microscope 102 may include two display devices viewable through binocular eyepieces 116 and a displayed image of the patient's eye 106 captured from different perspectives by two cameras to provide stereoscopic viewing. For example, the ophthalmic microscope 102 may be implemented as the NGENUITY 3D VISUALIZATION SYSTEM offered by Alcon Inc. of Fort Worth, Texas.

[0013] Additionally or alternatively, images from the ophthalmic microscope 102 may be displayed on one or more display devices. For example, the one or more display devices may include a display device 118 fastened to the support arm 110 above the ophthalmic microscope 102. The one or more display devices may include a display device 120 mounted on a cart or other structure. The display device 120 may be large, e.g., at least 48 inches diagonal, so that it can be viewed by a surgeon while the display device 120 is positioned at the foot of the operating table 114. The one or more display devices may include a display device mounted on the support base 112. Any of the display devices 118, 120, 122 may be implemented as a touchscreen for receiving input from the surgeon 104 or another operator. The one or more display devices may include a virtual reality headset or other type of three-dimensional viewing modality.

[0014] The illustrated ophthalmic viewing system 100 is exemplary only. Other configurations may be used. For example, the ophthalmoscope may be mounted on a stand or other fixture to view the eye of a patient sitting or standing in front of the ophthalmoscope. The ophthalmoscope may also include a camera for capturing RGB images, an MSI imaging device, and / or other types of imaging devices. The ophthalmoscope may include two cameras, and images from the two cameras may be viewed using two display devices and binocular eyepieces, a virtual reality headset, or other three-dimensional viewing modality.

[0015] FIG. 2 illustrates an ophthalmic visualization system 200 for enhanced ophthalmic visualization, according to certain embodiments. The system 200 includes an imaging device 202 that, during use, has within its field of view the eye 106 of a patient 108, such as the retina of the eye 106. The imaging device 202 may be implemented as an ophthalmic microscope 102, an ophthalmoscope, or other imaging device. The imaging device 202 may include a digital camera and a broadband light source, such as an RGB camera, an MSI imaging device, or other types of cameras. The imaging device 202 may capture images from a single perspective, or may capture images from multiple perspectives for stereoscopic or three-dimensional scene rendering. The imaging device 202 may capture images from multiple perspectives using multiple cameras with different viewing perspectives.

[0016] Imaging device 202 outputs image 204. In the following description of system 200, "image 204" shall be understood to refer to (a) a set of red, green, and blue (RGB) images that constitute a single color image, (b) a set of MSI images captured substantially simultaneously (e.g., within 10 milliseconds, 1 millisecond, or 0.1 milliseconds) and each corresponding to a different band of the electromagnetic spectrum, e.g., infrared, red, green, blue, and ultraviolet, or a greater or fewer bands of the electromagnetic spectrum, or (c) a set of images captured simultaneously by another imaging modality. If imaging device 202 outputs images from multiple viewpoints for stereoscopic viewing or rendering of a three-dimensional scene, "image 204" shall be understood to refer to a set of images according to (a), (b), or (c) captured from a single one of the multiple viewpoints defined in this paragraph.

[0017] Each image in the images 204 corresponds to a different band of wavelengths in the electromagnetic spectrum, whether that be red, green, blue, infrared, ultraviolet, or another wavelength band. Each image in the images 204 is itself a two-dimensional array of intensity values, i.e., a grayscale image. For example, each image in the images 204 is either (a) captured while the patient's eye 106 is illuminated with light having a peak intensity at a different wavelength than the other images in the images 204, and / or (b) captured by a sensor having a peak sensitivity at a different wavelength than the other images in the images 204. The wavelengths of peak intensity or sensitivity in the images 204 may be separated by a minimum separation distance, such as at least 10 nanometers (nm), 20 nm, 30 nm, 40 nm, or greater, to facilitate the capture of different information in each image in the images 204. The minimum separation distance between the wavelength of peak intensity or sensitivity of one image 204 and the wavelength of peak intensity or sensitivity of another image 204 may also be expressed as 1 to 10 percent of the wavelength. The wavelength peak intensity and / or peak sensitivity of image 204 is within a range of wavelengths that can safely illuminate the retina. For example, the wavelength range can be 100 nm to 100 μm. The intensity at which the retina is illuminated during capture of image 204 is selected to be within limits that will not cause damage.

[0018] Image 204 may be represented as separate objects, i.e., separate files or objects stored in memory or a storage device. Image 204 may be represented as a single file or object, for example, red, green, and blue fields stored at each pixel value of a color image that may be extracted by system 200 as a red image, a green image, and a blue image.

[0019] The system 200 may include an image weighting stage 206. The weighting stage 206 may perform some or all of the following processes: normalizing the image 204 so that the maximum and minimum pixel values ​​of the image 204 are the same; multiplying some or all of the images 204 by corresponding weights to compensate for differences in camera sensitivity when capturing each image of the images 204, e.g., to compensate for greater sensitivity to green light compared to read and blue light; and Multiplying part or all of image 204 by weights that are empirically selected to enhance the visibility of one or more features of the retina that are represented in image 204 after processing by system 100. The manner in which these weights may be determined is described in more detail below.

[0020] Note that in some embodiments, the weighting stage 206 is omitted so that references below to weighted images can be replaced with references to images 204 received from the imaging device 202.

[0021] The system 200 may also include a combining stage 208 in which the images output by the weighting stage 206 or the original image 204 are subtracted from one another. For example, let WIr, WIb, and WIg be the weighted red, blue, and green images output by the weighting stage 206. The subtraction stage may calculate some or all of WIrb = WIr - WIb, WIrg = WIr - WIg, WIgb = WIg - WIb, or other combinations or permutations. More complex combinations may include summations and additions, for example, WIrbg = (WIr + Wig) - Wib or WIrgb = WIr - (WIg + WIb), or some other combination. The weights used in the weighting stage 206 may be selected to enhance the visibility of one or more features of the retina after processing by the weighting stage 206.

[0022] As used herein, image addition and subtraction can be understood as pixel-wise addition and subtraction, with respect to images A and B, such that the value of pixel D(x,y) in difference (or sum) image D is equal to the difference (or sum) of pixel value A(x,y) in image A and pixel value B(x,y) in image B, where x and y are index values ​​in the two-dimensional arrays of pixels that make up images A, B, and D.

[0023] Although the examples described herein refer to addition and subtraction, other pixel-wise operations may be performed in a similar manner, for example, D(x,y)=A(x,y) / B(x,y), D(x,y)=A(x,y)*B(x,y), or D(x,y)=F(A(x,y),B(x,y)), where F() is a mathematical function selected to improve the visibility of retinal features.

[0024] 3A-3C illustrate the benefits of the combination step 208. The retina of the eye is composed of many layers, including a layer 302 composed of superficial nerve fibers and blood vessels, a layer 304 composed of photoreceptors (i.e., rods and cones), the retinal pigment epithelium (RPE) 306, Bruch's membrane 308, the choroid 310, and the sclera 312. Each of these layers is vascularized and may exhibit physiological changes due to pathological conditions.

[0025] 3A, detectable levels of higher wavelength light LB, such as ultraviolet or blue light, penetrate to a certain depth within the retina and reflect back to the sensor, but the higher wavelength light LB either does not penetrate further or is absorbed in deeper layers without being reflected. In the illustrated example, detectable levels of light LB are reflected only from a level as deep as the RPE 306.

[0026] 3B, detectable levels of low wavelength light LR, such as red or infrared light, penetrate to greater depths relative to light LB and reflect back to the sensor before being reflected or absorbed. In the illustrated example, detectable levels of light LR are reflected from Bruch's membrane 308. Light LR is also reflected from shallower layers of the retina. Often, light reflected from shallower layers has a greater intensity than light reflected from deeper layers of the retina.

[0027] 3C, the difference between the reflected light LR and the reflected light LB (LD=LR-LB) at least partially filters out the portion of the light LR that is reflected from shallower layers, allowing visualization of deeper layers. In the illustrated example, light that is reflected from the surface of the RPE 306 is filtered out, thereby allowing visualization of the RPE 306 itself and Bruch's membrane 308.

[0028] Using the principles illustrated in Figures 3A-3C, variations in image WIr caused by reflections from relatively deeper layers of the retina can therefore be enhanced by subtracting image WIb to obtain WIrb = WIr - WIb.

[0029] The operations of the image weighting step 206 and the combining step 208 can be expressed according to (1), where I out is the output of the combination stage, and W is the weight W i (i=1~N) vector and I i (i=1 to N) are images 204, and N is the number of images 204, for example N=3 for RGB images 204, or a larger number for MSI images 204.

number

[0030] In equation (1), each value Wi can be either 0 (no contribution of image Ii to Iout), a positive non-zero number (add an image), or a negative non-zero number (subtract an image). The value of W can be selected according to (2), where FOM is the figure of merit function and the I for a given value of W out Evaluate how well (W) represents retinal features such as vasculature, drusen, cotton wool spots, hemorrhage, internal limiting membrane (ILM), epiretinal membrane (ERM), or other features corresponding to pathological conditions.

number

[0031] Obtaining W according to equation (2) can be performed using any approach for performing optimization, such as the Nelder-Mead method, the Simplex method, or other numerical search methods.

[0032] The performance index function FOM() is out (W) is calculated based on common criteria for image quality such as contrast and sharpness, e.g., I out (W) can be evaluated for the acquired image 204 (either before or after processing by the weighting stage 206). Alternatively, the figure of merit function FOM() can be calculated based on the degree of enhancement of known features represented in the image 204 processed according to equation (1) for a given value of W. out (W) can be evaluated.

[0033] For example, I out The image 204 processed to obtain (W) may have one or more segmentation masks that mark regions within the image 204 whose non-zero pixels represent one or more features of interest to be enhanced. This region may include non-zero pixels only for pixels of the image 204 that include the feature of interest, or may include non-zero pixels for an extended region that includes both the pixel of the image 204 that includes the feature of interest and a region of surrounding pixels, such as in the form of an orthogonal or oriented bounding box. The segmentation mask may be generated by a human labeler or may utilize information obtained from sources other than the image 204, such as images 204 from a later stage in the disease progression when features are more visible, images obtained using another imaging modality, such as optical coherence tomography (OCT) or scanning laser ophthalmoscopy (SLO), or other sources.

[0034] The figure of merit function FOM() is calculated for each segmentation mask as I out (W) can be evaluated and assigned an output value of FOM() that is a function of some or all of the following: the contrast of one or more regions of Iout(W) marked by the segmentation mask with respect to the surrounding pixels, and Pixel intensity of one or more regions of Iout(W) marked by a segmentation mask relative to the surrounding pixels (the score increases with the intensity of pixels marked by the segmentation mask and decreases with the intensity of pixels not marked by the segmentation mask).

[0035] The output of the figure of merit function FOM() is I out The output of the figure of merit function FOM() may be a function (e.g., a sum) of values ​​obtained for multiple segmentation masks with respect to (W). Similarly, the output of the figure of merit function FOM() may be a function (e.g., a sum) of values ​​obtained by processing one or more segmentation masks on one or more sets of images 204.

[0036] Multiple weight vectors W may be obtained, e.g., one weight vector for enhancing visualization of each of a plurality of pathological conditions, one weight vector for enhancing visualization of each of a plurality of features (e.g., vascularization, hemorrhage, drusen, ILM, ERM, etc.), and / or one weight vector for enhanced imaging of each layer or group of layers within the retina (e.g., any of layers 302-312). Thus, each weight vector W may be obtained using one or more sets of images 204 and a corresponding segmentation mask corresponding to the pathological condition or feature being enhanced using weight vector W.

[0037] Examples of pathological conditions for which the vector W may be generated may include any of the pathological conditions listed below and / or retinal features corresponding to any of the pathological conditions listed below. ·Retinal tear Retinal detachment ·Diabetic retinopathy Hypertensive retinopathy Sickle cell retinopathy Central retinal vein occlusion Epiretinal membrane ·Macular hole Macular degeneration (including age-related macular degeneration) Retinitis pigmentosa Glaucoma Alzheimer's disease Parkinson's disease

[0038] The image I output by the combination step 208 out (W) may be used in a variety of ways. In some embodiments, I out (W) is displayed to the surgeon or operator. out (W) is processed by a feature extraction stage 212. The feature extraction stage 212 extracts I corresponding to one or more features or pathological conditions. out The feature extraction stage 212 may be a machine vision algorithm or a machine learning model that outputs one or more segmentation masks that label portions of (W). If implemented as a machine learning model, the feature extraction stage 212 may be implemented as a neural network, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a region-based CNN (R-CNN), an autoencoder (AE), or other type of neural network. The feature extraction stage may also take as input part or all of the image 204 and a weighted image as output from the weighting stage 206.b.

[0039] In some embodiments, the operator interface 210 displays the images 204, I out (W), and some or all of the segmentation mask from feature extraction stage 212. Operator interface 210 may also receive user input from an operator, such as a surgeon. Operator interface 210 may receive and execute instructions to perform some or all of the following tasks: Displaying only the image 204 (i.e., combining the image 204 into a single RGB image or an MSI image), Displaying the image 204 with one or more segmentation masks superimposed on it, e.g., operator-selected segmentation masks. The segmentation masks may be displayed as pixels with easily recognizable colors (red, black, purple, etc.); I out( W) to display I out (W) overlaid on one or more images 204, e.g., added to the red pixels of an RGB color image to enhance visualization of vasculature or hemorrhage; Displaying Iout(W) (alone or overlaid on the image 204) for a weight vector W of a plurality of weight vectors corresponding to a selection received from the operator, e.g., the weight vector W is selected to enhance the visualization of a feature or pathological condition of interest to the operator; and For features identified by feature extraction stage 212, overlaying a segmentation map onto any of the above options in response to a selection from multiple segmentation maps representing, for example, multiple types of features (vasculature, hemorrhage, drusen, ILM, ERM, etc.), multiple layers or groups of layers within the retina, and / or multiple pathological conditions.

[0040] System 200 may display any of the above images selected by the operator on display device 214, for example, on one of the display devices described above with respect to Figure 1 or some other display device. Operator interface 210 may receive instructions from the operator regarding which images to display by way of a touch screen, a keyboard, a mouse, voice commands, gestures detected by a camera, or other input device implementing one of the imaging devices described above with respect to Figure 1.

[0041] 4A and 4B show exemplary images that may be acquired using system 200. Fig. 4A shows the red, green, and blue images that make up the captured RGB original image of a retinal sample. Fig. 4B shows the difference images: red-green, red-blue, and green-blue. As can be seen, features such as veins are highly visible, especially in the red-green image.

[0042] 4B also includes a plot of the camera quantum efficiency of a typical digital camera. As can be seen, the peak sensitivity (red, green, and blue) of each sensor is not equal. Therefore, this variation in peak sensitivity can be accounted for by the weighting step 206.

[0043] 5 illustrates an exemplary method for processing the image 204 as defined above. The method 500 may be performed by the system 200. The system 200 itself may be implemented using the computing power of the imaging device 202 or a computing device that receives the image from the imaging device 202.

[0044] Method 500 includes receiving images 204 from imaging device 202 in step 502 and weighting images 204 to obtain a weighted image in step 504. Weighting may include some or all of normalizing, compensating for differences in camera sensitivity, and multiplying by weights to enhance feature visibility. The weighted images may then be combined in step 506 by adding, subtracting, multiplying, or performing some other operation on two or more of images 204 to obtain a combined image. Steps 504 and 506 result in a combined image I. out This can be done according to equation (1) above to obtain (W).

[0045] Method 500 may include performing feature extraction on the combined image at step 508. Feature extraction may include processing the combined image using machine learning models or machine vision algorithms as described above with respect to feature extraction stage 212.

[0046] Method 500 may include, at step 510, overlaying a representation of the features identified in step 508, such as a segmentation mask, onto image 204, the combined image, or one or more of any of the options described above. The result of the overlay at step 510 may then be displayed on a display device at step 512. Note that in some embodiments, feature extraction step 508 and overlay step 510 are omitted, and the combined image is displayed alone or overlaid on image 204.

[0047] If stereoscopic imaging or three-dimensional rendering is used, step 512 may use the results of processing two or more image sets from two or more different camera viewpoints according to method 500 and use the combined image or overlay results for each of the two or more image sets to provide the stereoscopic vision or three-dimensional rendering.

[0048] Step 512 may be performed repeatedly or periodically during surgery to provide real-time feedback to the surgeon. For example, step 512 may be performed to provide feedback regarding membrane detachment (ILM, ERM) by enhancing visibility of the portion of the membrane that remains detached and / or detecting traction on other layers of the retina. Step 512 may be used to assess the condition of the retina while performing a vitrectomy, pneumatic retinopexy, scleral buckle, or other ophthalmic surgery.

[0049] Step 512 may be used to diagnose a pathological condition. Images with enhanced or labeled features as described above may be displayed to a surgeon to enable diagnosis of any of the above pathological conditions. Such images captured at two or more different time points may be analyzed by a human operator or a machine learning model to assess changes in features (e.g., drusen growth, changes in vasculature, etc.).

[0050] Figure 6 illustrates an exemplary computing system 600 7700 00 that at least partially implements one or more functions described herein with respect to Figures 2 and 5. Computing system 600 may be integrated with an imaging device that captures images according to one or more of the imaging modalities described herein, or may be a separate computing device.

[0051] As shown, computing system 600 includes a central processing unit (CPU) 602, one or more input / output device interfaces 604 that enable various input / output devices 614 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to computing system 600, a network interface 606 that connects computing system 600 to a network 690, memory 608, storage 610, and an interconnect unit 612.

[0052] If the computing system 600 is an imaging system such as an SLO, OCT, or fundus camera, the computing system 600 may further include one or more optical components for performing ophthalmic imaging of a patient's eye, and any other components known to those skilled in the art.

[0053] CPU 602 may retrieve and execute programming instructions stored in memory 608. Similarly, CPU 602 may retrieve and store application data in memory 608. Interconnect 612 transfers programming instructions and application data between CPU 602, input / output device interface 604, network interface 606, memory 608, and storage 610. CPU 602 is included as representative of a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc.

[0054] Memory 608 represents volatile memory, such as random access memory, and / or non-volatile memory, such as non-volatile random access memory, phase change random access memory, etc. As shown, memory 608 may store executable code to implement weighting stage 206, combining stage 208, and / or feature extraction stage 212.

[0055] Storage 610 may be non-volatile memory such as a disk drive, solid state drive, or a collection of storage devices distributed across multiple storage systems. Storage 610 may optionally store images 204 or processed versions 624 of images 204 resulting from weighting, combining, or other processing described herein. Storage 610 may also store feature labels 626 extracted from processed images 204 as described herein, such as a segmentation map obtained from the combined image as described above.

[0056] Additional matters The above description is provided to enable those skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of elements discussed without departing from the scope of the disclosure. In various examples, various actions or elements may be omitted, substituted, or added as appropriate. Also, features described with respect to some examples may be combined in several other examples. For example, an apparatus may be implemented or a method may be practiced using any number of aspects described herein. Furthermore, the scope of the disclosure is intended to cover similar apparatuses or methods that are practiced using structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements recited in the claims.

[0057] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single elements. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or a, b, and c in any other order).

[0058] As used herein, the term "determining" encompasses a wide variety of acts. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. "Determining" can also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0059] The methods disclosed herein include one or more steps or actions for achieving the method. Method steps and / or actions may be interchangeable with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the claims. Furthermore, various actions of the methods described above may be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software elements and / or modules, including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. In general, where there are actions illustrated in figures, those actions may include corresponding means-plus-function elements with similar numbering.

[0060] The various illustrated logic blocks, modules, and circuits described in connection with this disclosure may also be implemented or embodied in a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0061] The processing system may be implemented with a bus architecture. The bus may include any number of interconnecting buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may interconnect various circuits, including, among other things, a processor, machine-readable media, and input / output devices. A user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also connect various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are known in the art and will not be described further. The processor may be implemented by one or more general-purpose and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Those skilled in the art will recognize how to best implement the described functionality of a processing system depending on the particular application and the overall design constraints imposed on the overall system.

[0062] If implemented in software, the functions described above may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Software should be broadly construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Computer-readable media includes both computer storage media and communication media, such as any medium that facilitates transfer of a computer program from one place to another. A processor may be responsible for managing the bus and general processing, including the execution of software modules stored on the computer-readable storage medium. The computer-readable storage medium may be coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integral to the processor. By way of example, computer-readable media may include computer-readable storage media on which instructions are stored separately from a transmission line, a carrier wave modulated with data, and / or a wireless node, all of which may be accessed by the processor via a bus interface. Alternatively or additionally, the computer-readable medium, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or general-purpose register file. Examples of machine-readable storage media include, for example, RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium or any combination thereof. The machine-readable medium may be embodied in a computer program product.

[0063] A software module may include a single instruction or many instructions and may be distributed across several different code segments, among different programs, and across multiple storage media. A computer-readable medium may include many software modules. A software module includes instructions that, when executed by a device such as a processor, cause a processing system to perform various functions. A software module may include a transmitting module and a receiving module. Each software module may reside on a single storage device or may be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. During execution of a software module, a processor may load some of the instructions into a cache to speed access. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to the functionality of a software module, it is understood that such functionality is realized by the processor when executing instructions from that software module.

[0064] The following claims are not limited to the embodiments set forth herein but are to be accorded the full scope consistent with the language of the claims. In the claims, when an element is referred to in the singular, it means "one or more," not "only one," unless specifically stated otherwise. The term "some" refers to one or more, unless specifically stated otherwise. No element of a claim is to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step of." All structural and functional equivalents of the elements of various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, the disclosure herein is not intended to be made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. one or more processing devices; and one or more memory devices coupled to the one or more processing devices, the one or more memory devices storing executable code that, when executed by the one or more processing devices, causes the one or more processing devices to: receiving a plurality of images of the patient's eye, each image corresponding to a different portion of the electromagnetic spectrum; performing a pixel-by-pixel combination of two or more images of the plurality of images to obtain a combined image; outputting a representation of the combined image to a display device; Ophthalmic visualization system.

2. The ophthalmic visualization system of claim 1 , wherein the plurality of images includes a red image, a green image, and a blue image that form a color image.

3. The ophthalmic visualization system of claim 1 , wherein the plurality of images are obtained using a multispectral imaging (MSI) device.

4. The ophthalmic visualization system of claim 1 , wherein the plurality of images includes an infrared image.

5. 2. The ophthalmic visualization system of claim 1, wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to obtain the combined image by performing pixel-by-pixel subtraction of one or more first images of the plurality of images from one or more second images of the plurality of images.

6. 2. The ophthalmic visualization system of claim 1, wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to weight the two or more images with two or more weights to obtain two or more weighted images, and obtain the combined image by obtaining a pixel-wise combination of the two or more weighted images.

7. 7. The ophthalmic visualization system of claim 6, wherein the two or more weights are selected to enhance representation of one or more features selected from the group consisting of vasculature, drusen, hemorrhage, and cotton wool spots.

8. The ophthalmic visualization system of claim 6 , wherein the two or more weights are selected to enhance representation of one or more layers of a retina of the eye of the patient.

9. The two or more weights are: ・Retinal tear, - retinal detachment, ・Diabetic retinopathy, - hypertensive retinopathy, sickle cell retinopathy, - Central retinal vein occlusion, - epiretinal membrane, ・Macular hole, -Macular degeneration (including age-related macular degeneration), - retinitis pigmentosa, ・Glaucoma, Alzheimer's disease, 7. The ophthalmic visualization system of claim 6, wherein the imaging device is selected to enhance the representation of one or more features corresponding to a pathological condition selected from the group consisting of:

10. When executed by the one or more processing devices, the executable code further causes the one or more processing devices to: processing the combined image to obtain a segmentation mask that identifies one or more features represented in the combined image; (a) overlaying the segmentation mask on one or more of the plurality of images and (b) one of the combined images to obtain the representation of the combined image. outputting the representation of the combined image to the display device by The ophthalmic visualization system of claim 1 .

Images