Patient-tailored ophthalmic imaging system with single-exposure multi-species imaging, improved focusing, and improved angiographic image sequence display

Abstract

A tool is provided to facilitate focusing of an ophthalmic imaging or examination system. The ophthalmic imaging system provides an autofocus mechanism based on the difference between successive scan lines. The system also provides user selection of focus within fundus images. A neural network automatically identifies the optic disc within FA or ICGA images, which can be used to determine the fixation angle. The system also provides additional scan tables for multiple imaging modalities to accommodate photophobic patients and multispectral imaging options.

Description

[Technical Field]

[0001] The present invention is directed generally to the field of ophthalmic imaging systems, and more particularly to techniques for facilitating user operation of ophthalmic imaging systems. [Background technology]

[0002] There are various types of ophthalmic examination systems, including ophthalmoscopes, optical coherence tomography (OCT), and other ophthalmic imaging systems. One example of ophthalmic imaging is slit-scanning fundus imaging or wide-line fundus imaging (see, for example, U.S. Patent Nos. 6,229,999; 6,230,263; 6,230,263; 6,230,264; and 6,230,265, the entire contents of which are incorporated herein by reference), which is a promising technology for achieving high-resolution in vivo imaging of the human retina. The imaging approach is a hybrid of confocal and wide-field imaging systems. By illuminating a narrow, elongated portion of the retina during scanning, the illumination remains outside the viewing path, allowing for clearer viewing of a larger portion of the retina compared to the annular ring illumination used in conventional fundus cameras.

[0003] To obtain a good image, it is desirable for the illumination strip to be well focused and pass through the pupil of the eye without attenuation to the fundus of the eye. This requires careful focusing of the system and alignment of the eye with the ophthalmic imaging system. Further complicating ophthalmic imaging is that it can be difficult for the patient to remain still during imaging. This can be particularly problematic when multiple different types of images are required or when the patient has an aversion to high-intensity illumination, e.g., photophobia. As a result, it generally requires significant training to achieve a high level of proficiency when using such systems. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 4,170,398 [Patent Document 2] U.S. Patent No. 4,732,466 [Patent Document 3] International Publication No. 2012059236 [Patent Document 4] US Patent Application Publication No. 2014 / 0232987 [Patent Document 5] US Patent Application Publication No. 2015 / 0131050 Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present invention to provide a tool to facilitate focusing of an ophthalmic imaging or examination system. Another object of the present invention is to provide various methods for speeding up the capture of ophthalmic images.

[0006] It is a further object of the present invention to provide a method for reducing patient anxiety or discomfort during the capture of ophthalmic images. [Means for solving the problem]

[0007] The above objectives are achieved with a system / method having improved focus and functionality. First, a focus mechanism based on the difference between successive line scans effectively converts broad lines into thin lines for the purpose of determining defocus measurements. These thin lines can then be used to determine the topology of the retina.

[0008] In a preferred embodiment, the system operator is presented with a preview screen with multiple pre-specified focus assist positions. An image may be displayed on the preview screen, and the system operator may freely select any point on the preview screen to achieve tighter focus. The preview screen then focuses the image at the selected point by combining focus information from the focus assist positions.

[0009] In some embodiments, the focus is further adjusted based on the patient's fixation angle, which may be determined by locating the patient's optic nerve head (ONH). Provided herein are techniques for locating the ONH in an infrared (IR) preview image so that the patient's true fixation angle can be ascertained.

[0010] It should be noted that locating the ONH in any type of image is beneficial, as locating the ONH has additional applications. Presented herein is a method for locating the ONH in FA (fluorescein angiography) and / or ICGA (indocyanine green angiography) images and / or any other imaging modality. The system may continuously capture an IR (infrared) preview image before capturing the FA or ICGA image, so that the system locates the ONH in the IR preview image and then transfers the located ONH to the captured FA or ICGA image.

[0011] To achieve this, the system provides a U-Net (an overview of the U-Net can be found in Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation," Computer Vision and Pattern Recognition, 2015, Archives (arXiv):1505.04597 [cs.CV]), which uses transfer training to utilize a large library of training color images to learn how to locate the ONH in infrared images. First, the U-Net is trained using only color images. Then, a predetermined number of initial layers of the U-Net are fixed, and the remaining layers are trained with a smaller training set of infrared images. Localizing the ONH in infrared images can still be problematic, and additional steps are provided to optimize ONH localization.

[0012] To further improve FA and ICGA examinations, the system provides simplified capture of FA or ICGA image sequences. The system provides a very high dynamic range so that the operator does not need to adjust image brightness while capturing the FA or ICGA image sequence. Raw images are stored, and relative brightness information between images in the sequence can be determined from the raw data and accurately provided to the user, for example, as a plot. In addition, the operator is given the option to brighten the stored images for viewing while maintaining the relative brightness between the acquired image sequence.

[0013] To further accommodate patients, the system provides patient-tailored imaging. The system examines the physical characteristics of the patient's eye (and / or the patient's medical records) for indicators of photophobia. If the patient is suspected of being a photophobic candidate or has previously been diagnosed with photophobia, the system operator is alerted to this fact. The system operator can then select to have the system use a dimming imaging scheme based on the patient's degree of photophobia.

[0014] Other improvements to the system include various mechanisms for performing multiple imaging modalities with a single capture command. This not only reduces the time required for the examination, but also provides additional information that can be used for further diagnosis. For example, the system can automatically capture IR images along with all color images, which can enhance the visibility of some tissues.

[0015] Other objects and achievements of the present invention, together with a fuller understanding of the invention, will become apparent and appreciated by reference to the following description and claims taken in conjunction with the accompanying drawings.

[0016] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Features of any embodiment described in one claim category, e.g., a system, may also be claimed in other claim categories, e.g., a method. Dependencies or back-references in the appended claims are selected for formality reasons only. However, any subject matter available from a careful back-reference to a previous claim may also be claimed, thereby disclosing any combination of claims and their features and may be claimed regardless of the dependencies selected in the appended claims. [Brief explanation of the drawings]

[0017] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee. [Figure 1] 1 illustrates a simplified pattern of scan lines as they may be generated on an object being scanned. [Figure 2A] 1 illustrates an overlapping line scan sequence where the step size between scan lines is less than the width of a scan line. [Figure 2B-2C] 2B illustrates the subtraction effect of two successively captured images of consecutive scan lines in FIG. 2A, and 2C illustrates the creation of a single thin line from the two scan lines and the determination of the second moment σ, which defines the line width. [Figure 3A-3B] 3A illustrates how the displacement of the center of gravity can correspond to a change in eye depth (or height) (e.g., a deconvolution measurement), and 3B illustrates how the displacement of the center of gravity can correspond to a change in eye depth (or height) (e.g., a deconvolution measurement). [Figure 4A-4B] 4A illustrates a preview fundus image in a preview window on an electronic display with multiple focus assist positions superimposed on the fundus image, and 4B provides an example of how a user may freely select any point on the preview image on which to focus. [Figure 5]Two images of blood vessels taken at different stages of an angiography procedure are provided. [Figure 6] In accordance with the present invention, a segmentation and localization framework is provided for locating ONH in FA / ICGA images during operation. [Figures 7A-7B] 7A illustrates the location of the ONH region in an IR preview image identified using the system of FIG. 6, and 7B illustrates an FA image captured immediately after capturing the IR preview image of FIG. [Figure 8] It provides a framework for processing individual images within a sequence of images captured as part of an angiographic examination. [Figure 9] 1 shows a FA time-lapse image sequence in which each individual image has been adjusted according to its respective individual white level. [Figure 10] 1 shows a raw FA image sequence (corresponding to the image sequence of FIG. 9) in which no brightness adjustment has been applied to any of the images. [Figure 11] 10 shows a globally adjusted FA image sequence (corresponding to the image sequence of FIG. 10) in which all images are brightened by the same global scaling factor (eg, 255 / wmax). [Figure 12A] 1 illustrates the effect of setting the compression parameter ∝ to zero, which is equivalent to brightening all images by the same global scaling factor. [Figure 12B] 1 illustrates the effect of the compression parameter ∝ being set to 1, which is equivalent to automatically brightening each image individually according to its stored white level. [Figure 12C] 1 illustrates how modifying the compression parameter ∝ affects the luminance of an image sequence for different values ​​of ∝. [Figure 12D] 1 illustrates how modifying the compression parameter ∝ affects the luminance of an image sequence for different values ​​of ∝. [Figure 12E] 1 illustrates how modifying the compression parameter ∝ affects the luminance of an image sequence for different values ​​of ∝. [Figure 12F] 1 illustrates how modifying the compression parameter ∝ affects the luminance of an image sequence for different values ​​of ∝. [Figure 13] An example of a suitable graphical user interface (GUI) for an ophthalmic imaging device is provided. [Figure 14] Exemplary steps are provided for adjusting (eg, adjusting) the amount of light dimming depending on the patient's relative light sensitivity. [Figure 15] 10 illustrates how scan tables (e.g., intensity levels used for image acquisition) can be selected and / or modified based on pupil size and / or iris color. [Figure 16] 1 illustrates an exemplary series of multiple images across multiple wavelengths that may be captured in response to a single capture instruction input with the multispectral option selected. [Figure 17] The present invention is compared to two previous methods that provide FA+ICGA functionality in response to a single capture instruction. [Figure 18] A four-channel color fundus image is shown. [Figure 19] 10 shows an example of a "Channel Split" review screen that provides options for viewing red, green, blue, and infrared channels separately. [Figure 20] 1 shows the absorbance spectra of oxygenated and deoxygenated hemoglobin. [Figure 21] 1 provides an example of an oxygen saturation map. [Figure 22] The principle of MPOD measurement (left) and MPOD profile (right) are illustrated. [Figure 23] A light box design with eight light sources (two each of red, green, blue, and IR) is illustrated. [Figure 24] 1 illustrates an example of a slit-scanning ophthalmic system for photographing the fundus. [Figure 25] 1 illustrates a generalized frequency-domain optical coherence tomography system used to collect 3D image data of the eye suitable for use with the present invention. [Figure 26]An example of an en face vascular image is shown. [Figure 27] Illustrate an example of a multi-layer perceptron (MLP) neural network. [Figure 28] A simplified neural network consisting of an input layer, a hidden layer, and an output layer is shown. [Figure 29] 1 illustrates an exemplary convolutional neural network architecture. [Figure 30] 1 illustrates an exemplary U-Net architecture. [Figure 31] 1 illustrates an exemplary computer system (or computing device or computer apparatus). DETAILED DESCRIPTION OF THE INVENTION

[0018] There are various types of ophthalmic imaging systems, as discussed below in the sections on fundus imaging systems and optical coherence tomography (OCT) imaging systems. Aspects of the present invention may be applied to any or all of such ophthalmic imaging systems. Generally, the present invention provides various enhancements to the operation and user interface of ophthalmic imaging systems.

[0019] Fine line scanning for focus and depth analysis. One aspect of the present invention provides improved methods for determining image measurements for focusing applications (e.g., autofocus) and deconvolution applications (e.g., topography). As a particular example, the improved focusing (and deconvolution) techniques and applications of the present invention are described as applied to ophthalmic imaging systems that use a linear light beam, such as a wide line, that is scanned across a sample to create a series of image segments that can be combined to construct a composite image of the sample, although it should be understood that the present invention may also be applied to other types of ophthalmic imaging systems.

[0020] For example, FIG. 1 illustrates a simplified scan line pattern (e.g., a slit or a wide line) that may be generated on a scanned object. In this example, the scan line is scanned vertically (e.g., traversed) to generate multiple scan lines L1-Li in a vertical scan pattern, a V-scan. Such a scan pattern may be used by a line-scan fundus imaging system (or an OCT-based system), and generally, the scan line may maintain some confocal suppression of out-of-focus light orthogonal to the scan lines (L1-Li) (e.g., along the Y axis) but may lack confocal suppression along the line (e.g., along the X axis). The scan line may also be used to improve imaging. For example, the edge definition of the illumination strip may be used to find an optimized focus for a line-scan system when the illumination does not move significantly during acquisition by the detector (typically, when the scanning beam is scanned in steps and remains relatively stationary during acquisition). A focusing method suitable for line-scanning fundus imaging systems is disclosed in International Publication No. 2018178269, which is incorporated herein by reference in its entirety. As another example of an improvement, to assess background light levels, e.g., stray light levels, from out-of-focus regions of the eye, locations on the retina that are not directly illuminated by the scanning line can be detected (e.g., images captured), and this background level can then be subtracted from the captured line image. Line-scanning imagers have also been combined with pupil splitting (see, e.g., U.S. Patent No. 8,488,895 to Muller et al., which is incorporated herein by reference in its entirety).

[0021] In summary, line-scanning imaging systems construct fundus images by recording images of individual scan lines on the retina of the eye, but the recorded scan lines can be used to extract more information. For example, the line width is a direct measure of focus. This information can be used for (e.g., automatic) focusing of the system or as additional input to a deconvolution algorithm. Furthermore, the line position (centroid) on the detector (e.g., camera) is a direct measure of the height of the sample (e.g., the depth of the retina at a particular point). This topographic information can be used, for example, to analyze blood vessels, vascular intersections, or other structures on the retina (such as tumors whose volume is to be measured).

[0022] One of the crucial technical details of line scanning is the line width. If the line is too wide, it is difficult, if not impossible, to distinguish reflectivity from position and line width, increasing the uncertainty of line localization; for example, a wide line may automatically average information over a wide area. While there are advantages to using a wide line to scan an image, as explained below, a wide line may not be ideal for localization or width analysis. The present invention provides a method for utilizing a scan sequence to narrow the effective line width, which can be used for focus and / or topography analysis.

[0023] FIG. 2A illustrates an overlapping line scan sequence in which the step size between scan lines is smaller than the width of the scan line. For example, the scan step size S1 between the first scan line l1 and the second scan line l2 is smaller than the width W of the scan line l1 (and scan line l2) so that the second scan line l2 overlaps the scan area defined by the first scan line l1. FIG. 2B illustrates the subtraction effect (e.g., l2 subtracted from l1) of two consecutively captured images (e.g., scan images) of consecutive scan lines (e.g., adjacent scan lines) in FIG. 2A. The adjacent scan lines can be modeled as multiple rectangles with a small displacement S1, resulting in a double line D1 / D2 with a displacement width S1. Here, the two lines D1 / D2 (one positive and the other negative) can be analyzed based on the width S1 and position of the two lines D1 / D2. This effectively converts a scan image of a wide line (eg, l1 and / or l2) with a small step size S1 into a scan image of a (double) thin line D1 / D2 with the same step size.

[0024] To determine the line position of the captured scan image, the (light) intensity centroid (e.g., target region) of the captured scan image can be calculated. This is a very fast algorithm, but can be susceptible to noise. To improve robustness, the scan image can be segmented into foreground and background (e.g., line and noise) segments, and the centroid can be calculated for the foreground segment. The centroid position can be calculated with sub-pixel accuracy, which increases depth sensitivity.

[0025] 3A and 3B illustrate how the displacement of the center of gravity can correspond to changes in the depth (or height) of the eye (e.g., a deconvolution measurement). The displacement (of the center of gravity relative to the predicted position) is a direct measurement of retinal topography and can be converted to an actual height (measurement) with a calibration system. That is, the misalignment of the line position (from the predicted position) can be due to changes in the height of the scanned object or a protrusion (e.g., a change in the depth of the eye due to two blood vessels crossing each other, etc.). FIG. 3A shows a scan image of scan line l3 without any displacement, such as from a scan of a planar object with no irregularities in depth. FIG. 3B shows a scan image of the same scan line l3 with a protrusion B1 due to the scanned object having an irregularity (e.g., a protrusion) on its surface. The height Δz of the protrusion B1, which can be determined by tracking the change in the position of the center of gravity along the length of the captured scan image, corresponds to the height of the difference (protrusion) in the surface of the scanned object (e.g., is a height measurement). This approach may benefit from calibration of the camera and illumination optics.

[0026] Similar to locating the line position, the line width can be determined from the foreground segment using the second moment of the intensity distribution (note: the centroid corresponds to the first moment). The width is a direct measure of image blur (e.g., defocus), at least in the direction perpendicular to the scan line. This width measurement can be determined from any of the constructed thin lines illustrated in FIG. 2B.

[0027] For example, double thin lines D1 / D2 can be converted to a single line by selecting either the positive line (e.g., D1) or the negative line (e.g., D2) for analysis. For example, thin line D1 can be defined alone by selecting the maximum value of (l1-l2) and setting the remainder to zero (e.g., max[(l1-l2),0]), or thin line D2 can be defined alone by selecting the minimum value of (l1-l2) and setting the remainder to zero (min[(l1-l2),0]). FIG. 2C illustrates the creation of a single thin line D1' from two scan lines l1' and l2' by defining max[(l1'-l2'),0] and identifies the second moment σ, which defines the line width of D1' and its position. Alternatively, or in addition to using the second moment σ, the line position and width can be defined by fitting a functional shape (e.g., a Gaussian distribution) to the observed data (e.g., D1'). The parameters of this fitting then give the position and width. The advantage of this approach is that it can take into account prior knowledge about the shape of the scan line (e.g., rectangular illumination). In addition, robust fitting methods can be used to reduce noise sensitivity.

[0028] In this way, broad lines in a fundus imaging system (such as those shown in FIG. 2A) can be converted (e.g., digitally) into thin lines as the difference between small step scan images, which can then be analyzed for width (defocus) and position (depth of the object).

[0029] The line width and line displacement are direct measurements of defocus and can be considered equivalent to retinal depth, i.e., retinal height. These measurements can be used in imaging systems, for example, to provide autofocus. At higher resolutions, such as those described with reference to FIG. 2, this information can also be used for other purposes (e.g., depth can be used to segment blood vessels, segment optical disks, estimate tumor volume, etc.). Essentially, the method provides improved resolution.

[0030] This increased resolution can be explained using the following imaging model.

[0031]

number

[0032]

number

[0033]

number

[0034]

number

[0035]

number

[0036] Alternatively, the weights can be used to approximate the sinusoidal illumination.

[0037]

number

[0038] Using this technique, a (unidirectional) defocus map can be constructed by determining the focus (and / or centroid position) at multiple points along the scanned image, and the defocus map can then be used as an autofocus tool or to improve deconvolution algorithms.

[0039] Alternatively, instead of analyzing a single scan line, a structured light pattern, such as that used in 3D scanning, can be generated. Because the rectangular shape of the illumination is blurred (i.e., more Gaussian bell (or convex) shaped), sinusoidal illumination can be resampled in combination with alternating signs (e.g., positive and negative signs). Three such patterns can be used for phase shifting to extract depth information. Another example is the use of a triangular illumination pattern, which can be approximated from a rectangular pattern by correlation.

[0040] In the above example, we removed the alpha (α) scaling factor, which may be a measure of amplitude (e.g., intensity), to remove (or reduce) illumination blur. However, it has been found that the alpha (α) scaling factor varies with (e.g., represents or is related to) illumination defocus and therefore may provide a direct measure of system defocus. For example, the defocus measurement may be determined based on the amplitude of the sinusoidal illumination. Furthermore, it has been found that the alpha (α) scaling factor is frequency dependent. As a result, using sinusoidal waves of different frequencies results in a series of alpha values ​​that can be used to represent the complete (or substantially complete) point spread function of the system. Because the amplitude depends on the frequency used, the amplitude information can also be used to generate a pseudo modulation transfer function (MTF), which may include the system optics, eye, and stripe width. These alpha (α) values ​​may constitute an “alpha map” that can be used for autofocus purposes and / or depth measurement purposes (e.g., surface topography / topology). For example, by sweeping the frequency, a topographical map can be obtained, since different frequencies result in different alpha (α) values ​​(e.g., different amplitudes). Topographical information can be determined by performing a Fast Fourier Transform (FFT) of the amplitude data for each scan. Thus, a retinal topography and / or defocus map can be determined using a linear combination of image stripes (or scans).

[0041] Balanced multi-point autofocus Any of the above-described methods for determining defocus at multiple points or locations on a fundus image (such as on a preliminary or preview image before imaging / scanning the eye) (or other known methods for measuring defocus) may be used in conjunction with a graphical user interface (GIU) to improve a user's (e.g., a human operator's) ability to capture a usable image of the fundus.

[0042] The focal depth of a wide-field fundus camera can be smaller than the retinal depth variation, causing peripheral defocusing of the image. This is very common in images of myopic retinas. Also, in cases of retinal detachment and tumors, users may struggle to obtain a focused image of the tumor or bulging retina or of the surrounding retina. In conventional autofocus methods used in ophthalmic imaging systems, the (fundus) image is focused at the center of the image, but this method is not useful when a clinician wants to focus on a specific portion of the retina (e.g., off-center) or simply wants to obtain the best overall focused image. The present invention addresses this problem.

[0043] Typically, autofocus is generally provided in the center of the camera's field of view (FOV) using a number of different techniques. A limitation of this approach is that the focus is selected purely based on the center of the image, and therefore cannot be used for areas other than the center of the field of view. This approach also prevents the imaging system from being able to focus on more of the image. As can be understood, the back of the eye is not flat, and therefore different parts of the fundus of the eye within the camera's FOV may require different focus settings to focus on them.

[0044] The present invention can determine focus at different portions of the eye, each portion defining a focus assist position. The focus assist positions can then be used to determine the optimal focus for capturing a new image. The present invention can obtain focus readings at multiple distinct positions (focus assist positions) within the eye (e.g., determine defocus measurements for multiple distinct positions (focus assist positions)) using any of multiple focus assists / mechanisms, for example, as described above and / or in WO2018178269. The focus at each position can be adjusted to compensate for various defocus conditions (such as areas of astigmatism in the eye (irregularly shaped cornea), floaters, eye curvature, etc.) taking into account the patient's direction of gaze. Additionally, the system can adjust how it focuses the image according to input commands from a user (imaging system operator). For example, a weight for each of multiple focus assist positions can be calculated depending on how the user wants the image to be focused (e.g., balanced focus or point focus). Optionally, the collected information can be combined to calculate a single focus reading. In this way, the present invention can focus on any user-specified portion of the eye, or on a wider region of the imaging system's FOV.

[0045] 4A illustrates a preview fundus image 11 in a preview window (or display area) 15 on an electronic display, with multiple focus assist positions f1-f7 superimposed on the fundus image 11. It should be understood that seven focus assist positions are shown for illustrative purposes, and that any number of multiple focus assist positions may be provided. It should further be understood that focus assist positions f1-f7 may be positioned at pre-specified (optionally fixed) locations and span a predetermined area (fixed or variable) of the fundus image 11 or display area 15. Optionally, focus assist positions f1-f7 may be displayed on the fundus image 11 (e.g., superimposed on the fundus image 11) or may be hidden from view in the display area 15. That is, a user may be permitted to view and select any number of focus assist positions f1-f7, or focus assist positions f1-f7 may be hidden from the user's view. The user preferably selects one or more areas (or points) within the fundus image 11 to focus on for image capture. The user may input this selection using any known input device or mechanism (e.g., keyboard, pointer, roller, touch screen, voice, input script, etc.).

[0046] In operation, each focal assist location on the retina (fundus) is illuminated (e.g., by a respective focal assist light, which may be a special scan line or other shaped light beam) through a light path entering the patient's pupil that is offset relative to the viewing path (e.g., collection path) of the imaging system's camera, as described below. Refractive errors in the patient's eye bend the focal assist light entering the eye differently than the viewing path exiting the eye, resulting in a refractive error-dependent position of each focal assist location (e.g., a position different from the expected position) that is converted into a defocus measurement of the system's camera relative to the focus setting required to clearly image the focal assist light's location on the retina.

[0047] Each focus aid f1-f7 senses the component of refraction in an offset direction within the pupil, which may differ from refraction in other directions, and then determines the focus for best image clarity at each focus aid position, especially in the presence of astigmatism. The average human eye has several diopters of astigmatism, and an adjustable, astigmatism-free fundus camera (or other ophthalmic imaging system) typically only corrects the astigmatism of the average eye when the eye is rotated to look at the center of the camera. When the eye is rotated relative to the camera, such as at any given fixation target, the system can take into account the mismatch between the eye's astigmatism and the fundus camera's astigmatism correction. This can be done using the following equation:

[0048]

number

[0049] Thus, a user may choose to optimize the system for focusing at any given point / region within the system's FOV, such as by selecting a given focus assist position or any position on the preview fundus image 11. If the user-selected point / region for focusing does not correspond to a particular focus assist position (f1-f7), which may or may not be displayed on the preview image 11, the optimal focus for the user-selected point may be determined by an appropriate combination of the focus assists (f1-f7) weighted relative to the user-selected point's location. Alternatively, a user may choose to use a predetermined focus balance across the preview image, which may apply a predetermined weight to each of the focus assist positions. For example, balanced focus may be calculated as follows:

[0050]

number

[0051] FIG. 4B provides an example of how a user can freely select any point on the preview image 11 to focus on. In this example, the user-inputted focus point is identified by a star. The user can input the desired focus point using an input device (e.g., a click point) or using a touch screen. In this case, the optimal focus point for the user-inputted point can be determined as a weighted combination of focus assist positions for the user-inputted point. For example, individual distances d1 to d7 from the input click point to each of focus assist positions f1 to f7 are determined. Then, the focus point for the click point (star) can be determined as follows:

[0052]

number

[0053] In summary, the technique takes focus readings at multiple distinct positions on the eye (e.g., f1-f7). The focus readings at each position are then adjusted to account for the fixation angle. A weight is calculated for each focus position relative to the user-input focus (the weight may depend on the distance from a given distinct focus position to the user-selected focus). Finally, a focus adjustment for image capture may be calculated for the user-selected focus based on the weighted focus of each distinct focus position.

[0054] It should be noted that while the system operator may provide a fixation point for the patient, the patient may not be able to maintain focus on the provided fixation point. That is, the patient's true fixation point may differ from the fixation point provided by the system. Therefore, the present system further provides a mechanism for determining the patient's true fixation direction when determining the optimal focus setting, as described above. For example, the system may use infrared imaging to acquire a test image, which may be preview image 11. The system may then identify the optic disc in the test (e.g., preview) image and determine the patient's gaze angle based on the location of the identified optic disc. Identifying the optic nerve may be complicated if the patient suffers from optic nerve hypoplasia, which is typically associated with an underdeveloped optic nerve. Therefore, the system may further search for other landmarks indicative of gaze angle, such as the fovea or distinctive vasculature. The patient's gaze may then be determined based on the location of the identified landmarks in the IR preview image and the known imaging settings of the system relative to the patient's eye (e.g., lighting and viewing angle). The use of IR preview images is advantageous because the eye is not sensitive to IR wavelengths, and IR preview images can be captured continuously without disturbing the eye, such as for system alignment purposes. Specific landmarks can be identified using specific image processing algorithms or machine learning techniques (e.g., deep learning). Because landmarks in visible light images can be more difficult to locate when certain types of images are captured, such as fluorescein angiography (FA) or indocyanine green angiography (ICGA) imaging, infrared (preview) images are preferred over visible light (preview) images for this task. However, the identified location of the optic disc determined from the IR preview image can be directly applied to the captured FA and / or ICGA images if the IR preview image is captured substantially close in time to the captured FA and / or ICGA images (e.g., immediately before or simultaneously with the captured FA and / or ICGA images). As will be appreciated, this use of IR imaging to determine the eye's fixation angle can be used with other ophthalmic imaging systems, such as OCT.Furthermore, it should be noted that IR images are used herein for fixation detection and therefore are not used for eye tracking, which images the fundus (or posterior part) of the eye and not the pupil (or anterior part) of the eye.

[0055] U-net for locating the center of the optic disc in wide-field fundus images Provided herein is an example of using deep learning to identify the center of the optic disc, such as in a wide-field fundus image. As mentioned above, the center of the optic disc can be used to identify the patient's gaze direction.

[0056] The optic nerve head (ONH) is one of the most prominent landmarks observed in fundus images, and the ability to automatically locate the ONH is highly desirable in fundus image processing. ONH localization algorithms can help locate the fovea or other landmarks by providing a reference for applying a standardized offset or region of interest (ROI) to begin searching for lesions. Most of the available literature on ONH localization addresses locating the ONH in color (e.g., visible light) fundus images. However, while there are several types of fundus images and ONH localization techniques applicable to color images, many of these techniques are less suitable for other types of images. For example, locating the optic nerve in fluorescein angiography (FA) and indocyanine green angiography (ICGA) angiography images is extremely challenging because the appearance of the ONH changes as dye progresses through the ocular vasculature during various stages of the angiography procedure. This paper presents a U-net-based deep learning architecture applied to infrared (IR) preview images to reliably locate the ONH in all stages of FA and ICGA images.

[0057] Traditional ONH localization algorithms use handcrafted features / filters. For example, Sinthanayothin et al., "Automatic Localization of the Optic Disk, Fovea, and Retinal Blood Vessels from Digital Color Fundus Images," Br. J. Ophthalmol., Vol. 83, No. 8, pp. 902-910, 1999, uses a dispersion filter to exploit the rapid intensity changes due to the presence of blood vessels in the optic disc. However, this method is only effective for color fundus images and was tested on a small dataset. It has also been shown that this method is likely to fail on fundus images with white lesions and prominent choroidal vessels. Another approach, presented by Akita et al., "A Computer Method of Understanding Ocular Fundus Images," Pattern Recognition, Vol. 15, No. 6, 1982, pp. 431-443, uses vessel tracking to locate the ONH. However, this algorithm relies on successful application of a vessel segmentation algorithm. Because the algorithm relies on the vasculature, abnormal vasculature can lead to false detections. Mendels et al., "Identification of the Optic Disk Boundary in Retinal Images Using Active Contours," Proc. Irish Machine Vision Image Processing Conf., September 1999, pp. 103-115, uses active contours for ONH detection. The drawback of this approach is its dependency on the initial contour generated by morphological filtering, which is not robust to changes in image quality.Because the algorithm is based on active contours, it is not time-efficient. Yet another approach, presented in Sekar et al., "Automatic Localization of the Optic Disc and Fovea in Retinal Fundus Images," 2008 16th European Signal Processing Conference, Lausanne, 2008, pp. 1-5, uses morphological operations followed by a Hough transform to locate the ONH. In this algorithm, the Hough transform is used to fit a circle with a given limiting radius. The algorithm finds image regions with large variations in gray-level intensity and uses these regions as a reference for locating the ONH. While this may not be a problem in normal images (e.g., images of a healthy eye), in images with a pathology, the optic disc may not be the only structure with large variations in intensity, which may lead to errors. Zhu et al., "Detection of the Optic Nerve Head in Fundus Images of the Retina Using the Hough Transform for Circles," J Digit Imaging, 2009, Vol. 23 (No. 3), pp. 332-41, uses edge information and the Hough transform to locate the ONH. However, because this algorithm relies on extracting good edge information (image quality) and fitting it to a circle of a certain radius, it is likely not robust to varying image quality and ONH-related pathologies.Another approach, described by Rangayyan et al. in "Detection of the Optic Nerve Head in Fundus Images of the Retina with Gabor Filters and Phase Portrait Analysis," Journal of Digital Imaging, Vol. 23 (No. 4), pp. 438-53, uses Gabor filters for vessel detection and attempts to localize the ONH by applying phase portrait modeling. This algorithm is specifically based on the characteristics of the ONH as a collection point of retinal vessels. The performance of this algorithm is affected when the vessels or collection center points are obscured by image quality or pathology. Several approaches using deep learning algorithms to localize the optic nerve using convolutional neural networks (CNNs) have also been reported. A description of CNNs is provided below.Examples of using CNNs to locate the optic nerve include D. Niu et al., "Automatic Localization of Optic Disc Based on Deep Learning in Fundus Images," 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), Singapore, 2017, pp. 208-212; and Gonzalez-Hernandez D. et al., "Segmentation of the Optic Nerve Head Based on Deep Learning to Determine its Hemoglobin Content in Normal and Glaucomatous Subjects," J Clin Exp Opthamol, Vol. 9:760, and H.S. Alghamdi et al., "Automatic Optic Disc Abnormality Detection in Fundus Images: A Deep Learning Approach," Proceedings of the Ophthalmic Medical Image Analysis International Workshop (OMIA 2016, held simultaneously with MICCAI 2016 in Athens, Greece), Iowa Research Online, pp. 17-24, October 2016. All of the above published works are incorporated herein by reference in their entirety. However, it should be noted that in all of these published works, the optic disc is detected in color fundus images.That is, all of these approaches use features or use color fundus images (e.g., of the ONH) to train (deep learning) algorithms.

[0058] The ONH localization technique of the present invention differs from the above examples in that it is suitable for localizing the ONH in image types (eg, imaging modalities) other than color images, such as in FA and ICGA images.

[0059] In FA and ICGA angiography, a series of time-lapse images are captured after a photoreactive dye (e.g., a fluorescent dye in FA and indocyanine green dye in ICGA) is injected into the subject's bloodstream. High-contrast images are captured using specific light frequencies selected to excite the dye. As the dye flows through the eye, it causes various parts of the eye to glow brightly (e.g., fluoresce), making it possible to identify the progression of the dye and, therefore, blood flow through the eye. Typically, the captured series of images is characterized by a dark color when there is no (or very little) dye in the eye, increasing in brightness as more dye enters and flows through the eye, and then darkening again as the dye leaves the eye, resulting in brighter areas containing more dye. As a result, the appearance of the ONH changes over time as the dye progresses through the blood vessels during various stages of the angiography procedure. This change generally makes it very difficult to identify the location of the ONH in fluorescein angiography (FA) and indocyanine green angiography (ICGA) images. For illustrative purposes, FIG. 5 provides two vascular images, A1 and A2, captured at different stages of an angiographic examination (or at different times during an angiographic examination). As shown, the ONH in image A1 appears very different from the ONH in image A2, which complicates the creation of a filter suitable for all stages of an angiographic examination. Therefore, most of the available literature on ONH localization generally limits itself to color fundus images, avoiding the challenge of locating the ONH in FA or ICGA images. However, herein, a U-net-based deep learning architecture applied to IR preview images is presented to reliably locate the ONH in all stages of FA and ICGA images or any other type of image / scan.

[0060] Generally, an IR preview image of the fundus / retina is collected before scanning (imaging) for the examination. As explained above, the IR preview image can be used to adjust focus, and as explained below, the IR preview image can be used to ensure accurate alignment of the system with the patient and ensure the correct area of ​​the eye is imaged. Therefore, the IR preview image is collected / recorded immediately before capturing the FA and / or ICGA images. It is stated herein that the location of the ONH, which is immediately found in the IR preview image before capturing the FA or ICGA image (or any other type of image), remains the same (unchanged) in the captured FA or ICGA image. That is, the location of the ONH in the IR preview image is used to locate the ONH in the FA and ICGA images. The present invention indirectly locates the ONH in the FA and / or ICGA images by locating the ONH in the IR preview image of the retina taken immediately before capturing the FA and / or ICGA image (using an algorithm trained using the IR image) and transferring the location of the ONH from the IR preview image to the FA and / or ICGA image.

[0061] Previous conventional techniques were limited to locating the ONH in color fundus images, which is relatively easy (given a substantial number of training images) compared to locating the ONH in dynamic imaging modalities such as FA and ICGA images. However, collecting a substantial training image set (e.g., more than 1,000 images) for each of the various stages of FA and / or ICGA imaging is extremely challenging, especially since it involves the injection of a photoreactive dye. The present invention circumvents some of these difficulties.

[0062] The present invention manages to locate the ONH (or any other predetermined structure or structure) without requiring a large number of difficult-to-acquire angiography images (e.g., FA and / or ICGA images) to train a machine learning model. In one embodiment of the present invention, transfer training can be used based on an existing machine learning model trained to locate the ONH in color images by extending the operation of the machine learning model to include IR images. Transfer training is a machine learning method in which a machine learning model developed for a first task is reused as a starting point to train another machine learning model for a second task. Given the extensive computational and time resources required to develop neural network models for these problems, a pre-trained model is used as the starting point. For example, the machine learning model is first trained using color images (or individual color channels extracted from the color images), followed by transfer training on IR images. The use of transfer training is another novel feature of the present invention. For example, the algorithm can be trained in two steps: first, using color images and / or color channels extracted from color images (e.g., about 2000 training images), and second, using IR images (e.g., about 1000 images). That is, the algorithm can be first trained on color images and / or individual color channel images (e.g., monochrome images), and then finished training on IR images. For this reason, this two-step approach can be used when a large number of color images are readily available (e.g., 2000 images is sufficient to train all parameters of a neural network), and the number of IR images available for training is small, as is often the case. For example, red channel images extracted from color images can be used to first train the network in the first step, and IR preview images can be used to train a machine model on the same network.In this two-step approach, for example, after training a neural network using color images, a given number of initial layers are fixed (e.g., the weights / parameters are fixed so that they do not change in subsequent training), and the neural network is then trained again using IR preview images with only the remaining, unfixed layers (or with only the last, e.g., two, layers). The weight / parameter training in this second step is more limited than the weight / parameter training in the first step. However, because the availability of IR preview images is generally lower than that of color images, there may not be enough IR preview images to train all the weights / parameters in the network. For this reason, it is beneficial to train a machine model using commonly available modality images (e.g., color images), fix some layers, and then transitionally train only a few layers of the neural network using available IR preview images.

[0063] Alternatively, if a sufficient number of IR (preview) images (e.g., more than 5,000 images) are available for training, the machine model can be trained with IR images alone or with a combination of IR and color (or color channel) images in one go (e.g., one training step). However, because color fundus images are generally more widely available than IR images, transitional training can still be used to improve performance. Experiments have shown a measurable improvement in accuracy when using this stepwise process compared to a one-step process.

[0064] 6 provides a segmentation and localization framework for locating ONH in FA / ICGA images during operation (in a deployed system) in accordance with the present invention. The system may begin in block B1 by capturing an IR (e.g., preview) image, followed by capturing an FA and / or ICGA image in block B3. Alternatively, in block B3, another type of image different from the IR image (e.g., FAF, color, etc.) may be taken instead of (or in addition to) the FA and / or ICGA image, and the framework is similarly effective for locating ONH in other types of images. The captured IR image from block B1 may be sent to an optional pre-processing block B5, which may apply an anti-aliasing filter and downsampling, before being sent (as an input test image) to a previously trained U-Net, a deep learning machine model (block B7) for processing.

[0065] An example of a suitable U-Net architecture is provided below. The U-Net in block B7 is trained to receive test (IR) images and identify ONHs within the test images. Preferably, the U-Net is trained in a two-step process, as described above, using a training set of IR fundus images (e.g., IR preview images) and a set of color fundus / channel-extracted color fundus images. The U-Net can first be trained using channel-channel extracted color images using color images and / or optic disc segments (e.g., image segments with ONHs demarcated by manual marking or using an automated algorithm to identify ONHs within color images). This first training step assigns initial weights / parameters to all layers of the U-Net. The resulting weights are then transitionally trained using IR preview images with optic disc segments generated from manual marking or an automated algorithm. That is, the U-Net can be trained using some of the initial layers fixed with the initial weights / parameters and the remaining, unfixed layers (e.g., the last two layers) trained using IR preview images. Thus, the U-Net receives an inspection (IR) image and outputs a number of possible ONH segments identified in the inspection image.

[0066] Block B9 receives all possible ONH segmentations identified by the U-Net and selects candidate ONH segments based on size criteria, which may include filtering out all received ONH segments that are larger than a first threshold and smaller than a second threshold (e.g., in area or diameter). An acceptable size range may be determined according to the criteria range.

[0067] The selected candidate ONH segments are then sent to block B11, where one of the candidate ONH segments is selected as the true ONH segment. This selection may be based on the shape of the candidate ONH segments. For example, the candidate ONH segment having the roundest shape may be selected as the true ONH segment. The true ONH segment may be determined by measuring the circularity of each candidate ONH region, such as by determining the difference between the minor and major axes (radii) of each candidate ONH segment. The ONH segment with the smallest difference between the minor and major axes may be selected as the true ONH segment. If multiple ONH segments are found to have the same circularity, the largest of these ONH segments may be designated as the true ONH region.

[0068] With the true ONH segment thus identified, the next step is to determine the location of the true ONH segment in the IR (preview) image. Locating the candidate ONH segment in the IR preview image can be done by finding the centroid of the candidate ONH segment (Block B13).

[0069] Finally, in block B13, the determined ONH segmentation and localization may be applied to the captured IR preview image of block B1 and / or the captured FA and / or ICGA images of block B3.

[0070] FIG. 7A illustrates the location of the ONH region ONH-1 in the IR preview image IR-1 identified using the system of FIG. 6 . FIG. 7B illustrates an FA image FA-1 (e.g., an angiographic image) captured immediately after the capture of the IR preview image IR-1 of FIG. 6 , e.g., image FA-1 is captured within a time period subsequent to the capture of image IR-1 that is determined to be short enough to prevent substantial movement of the ONH by the eye. FIG. 7B shows the located ONH region ONH-1 transferred from image IR-1 onto the FA image FA-1. In this manner, the ONH at any stage of an angiographic examination (FA or ICGA) can be quickly and efficiently identified. Optionally, if supported by the imaging system, additional IR preview images can be captured continuously throughout multiple (or all) stages of the angiographic examination so that the location of the ONH can be continuously updated, as needed, from successively captured IR images.

[0071] The captured FA or ICGA image sequence can then be displayed with or without the identified ONH region highlighted. Regardless, there are difficulties associated with displaying a series of captured FA or ICGA images.

[0072] Automatic adjustment of angiographic image sequences. Angiographic imaging, both fluorescein angiography (FA) and indocyanine green angiography (ICGA), requires capturing a wide range of light intensities, from dark to bright, as injected dye passes through the eye. Setting the camera to capture this intensity variation in an image sequence can be difficult, resulting in some images being saturated (e.g., too much signal gain resulting in an overly bright or washed-out image) or overly dim. Therefore, system operators typically manually adjust the camera's capture settings as an image sequence for an FA or ICGA exam is being captured. For example, the system operator may monitor the currently captured image and adjust the camera to increase or decrease the gain for subsequent images in the image sequence. That is, most fundus cameras require the operator to adjust illumination brightness, sensor gain, or aperture size during angiography to accommodate the variability in signal levels and attempt to capture images with good brightness for display (i.e., images that are not saturated or overly dim). This has several distinct disadvantages. The first drawback is that the need to adjust camera settings (e.g., to control brightness) distracts the operator. The second drawback is that operator error or omission can lead to underexposure or overexposure of images. The third drawback is that the resulting angiographic image sequences may be differently brightened from one another, meaning that true changes in image intensity over time (which may indicate blood flow and be relevant to disease assessment) are partially or completely obscured. Some ophthalmic imaging systems may provide automatic brightness control (or sensitivity control or automatic gain control) to relieve the system operator of the burden of adjusting the camera. While this type of automatic control can help overcome the first and second drawbacks, it generally does not address the third drawback. In practice, because automatic brightness control is typically applied on an individual image basis, all images in a captured sequence may be individually adjusted to provide similarly bright images. This automatic brightness control exacerbates the third drawback because it further hides naturally occurring relative changes in brightness between images in the sequence.

[0073] The present invention provides automatic adjustment of image brightness while maintaining true proportional changes in signal intensity (between different images) during an angiographic examination. Additionally, the present invention can also provide trends in light intensity levels over time, which may have clinical relevance for certain ocular diseases.

[0074] First, the ophthalmic imaging system of the present invention preferably uses a high dynamic range (optical) sensor to capture an image sequence in an angiographic examination (e.g., FA or ICGA). A novel feature of the present invention is the use of a sensor with a sufficiently wide dynamic range to avoid underexposure and saturation during the capture of all images in an image sequence during an angiographic examination without system / hardware adjustments (e.g., adjustments to illumination brightness, sensor gain, and / or aperture size). The system stores the captured raw images (e.g., for future processing) but may optionally adjust each image for optical display without changing the image's raw data. The system may examine regions of the image to determine an "image brightening factor" that optimally adjusts the image for optimal display utilization. The system may further examine the "image brightening factor" of all images acquired as part of an angiographic series to determine a "sequence brightening factor" that adjusts all images in the series for optimal display utilization, within the constraint that the relative brightness between images within the series is maintained. In this case, the same "sequence brightening factor" can be applied to each image for display, rather than an individual "image brightening factor" for each image. Using the "sequence brightening factor" or raw data, the system can further determine and record trends in image intensity levels versus time since dye injection.

[0075] The system is optimized for angiography. It provides a dynamic range sufficient to accommodate the fluorescence range required for angiography. As a result, there is minimal risk of image overexposure, eliminating the need to constantly adjust illumination intensity or detector sensitivity during angiography. Therefore, the operator is never distracted by the need to monitor or control the brightness or sensitivity of the image capture sequence. Because no adjustments are made to the optical sensor, the captured image sequence faithfully represents the true relative changes in image intensity acquired throughout angiography, which is important for many diseases / pathologies when reviewed by a physician. For example, venous or arterial blockages can slow the passage of dye through the retina, potentially reducing the rate of image intensity decay over time. Relative differences in fluorescence between images are also important for diagnosing inflammatory conditions. In contrast, conventional ophthalmic imaging systems obscure the true relative changes in image intensity due to manual or automatic adjustments of brightness or sensitivity settings during data acquisition, and there is a high likelihood that a physician will not be present to review the images.

[0076] The intensity of angiographic images varies greatly depending on the dye transit stage, dye dosage, retinal pigmentation, ocular media condition, and disease state. Most fundus imaging systems have limited dynamic range and may therefore require adjustment of illumination brightness or detector sensitivity settings during angiography to avoid underexposure or overexposure. The natural variation of the fluorescent signal during angiography (usually strongest in the early stages and gradually decreasing toward the later stages) contains information about the dynamics of blood transit through the retina, which can be useful for clinicians in assessing ocular health.

[0077] FIG. 8 provides a framework for processing individual images within a series of images captured as part of an angiographic examination. During acquisition, if enabled by the system operator, such as by checking the “Auto Adjust” checkbox in the acquisition window, the captured images are optimized (in terms of gray-level range) for individual display (e.g., rather than for display as part of a group or sequence of images representing the elapsed time of the angiographic examination). This optimization prevents individually displayed images from being too dark or oversaturated, which could prevent a human expert from properly assessing image quality. First, an image 20 is captured. Capturing the image 20 may include capturing multiple scanlines (e.g., stripe data) 22 and synthesizing (combining) the stripe data to create an (unadjusted) raw image 20 24. For each image 20, the captured raw pixel intensities are stored. A white level optimized for displaying the image is determined for the raw image 20. The white level may be determined by examining some 26 or all of the acquired image data 20 and calculating the 99.9th percentile (or other appropriately selected percentile) pixel intensity. The determined optimal white level is stored in image metadata 28 associated with the raw image 20. The metadata 28 also includes a black level for displaying the image, which may be preset at a level that masks the electronic noise of the (light) sensor and may be provided by a configuration file. The image metadata 28 effectively provides auto-brightening information for each image 20 (e.g., according to the white level of the image 20). The "image brightening factor" of each image 20 may be based on the individually determined white level and / or black level of the image 20.

[0078] When an individual image 20 is selected for display, it undergoes image adjustment 30 (e.g., automatic brightness) according to its associated metadata 28 (e.g., according to stored white and / or black levels). The adjusted image may then be output to a review screen 32 on an electronic display 34. Optionally, the adjusted image may also be sent to review software 36 for further processing.

[0079] After all imaging for a given angiography study is complete (e.g., all time-lapse images have been captured), the time-lapse images can be displayed as a group. For example, the group images can be displayed on a review screen 32. While each image's corresponding metadata provides information for individual display adjustments, by displaying each image according to individually optimized display settings, many of the images will have a similar appearance, such that visual changes due to migration of the injected dye over time are not easily discernible from one image to another. For example, Figure 9 shows an FA time-lapse image sequence 40 in which each individual image has been adjusted (e.g., brightened) according to its respective individual white level. Note that the later images should appear similarly bright to the earlier images, despite the true fluorescent signal decreasing over time; therefore, the later images should appear darker than the earlier images.

[0080] The visual changes from one image to another resulting from dye migration are available in the images' respective raw data but not in their respective display-optimized formats. Displaying each image in a time-lapse sequence according to its raw data format preserves the dye migration information, but because no display optimization is applied, it can be difficult to discern details in each image. For example, Figure 10 shows a raw FA image sequence 42 (corresponding to the image sequence of Figure 9) in which no brightness adjustments have been applied to any images (e.g., displaying the raw signal levels of each image). As a result, many of the images are dim and difficult to evaluate.

[0081] For group display, the objective of the present invention is to adjust all images in a time-lapse sequence as a group to display as much detail as possible while preserving the relative differences (e.g., visual changes) between the individual images in terms of overall brightness due to dye transitions. Rather than automatically brightening all images in the sequence individually according to their respective white levels (as illustrated in FIG. 9), one embodiment of the present invention examines the white levels of all images in the sequence (e.g., the white levels stored in each image's corresponding metadata) to find the maximum white level among those white levels, which is referred to herein as "w max The global scaling factor (i.e., the scaling factor across all images in a sequence) is used to set the maximum number of available image intensity levels to w max (e.g., 255 / w max ) is defined as the sum of the image brightnesses of all images in the sequence. This same global scaling factor is then applied to all images in the sequence so that the relative differences in brightness between images are preserved. Figure 11 shows how all images are scaled using the same global scaling factor (e.g., 255 / w max 11 shows a globally adjusted FA image sequence 44 (corresponding to the image sequence of FIG. 10) that has been brightened with FA. As shown, the images retain the relative intensity variations indicative of true fluorescent signals, but unlike the raw FA image sequence 42 of FIG. 10, the images in FIG. 11 have been brightened to optimize visible detail.

[0082] In an alternative embodiment, rather than applying the same global scaling factor to all images in the sequence, each image may be adjusted individually based on its respective stored white level, but the brightness setting of each image may be adjusted based on the w of its group. maxIn this way, the relative brightness of the images in a sequence (e.g., a group) is compressed so that the images are generally brighter and easier to interpret, but still retain some similarity in the changes in the fluorescence signal during angiography. This compression-brightening technique can be implemented by applying a customized brightness scaling factor to each image in the sequence, which can be calculated as follows:

[0083]

number

[0084] The compression parameter ∝ may be set to a fixed value during an individual angiographic examination (e.g., the compression parameter ∝ preferably does not change during the acquisition of an image sequence), and ideally may be set to a value selected by a clinician (e.g., a system operator). That is, input signals from the system operator may modify the amount of relative intensity compression between images in a sequence by adjusting the value of the compression parameter ∝. For example, the system operator may adjust the compression parameter ∝ using an intensity slide bar, using increment / decrement buttons, and / or by directly entering the value of ∝.

[0085] 12A-12F provide some examples of how this compression-brightening technique affects the brightness of individual images in a time-lapse sequence for different values ​​of the compression parameter ∝. In each of FIGS. 12A-12F, the resulting brightened image sequence is shown on the left, and a plot of brightness versus angiographic time is shown on the right. In FIG. 12A, the compression parameter ∝ is set to zero, which means that the same overall scaling factor, e.g., "w" in this case, is used, as described above with reference to FIG. 11. max " is equivalent to brightening all images. For comparison purposes, the captured raw data is also stored, so multiple plots are shown. Plot 50 illustrates the relative brightness between raw images over time (e.g., from zero to 10 minutes) within a series of captured images. Plot 52 illustrates the relative brightness between images when each image is automatically brightness adjusted according to each image's stored white level, which generally results in images with similar brightness. Plot 54 illustrates the relative brightness between images captured at the same w max corresponds to the relative brightness between the images when all images are brightness adjusted according to the value. max The relative difference in plot 54 follows the relative difference in raw image plot 50 .

[0086] In Figure 12B, the compression parameter ∝ is set to 1, which is equivalent to automatically brightening each image individually according to its stored white level, as described above with reference to Figure 9. Again, plot 50 illustrates the relative brightness between the raw images, and plot 54 illustrates the brightness of the raw images with the same brightness factor w max 1 illustrates the relative brightness between images using , and plot 52 shows the relative brightness between images over time, with each image brightened based on its own individual white level. As shown by plot 52, by automatically brightening each image according to (only) its respective white level, the differences in relative brightness between images over time are hidden.

[0087] 12C-12F illustrate how modifying the compression parameter ∝ affects the brightness of an image sequence for different values ​​of ∝. In FIGS. 12C-12F, the compression parameter ∝ increases in increments of 0.2, from ∝=0.2 in FIG. 12C to ∝=0.8 in FIG. 12F. Each of FIGS. 12C-12F provides a plot of the resulting relative brightness between images in a time-lapse sequence (56C-56F, respectively). For comparison purposes, each of FIGS. 12C-12F shows plot 50 (raw image), plot 52 (maximum automatically brightened image based on stored white level), and plot 54 (w max As plots 56C-56F show, increasing ∝ has the effect of increasing the brightness of the displayed image sequence compared to raw image plot 50, while maintaining a similar relative difference between the images as raw image plot 50.

[0088] Graphical User Interface As described above, the present system provides enhanced focusing capabilities. In addition to allowing a user to select from among multiple predetermined points / regions for focusing or allowing the user to freely select any random point for focusing, the present invention provides additional features for capturing multiple imaging types (imaging modalities) and / or for tailoring a particular imaging type to a particular patient. By way of illustration, FIG. 13 provides an example of a graphical user interface (GUI) suitable for an ophthalmic imaging device, highlighting some of these additional features. The GUI provides an acquisition (interface) screen 21 that can be used to acquire (e.g., capture) images / scans of a patient. Optionally, patient data 23 identifying the current patient may be displayed at the top of the acquisition screen 21. The acquisition screen 21 may provide multiple display areas (e.g., boxed areas on the display screen for viewing information), as well as display elements and icons to guide the equipment operator through setting acquisition parameters, capturing and reviewing images, and analyzing data. For example, a preview window 15 may be provided within the display area, with multiple focus assist positions f1-f7 superimposed on a preview (fundus) image (e.g., an IR preview image) 11. As explained above, the system operator may select any point (e.g., target focal area) on the preview image 11 for focusing using a user input device (e.g., electronic pointer device, touch screen, etc.).

[0089] If desired, additional focus options may be provided, such as a (step adjustment) focus button 33 or focus slide bar 35. In an exemplary embodiment, the target focus area of ​​the preview window 15 and the focus button 33 or focus slide bar 35 may be mutually exclusive, such that use of one overrides the other. For example, use of the focus button 33 or focus slide bar 35 may override any previously manually selected target focus area on the preview window 15 and apply a conventional global focus adjustment to the entire fundus image 11 according to the focus button 33 and / or focus slide bar 35. Similarly, assigning (e.g., selecting) a target focus area on the review screen 15 may override any focus setting of the focus button 33 or focus slide bar 35. Alternatively, the target focus area of ​​the preview window 15 and the focus button 33 or slide bar 35 may function in conjunction. For example, if a user selects a target focus area in the fundus image 11 to focus on, the system may adjust the focus at the selected focus area accordingly, as described above. However, if the user desires to further adjust the focus setting for a selected target focal region other than the focus region automatically provided by the system, the user may further adjust the focus at the selected target focal region using the focus button 33 and / or slide bar 35. Thus, the system may provide manual focus for any selected target focal region within the fundus image 11 (e.g., within the preview screen 15).

[0090] The acquisition screen 21 may display one or more pupil streams 39 of live images (e.g., infrared images) of the eye's pupil from different viewing angles, which may be used to facilitate alignment between the ophthalmic imaging system and the patient. An overlay guide, such as a semi-transparent band 47 or crosshairs 49, may be superimposed on the live stream to identify a target position or range of target positions where the patient's pupil should be positioned for good image acquisition. The overlay guide informs the system operator as to the current offset between the target pupil position for imaging and the current position of the patient's pupil center so that corrective action can be taken.

[0091] Once the operator is satisfied with the capture settings, an image may be captured using a capture activation input, such as capture button 37 . As images are captured, they may be displayed in thumbnail format in the capture bin section 41 of the acquisition screen 21. A display filter 43, which may provide a drop-down list of filter options (e.g., laterality, scan (or image) type, etc.), may be used to narrow (select or limit) the thumbnails displayed in the capture bin 41. If an imaging mode is selected that requires the acquisition of multiple images (e.g., an ultra-wide field option (UWF), which may combine two captured images, or auto-compositing, which may combine a predetermined number of captured images offset from one another), the capture bin 41 may be pre-populated with a placeholder icon 45 (e.g., a dashed circle) that indicates a placeholder for the image to be captured. As each required image is captured, the placeholder icon 45 may be replaced with a thumbnail of the captured image to provide the user with an indication of the execution of the imaging mode.

[0092] However, it will be appreciated that the system operator may be provided with multiple options to set before capturing an image (e.g., activating a scanning operation). For example, additional display elements and icons may be provided to enable the system operator (user) to select the type of image (scan) to be acquired to ensure proper focus and proper alignment of the patient with respect to the ophthalmic imaging system. For example, the laterality of the eye being examined may be displayed / selected, for example, by a laterality icon 25, highlighting whether the right or left eye (e.g., right eye (oculus dexter) OD) or left eye (oculus sinister) OS) is being examined / imaged.

[0093] Various user-selectable scanning (e.g., imaging) options may be displayed on the acquisition screen 21. The scanning options may include an FOV button 27 for the user to select the imaging FOV of one or more images to be acquired. The FOV option buttons may include a wide field of view (WF) option for standard single-image operation, an ultra-wide field of view (UWF) option that captures and combines two images, an “auto-combine” option that provides a pre-set combination sequence that collects and combines a predetermined number of images (e.g., four), and / or a user-definable “combine” option that allows the user to present a user-specified combination sequence. Additionally, check boxes may be provided for the user to indicate whether they wish to perform stereo imaging and / or use an external fixation target. Selecting an external fixation target may disable the internal fixation target within the system during imaging.

[0094] Fundus imaging systems may support multiple scan types (e.g., imaging modalities). An example of a scan type may include color imaging, which provides a full-color view of the fundus and requires bright light, which may be uncomfortable for the patient. Another type of image is IR imaging, which is invisible to the patient and therefore more comfortable. IR images are generally monochrome and may be more useful for identifying finer details than color images. Some types of tissue have natural light-sensing molecules that can be made to fluoresce at specific wavelengths, e.g., 500-800 nm. These tissues can be used to identify potential areas / structures of pathology that may not be easily defined with color or IR frequencies. Imaging the eye using specific wavelengths selected to cause targeted tissues to fluoresce is called fundus autofluorescence imaging (FAF). FAF imaging at different frequencies may automatically fluoresce different types of tissue, thereby providing different diagnostic tools for different types of pathology. Other imaging modalities include FA and ICGA, which utilize injected dyes to track blood flow in a series of images, as described above.

[0095] In this example, acquisition screen 21 provides a scan type section 29 for selecting from among multiple imaging modalities. Examples of selectable scan types may include color (e.g., true color imaging using visible light, such as red, green, and blue light components / channels), IR (imaging using non-visible infrared light, for example, using an IR laser), FAF-green (fundus autofluorescence with green excitation), and FAF-blue (fundus autofluorescence with blue excitation), FAF-NIA (fundus autofluorescence with near-infrared light), FA (e.g., fluorescein angiography by injecting a fluorescent dye into the patient's bloodstream), FA-NIA (fluorescein angiography with near-infrared light), and ICGA (e.g., indocyanine green angiography by injecting indocyanine green dye into the patient's bloodstream).

[0096] Acquisition screen 21 may further provide input for various image capture settings, such as for dilated eyes (mydriatic mode), non-dilated eyes (non-mydriatic mode), and / or eyes that may be light-sensitive (e.g., patients suffering from photophobia). A user may select between mydriatic (Myd) and non-mydriatic (Non-Myd) imaging modes by selecting button 31 for the respective mode. Optionally, the system may automatically select (e.g., auto-select) which mode is appropriate based on the condition of the eye.

[0097] The operator may utilize multiple photophobia-related settings before initiating an image capture sequence. Optionally, the system may identify / flag a patient as having previously been diagnosed as suffering from photophobia or as a candidate for light sensitivity. In either case, the system may alert the operator to that fact and suggest that the imaging sequence be adjusted accordingly. For example, if the patient is a candidate for light sensitivity, the warning icon / button 61 may illuminate, change color, and / or flash to attract the operator's attention. The operator may then select a default photopic setting by selecting a button and / or manually selecting from a list of light intensity modification options 63, each of which may adjust the applied light intensity during image acquisition. Adjusting (tuning) image acquisition settings according to the patient's medical condition (e.g., photophobia) is an example of a patient-tailored diagnosis.

[0098] Tailored diagnosis for patients Ophthalmic imaging systems may use flashes of light, such as in a scanning pattern, to acquire fundus images. For example, the CLARUS 500™ from Carl Zeiss Meditec Inc. can capture high-resolution, wide-field images using flashes of light. Some types of flashes of light, such as those used for FAF-blue mode imaging, can be bright and can have short-term visual effects on patients, including blurred vision, amblyopia, and / or reduced color vision. For patients with photophobia, e.g., eye discomfort or pain due to light exposure, the light source of an ophthalmic imaging system can be quite disruptive and induce short-term effects, including pain, migraines, nausea, tearing, and other symptoms.

[0099] Typically, when imaging a patient with known light sensitivity, the system operator may take steps to reduce the patient's exposure to light. For example, the system operator may perform a diagnosis (e.g., image the patient's eye) on a patient with light sensitivity by preventing dilation of the patient's eye and performing the examination (e.g., imaging) in a darkroom setting. Where possible, the system operator may attempt to reduce the system's light intensity and use a smaller FOV of imaging, but this may result in reduced image quality.

[0100] Presented herein is an ophthalmic imaging system having a diagnostic mode (e.g., an imaging mode) for patients with light sensitivity and / or a system capable of identifying patients who may be candidates for being light sensitive.

[0101] The system may provide a “tailored” (e.g., photophobic or light-sensitive) imaging mode for light-sensitive patients that reduces the light output of the ophthalmic imaging system, while still providing (e.g., minimally) feasible image quality for diagnosis. FIG. 14 provides an exemplary process for adjusting (e.g., adjusting) the amount of light reduction depending on the patient's relative light sensitivity. This process may include providing an optional automated mode for patients identified as light-sensitive (e.g., in the patient's medical record) and a manual select / deselect option to the system operator within a graphical user interface (GUI), such as within the image acquisition screen (e.g., buttons 61 and / or 63 in FIG. 13 ). Thus, the process may begin by automatically or manually setting the light-sensitive mode (step 60), such as by reviewing the patient's record. Optionally, an assessment of the patient's light sensitivity level may be adjusted / determined by providing additional information / parameters (step 62), such as iris size, iris color, and / or light / dark retinal detection.

[0102] Scan-based ophthalmic imaging systems may use scan tables that specify the light frequency, intensity, and / or duration of scan lines applied to a given location on the fundus. Typically, one scan table is constructed / defined for each imaging type. However, embodiments of the present photophobia mode may include generating one or more scan tables using a subset of current / standard light settings (e.g., default light settings) for a given imaging type. The scan tables may provide both reduced light intensity and reduced duration (e.g., shorter flashes). The scan tables may also include specific scan settings (e.g., additional intensity and duration adjustments) based on the patient's pupil size and / or iris color. The minimum achievable image quality based on the minimum exposure for each imaging mode / type (color, FAF-B, FAF-G, IGC, FA) may be predetermined based on clinical testing. For example, a patient with light sensitivity may first be examined (imaged) using the light-sensitive mode by having the system automatically select one of the alternative scan tables based on the patient's light sensitivity (step 64). Alternatively, the user may select a predetermined alternate scan table. The system then scans (images) the patient using the selected scan table (step 66), after which the acquired image is evaluated (step 68). This evaluation may be performed automatically by the system, such as by determining a quality measurement of the acquired image, or may be performed by a system operator. If the resulting image does not have sufficient quality, the image may be additionally acquired at a step function increase in intensity up to the normal mode intensity, if necessary (step 70). This manual selection of the step increase may be performed, for example, using input 63 of FIG. 13.

[0103] The system may include optional elements / parameters (e.g., as shown in optional step 62), such as automatic detection of pupil size, iris color, and automatic light-dark retina detection (e.g., using an IR preview image of the retina). Light-colored retinas tend to be more photophobic than dark-colored retinas. A direct table correlation may be created between different combinations of photophobia indices and corresponding light / duration settings (e.g., scan table selection) for a given image acquisition operation. Alternatively, a mathematical equation may be created for optimal (minimum) power settings based on a combination of parameters that affect image quality. Additionally, the system may further integrate DNA data to automatically adjust diagnostic device settings based on a marker or set of markers that may indicate possible light sensitivity. This can be further extended to mathematical evaluation of a marker set for optimal settings.

[0104] 15 illustrates how a scan table (e.g., intensity levels used for image acquisition) may be selected and / or modified based on pupil size and / or iris color. Optionally, the system's default intensity and duration settings may be set to a minimum setting associated with light-colored eyes, and the intensity (and / or duration) may be increased for non-photophobic eyes. Further optionally, the system's default intensity and duration settings may be set to a minimum setting associated with light-colored eyes, and if the captured image quality is below a minimum value, the intensity and / or duration may be automatically increased in predetermined step increments to capture additional images in the current image capture sequence until an image of minimum image quality is obtained.

[0105] The patient-tailored diagnostics of the present invention may augment existing ophthalmic imaging systems with multiple operating modes based on a patient's known condition or disease, such as adding a dry eye mode for OCT imaging. The approach may also be applied to patients with mental health conditions where discomfort, such as stress or anxiety, may be triggered by flashing light or luminance intensity. For example, the patient-tailored diagnostics of the present invention may be applied to patients with post-traumatic stress disorder (PTSD) to minimize the patient's discomfort and adverse symptoms.

[0106] Returning to FIG. 13 , as indicated by scan type section 29, the system may further provide functionality for acquiring multiple image types (different imaging modalities) with a single button. As described more fully below, this may be achieved by providing a scan table that supports multiple imaging modalities within a single image capture sequence. For example, a multispectral option / button 63 may be provided to capture multiple images across a spectrum of wavelengths. FIG. 16 illustrates an exemplary series of multiple images across multiple wavelengths that may be captured in response to a single capture instruction input with multispectral option 63 selected. Another example may be an oximetry option / button 65 that determines a measurement of oxygen in the blood by capturing multiple images at different selected wavelengths, as described more fully below. Also provided is a color+IR option / button 67 that captures both color and IR images in a single image capture sequence in response to a single image capture button. Another option may be an FA+ICGA option / button 69 to take both FA and ICGA images or a sequence of images in response to a single image capture instruction. A preferred method for implementing the FA+ICGA option is provided herein.

[0107] Sequential acquisition of FA and ICGA images in response to a single "capture" command Both fluorescein angiography (FA) and indocyanine green angiography (ICGA) can be performed as part of a single exam. ICG (indocyanine green) dye can be injected immediately before or after the fluorescent dye, or a mixed bolus can be used. FA images can be acquired using a blue / green illumination source, and ICGA images can be acquired using a near-infrared illumination source. Incorporating both modalities into a single exam reduces patient chair time. However, performing sufficient acquisitions of both FA and ICGA images throughout the different phases of angiography can be taxing for both the patient (especially those with more severe photophobia) and the operator. This burden can be alleviated by using an imaging device capable of acquiring both FA and ICGA images in a single exposure sequence.

[0108] Simultaneous acquisition of both FA and ICGA images presents challenges in system design due to the different light source and optical filter requirements for each modality. Due to the risk of eye movement, exposures must be completed in a short time (approximately 100 ms) while still achieving sufficient image quality for physicians to assess the ocular condition.

[0109] With any device that delivers visible light to the retina, patient comfort is an important consideration in ensuring compliance; patients with photophobia may move or blink excessively during imaging or even refuse to complete the test. As explained in JM Stringham et al., "Action spectrum for photophobia," Journal of the Optical Society of America A (JOSA A), 2003, Vol. 20 (No. 10), pp. 1852-1858, photophobia increases with shorter wavelengths. The short wavelengths of light typically used for FA (470-510 nm) are much more uncomfortable than the near-infrared light used in ICGA. Furthermore, it has been shown that perceived flash duration increases as a function of flash duration (Osaka N., "Perceived brightness as a function of flash duration in the peripheral visual field," Perception & Psychophysics, 1977, Vol. 22 (No. 1), pp. 63-69). Therefore, to promote patient comfort, it is desirable to minimize the duration of potentially painful exposures (as well as minimize flash energy).

[0110] FIG. 17 compares the present invention with two previous methods for providing FA+ICGA functionality in response to a single capture command. The first previous approach 82 uses both blue and infrared laser sources to simultaneously transmit light into the eye and simultaneously collect the emitted fluorescence using two separate photodetectors. The second previous approach 84 interleaves (or alternates) illumination and detection. That is, the laser sources are alternated such that one line of the FA image is scanned, then one line of ICGA, then a second line of FA, and so on. These two approaches have the advantage that the acquired FA and ICGA images are accurately registered together, even in the presence of eye movement. This advantage can be beneficial for image post-processing and visualization tools.

[0111] However, these two conventional approaches also have drawbacks. The fully simultaneous approach 82 requires relatively complex hardware, e.g., separate detectors and filters are required for each of the FA and ICGA acquisition paths. The interleaved approach 84 may have drawbacks in terms of patient comfort, as the blue / green FA illumination source (which is much more bothersome to the patient than the ICGA source) is perceived by the patient as continuing for the entire duration of the exposure (i.e., a long flash of light). This approach may induce photophobic responses (e.g., blinking, miosis, eye movements, Bell's phenomenon).

[0112] A presently preferred technique 86 captures FA and ICGA images sequentially in response to a single "capture" command from the user, i.e., a complete ICGA image (or image sequence) is captured, followed immediately by a complete FA image (or image sequence) in a single image capture sequence (e.g., within one flash period).

[0113] In a preferred embodiment, the ICGA image is scanned first because the required infrared source is not uncomfortable for the patient. Once the ICGA image acquisition scan is complete, the FA scan begins. This differs from two previous approaches for simultaneous FA+ICGA acquisition, which are to illuminate the eye with both light sources at once (82) or to interleave multiple FA and ICGA acquisitions to construct both images (84).

[0114] Because this sequential FA+ICGA technique 86 shortens the time interval between perceptible visible light exposures (compared to the interleaved FA+ICGA technique 84), it may cause less patient photophobic discomfort and reduce the probability of blink and / or eye movement artifacts. That is, the time from onset to end of visible light (e.g., in the case of FA) is much shorter than in the case of interleaved flashes delivering the same energy, thus minimizing the interval during which photophobia may be induced in the patient. This may lead to improved patient comfort and image quality from angiographic examinations. This technique 86 also avoids the complex hardware requirements of the first technique 82. In practice, this technique 86 may be implemented using a properly defined scan table.

[0115] In addition to this FA+ICGA, the system may provide additional multispectral fundus imaging modes 63-67, as shown with reference to FIG. Single-capture multispectral imaging for scanning slit ophthalmoscopes Multispectral fundus imaging can provide improved visibility and differentiation of retinal structure and function by integrating information derived from images acquired using multiple wavelength bands of light (e.g., visible and non-visible reflectance and autofluorescence).

[0116] True color fundus imaging is the standard for retinal examinations. Near-infrared light can penetrate deeper into the retina because its longer wavelengths are less absorbed by blood and melanin. Therefore, infrared fundus imaging may be able to provide complementary information to true color imaging by enabling better visibility of subretinal features, such as small drusen beneath the retinal pigment epithelium (RPE).

[0117] Blood oxygenation can be investigated noninvasively in the retina using multispectral imaging techniques. As described in JM Beach et al., "Oximetry of retinal vessels by dual-wavelength imaging: calibration and influence of pigmentation," Journal of Applied Physiology, Vol. 86, No. 2 (1999), pp. 748-758, two or more excitation wavelengths (those whose blood absorption is insensitive to oxygenation) can be used to capture reflectance images that, when appropriate corrections are made for vessel diameter and retinal pigmentation, can be used to infer blood oxygen saturation.

[0118] In fundus autofluorescence (FAF) imaging, the acquired image information differs depending on the spectral content of the excitation and collection wavelength bands. These differences can be used to extract clinical information about the state of the retina. Focusing FAF at different wavelength bands can be used, for example, to distinguish between fluorophores within the retina, as described in M. Hammer, "Color Autofluorescence Imaging in Age-Related Macular Degeneration and Diabetic Retinopathy," Investigative Ophthalmology & Visual Science, Vol. 49, No. 13 (2008), pp. 4207-4207. Similarly, varying the FAF excitation wavelength band can be used to estimate macular pigment optical density, as described in Delori, Francois C. et al., "Macular pigment density measured by autofluorescence spectrometry: comparison with reflectometry and heterochromatic flicker photometry," Journal of the Optical Society of America A (JOSA A), Vol. 18, No. 6 (2001), pp. 1212-1230.

[0119] All references mentioned herein are incorporated by reference in their entirety. In most commercially available fundus imaging systems, acquiring multispectral data typically requires multiple acquisitions, which is time-consuming for the operator and uncomfortable for the patient. The present system provides a path to extending wide-field slit-scanning fundus imaging to perform multispectral fundus imaging using only a single acquisition protocol.

[0120] Some conventional fundus imaging systems offer imaging at multiple wavelengths, but these are typically provided as separate scanning modes, with each mode requiring a dedicated acquisition input. Thus, for example, acquiring both an RGB color image and an infrared image requires two separate scans, and the resulting images are likely to be misregistered (due to unavoidable eye movement / realignment between scans). Conventional oximetry techniques use simultaneous dual-wavelength oximetry. Bayer filters in the color sensor are used to separate the wavelength channels, and analysis software calculates vascular hemoglobin oxygen saturation. Academic research has used conventional systems that use two separate FAF imaging modalities for macular pigment assessment. The CLARUS™ 500 also offers two distinct FAF excitation / detection bands. However, each mode requires a separate, dedicated acquisition.

[0121] Techniques for simultaneous color and infrared imaging are presented herein. In some applications, sequenced multispectral scans eliminate the need to acquire an IR image as an additional scan. Instead, a fully registered infrared image is automatically provided with every true-color scan. Additional image information can be made available to the user through a four-band channel splitting feature in the instrument software's review screen. For example, FIG. 18 shows a four-channel (e.g., true-color) fundus image 88-C. That is, in addition to the typical red channel 88-R, green channel 88-G, and blue channel 88-B, this color image 88-C additionally has an infrared channel 88-IR.

[0122] The ophthalmic imaging system of the present invention images the retina using sequenced light sources and a monochrome camera. This imaging is achieved using a scan table that instructs the hardware to switch on specific light sources for each slit acquisition. This method allows great flexibility in both scanning and sequencing different types of light.

[0123] The scan table instruction sequence is used to group the acquired slit images by individual color so that they can be combined into separate images that are then combined together in a process to form a true color image. The scan table may be arranged into rows and tables of parameters used by the system to identify specific scan positions, excitation wavelengths (e.g., channels), etc. For example, in a typical scan table for true color mode, each row in the table indicates a slit acquisition, and the "Channel" column tracks the color of light for that particular acquisition and is later used to combine the image color channels separately. In this case, the channels may be R, G, and B to represent the acquisition of red, green, and blue LEDs. To expand color image acquisition to include infrared image acquisition, the scan table may be modified to incorporate an additional IR acquisition (denoted as "I" in the "Channel" column of the scan table), so that the R, G, and B sequences in the color image "Channel" column are expanded to an R, G, B, and I sequence. This effectively provides a four-channel multispectral image. Therefore, all true-color images also provide fully registered IR images as part of the scan. Note that IR light is not uncomfortable for the patient and does not significantly affect phosphene brightness. For FAF with multiple excitations, the green and blue illumination are similarly ordered in the scan table using a long-pass barrier filter (e.g., >650 nm, used to collect the transmitted FAF signal).

[0124] All color channel component images are acquired in a single flash sequence and are therefore perfectly registered (barring eye movement during the scan). Selection of each channel view can be provided in a "Channel Split" review screen. For example, FIG. 19 shows an example of a "Channel Split" review screen 90 that provides an option 92 for use by a system operator, allowing the operator to view the red, green, blue, and infrared channels separately. In another embodiment, the IR image information can be combined with the color image to form a composite image. For example, the red channel of the color image can be linearly combined with the IR image information to provide a "depth-enhanced" true-color fundus image.

[0125] Unlike previous systems that offer IR imaging as a separate acquisition mode, the present technology can automatically provide IR images as part of every true-color acquisition. This has the advantages of requiring no additional chair time, causing no discernible additional discomfort to the patient, and the resulting IR images being perfectly registered to the color images. This feature may help increase interest in IR imaging among retinal specialists, as it provides them with the additional information afforded by IR at no discernible cost.

[0126] A technique for measuring retinal vascular oxygenation is also presented herein. Blood oxygen saturation can be noninvasively examined within the retina using multispectral reflectance imaging. In the basic technique, only two excitation wavelengths are required. One excitation wavelength is chosen so that its absorption by hemoglobin (to which oxygen in blood binds) is independent of oxygen saturation (isobestic). The other excitation wavelength should show a significant difference in absorption between oxyhemoglobin and deoxyhemoglobin.

[0127] Figure 20 shows the absorbance spectra of oxygenated and deoxygenated hemoglobin. Suitable isosbestic points are found at 390, 422, 452, 500, 530, 545, 570, 584, and 797 nm. The wavelengths used should be as close as possible to minimize wavelength-dependent differences in light transport (e.g., scattering) within the retina. The currently preferred approach for oximetry provides a wide field of view and uses slit-scanning techniques in addition to sequenced illumination (e.g., rather than Bayer filter separation as used by conventional methods).

[0128] The system can achieve oxygen measurements using a dedicated excitation filter on a slit wheel. The filter can be a dual bandpass filter or a stacked filter with a 5-10 nm notch to pass two wavelengths of light. The wavelength notch center can be selected so that it is suitable for dual wavelength oxygen measurements and can be provided by separate LEDs (e.g., 570 nm and 615 nm) in the light box. The method for performing oxygen measurements can then be as follows:

[0129] a) Sequence green and red light in a single scanning pattern, then collect two images at 570 nm and 615 nm (these do not need to be registered) in a single acquisition. b) Segment the blood vessels.

[0130] c) Applying the method used by Beach et al. [1] (with some correction for variations in vessel diameter and retinal pigmentation). d) Display of oxygen saturation map. Figure 21 provides an example of an oxygen saturation map.

[0131] This system can also be used to measure macular pigment optical density. In this case, autofluorescence images with dual excitation wavelengths are acquired in a single scan by sequencing blue and green excitation. The acquired data allows for objective measurement of macular pigment optical density (MPOD). MPOD may be clinically useful in determining a patient's risk of developing age-related macular degeneration (AMD). AMD is the leading cause of blindness in Western countries. AMD is a progressive, incurable disease that affects the macular region of the retina, resulting in the loss of high-resolution color vision in central vision. The disease primarily affects people over the age of 50. Incidence rates are generally higher in Western countries due to age profiles, dietary habits, and milder average pigmentation of the eye. Measuring MPOD may offer several benefits to physicians, especially in screening settings. MPOD measurement opens the possibility for healthcare providers to offer nutritional supplements, many of which are commercially available, to expand their reach on patients' dietary health. A currently preferred method may use sequenced FAF acquisition using multiple excitation bands to allow for objective measurement of macular pigment optical density using a single image acquisition.

[0132] In a preferred embodiment, autofluorescence images with dual wavelength excitation are obtained in a single scan by sequencing blue and green excitation with a >650 nm barrier filter typically used for FAF-green. A non-mydriatic mode can be used to enable small-pupil imaging in OD screening settings, and patient comfort with respect to light flashes can be optimized by limiting scanning to the macula and surrounding area. The acquired data should allow for objective measurement of macular pigment optical density.

[0133] Figure 22 illustrates the principle of MPOD measurement (left) and the MPOD profile (right). The FAF-green image has an overlay (of a predetermined color, e.g., yellow) of the normalized difference between FAF-green and FAF-blue. This normalized difference can be used to infer the MPOD profile.

[0134] Some considerations regarding multispectral imaging are provided herein. Additional light sources of various wavelengths can be incorporated into the light box design for single-exposure, sequenced, multispectral imaging. This can provide multispectral imaging capabilities with a single exposure (rather than many exposures) and fully registered image output.

[0135] FIG. 23 illustrates a light box design with eight light sources (two each of red, green, blue, and IR). Typically, only one of each light source is required for imaging in non-mydriatic mode, although there are some advantages to using both light sources of each color if possible for larger pupils (to obtain better image SNR (signal-to-noise ratio) and more effectively suppress reflections). However, the light box design can be modified to incorporate additional light sources. This design can provide the multispectral capability 63 of FIG. 13, resulting in imaging using multiple combinations of LED (light-emitting diode) light sources and filters at wavelengths ranging from 500 to 940 nm, as illustrated in FIG. 16.

[0136] Returning to FIG. 13 , multiple multifunction (combination) keys 63-69 are provided that capture two or more types of images in response to a single control input from the system operator. These combinations are predetermined and presented as fixed multifunction keys 63-69, although the operator may optionally define custom multifunction keys that capture multiple images of different imaging modalities in response to a single capture command. For example, the operator may be provided with a settings window for linking additional types of images to a single image type selection. Note that different types of images may require different types of filters, and if the system only supports the use of one filter at a time, this may limit the number of image types that can be linked together (e.g., only image types that require similar filters may be linked together). The system may limit the available options accordingly. Alternatively, if the system has multiple detectors, each with its own filter, an optical splitter may be used to direct light of different wavelength bands to different detectors through the respective filters. This increases the number of image types that can be linked and captured in response to a single capture command.

[0137] Below, a description of various hardware and architectures suitable for the present invention is provided. Fundus Imaging System Two categories of imaging systems used to image the fundus are flood-illumination imaging systems (or flood-illumination imagers) and scanning-illumination imaging systems (or scanning imagers). Flood-illumination imagers simultaneously flood the entire field of view (FOV) of interest of the object with light, such as with a flash lamp, and capture a full-frame image of the object (e.g., the fundus) using a full-frame camera (e.g., a camera having a two-dimensional (2D) photosensor array sized collectively to capture the desired FOV). For example, a flood-illumination fundus imager floods the fundus of the eye with light and captures a full-frame image of the fundus in a single image capture sequence of the camera. Scanning imagers provide a scanning beam that is scanned across the object, e.g., the eye, and the scanning beam is imaged at different scanning locations as the scanning beam is scanned across the object, creating a series of image segments that can be reconfigured, e.g., combined, to create a composite image of the desired FOV. The scanning beam can be a point, a line, or a two-dimensional region such as a slit or a wide line.

[0138] FIG. 24 illustrates an example of a slit-scanning ophthalmic system SLO-1 for imaging the fundus F, which is the inner surface of the eye E opposite the ocular lens (or crystalline lens) CL and may include the retina, optic disc, macula, fovea, and posterior pole. In this example, the imaging system is in a so-called “scan-descan” configuration, in which a scanning line beam SB traverses the optical components of the eye E (including the cornea Crn, iris Irs, pupil Ppl, and lens) to scan across the entire fundus F. In the case of a projected light fundus imaging device, no scanner is required, and light is irradiated across the entire desired field of view (FOV) at once. Other scanning configurations are known in the art, and the specific scanning configuration is not critical to the present invention. As shown, the imaging system includes one or more light sources LtSrc, preferably a polychromatic LED system or a laser system with a suitably adjusted etendue. An optional slit Slt (adjustable or stationary) may be positioned in front of the light source LtSrc and used to adjust the width of the scanning line beam SB. Additionally, the slit Slt can remain stationary during imaging or can be adjusted to different widths to allow for different confocal levels and different applications for a particular scan or during scanning used for reflection suppression. An optional objective lens ObjL can be placed before the slit Slt. The objective lens ObjL can be any one of several state-of-the-art lenses, including, but not limited to, refractive, diffractive, reflective, or hybrid lenses / systems. Light from the slit Slt passes through a pupil-splitting mirror SM and is directed to the scanner LnScn. It is desirable to keep the scan plane and pupil plane as close as possible to reduce vignetting in the system. An optional optical system DL can be included to manipulate the optical distance between the images of the two components. The pupil-splitting mirror SM can pass the illumination beam from the light source LtSrc to the scanner LnScn and reflect the detection beam from the scanner LnScn (e.g., reflected light returning from the eye E) toward the camera Cmr. The task of the pupil-splitting mirror SM is to split the illumination beam from the scanner LnScn and assist in suppressing system reflections.Scanner LnScn can be a rotating galvo scanner or other type of scanner (e.g., piezoelectric or voice coil, microelectromechanical system (MEMS) scanner, electro-optic deflector, and / or rotating polygon scanner). Depending on whether pupil splitting occurs before or after scanner LnScn, the scan can be split into two steps with one scanner in the illumination path and a separate scanner in the detection path. Specific pupil splitting arrangements are described in detail in U.S. Pat. No. 9,456,746, the entire contents of which are incorporated herein by reference.

[0139] From the scanner LnScn, the illumination beam passes through one or more optical systems—in this case, a scan lens SL and an ophthalmic or ocular lens OL—that enable the pupil of the eye E to be imaged onto the system's image pupil. Generally, the scan lens SL receives the scanning illumination beam from the scanner LnScn at any of a number of scan angles (angles of incidence) and generates a scan line beam SB with a substantially planar focal plane (e.g., a collimated optical path). The ophthalmic lens OL focuses the scan line beam SB onto the fundus F (or retina) of the eye E, allowing it to image the fundus. In this way, the scan line beam SB creates a transverse scan line that moves across the fundus F. One possible configuration of these optical systems is a Keplerian telescope, in which the distance between the two lenses is selected to generate a nearly telecentric intermediate fundus image (4-f configuration). The ophthalmic lens OL can be a single lens, an achromatic lens, or an arrangement of different lenses. All lenses can be refractive, diffractive, reflective, or hybrid, as known to those skilled in the art. The size and / or shape of the ophthalmic lens OL, the focal length of the scanning lens SL, the pupil-splitting mirror SM, and the scanner LnScn can vary depending on the desired field of view (FOV). Therefore, an arrangement is conceivable in which multiple components can be switched in and out of the beam path depending on the field of view, for example, by using a flip of the optical system, a motorized wheel, or removable optical elements. Because a change in field of view results in a different beam size on the pupil, the pupil splitting can also be changed to accommodate a change in FOV. For example, a field of view of 45° to 60° is a typical or standard FOV for a fundus camera. Higher fields of view, such as wide-field FOVs of 60° to 120° or more, may also be feasible. A wide-field FOV may be desirable for combining a wide-line fundus imager (BLFI) with another imaging modality, such as optical coherence tomography (OCT). The upper limit of the field of view may be determined by the accessible working distance combined with the physiological conditions around the human eye. Since a typical human retina has an FOV of 140° horizontally and 80°-100° vertically, it may be desirable to have an asymmetric field of view with as high an FVO as possible on the system.

[0140] The scanning beam SB passes through the pupil Ppl of the eye E and is directed onto the retina or fundus, i.e., surface F. The scanner LnScn1 adjusts the position of the light on the retina or fundus F so that a range of lateral positions of the eye E is illuminated. The reflected or scattered light (or emitted light in the case of fluorescence imaging) is directed along a path similar to the illumination, defining a focused beam CB on a detection path to the camera Cmr.

[0141] In the “scan-descan” configuration of the exemplary slit-scanning ophthalmic system SLO-1 of the present invention, light returning from eye E is “descanned” by scanner LnScn on its way to pupil-splitting mirror SM. That is, scanner LnScn scans illumination beam SB from pupil-splitting mirror SM to define a scanning illumination beam SB across eye E, but because scanner LnScn also receives returning light from eye E at the same scanning position, it effectively descans the returning light (e.g., cancels the scanning motion) to define a non-scanning (e.g., stationary) focused beam from scanner LnScn to pupil-splitting mirror SM, which folds the focused beam toward camera Cmr. At pupil-splitting mirror SM, reflected light (or emitted light, in the case of fluorescence imaging) is separated from the illumination light onto a detection path that is directed to camera Cmr, which may be a digital camera with a photosensor for capturing an image. An imaging (e.g., objective) lens ImgL may be positioned in the detection path so that the fundus is imaged onto camera Cmr. As with the objective lens ObjL, the imaging lens ImgL can be any type of lens known in the art (e.g., refractive, diffractive, reflective, or hybrid lens). Additional operational details, particularly methods for reducing artifacts in images, are described in International Publication WO 2016 / 124644, the entire contents of which are incorporated herein by reference. The camera Cmr captures the received images and, for example, creates an image file, which can be further processed by one or more (electronic) processors or computing devices (e.g., the computer system shown in FIG. 31). Thus, focused beams (returned from all scanning positions of the scanning line beam SB) are collected by the camera Cmr, and a full-frame image Img can be constructed, such as by montage, from a combination of the individually captured focused beams. However, other scanning configurations are also envisioned, including those in which the illumination beam is scanned across the eye E and the focused beam is scanned across the camera's photosensor array.WO 2012 / 059236 and U.S. Patent Application Publication No. 2015 / 0131050, which are incorporated herein by reference, describe several embodiments of scanning slit ophthalmoscopes, including various designs, such as designs in which the returning light is swept across the camera's photosensor array, and designs in which the returning light is not swept across the camera's photosensor array.

[0142] In this example, the camera Cmr is connected to a processor (e.g., processing module) Proc and a display (e.g., display module, computer screen, electronic screen, etc.) Dspl, both of which may be part of the imaging system itself or may be part of separate, dedicated processing and / or display units, such as a computer system, with data passed from the camera Cmr to the computer system via a cable or computer network, including a wireless network. The display and processor may be an integrated unit. The display may be a traditional electronic display / screen or may be a touchscreen and may include a user interface for displaying information to and receiving information from an equipment operator or user. A user may interact with the display using any type of user input device known in the art, including, but not limited to, a mouse, knob, button, pointer, and touchscreen.

[0143] It may be desirable for the patient's gaze to remain fixed while imaging is being performed. One way to achieve gaze fixation is to provide a fixation target to which the patient can be instructed to gaze. The fixation target can be internal or external to the device, depending on which region of the eye is being imaged. One embodiment of an internal fixation target is shown in FIG. 24. In addition to the primary light source LtSrc used for imaging, an optional second light source FxLtSrc, such as one or more LEDs, can be positioned to image a light pattern onto the retina using a lens FxL, scanning elements FxScn, and a reflector / mirror FxM. The fixation scanner FxScn can move the position of the light pattern, and the reflector FxM guides the light pattern from the fixation scanner FxScn to the fundus F of the eye E. Preferably, the fixation scanner FxScn is positioned at the pupil plane of the system so that the light pattern on the retina / fundus can be moved depending on the desired fixation position.

[0144] Slit-scanning ophthalmoscope systems can operate in different imaging modes depending on the light source and wavelength-selective filtering elements used. True-color reflectance imaging (similar to that observed by clinicians when examining the eye using a handheld or slit-lamp ophthalmoscope) can be achieved when imaging the eye using a series of colored LEDs (red, blue, and green). Images for each color can be built up stepwise with each LED turned on at each scanning position, or each color image can be captured completely separately. The three color images can be combined to display a true-color image, or displayed individually to highlight different features of the retina. The red channel best highlights the choroid, the green channel highlights the retina, and the blue channel highlights the pre-retinal layers. Additionally, specific frequencies of light (e.g., individual colored LEDs or lasers) can be used to excite different fluorophores (e.g., autofluorescence) within the eye, and the resulting fluorescence can be detected by filtering out the excitation wavelengths.

[0145] Fundus imaging systems can also provide infrared (IR) reflectance images, such as by using an infrared laser (or other infrared light source). The infrared (IR) mode is advantageous in that the eye is not sensitive to IR wavelengths. This infrared (IR) mode may allow the user to continuously capture images without disturbing the eye (e.g., in preview / alignment mode) to assist the user during device alignment. IR wavelengths also have high penetration through tissue and may provide improved visualization of choroidal structures. Additionally, fluorescein angiography (FA) and indocyanine green angiography (ICG) imaging can be achieved by collecting images after a fluorescent dye is injected into the subject's bloodstream.

[0146] Optical coherence tomography system In addition to fundus photography, fundus autofluorescence (FA), and fundus fluorescein angiography (FA), ophthalmic images may be produced by other imaging modalities, such as optical coherence tomography (OCT), OCT fundus angiography (OCTA), and / or ocular echography. The present invention, or at least portions of the present invention, may also be applied to these other ophthalmic imaging modalities with minor modifications, as understood in the art. More specifically, the present invention may also be applied to ophthalmic images generated by OCT / OCTA systems that generate OCT and / or OCTA images. For example, the present invention may also be applied to en face OCT / OCTA images. Examples of fundus images are provided in U.S. Patent Nos. 8,967,806 and 8,998,411, examples of OCT systems are provided in U.S. Patent Nos. 6,741,359 and 9,706,915, and examples of OCTA imaging systems may be found in U.S. Patent Nos. 9,700,206 and 9,759,544, the contents of all of which are incorporated herein by reference in their entireties. For the sake of completeness, examples of exemplary OCT / OCTA systems are provided herein.

[0147] FIG. 25 illustrates a generalized frequency-domain optical coherence tomography (FD-OCT) system for collecting 3D image data of the eye suitable for use with the present invention. The FD-OCT system OCT_1 includes a light source LtSrc1. Typical light sources include, but are not limited to, a broadband light source with a short temporal coherence length or a swept laser source. A beam of light from the light source LtSrc1 is typically guided by an optical fiber Fbr1 to illuminate a sample, such as an eye E, a typical sample being human intraocular tissue. The light source LrSrc1 can be either a broadband light source with a short temporal coherence length in the case of spectral-domain OCT (SD-OCT) or a wavelength-tunable laser source in the case of swept-source OCT (SS-OCT). The light is typically scanned by a scanner Scnr1 between the output of the optical fiber Fbr1 and the sample E, such that the beam of light (dashed line Bm) is scanned laterally (in x and y) across the imaged region of the sample. In full-field OCT, no scanner is required; light is directed at the entire desired field of view (FOV) at one time. Light scattered from the sample is typically collected into the same optical fiber Fbr1 used to guide the illumination light. A reference beam, derived from the same light source LtSrc1, travels along a separate path, which in this case includes optical fiber Fbr2 and a retroreflector RR1 with an adjustable optical delay. As will be appreciated by those skilled in the art, a transmissive reference path can also be used, with an adjustable delay located in either the sample or the reference arm of the interferometer. The collected sample light is typically combined with the reference beam in a fiber coupler Cplr1 to form optical interference within the OCT photodetector Dtctr1 (e.g., a photodetector array, digital camera, etc.). While one fiber port is shown reaching the detector Dtctr1, those skilled in the art will appreciate that various interferometer designs can be used for balanced or unbalanced detection of the interference signal. The output from detector Dtctr1 is provided to processor Cmp1 (e.g., a computing device), which converts the observed interference into depth information of the sample, which may be stored in a memory associated with processor Cmp1 and / or displayed on a display (e.g., a computer / electronic display / screen) Scn1.The processing and storage functions may be located within the OCT device, or the functions may be performed on an external processing unit (e.g., the computer system shown in FIG. 20) to which the collected data is transferred. This unit may be dedicated to data processing or may perform other tasks that are general and not dedicated to the OCT device. Processor Cmp1 may include, for example, a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a system on a chip (SoC), a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a combination thereof, that performs some or all of the data processing steps before being provided to, or in parallel with, a host processor.

[0148] The sample and reference arms in an interferometer can be constructed of bulk optics, fiber optics, or hybrid bulk optics systems and can have different architectures, such as Michelson, Mach-Zehnder, or common-path designs, as known to those skilled in the art. Light beam, as used herein, should be interpreted as any carefully directed optical path. Instead of mechanically scanning the beam, a light field can illuminate a one-dimensional or two-dimensional area of ​​the retina to generate OCT data (e.g., U.S. Pat. No. 9,332,902; D. Hillmann et al., "Holoscopy-holographic optical coherence tomography," Optics Letters, Vol. 36(13), p. 2290, 2011; Y. Nakamura et al., "High-Speed ​​three dimensional human retinal imaging by line field spectral domain optical coherence tomography," Optics Express, 2011). Express, 15(12), p. 7103, 2007; Blazkiewicz et al., "Signal-to-noise ratio study of full-field Fourier-domain optical coherence tomography," Applied Optics, 44(36), p. 7722 (2005). In time-domain systems, the reference arm must have an adjustable optical delay to create interference. Balanced detection systems are typically used in TD-OCT and SS-OCT systems, while a spectrometer is used at the detection port for SD-OCT systems. The invention described herein can be applied to either type of OCT system.Various aspects of the present invention may be applied to any type of OCT system or to other types of ophthalmic diagnostic systems and / or to multiple ophthalmic diagnostic systems, including, but not limited to, fundus imaging systems, visual field testing devices, and scanning laser polarimeters.

[0149] In Fourier-domain optical coherence tomography (FD-OCT), each measurement is a real-valued spectrally controlled interferogram (Sj(k)). The real-valued spectral data typically undergoes several post-processing steps, including background subtraction, dispersion correction, etc. A Fourier transform of the processed interferogram yields a complex OCT signal output Aj(z) = |Aj|eiφ. The absolute value of this complex OCT signal, |Aj|, ​​reveals the scattering intensity at different path lengths and, therefore, the scattering profile with respect to depth (z-direction) within the sample. Similarly, the phase φj can also be extracted from the complex OCT signal. The scattering profile with respect to depth is called an axial scan (A-scan). A collection of A-scans measured at adjacent locations within the sample produces a cross-sectional image (tomogram or B-scan) of the sample. A collection of B-scans collected at different lateral locations on the sample constitutes a data volume or cube. For a particular data volume, the fast axis refers to the scanning direction along one B-scan, and the slow axis refers to the axis along which multiple B-scans are collected. The term "cluster scan" may refer to a unit or block of data generated by repeated acquisition at the same (or substantially the same) location (or region) to analyze motion contrast, which may be used to identify blood flow. A cluster scan can consist of multiple A-scans or B-scans collected at approximately the same location on a sample at a relatively short time interval. Because the scans in a cluster scan are of the same region, stationary structures remain relatively unchanged between scans in the cluster scan, while motion contrast between scans that meet predetermined criteria may be identified as blood flow. Various methods for generating B-scans are known in the art, including, but not limited to, along the horizontal or x-direction, along the vertical or y-direction, along the x and y diagonals, or in a circular or spiral pattern. B-scans may be in the xz dimension, but may also be any cross-sectional image including the z-dimension.

[0150] In OCT angiography or functional OCT, analysis algorithms may be applied to OCT data collected at different times (e.g., cluster scans) at the same or nearly the same sample location on the sample to analyze motion or flow (see, e.g., U.S. Patent Application Publication Nos. 2005 / 0171438, 2012 / 0307014, 2010 / 0027857, 2012 / 0277579, and U.S. Patent No. 6,549,801, all of which are incorporated by reference in their entireties). OCT systems may use any one of a number of OCT angiography processing algorithms (e.g., motion contrast algorithms) to identify blood flow. For example, motion contrast algorithms can be applied to intensity information derived from the image data (intensity-based algorithms), phase information from the image data (phase-based algorithms), or complex image data (complex-based algorithms). An en face image is a 2D projection of the 3D OCT data (e.g., by averaging the intensity of each individual A-scan, whereby each A-scan defines a pixel in the 2D projection). Similarly, an en face vascular image is an image that displays motion contrast signals in which the data dimension corresponding to depth (e.g., the z direction along the A-scan) is displayed as a single representative value (e.g., a pixel in the 2D projection image), typically by summing or integrating all or isolated portions of the data (see, e.g., U.S. Pat. No. 7,301,644, incorporated herein by reference in its entirety). OCT systems that provide angiography capabilities may be referred to as OCT angiography (OCTA) systems.

[0151] An example of an en face vasculature image is shown in Figure 26. After processing the data and highlighting motion contrast using any of the motion contrast methods known in the art, an en face (e.g., front view) image of the vasculature may be generated by summing pixel ranges corresponding to a tissue depth from the surface of the retinal internal limiting membrane (ILM).

[0152] Neural Networks As mentioned above, the present invention may use neural network (NN) machine learning (ML) models. For completeness, neural networks are generally described herein. The invention may use any of the following neural network architectures, alone or in combination: A neural network, or neural net, is a network of interconnected neurons (via nodes), with each neuron representing a node in the network. Collections of neurons may be arranged in layers, with the output of one layer being fed forward to the next layer in a multi-layer perceptron (MLP) arrangement. An MLP may be understood as a feed-forward neural network that maps a set of input data to a set of output data.

[0153] FIG. 27 illustrates an example of a multilayer perceptron (MLP) neural network. The structure may include multiple hidden (e.g., inner) layers HL1 through HLn, which map an input layer InL (which receives a set of inputs (or vector inputs) in_1 through in_3) to an output layer OutL, which generates a set of outputs (or vector outputs), e.g., out_1 and out_2. Each layer may have any number of nodes, which are shown here as circles within each layer for illustrative purposes. In this example, the first hidden layer HL1 has two nodes, and hidden layers HL2, HL3, and HLn each have three nodes. Generally, the deeper the MLP (e.g., the greater the number of hidden layers in the MLP), the greater its learning capacity. The input layer InL may receive vector inputs (shown for illustrative purposes as a three-dimensional vector consisting of in_1, in_2, and in_3) and feed the received vector inputs to the first hidden layer HL1 in the sequence of hidden layers. The output layer OutL receives the output from the last hidden layer in the multi-layer model, say HLn, and produces a vector output result (shown for illustration purposes as a two-dimensional vector consisting of out_1 and out_2).

[0154] Typically, each neuron (i.e., node) generates one output, which is fed forward to neurons in the immediately succeeding layer. However, each neuron in a hidden layer may receive multiple inputs, either from the input layer or from the outputs of neurons in the immediately preceding hidden layer. In general, each node may apply a function to its inputs to generate the output for that node. Nodes in a hidden layer (e.g., the training layer) may apply the same function to each of their inputs to generate their respective outputs. However, some nodes, e.g., nodes in the input layer InL, may receive only one input and be passive, meaning that they simply relay the value of that one input to their output, e.g., they provide a copy of that input to their output; this is indicated by the dashed arrows in the nodes of the input layer InL for illustrative purposes.

[0155] For illustrative purposes, Figure 28 shows a simplified neural network consisting of an input layer InL', a hidden layer HL1', and an output layer OutL'. The input layer InL' is shown to have two input nodes i1 and i2, which receive inputs Input_1 and Input_2, respectively (e.g., the input nodes of layer InL' receive a two-dimensional input vector). The input layer InL' feeds forward into one hidden layer HL1' with two nodes h1 and h2, which in turn feeds forward into an output layer OutL' with two nodes o1 and o2. The interconnections, or links, between neurons (shown with solid arrows for illustrative purposes) have weights w1 through w8. Typically, except for the input layer, a node (neuron) may receive as input the output of a node in the previous layer. Each node may calculate its output by multiplying each of its inputs by each input's corresponding interconnection weight, summing the products of the inputs, adding (or multiplying) a constant defined by other weights or biases that may be associated with that particular node (e.g., node weights w9, w10, w11, and w12 corresponding to nodes h1, h2, o1, and o2, respectively), and then applying a nonlinear or logarithmic function to the result. The nonlinear function may be referred to as an activation function or a transfer function. Multiple activation functions are known in the art, and the selection of a particular activation function is not important to this description. However, it should be noted that the operation of an ML model, the behavior of a neural net, depends on the values ​​of the weights, which the neural network may be trained to provide a desired output for a given input.

[0156] During a training, or learning, phase, a neural network learns (e.g., is trained to identify) appropriate weight values ​​to achieve a desired output for a given input. Before a neural network is trained, each weight may be individually assigned an initial (e.g., random, optionally non-zero) value, such as a random number seed. Various methods for assigning initial weights are known in the art. The weights are then trained (optimized) so that, for a given training vector input, the neural network produces an output that approximates a desired (predetermined) training vector output. For example, the weights may be gradually adjusted over thousands of iterative cycles by a method called backpropagation. In each backpropagation cycle, a training input (e.g., a vector input or training input image / sample) is passed forward through the neural network to provide its actual output (e.g., a vector output). The error of each output neuron, or output node, is then calculated based on the actual neuron's output and the supervised training output for that neuron (e.g., a training output image / sample corresponding to the current training input image / sample). It then propagates backward through the neural network (from the output layer back to the input layer), updating the weights based on how much influence each weight has on the overall error, thereby moving the neural network's output closer to the desired training output. This cycle is then repeated until the neural network's actual output is within an acceptable error range of the desired training output for that training input. As will be appreciated, each training input may require many backpropagation iterations to achieve the desired error range. Typically, an epoch refers to one backpropagation iteration (e.g., one forward pass and one backward pass) of all training samples, and training a neural network may require many epochs. Generally, the larger the training set, the better the performance of the trained ML model; therefore, various data augmentation methods may be used to increase the size of the training set.For example, if the training set includes pairs of corresponding training input images and training output images, the training images may be divided into multiple corresponding image segments (or patches). Corresponding patches from the training input images and training output images may be paired to define multiple training patch pairs from one input / output image pair, thereby expanding the training set. However, training a large training set increases the demands on computer resources, such as memory and data processing resources. The computational demands may be reduced by dividing the large training set into multiple mini-batches, the size of which determines the number of training samples in one forward / backward pass. In this case, one epoch may contain multiple mini-batches. Another problem is the possibility that the neural network may overfit the training set, reducing its ability to generalize from a specific input to different inputs. The overfitting problem may be mitigated by creating an ensemble of neural networks or by randomly dropping out nodes in the neural network during training, which effectively removes the dropped leads from the neural network. Various dropout adjustment methods, such as inverse dropout, are known in the art.

[0157] It should be noted that the operation of a trained NN machine model is not a simple algorithm of computation / analysis steps. Indeed, when a trained NN machine model receives an input, the input is not analyzed in the traditional sense. Rather, regardless of the purpose or nature of the input (e.g., vectors defining a live image / scan or vectors defining any other entity such as a demographic description or activity record), the input is subjected to the same architectural construction of the trained neural network (e.g., the same node / layer arrangement, trained weights and bias values, predetermined convolution / deconvolution operations, activation functions, pooling operations, etc.), and it may not be obvious how the architectural construction of the trained network generates its output. Furthermore, the values ​​of the trained weights and biases are not deterministic and depend on many factors, such as the amount of time given to the neural network for training (e.g., the number of epochs in training), the random starting values ​​of the weights before training begins, the computer architecture of the machine on which the NN is trained, the selection of training samples, the distribution of training samples among multiple mini-batches, the selection of activation functions, the selection of error functions that modify the weights, and even whether training is interrupted on one machine (e.g., with a first computer architecture) and completed on another machine (e.g., with a different computer architecture). The point is that the reasons why a trained ML model arrives at a particular output are not clear, and much research is currently being conducted to identify the factors on which ML models base their output. Therefore, the processing of neural networks on live data cannot be reduced to a simple algorithmic step. Rather, the operation depends on the training architecture, training sample set, training sequence, and various circumstances in the training of the ML model.

[0158] In general, constructing a neural network machine learning model may include a learning (or training) stage and a classification (or computation) stage. In the learning stage, a neural network may be trained for a specific purpose and provided with a set of training examples, including training (sample) inputs and training (sample) outputs, and optionally a set of validation examples for testing the training progress. During this learning process, various weights associated with nodes and node interconnections within the neural network are gradually adjusted to reduce the error between the neural network's actual output and the desired training output. In this way, a multi-layer feedforward neural network (such as those described above) may be able to approximate any measurable function to any desired accuracy. The result of the learning stage is a learned (e.g., trained) (neural network) machine learning (ML). In the computation stage, a set of test inputs (or live inputs) may be provided to the learned (trained) ML model, which may apply what it has learned to generate output predictions based on the test inputs.

[0159] Similar to the conventional neural networks of Figures 26 and 27, convolutional neural networks (CNNs) are also composed of neurons with learnable weights and biases. Each neuron receives an input and performs an operation (e.g., a dot product), optionally followed by a nonlinear transformation. However, CNNs receive raw image pixels at one end (e.g., the input) and provide a classification (or class) score at the other end (e.g., the output). Because CNNs expect images as input, they are optimized to handle volumes (e.g., image pixel height and width, and image depth, e.g., color depth, such as RGB depth defined by three colors: red, green, and blue). For example, CNN layers may be optimized for neurons arranged in three dimensions. Neurons in a CNN layer may connect to a small region of the previous layer rather than all of the neurons in a fully connected NN. The final output layer of a CNN may reduce the full image to a single vector (classification) arranged along the depth dimension.

[0160] FIG. 29 provides an exemplary convolutional neural network architecture. A convolutional neural network may be defined as a sequence of two or more layers (e.g., Layer 1 through Layer N), where a layer may include an (image) convolution step, a (resulting) weighted sum step, and a nonlinear function step. The convolution may be performed on the input data by, for example, applying a filter (or kernel) on a moving window over the input data to generate a feature map. Each layer and layer component may have different predetermined filters (from a filter bank), weights (or weighting parameters), and / or function parameters. In this example, the input data may be an image of a certain pixel height and width, or the raw pixel values ​​of this image. In this example, the input image is depicted as having a depth of three color channels, RGB (red, green, blue). Optionally, various preprocessing steps may be performed on the input image, and the results of the preprocessing steps may be input instead of or in addition to the raw image data. Some examples of image processing may include retinal vessel map segmentation, color space conversion, adaptive histogram equalization, connected component generation, etc. Within a layer, a dot product may be calculated between certain weights and their connected small regions within the input volume. While many methods for constructing CNNs are known in the art, by way of example, layers may be configured to apply element-wise activation functions, such as a max(0,x) threshold at zero. Pooling functions may be performed (e.g., along the x and y directions) to downsample the volume. Fully connected layers may be used to identify classification outputs and generate one-dimensional output vectors, which have proven useful in image recognition and classification. However, for image segmentation, CNNs must classify each pixel. Because each CNN layer tends to reduce the resolution of the input image, another stage is required to upsample the image to its original resolution. This may be achieved by applying a transposed convolution (or deconvolution) stage TC, which typically does not use any predetermined interpolation method but instead has learnable parameters.

[0161] Convolutional neural networks have been successfully applied to many problems in computer vision. As mentioned above, training a CNN generally requires a large training dataset. The U-Net architecture is based on a CNN and can generally be trained with a smaller training dataset than traditional CNNs.

[0162] FIG. 30 illustrates an exemplary U-Net architecture. This exemplary U-Net includes an input module (or input layer or stage) that receives an input U-in (e.g., an input image or image patch) of any size (e.g., 128×128 pixels in size). The input image may be a fundus image, an OCT / OCTA en face image, a B-scan image, etc. However, it should be understood that the input may be of any size and dimensionality. For example, the input image may be an RGB color image, a monochrome image, a volumetric image, etc. The input image passes through a series of processing layers, each of which is illustrated with exemplary dimensions, although these dimensions are for illustrative purposes only and may depend, for example, on the size of the image, convolutional filters, and / or pooling stages. This architecture consists of a convergence path (comprising four encoding modules) followed by an expansion path (comprising four decoding modules), and four copy-and-crop links (e.g., CC1-CC4) between corresponding modules / stages that copy the output of one encoding module in the convergence path and connect it to the input of the corresponding decoding module in the expansion path. The result is a characteristic U-shape, from which the architecture is named. The convergence path is similar to an encoder; its basic function is to capture context through compact feature maps. In this example, each encoding module in the convergence path may contain two convolutional neural network layers, followed by a max-pooling layer (e.g., a downsampling layer). For example, the input image U_in passes through two convolutional layers, each with 32 feature maps. The number of feature maps doubles with each pooling, starting with 32 feature maps in the first block, 64 in the second block, and so on. Therefore, the convergence path forms a convolutional network consisting of multiple encoding modules (or stages), each providing a convolutional stage, followed by an activation function (e.g., a rectified linear unit (ReLU) or sigmoid layer) and a max-pooling operation.The augmentation path is similar to the decoder; its function is to provide localization and preserve spatial information despite the downsampling and any max-pooling performed in the contraction stage. In the convergence path, spatial information is reduced and feature information is augmented. The augmentation path includes multiple decoding modules, each of which combines its current value with the output of the corresponding encoding module. That is, feature and spatial information are combined in the augmentation path through a series of upconvolutions (e.g., upsampling or transposed convolutions, i.e., deconvolutions) and combinations (e.g., via CC1-CC4) with high-resolution features from the convergence path. Therefore, the output of the deconvolution layer is combined with the corresponding (optionally cropped) feature map from the convergence path, followed by two convolutional layers and activation functions (optionally batch normalized). The output from the last module in the augmentation path may be fed to other processing / training blocks or layers, such as a classifier block, which may be trained together with the U-Net architecture.

[0163] The module / stage (BN) between the convergence path and the expansion path may be called the "bottleneck." The bottleneck BN may consist of two convolutional layers (with batch normalization and optional dropout).

[0164] Computing Devices / Systems FIG. 31 illustrates an exemplary computer system (or computing device). In some embodiments, one or more computer systems may provide functionality described or illustrated herein and / or perform one or more steps of one or more methods described or illustrated herein. The computer system may take any suitable physical form. For example, the computer system may be an embedded computer system, a system-on-chip (SOC), or a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, the computer system may reside in a cloud, which may include one or more cloud components within one or more networks.

[0165] In some embodiments, a computer system may include a processor Cpnt1, a memory Cpnt2, a storage Cpnt3, an input / output (I / O) interface Cpnt4, a communication interface Cpnt5, and a bus Cpnt6. The computer system may also optionally include a display Cpnt7, such as a computer monitor or screen.

[0166] The processor Cpnt1 includes hardware for executing instructions, such as those that constitute a computer program. For example, the processor Cpnt1 may be a central processing unit (CPU) or a general-purpose computing-on-graphics processing unit (GPGPU). The processor Cpnt1 may read (or fetch) instructions from an internal register, an internal cache, memory Cpnt2, or storage Cpnt3, decode and execute the instructions, and write one or more results to the internal register, the internal cache, memory Cpnt2, or storage Cpnt3. In particular embodiments, the processor Cpnt1 may include one or more internal caches for data, instructions, or addresses. The processor Cpnt1 may include one or more instruction caches and one or more data caches, for example, to hold data tables. Instructions in the instruction caches may be copies of instructions in memory Cpnt2 or storage Cpnt3, and the instruction caches may speed up retrieval of these instructions by the processor Cpnt1. Processor Cpnt1 may include any suitable number of internal registers and may include one or more arithmetic logic units (ALUs). Processor Cpnt1 may be a multi-core processor or may include one or more processors Cpnt1. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0167] The memory Cpnt2 may include a main memory that stores instructions for the processor Cpnt1 to execute processing or to hold intermediate data during processing. For example, the computer system may load instructions or data (e.g., a data table) from the storage Cpnt3 or from other sources (e.g., another computer system) into the memory Cpnt2. The processor Cpnt1 may load instructions and data from the memory Cpnt2 into one or more internal registers or internal caches. To execute an instruction, the processor Cpnt1 may read and decode the instruction from the internal register or internal cache. During or after the execution of an instruction, the processor Cpnt1 may write one or more results (which may be intermediate or final results) to an internal register, an internal cache, the memory Cpnt2, or the storage Cpnt3. The bus Cpnt6 may include one or more memory buses (each of which may include an ADDRESS bus and a DATA bus) and may couple the processor Cpnt1 to the memory Cpnt2 and / or the storage Cpnt3. Optionally, one or more memory management units (MMUs) facilitate data transfer between the processor Cpnt1 and the memory Cpnt2. The memory Cpnt2 (which may be a high-speed volatile memory) may include random access memory (RAM), such as dynamic RAM (DRAM) or static RAM (SRAM). The storage Cpnt3 may include long-term or high-capacity storage for data or instructions. The storage Cpnt3 may be internal or external to the computer system and may include one or more of a disk drive (e.g., a hard disk drive (HDD) or a solid-state drive (SSD)), flash memory, ROM, EPROM, optical disk, magneto-optical disk, magnetic tape, a universal serial bus (USB)-accessible drive, or other type of non-volatile memory.

[0168] The I / O interface Cpnt4 may be software, hardware, or a combination of both, and may include one or more interfaces (e.g., serial or parallel communication ports) for communicating with I / O devices, which may enable communication with a human (e.g., a user). For example, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, table, touch screen, trackball, video camera, other suitable I / O device, or a combination of two or more thereof.

[0169] The communication interface Cpnt5 may provide a network interface for communicating with other systems or networks. The communication interface Cpnt5 may include a Bluetooth interface or other types of packet-based communication. For example, the communication interface Cpnt5 may include a network interface controller (NIC) and / or a wireless NIC or wireless adapter for communication with a wireless network. The communication interface Cpnt5 may provide communication with a Wi-Fi network, an ad-hoc network, a personal area network (PAN), a wireless PAN (e.g., Bluetooth WPAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a cellular network (e.g., a Global System for Mobile Communications (GSM) network), the Internet, or a combination of two or more thereof.

[0170] Bus Cpnt6 may provide a communication link between the above-mentioned components of the computing system. For example, bus Cpnt6 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand bus, a low-pin-count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or any other suitable bus, or a combination of two or more thereof.

[0171] Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0172] As used herein, a computer-readable non-transitory storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, or any other suitable computer-readable non-transitory storage medium, or any suitable combination of two or more thereof, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0173] While the present invention has been described in conjunction with several specific embodiments, as will be apparent to those skilled in the art in light of the foregoing description, many other alternatives, modifications, and variations will be apparent. Accordingly, the invention as described herein is intended to embrace all such alternatives, modifications, applications, and variations that may fall within the spirit and scope of the appended claims.

Claims

1. 1. A method for capturing a fundus image, comprising: displaying a graphical user interface (GUI) on a screen, the GUI providing a plurality of imaging type options, each imaging type option being associated with a respective predefined set of imaging parameters; selectively setting the enhanced function options in response to the patient meeting a predetermined medical condition using an electronic processor; In response to an image capture input prompt when the extension is configured, Identifying a first imaging type option of the currently selected imaging type options; replacing the default set of imaging parameters associated with a first imaging type option with the adjusted set of imaging parameters specified by the advanced function option; executing an image capture sequence to capture an image of the fundus using said adjusted set of imaging parameters; A method comprising:

2. 10. The method of claim 1, wherein the medical condition is one or more of photophobia, post-traumatic stress disorder, and a predetermined DNA marker.

3. the predetermined set of imaging parameters includes one or more of a light intensity parameter that controls the intensity of light illuminated during an image capture operation and a light duration parameter that controls the duration for which light is illuminated during the image capture operation; The method of claim 1 , wherein the adjusted imaging parameters decrease one or both of the light intensity parameter and the light duration parameter.

4. The method of claim 1 , wherein the adjusted imaging parameters are based on physiological characteristics of the eye, including one or more of an iris size, an iris color, and a retinal brightness level of the eye.

5. 1. A method for capturing a fundus image, comprising: initiating a fundus image capture sequence using a scanning slit ophthalmoscope; evaluating a scan table specifying a plurality of scan locations on the fundus, each scan location being associated with a plurality of distinct light wave ranges; initiating a fundus scan sequence specified by the scan table, (a) sequentially applying, using at least one light source, each distinct range of light waves associated with a current scan position; (b) collecting, using at least one collector, return light from the current scan position for each sequentially applied range of light waves; and generating separate fundus images for each light wave range based on the collected return light from each light wave range; determining a ratiometric relationship between collected return light from at least two of the distinct light wave ranges within a predetermined region of each fundus image; processing or displaying the fundus image; including, including initiating, The method, wherein the ratiometric is a measure of at least one of macular pigment density and oxygenation.

Images