Patient-adjusted ophthalmic imaging system with single-exposure multi-type imaging, improved focus, and improved angiogram image sequence display

Abstract

An ophthalmic imaging system provides an autofocus mechanism based on differences in consecutive scan lines. The system also provides user selection of the focal point within the fundus image. A neural network automatically identifies the optic nerve head in FA or ICGA images, which can be used to determine the angle of gaze. The system also provides additional scan tables for multiple imaging modalities to accommodate photophobic patients and multi-spectral imaging options.

Description

Technical Field

[0001] This invention relates generally to the field of ophthalmic imaging systems. More specifically, this invention relates to techniques for facilitating user operation of ophthalmic imaging systems. Background Technology

[0002] Various types of ophthalmic examination systems exist, including ophthalmoscopy, optical coherence tomography (OCT), and other ophthalmic imaging systems. One example of ophthalmic imaging is slit scanning or wide-field-of-view fundus imaging (see, for example, U.S. Patent Nos. 4,170,398, 4,732,466, PCT Publication No. 2012059236, U.S. Patent Application No. 2014 / 0232987, and U.S. Patent Publication No. 2015 / 0131050, the entire contents of which are incorporated herein by reference), a promising technique for achieving high-resolution in vivo imaging of the human retina. The imaging method is a hybrid of confocal and wide-field-of-view imaging systems. By illuminating a narrow band of the retina during scanning, the illumination remains outside the observation path, allowing for a clearer view of more of the retina than the ring illumination used in conventional fundus cameras.

[0003] For good images to be obtained, the illuminating strip ideally needs to be well focused and pass through the pupil and reach the fundus without diminishing. This requires a careful focusing system and proper alignment of the eye with the ophthalmic imaging system. Adding to the complexity of ophthalmic imaging is the difficulty in keeping the patient still during imaging. This can be particularly problematic when multiple different types of images are needed, or when the patient dislikes high-intensity illumination, for example, exhibiting photophobia. Therefore, extensive training is typically required to develop a high level of proficiency in using such a system.

[0004] One object of the present invention is to provide a tool that facilitates focusing of ophthalmic imaging or examination systems.

[0005] Another object of the present invention is to provide various methods to accelerate the acquisition of ophthalmic images.

[0006] Another object of the present invention is to provide a method for reducing patient anxiety or discomfort during ophthalmic image capture. Summary of the Invention

[0007] The aforementioned objectives are achieved in a system / method with improved focusing and functionality. First, a focusing mechanism based on the difference between consecutive line scans effectively converts wide lines into fine lines to determine defocus metrics. These fine lines can then be used to determine the topology of the retina.

[0008] In a preferred embodiment, a preview screen with multiple pre-defined focus assist positions is presented to the system operator. An image can be displayed on the preview screen, and the system operator is free to select any point on the preview screen that should be brought into sharper focus. The preview screen then focuses the image on the selected point by combining focus information from the focus assist positions.

[0009] In some embodiments, focusing is further adjusted based on the patient's gaze angle. The gaze angle can be determined by locating the patient's optic nerve head (ONH). This article provides a technique for identifying the ONH in an infrared (IR) preview image, thereby determining the patient's true gaze angle.

[0010] It should be noted that ONH identification has other uses, and therefore it is beneficial to identify ONH in any type of image. This paper presents a method for identifying ONH in fluorescein angiography (FA) and / or ICGA (indocyanine green angiography) images and / or any other imaging modality. Since the system can continuously capture IR (infrared) preview images before capturing FA or ICGA images, the system locates the ONH in the IR preview images and then transfers the located ONH onto the captured FA or ICGA images.

[0011] To achieve this, the system provides U-Net (a general description of U-Net can be found in Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, Computer Vision and Pattern Recognition, 2015, arXiv: 1505.04597 [cs.CV]), which uses transfer learning to learn how to locate ONHs in infrared images using a large training color image library. Initially, U-Net is trained using only color images. Then, a predefined number of initial U-Net layers are frozen, and the remaining layers are trained on a smaller infrared image training set. Locating ONHs in infrared images still presents challenges, so additional steps are provided to optimize ONH localization.

[0012] To further improve FA and ICGA inspection, this system offers simplified capture of FA or ICGA image sequences. The system provides a very high dynamic range, eliminating the need for operators to adjust image brightness when capturing FA or ICGA image sequences. The original images are stored, and the relative brightness information between images in the sequence can be determined from the raw data and accurately provided to the user, for example, as a graph. Furthermore, the system provides operators with an option to brighten the stored images for viewing while maintaining the relative brightness between the acquired image sequences.

[0013] To further adapt to the patient, this system provides patient-adjustable imaging. The system examines the physical characteristics of the patient's eyes (and / or the patient's medical history) for indications of photophobia. If the patient is suspected of being a candidate for photophobia, or has been previously diagnosed with photophobia, the system operator is alerted to this fact. The system operator can then select a dimmable imaging scheme for the system based on the patient's degree of photophobia.

[0014] Other improvements to this system include various mechanisms for acquiring multiple imaging modalities using a single capture command. This not only reduces the time required for examinations but also provides additional information that can be used for further diagnostics. For example, along with each color image, the system can automatically capture an infrared image, which can enhance the visibility of some tissues.

[0015] Other objects and achievements, as well as a fuller understanding of the invention, will become apparent and readily understood by taking into account the accompanying drawings, the following description, and the appended claims.

[0016] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited thereto. Any embodiment feature mentioned in one claim class (e.g., system) may also be claimed in another claim class (e.g., method). Dependencies or references in the appended claims are chosen solely for formal reasons. However, any subject matter arising from intentional references to any prior claim may also be claimed, such that any combination of claims and their features may be disclosed and claimed, regardless of the dependency chosen in the appended claims. Attached Figure Description

[0017] The patent or application documents contain at least one drawing executed in color. A copy of the patent or patent application publication, along with the color drawing, will be provided by the Patent Office upon request and at the necessary cost.

[0018] In the accompanying drawings, the same reference numerals / characters denote the same parts:

[0019] Figure 1 Simplified patterns of scan lines are shown because they can be generated on the scanned object;

[0020] Figure 2A An overlapping line scan sequence is shown, wherein the step size between scan lines is smaller than the width of the scan line;

[0021] Figure 2B It shows the subtraction Figure 2A The effect of two consecutively captured images along a continuous scan line;

[0022] Figure 2CThis illustrates the creation of a single thin line from two scan lines and the identification of the second-order moment σ that defines the line width;

[0023] Figure 3A and Figure 3B This illustrates how the displacement of the centroid corresponds to changes in depth (or height) in the eye (e.g., deconvolution measurements);

[0024] Figure 4A The preview image of the fundus is shown in a preview window on an electronic display, with multiple focus assist positions overlaid on the fundus image;

[0025] Figure 4B Examples are provided showing how users can freely select any point in the preview image to focus on;

[0026] Figure 5 Two images of blood vessels taken at different stages of an angiography examination are provided;

[0027] Figure 6 A segmentation and localization framework for finding ONH in an operational FA / ICGA image is provided according to the present invention;

[0028] Figure 7A It shows the use of Figure 6 The system identifies the location of the ONH region in the infrared preview image;

[0029] Figure 7B It shows the capture Figure 6 The FA image was captured shortly after the IR preview image;

[0030] Figure 8 A framework is provided for processing individual images from a series of images captured as part of an angiography examination;

[0031] Figure 9 A time-shifted FA image sequence is shown, wherein each individual image is adjusted according to its corresponding individual white level;

[0032] Figure 10 This shows the original FA image sequence without any brightness adjustment applied to any images (corresponding to...). Figure 9 sequence);

[0033] Figure 11 The image sequence of FA with overlay adjustment is shown (corresponding to) Figure 10 (a sequence), where each image is brightened by the same overlay scaling factor (e.g., 255 / wmax);

[0034] Figure 12A This shows the effect of setting the compression parameter ∝ to zero, which is equivalent to brightening all images using the same overlay scaling factor.

[0035] Figure 12B The effect of setting the compression parameter ∝ to 1 is shown, which is equivalent to automatically brightening each image individually based on the white level stored in each image;

[0036] Figures 12C to 12F This demonstrates how modifying the compression parameter ∝ affects the brightness of an image sequence for different ∝ values;

[0037] Figure 13 Examples of graphical user interfaces (GUIs) for ophthalmic imaging devices are provided;

[0038] Figure 14 An example procedure is provided for adjusting (e.g., regulating) the amount of light reduction based on the patient's relative photosensitivity;

[0039] Figure 15 It demonstrates how to select and / or modify scan tables (e.g., intensity levels for image acquisition) based on pupil size and / or iris color;

[0040] Figure 16 An example series of images across multiple wavelengths are shown, captured in response to a single capture command input when the multispectral option is selected;

[0041] Figure 17 This invention is compared with two prior methods that provide FA+ICGA functionality in response to a single capture command;

[0042] Figure 18 A color fundus image including four channels is shown;

[0043] Figure 19 An example of a "channel splitting" viewing screen is shown, which provides the option to view the red, green, blue, and infrared channels separately;

[0044] Figure 20 The extinction spectra of oxidized and deoxygenated hemoglobin are shown;

[0045] Figure 21 An example of an oxygen saturation graph is provided;

[0046] Figure 22 The principles of MPOD measurement (left) and MPOD profile (right) are illustrated.

[0047] Figure 23 The design of the lightbox with eight light sources (two each of red, green, blue and infrared) is shown.

[0048] Figure 24 An example of a slit-scan ophthalmic system for fundus imaging is shown;

[0049] Figure 25 A generalized frequency-domain optical coherence tomography system for collecting three-dimensional image data of the eye, suitable for use in this invention, is shown.

[0050] Figure 26 An example of a frontal vascular system image is shown;

[0051] Figure 27 An example of a multilayer perceptron (MLP) neural network is shown;

[0052] Figure 28 A simplified neural network consisting of an input layer, hidden layers, and an output layer is shown.

[0053] Figure 29 An example convolutional neural network architecture is shown;

[0054] Figure 30 An example U-Net architecture is shown;

[0055] Figure 31 An example computer system (or computing device or computer apparatus) is shown. Detailed Implementation

[0056] Various types of ophthalmic imaging systems exist, such as those discussed below in the sections on fundus imaging systems and optical coherence tomography (OCT) imaging systems. Aspects of the present invention can be applied to any or all such ophthalmic imaging systems. In general, the present invention provides various enhancements to the operation and user interface of ophthalmic imaging systems.

[0057] Fine-line scanning for focusing and depth analysis

[0058] One aspect of the invention provides an improved method for determining image measurements for focusing applications (e.g., autofocus) and deconvolution applications (e.g., topology). As a specific example, this enhanced focusing (and deconvolution) technique and application are described as being applied to ophthalmic imaging systems using linear beams (e.g., wide-line beams) that scan a sample to create a series of image fragments that can be combined to construct a synthetic image of the sample; however, it should be understood that the invention can be applied to other types of ophthalmic imaging systems.

[0059] For example, Figure 1Simplified patterns of scan lines (e.g., slits or wide lines) are shown because they can be generated on the object being scanned. In this example, the scan lines are scanned vertically (e.g., across) to produce multiple scan lines L1 to Li in a vertical scan mode (V scan). This scan mode can be used by line-scan fundus imaging systems (or OCT-based systems), and typically, the scan lines can maintain some degree of confocal suppression of defocused light perpendicular to (e.g., along the Y-axis) the scan lines (L1 to Li), but may lack confocal suppression along the line (e.g., along the X-axis). Scan lines can also be used to enhance imaging. For example, the sharpness of the illumination band edge can be used to find the optimal focus for a line-scan system in cases where there is no significant movement of illumination during detector acquisition (typically when the scan beam is scanned in steps and is relatively stationary during acquisition). International Publication WO2018178269A1 discloses a focusing method suitable for line-scan fundus imaging systems, the entire contents of which are incorporated herein by reference. As another example of enhancement, areas on the retina not directly illuminated by the scan lines can be detected (e.g., in the captured image) to assess the background light level, such as stray light level, from the defocused areas of the eye, which can then be subtracted from the captured line image. Line scan imagers have also been combined with pupil dilation (see, for example, U.S. Patent No. 8,488,895 to Muller et al., the entire contents of which are incorporated herein by reference).

[0060] In summary, although line-scan imaging systems construct fundus images by recording images of single scan lines on the retina of the eye, the recorded scan lines can be used to extract more information. For example, line width is a direct measure of focus. This information can be used (e.g., for automatic) focusing systems or as additional input to deconvolution algorithms. Furthermore, the line position (centroid) on the detector (e.g., camera) is a direct measure of sample height (e.g., the depth of the retina at a specific point). For example, this topological information can be used to analyze blood vessels, vascular intersections, or other structures on the top of the retina (e.g., measuring the volume of a tumor).

[0061] One of the key technical details of line scanning is linewidth. If the line is too wide, it becomes difficult (if not impossible) to separate reflectance from position and linewidth, and the uncertainty in line positioning increases; for example, wide lines can automatically average information over a large area. While there are advantages to using wide-line scanning images, as described below, wide lines may not be well-suited for positioning or width analysis. This invention provides a method for reducing the effective linewidth using a scanning sequence, which can be used for focusing and / or topology analysis.

[0062] Figure 2AAn overlapping line scan sequence is shown, wherein the step size between scan lines is smaller than the width of the scan lines. For example, the scan step size S1 between the first scan line l1 and the second scan line l2 is smaller than the width W of scan line l1 (and scan line l2), such that the second scan line l2 overlaps with the scan area defined by the first scan line l1. Figure 2B It shows from Figure 2A The effect of subtracting two consecutively captured images (e.g., a scan image) from consecutive scan lines (e.g., adjacent scan lines) is equivalent to subtracting l2 from l1. Adjacent scan lines can be modeled as rectangles with small displacements S1, resulting in bilinear lines D1 / D2 with a width of displacement S1. The two lines D1 / D2 (one positive and one negative) can now be analyzed based on their width S1 and position. This effectively transforms a scan image with wide lines (e.g., l1 and / or l2) with small step sizes S1 into a scan image with (bilinear) thin lines D1 / D2 of the same step size.

[0063] To determine the line position in a captured scanned image, the intensity of the (light) at the centroid (e.g., the target region) of the captured scanned image can be calculated. This is a very fast algorithm, but it can be susceptible to noise. To improve robustness, the scanned image can be segmented into foreground and background (e.g., lines and noise) parts, and the centroid can be calculated for the foreground part. The position of the centroid can be calculated with sub-pixel precision, which increases depth sensitivity.

[0064] Figure 3A and Figure 3B This illustrates how the displacement of the centroid corresponds to changes in depth (or height) within the eye (e.g., deconvolution measurement). The displacement (of the centroid relative to the expected position) is a direct measurement of retinal topology and can be converted to actual height (metric) in a calibration system. That is, a positional shift of the line position (from the expected position) may be due to changes in the height or bumps of the scanned object (e.g., changes in eye depth, such as due to two blood vessels crossing each other). Figure 3A A non-displacement scan image of scan line 13 is shown, for example, by scanning a planar object without irregularities in depth. Figure 3B A scanned image of the same scan line l3 with a protrusion B1 is shown, due to irregularities (e.g., protrusions) on the surface of the scanned object. The height Δz of the protrusion B1 can be determined by tracking the change in the centroid position along the length of the captured scanned image, which corresponds to the height (e.g., a measure) of the difference (protrusion) on the surface of the scanned object. This method can benefit from the calibration of the camera and illumination optics.

[0065] Similar to line positioning, the linewidth can be determined from the foreground portion using the second moment of the intensity distribution (note: the centroid will correspond to the first moment). Width is a direct measure of the image's insufficiency (e.g., defocus), at least in the direction perpendicular to the scan lines. This width measure can be derived from... Figure 2B Determined within any of the thin lines created as shown.

[0066] For example, the double thin lines D1 / D2 can be converted into a single line by selecting either a positive line (e.g., D1) or a negative line (e.g., D2). For example, thin line D1 can be defined separately by selecting the maximum value of (l1-l2) and setting the rest to zero (e.g., max[(l1-l2), 0]), or thin line D2 can be defined separately by selecting the minimum value of (l1-l2) and setting the rest to zero (min[(l1-l2), 0]). Figure 2C The diagram illustrates the creation of a single thin line D1' from two scan lines l1' and l2' by defining max[(l1'-l2'), 0] and identifying the second moment σ, which defines the line width and position of D1'. Alternatively, or in addition to using the second moment σ, the line position and width can be defined by fitting a function shape (e.g., Gaussian) to the observed data (e.g., D1'). The position and width are then given by the parameters of this fit. The advantage of this approach is that existing knowledge about the scan line shape (e.g., rectangular illumination) can be taken into account. Furthermore, robust fitting methods can be used to reduce noise sensitivity.

[0067] In this way, the broad lines in the fundus imaging system (e.g., Figure 2A Those shown can be converted (e.g., digitally) into fine lines as differences in small-step scan images. The width (defocus) and position (object depth) of the fine lines can then be analyzed.

[0068] Line width and line shift are direct measures of defocus, which can be equated to the depth, or height, of the retina. These measures can be used internally by the imaging system, for example, to provide autofocus. Utilizing higher resolution, for example, reference... Figures 2A to 2C As described, this information can also be used for other purposes (e.g., depth can be used to segment blood vessels, segment optical discs, estimate tumor volume, etc.). Essentially, this method provides resolution enhancement.

[0069] This resolution enhancement can be described using the following imaging model:

[0070]

[0071] Wherein, the image represents the observed intensity, p illum It is the point spread function of illumination, P detect It is the point spread function of the detection. This represents the rectangular band of illumination. An asterisk (*) indicates convolution, and a dot (·) indicates multiplication. The model is linear in illumination, which means:

[0072]

[0073] To improve resolution, you can remove items. The basic idea behind subtracting bands is to convert the bands into Δ peaks: A simple subtraction yields two Δ values, but as mentioned above, this can be easily solved using a min-max operation. This minimizes the illumination width of the input imaging system, but still treats illumination blur as a resolution limiting factor (δ*p). illum =p illuim ).

[0074] Alternatively, weights can be used to approximate sinusoidal illumination.

[0075]

[0076] This will result in:

[0077] ∑w i Image = (((sin i )*p illum )·O)*p detect =((α) i (sin i ))·O)*p detect =α i (((sin i ))·O)*p detect

[0078] This is because the (complex) sine curve is a characteristic function of convolution (e.g., convolving sine / cosine with scaled sine / cosine). Now, to eliminate the lighting blur, we can remove the α scaling factor, which will make P... illum Deconvolution. After removing the α scaling factor, for example by intensity normalization, the sine curve (with reverse weights) can be recombined and returned to Δ or rectangle as illumination.

[0079] Using this technique, a (unidirectional) defocus map can be constructed by determining the focal point (and / or centroid location) at multiple points along a scanned image, and the defocus map can then be used as an autofocus tool or to improve deconvolution algorithms.

[0080] Alternatively, instead of analyzing individual scan lines, structured light patterns can be generated for use in 3D scanning. Since the rectangular shape of the illumination is blurred (i.e., more of a Gaussian bell shape (or convex shape)), it can be combined with alternating signs (e.g., positive and negative signs) to resample sinusoidal illumination. Three such patterns can be used for phase shifting to extract depth information. Another example is using a triangular illumination pattern, which can be approximated from the rectangular pattern via correlation.

[0081] In the example above, the removal of the α scaling factor, which can be a measure of amplitude (e.g., intensity), is used to remove (or mitigate) illumination blur. However, it has been found that the α scaling factor changes (e.g., represents or is related to) the defocus of the illumination, and therefore can provide a direct measure of the system's defocus. For example, the defocus metric can be determined based on the amplitude of sinusoidal illumination. Furthermore, the α scaling factor has been found to be frequency-dependent. Therefore, using sine curves of different frequencies will produce a series of α values ​​that can be used to represent the full (or substantially full) point spread function of the system. Since the amplitude depends on the frequency used, amplitude information can also be used to generate a pseudo modulation transfer function (MTF), which can include system optics, eye, and bandwidth. These α values ​​can form an "α map," which can be used for autofocusing purposes and / or depth measurement purposes (e.g., surface topology / topology). For example, since different frequencies result in different α values ​​(e.g., different amplitudes), a topology map can be obtained by scanning frequencies. Topology information can be determined by performing a Fast Fourier Transform (FFT) on the amplitude data on a scan-by-scan basis. Therefore, a linear combination of image bands (or scans) can be used to determine the topological map and / or defocus map of the retina.

[0082] Multi-point balanced autofocus

[0083] Any of the aforementioned methods (or other known methods for measuring defocus) that determine defocus at multiple points or locations on a fundus image (e.g., on a preliminary or preview image before imaging / scanning the eye) can be used in conjunction with a graphical user interface (GIU) to enhance the ability of a user (e.g., a human operator) to capture available images of the fundus of the eye.

[0084] Wide-field-of-view fundus cameras may have a depth of focus smaller than the variation in retinal depth, resulting in peripheral defocusing of the image. This is common in images of myopic retina. Furthermore, in cases of retinal detachment and tumors, users may have difficulty obtaining a focused image on or around the tumor or bulging retina. In conventional autofocusing methods used in ophthalmic imaging systems, the (fundus) image is focused at the center of the image, but this is unhelpful when clinicians want to focus on a specific portion of the retina (e.g., off-center) or simply want an image with optimal overall focus. This invention addresses this problem.

[0085] Typically, autofocus uses several different techniques to perform at the center of the camera's field of view (FOV). The limitation of this approach is that the focus is selected purely based on the center of the image, and therefore cannot be used for areas not at the center of the field of view. This also prevents the imaging system from focusing more of the image. Understandably, the back of the eye is not flat, so different parts of the retina within the camera's FOV may require different focus settings to focus them.

[0086] This invention can determine the focal point of different parts of the eye, each defining a focus assist position. The focus assist positions can then be used to determine the optimal focal point for capturing a new image. This invention can use any of the multiple focus assists / mechanisms to acquire focus readings (e.g., determine their defocus metrics) at multiple discrete locations (focus assist positions) in the eye, for example, as described above and / or in International Publication WO2018178269A1. The focal point at each position can be adjusted to compensate for various defocus conditions (e.g., astigmatic areas of the eye (irregularly shaped cornea), floaters, eye curvature, etc.) to indicate the patient's gaze direction. Furthermore, the system can adjust how the image is focused based on input instructions from the user (imaging system operator). For example, the weight of each of the multiple focus assist positions can be calculated based on how the user wants the image focused (e.g., balanced focus or single-point focus). Optionally, the collected information can be combined to calculate a single focal point reading. In this way, the invention can focus on any user-specified part of the eye or focus a larger area of ​​the imaging system's field of view (FOV).

[0087] Figure 4A A preview fundus image 11 is shown within a preview window (or display area) 15 on an electronic display, with multiple focus assist positions f1 to f7 overlaying the fundus image 11. It should be understood that, for illustrative purposes, seven focus assist positions are shown, and any number of multiple focus assist positions may be provided. It should also be understood that the focus assist positions f1 to f7 may be located in pre-specified (and optionally fixed) positions and span a predetermined area (fixed or variable) of the fundus image 11 or display area 15. Optionally, the focus assist positions f1 to f7 may be displayed (e.g., overlay) on the fundus image 11, or may be hidden within the display area 15. That is, the user may be allowed to view and select any number of focus assist positions f1 to f7, or the focus assist positions f1 to f7 may be hidden from the user's field of vision. The user preferably selects one or more areas (or points) within the fundus image 11 that should be focused to capture an image. The user may input this selection using any known input device or mechanism (e.g., keyboard, pointer, scroll wheel, touchscreen, voice, input script, etc.).

[0088] During operation, each focus-assist position on the retina (fundus) is illuminated via a light path entering the patient's pupil (e.g., by a corresponding focus-assist illumination, which may be a special scan line or other shaped beam of light), which is offset relative to the observation path (e.g., collector path) of the imaging system's camera, as described below. Errors in the patient's eye's refractive power cause the focus-assist illumination entering the eye to bend differently from the observation path leaving the eye, resulting in the position of each focus-assist position depending on the refractive error (e.g., different from the expected position), and can be translated into a defocus metric for the system camera's focus setting required for a sharp image of the focus-assist illumination position on the retina.

[0089] Each focusing assist f1 through f7 senses the refractive component in the pupil in its offset direction. This refractive component may differ from the refractive component in other directions, thus determining the focal point for optimal image sharpness at its location, especially when astigmatism is present. The ordinary eye has multiple degrees of astigmatism, and fundus cameras (or other ophthalmic imaging systems) without adjustable astigmatism typically only correct for astigmatism when the ordinary eye is rotated to the center of the camera. When the eye rotates relative to the camera, for example, at any given fixed target, this system can resolve the misalignment between the eye's astigmatism and the fundus camera's astigmatism correction. This can be accomplished using the following formula:

[0090]

[0091] in:

[0092] f k =The focus point given the focus auxiliary position,

[0093] H = Horizontal eye orientation coefficient

[0094] X Fixation = Fixed target's X position,

[0095] X focus aid =Focus on the X position of the auxiliary,

[0096] V = Vertical eye orientation coefficient

[0097] Y Fixation = Fixed target's Y position,

[0098] Y focus aid =Focus on the auxiliary Y position,

[0099] Therefore, the user can choose to optimize the system to focus on any given point / region within the system's FOV, for example, by selecting a given focus assist position or any position on the preview fundus image 11. If the point / region selected by the user for focusing does not correspond to a specific focus assist position (f1 to f7), which may or may not be displayed on the preview image 11, then the optimal focus for the user-selected point can be determined by an appropriate combination of weighted focus assists (f1 to f7) relative to the position of the user-selected point. Alternatively, the user can choose to use a predefined focus balance on the preview image, which can apply predefined weights to each focus assist position. For example, the balanced focus can be calculated as follows:

[0100]

[0101] Where: w k = Weight of the focal position,

[0102] f′ k =The astigmatism correction focus at the location,

[0103] Figure 4B An example is provided of how a user can freely select any point in preview image 11 to focus on. In this example, the user-input focus is indicated by an asterisk. The user can input the desired focus using an input device (e.g., a click point) or by using a touchscreen. In this case, the optimal focus of the user-input point can be determined as a weighted combination of focus aid positions relative to the user-input point. For example, the individual distances d1 to d7 from the input click point to the respective focus aid positions f1 to f7 are determined. The focus of the click point (star) can be determined as follows:

[0104] in

[0105] Where dk = the distance between the desired focusing position and the given focusing auxiliary position.

[0106] In summary, this method acquires focus readings at multiple discrete locations on the eye (e.g., f1 to f7). The focus readings at each location are then adjusted to account for the gaze angle. A weight is calculated for each focus location relative to the user-input focus (the weight can depend on the distance from the given discrete focus location to the user-selected focus). Finally, focus adjustment for the image capture can be calculated based on the weighted focus at each discrete focus location for the user-selected focus.

[0107] Note that while the system operator can provide a fixation point to the patient, the patient may not be able to maintain focus on the provided fixation point. That is, the patient's true fixation point may differ from the fixation point provided by the system. Therefore, as described above, when determining the optimal focusing settings, the system can further provide a mechanism for determining the patient's true gaze direction. For example, the system can use infrared imaging to obtain a test image, which may be a preview image 11. The system can then identify the optic nerve in the test (e.g., preview) image and determine the patient's gaze angle based on the location of the identified optic disc. If the patient has optic dysplasia, which is often associated with optic nerve hypoplasia, optic nerve identification can be complex. Therefore, the system can further search for other landmarks indicating the gaze angle, such as the fovea or a distinctive vascular system. The patient's gaze can then be determined based on the location of the landmarks identified in the infrared preview image and the system's known imaging settings relative to the patient's eye (e.g., illumination and viewing angle). Using infrared preview images is advantageous because the eye is insensitive to infrared wavelengths and infrared preview images can be captured continuously without interfering with the eye, for example, for system alignment purposes. Specific landmarks can be identified by using specific imaging processing algorithms or machine learning techniques (e.g., deep learning). Infrared (preview) images are superior to visible (preview) images because landmarks in visible images can be more difficult to discern when capturing certain types of images, such as fundus fluorescein (FA) or indocyanine green angiography (ICGA) imaging. However, if the infrared preview image is captured substantially close in time to (e.g., immediately before or simultaneously with) the capture of FA and / or ICGA images, the identification location of the optic disc determined from the infrared preview image can be directly applied to the captured FA and / or ICGA images. It should be understood that the current use of infrared imaging for determining eye fixation angles can be used in conjunction with other ophthalmic imaging systems, such as OCT. Further note that the infrared images used herein are for fixation detection, thereby imaging the fundus (or back) of the eye, and not for eye tracking imaging the pupil (or front) of the eye.

[0108] U-Net locates the center of the optic disc in wide-field fundus images.

[0109] This article provides an example of using deep learning to identify the center of the optic disc, for instance, in a wide-field-of-view fundus image. As mentioned above, the center of the optic disc can be used to identify the patient's gaze direction.

[0110] The optic nerve head (ONH) is one of the most prominent landmarks observed in fundus images, and the ability to automatically locate the ONH in fundus image processing is highly desirable. ONH localization algorithms can help locate the fovea or other landmarks by providing a basis for pathological searching by applying a normalized offset or region of interest (ROI). Most available literature on ONH localization involves locating the ONH in color (e.g., visible light) fundus images. However, there are various types of fundus images and ONH localization techniques suitable for color images, many of which are not applicable to other types of images. For example, locating the optic nerve in fluorescein angiography (FA) images and indocyanine green angiography (ICGA) images is very difficult because the appearance of the ONH changes at different stages of the angiography examination as the dye passes through the ocular vascular system. This paper presents a U-Net-based deep learning architecture applied to infrared (IR) preview images to robustly locate the ONH at all stages of FA and ICGA images.

[0111] Traditional ONH localization algorithms use handcrafted features / filters. For example, in "Automatic Localization of the Optic Disk, Fovea, and Retinal Blood Vessels from Digital Colour Fundus Images," Br. J. Ophthalmol., Vol. 83, No. 8, pp. 902-910, 1999, Sinthanayothin et al. used variance filters to explore rapid changes in intensity due to the presence of blood vessels in the optic disc. However, this method is only applicable to color fundus images and has been tested on small datasets. Studies have also shown that this method may fail for fundus images with leukoplakia and prominent choroidal vessels. Another method, presented by Akita et al. in "A Computer Method of Understanding Ocular Fundus Images," Pattern Recognition, Vol. 15, No. 6, 1982, pp. 431-443, uses vessel tracking to localize ONH. However, this algorithm relies on the successful application of vessel segmentation algorithms. Since the algorithm depends on the vascular system, abnormal vascular systems may lead to false detections. Mendels et al. (in “Identification of the Optic Disk Boundary in Retinal Images Using Active Contours”, Proc. Irish Machine Vision Image Processing Conf., September 1999, pp. 103-115) used active contours for ONH detection. A drawback of this method is its reliance on initial contours generated by morphological filtering, which is not robust to changes in image quality. The algorithm also suffers from reduced time efficiency due to its reliance on active contours. Sekar et al., in “Automated Localisation of Optic Disk and Fovea in Retinal Fundus Images”, 16th European Signal Processing Conference, Lausanne, 2008, pp. 1-5, presented another method using morphological localization followed by a Hough transform to find the ONH. This algorithm uses the Hough transform to fit a circle within a given radius range. It identifies image regions with high grayscale intensity variations and uses these as the basis for finding the ONH.While this may not be a problem in normal images (e.g., images of a healthy eye), in images with pathology, the optic disc may not be the only structure with high intensity variations, which could lead to errors. In Zhu et al.'s "Detection of the Optic Nerve Head in Fundus Images of the Retina Using the Hough Transform for Circles," J Digit Imaging, 2009; 23(3): 332-41, edge information and the Hough transform are used to find the ONH. However, this algorithm may not be robust to variable image quality and ONH-related pathology because it relies on being able to extract good edge information (image quality) and match it to a circle of a certain radius. Another approach described by Rangayyan et al. in "Detection of the Optic Nerve Head in Fundus Images of the Retina with Gabor Filters and Phase Portrait Analysis," Journal of digital imaging, 23(4), 438-53, uses Gabor filters to detect vessels and applies phase portrait modeling to attempt to locate the ONH. This algorithm is specifically based on the characteristic of the ONH as a convergence point of retinal vessels. The performance of this algorithm will be affected if blood vessels or convergence nodes are obscured by image quality or pathology. Several methods using deep learning algorithms to locate the optic nerve using convolutional neural networks (CNNs) have also been reported. CNNs are described below.Examples of using CNNs to find the optic nerve are described below: D. Niu et al., “Automatic Localization of Optic Disc Based on Deep Learning in Fundus Images,” 2nd IEEE International Conference on Signal and Image Processing (ICSIP), Singapore, 2017, pp. 208-212; Gonzalez-Hernandez D. et al., “Segmentation of the Optic Nerve Head Based on Deep Learning to Determine its Hemoglobin Content in Normal and Glaucomatous Subjects,” J. Clin Exp Opthamol 9:760; and H.S. Alghamdi et al., “Automatic Optic Disc Abnormality Detection in Fundus Images: A Deep Learning Approach,” Proceedings of the Ophthalmic Medical Image Analysis International Workshop (OMIA 2016 and MICCAI 2016 Athens Greece), Iowa Research Online, pp. 17-24, October 2016. The full contents of all the aforementioned published works are incorporated herein by reference. However, it is noteworthy that in all of these published works, the optic nerve head is detected in color fundus images. That is, all of these methods utilize features derived from color fundus images (e.g., ONH) or training algorithms (in deep learning).

[0112] The ONH localization method of the present invention differs from the above examples in that it is applicable to localizing ONHs in image types other than color images (e.g., imaging modalities), such as in FA and ICGA images.

[0113] In FA and ICGA angiography, a series of time-lapse images are captured after a photoactive dye (e.g., the fluorescent dye for FA and the indocyanine green dye for ICGA) is injected into the subject's bloodstream. A specific light frequency of selection is used to excite the dye, thus capturing high-contrast images. As the dye flows through the eye, different parts of the eye emit bright light (e.g., fluorescence), allowing the progression of the dye to be discerned, thus revealing the blood flow through the eye. Parts with more dye will glow brighter, making the captured image series typically dark initially with little or no dye in the eye, increasing in brightness as more dye enters and flows through the eye, and then dimming again as the dye leaves the eye. Therefore, the appearance of the ONH changes over time as the dye passes through the blood vessels at different stages of the angiography examination, which often makes it very difficult to locate the ONH in FA and ICGA images. For illustrative purposes, Figure 5 Two vascular images, A1 and A2, taken at different stages (or during) of angiography are provided. As shown, the ONH in image A1 looks very different from the ONH in image A2, which complicates the creation of filters applicable to all stages of angiography. Therefore, most available literature on ONH localization is usually limited to color fundus images and avoids the difficulty of locating ONH in FA or ICGA images. However, this paper proposes a U-Net-based deep learning architecture applied to infrared preview images to robustly locate ONH in all stages of FA and ICGA images or any other type of image / scan.

[0114] Typically, an infrared preview image of the fundus / retina is collected before performing an examination scan (image). As mentioned above, the infrared preview image can be used to adjust the focus and, as described below, to ensure precise alignment of the system with the patient and to ensure that the correct area of ​​the eye is imaged. Therefore, the infrared preview image is collected / recorded just before capturing FA and / or ICGA images. It is proposed here that the location of the ONH found in the infrared preview image before capturing the FA or ICGA image (or any other type of image) will be the same (unchanged) in the captured FA or ICGA image. That is, the location of the OHN in the infrared preview image is used for ONH localization in the FA and ICGA images. This invention indirectly localizes the OHN in the FA and / or ICGA images by identifying the ONH in an infrared preview image of the retina (using an algorithm trained using infrared images) taken immediately before the start of capturing FA and / or ICGA images and by migrating the location of the ONH from the infrared preview image to the FA and / or ICGA images.

[0115] Previous conventional methods were limited to locating ONHs in color fundus images, and identifying ONHs in color fundus images is relatively straightforward (given a large number of training images) compared to locating ONHs in dynamic imaging modalities (imaging types) (e.g., FA and ICGA images). Collecting a large set of training images (e.g., more than 1000 images) for each different stage of FA and / or ICGA imaging is very difficult, especially since it involves the injection of photoactive dyes. This invention circumvents some of these difficulties.

[0116] This invention aims to locate ONHs (or any other predefined configuration or structure) without requiring a large number of hard-to-obtain angiographic images (e.g., FA and / or ICGA images) to train a machine learning model. In one embodiment of the invention, transfer training can be used to build an existing machine learning model trained to locate ONHs in color images by extending its operation to include infrared images. Transfer learning is a machine learning method in which a machine learning model developed for a first task is reused as a starting point for training another machine model on a second task. Considering the large computational and time resources required to develop neural network models for these problems, a pre-trained model is used as a starting point. For example, the current machine learning model can be trained first using color images (or a single color channel extracted from a color image), and then the training can be transferred to infrared images. Using transfer training is another novel feature of this invention. For example, the algorithm can be trained in two steps: first, training is performed using color images and / or color channels extracted from color images (e.g., approximately 2000 training images); second, the training is transferred to infrared images (e.g., approximately 1000 images). That is, the algorithm can be trained first on color images and / or individual color channel images (e.g., monochrome images), and then the training can be completed on infrared images. One reason for this two-step approach is that a large number of color images are readily available, which is a common occurrence (e.g., 2000 images is a good number for training all parameters in a neural network), and a small number of infrared images are available for training. For example, red channel images extracted from color images can be used first to train the network in the first step, and infrared preview images can be used to transfer training of the machine model on the same network. In this two-step approach, for example, after training the neural network using color images, a given number of early layers are frozen (e.g., their weights / parameters are frozen so that they do not change in subsequent training), and the neural network is again trained only on its remaining unfrozen layers (or only on a given number of the last layers, e.g., two layers) using infrared preview images. The weight / parameter training in the second step is more constrained than in the first step. However, since the availability of infrared preview images is generally lower than that of color images, there may not be enough infrared preview images to train all weights / parameters in the network. For this purpose, it is beneficial to train the machine model using commonly available modal images (e.g., color images) and freeze some layers, and then use available infrared preview images to transfer only a few layers of the trained neural network.

[0117] Alternatively, if a sufficient number of infrared (preview) images are available for training (e.g., over five thousand images), the machine model can be trained individually on infrared images, or in a single training step on a combination of infrared and color (or color channel) images. However, since color fundus images are generally more widely available than infrared images, transfer training can still be used to improve performance. Experiments show that using this step-by-step process results in a significant improvement in accuracy compared to a one-step process.

[0118] Figure 6 A segmentation and localization framework according to the invention is provided for finding ONHs in FA / ICGA images during operation (in a deployed system). The system may begin by capturing an infrared (e.g., preview) image in block B1, followed by capturing a fluorescence and / or ICGA image in block B3. Alternatively, in block B3, an image of a different type than the infrared image (e.g., FAF, color, etc.) may replace (or be appended to) the FA and / or ICGA image, and the framework can equally effectively identify ONHs in other types of images. The captured infrared image from block B1 may be submitted to an optional preprocessing block B5, which may apply anti-aliasing filters and downsampling before being submitted (as input test images) to a trained U-Net deep learning machine model for processing (block B7).

[0119] Below is an example of a suitable U-Net architecture. The U-Net in block B7 is trained to receive a test (IR) image and identify ONHs in the test image. Preferably, as described above, the U-Net is trained in a two-step process using a set of training infrared fundus images (e.g., infrared preview images) and a set of color fundus images / channel-extracted color fundus images. The U-Net can first be trained using optic nerve sac segments with color images and / or channels extracted from color images (e.g., by manually labeling or using an automated algorithm to identify OHHs in color images to segment image segments for ONHs). This first training step assigns initial weights / parameters to all layers of the U-Net. The trained weights are then transferred using the infrared preview images with optic nerve sac segments generated by manual labeling or automated algorithms. That is, the U-Net can be trained using some initial layers frozen with the initial weights / parameters and the remaining unfrozen layers (e.g., the last two layers) trained with the infrared preview images. Thus, the U-Net receives a test (IR) image and outputs multiple possible ONH segments identified in the test image.

[0120] Block B9 receives all possible ONH segments identified by U-Net and selects candidate ONH segments based on size criteria. These criteria may include excluding all received ONH segments (e.g., area or diameter) that are larger than a first threshold and smaller than a second threshold. Acceptable size ranges can be determined based on the criterion range.

[0121] The selected candidate ONH segments are then submitted to block B11, where one candidate ONH segment is chosen as the actual ONH segment. This selection can be based on the shape of the candidate ONH segment. For example, the candidate ONH segment with the most rounded shape can be selected as the actual ONH segment. This can be determined by performing a roundness measurement on each candidate ONH region, for example, by determining the difference between the minor axis and the major axis (radius) of each candidate ONH segment. The ONH segment with the smallest difference between the minor axis and the major axis can be selected as the actual ONH segment. If multiple ONH segments are found to have the same roundness measurement value, the segment with the largest roundness measurement value can be designated as the actual ONH region.

[0122] After identifying the true ONH segment, the next step is to determine its location in the infrared (preview) image. The location of the candidate ONH segment in the infrared preview image can be determined by finding the centroid of the candidate ONH segment (block B13).

[0123] Finally, in block B15, the identified ONH segmentation and positioning can be applied to the captured infrared preview image of block B1 and / or the captured FA and / or ICGA image of block B3.

[0124] Figure 7A It shows the use of Figure 6 The system identifies the location of the ONH region ONH-1 in the infrared preview image IR-1. Figure 7B It shows the capture Figure 6 An FA image FA-1 (e.g., an angiographic image) is captured shortly after an infrared preview image IR-1, for example, within a time period after capturing image IR-1, which is determined to be short enough that the eye will not make substantial movements of the ONH. Figure 7B The local ONH region ONH-1 is shown migrated from image IR-1 to FA image FA-1. In this way, ONHs can be quickly and efficiently identified at any stage of angiography (FA or ICGA). Optionally, if the imaging system supports it, additional infrared preview images can be continuously captured through multiple (or all) stages of angiography, so that the location of the ONH can be continuously updated based on the continuously captured infrared preview images if needed.

[0125] The captured FA or ICGA image sequence can then be displayed, with or without highlighting the identified ONH regions. However, displaying a series of captured FA or ICGA images is challenging in any case.

[0126] Automatic adjustment of angiography image sequences

[0127] Angiographic imaging, whether fluorescein angiography (FA) or indocyanine green angiography (ICGA), requires capturing a range of light intensities from dark to bright as the injected dye passes through the eye. Setting up the camera to capture such intensity variations in an image sequence can be difficult, resulting in some images being saturated (e.g., too much signal gain causing the image to be too bright or faded) or too dark. Therefore, when acquiring image sequences for FA or ICGA examinations, the system operator typically adjusts the camera acquisition settings manually. For example, the operator may monitor the currently captured image and adjust the camera to increase or decrease the gain in subsequent images in the image sequence. That is, most fundus cameras require the operator to adjust the illumination brightness, sensor gain, or aperture size during angiography to accommodate the diversity of signal levels and attempt to capture images with good brightness for display (i.e., unsaturated or excessively dark images). This has several obvious drawbacks. The first drawback is that the need to adjust camera settings (e.g., control brightness) can be distracting for the operator. The second drawback is that operator errors or inaction can lead to underexposed or overexposed images. The third drawback is that the resulting sequence of angiographic images may be brighter than each other, meaning that the true changes in image intensity over time (which indicate blood flow and may be relevant to disease assessment) are partially or completely blurred. Some ophthalmic imaging systems offer automatic brightness control (or sensitivity control or automatic gain control) to reduce the burden on the system operator in adjusting the camera. This type of automatic control can help overcome the first and second drawbacks, but it generally does not solve the third. In fact, because automatic brightness control is usually applied on an individual image basis, all images in the capture sequence can be individually adjusted to provide similarly bright images. This exacerbates the third drawback because it further obscures the naturally occurring relative changes in brightness between images in the sequence.

[0128] This invention provides automatic adjustment of image brightness while maintaining the true proportional change in signal intensity (between different images) during angiography. Furthermore, this invention can provide a trend of light intensity levels relative to time, which may be clinically relevant for certain eye diseases.

[0129] First, this ophthalmic imaging system preferably uses a high dynamic range (light) sensor to capture image sequences during angiography (e.g., FA or ICGA). A novel feature of this invention is the use of a sensor with a sufficiently wide dynamic range to avoid underexposure and saturation during the capture of all images in an angiography sequence, without requiring adjustments to the system / hardware (e.g., no adjustment to illumination brightness, sensor gain, and / or aperture size). The system stores the original captured images (e.g., for future processing), but each image can optionally be adjusted for optical display without altering its original data. The system can query regions of an image to determine an “image brightness factor” that optimally adjusts the image for optimal display. The system can further query the “image brightness factor” of each image acquired as part of an angiography series to determine a “sequence brightness factor” that adjusts all images in the series for optimal display, constrained by maintaining relative brightness between images in the series. In this case, the same “sequence brightness factor” can be applied to each image to be displayed, rather than a separate “image brightness factor” for each image. Using either the "sequence brightening factor" or the raw data, this system can further determine and record the trend of image intensity levels relative to elapsed time since dye injection.

[0130] This system is optimized for angiography. It provides sufficient dynamic range to accommodate the fluorescence range required for angiography. Therefore, the risk of overexposed images is negligible, eliminating the need to adjust illumination intensity or detector sensitivity during angiography. The operator is no longer distracted by the need to monitor the brightness or sensitivity of the image capture sequence. Because there is no adjustment to the light sensor, the captured image sequence accurately represents the true relative changes in image intensity acquired throughout the angiographic images, which is important in many diseases / pathologies. For example, venous or arterial occlusion may slow the rate at which dye passes through the retina, meaning that the rate of decrease in image intensity over time may be slower. The relative differences in fluorescence between images are also important for diagnosing inflammatory conditions. In contrast, existing ophthalmic imaging systems obscure the true relative changes in image intensity due to manual or automatic adjustments to brightness or sensitivity settings during data acquisition, changes that the examining physician may not be aware of.

[0131] The intensity of angiographic images varies considerably depending on the dye's transition period, the dye dosage, retinal pigmentation, the condition of the ocular media, and the disease state. Most fundus imaging systems have limited dynamic range, therefore adjustments to illumination or detector sensitivity settings may be necessary during angiography to avoid underexposure or overexposure. The natural variations in fluorescence signal during angiography (typically strongest early and gradually diminishing late) contain information about the dynamics of blood transport through the retina, which can be useful for clinicians assessing eye health.

[0132] Figure 8 A framework is provided for processing individual images from a series of images captured as part of an angiography examination. If enabled by the system operator, for example by selecting the "Auto-adjust" checkbox in the acquisition window, the captured images are optimized (in terms of grayscale range) during acquisition for individual display (e.g., not as part of a group or sequence of images representing the passage of time in an angiography examination). This optimization prevents individually displayed images from being too dark or oversaturated, which could hinder a technician's proper assessment of image quality. First, image 20 is captured. This may include capturing multiple scan lines (e.g., band data) 22 and stitching (assembling) the band data 24 into a raw (unadjusted) image 20. For each image 20, the pixel intensity of the original acquisition is stored. An optimized white level for displaying the image is determined for the raw image 20. This white level can be determined by querying a portion 26 or all of the acquired image data 20 and calculating the 99.9th percentile (or some other appropriately chosen percentile) pixel intensity. The determined optimal white level is stored in image metadata 28 associated with the raw image 20. The metadata 28 also includes a black level for displaying the image, which can be preset to a level that shields the electronic noise of the (light) sensor and can be provided by a configuration file. The image metadata 28 effectively provides automatic brightening information (e.g., based on its white level) for individual images 20. The "image brightening factor" of an individual image 20 can be based on its individually determined white and / or black levels.

[0133] When a single image 20 is selected for display, it undergoes image adjustment 30 (e.g., automatic brightening) based on its associated metadata 28 (e.g., based on its stored white and / or black levels). The adjusted image can then be output to a viewing screen 32 on an electronic display 34. Optionally, the adjusted image can also be sent to inspection software 36 for further processing.

[0134] After all imaging for a given angiography is completed (e.g., capturing all time-lapse images), the time-lapse images can be displayed as a group. For example, a group of images can be displayed on viewing screen 32. Although the corresponding metadata for each image provides information for individual display adjustments, displaying each image according to its individually optimized display settings will result in many of them having a similar appearance, making it difficult to distinguish visual changes due to the injected dye over time between the images. For example, Figure 9 A sequence 40 of time-lapse images of FA is shown, in which each individual image is adjusted (e.g., brightened) according to its corresponding individual white level. Note that although the actual fluorescence signal decreases over time, the later images appear as bright as the earlier images, therefore the later images should be darker than the earlier images.

[0135] The visual changes from one image to another caused by dye transitions are available in their respective raw data, but not in their corresponding display-optimized forms. Displaying each image in chronological order according to the raw data form will preserve the diurnal transition information, but details in each image may be difficult to discern due to the lack of display optimization. For example, Figure 10 The original FA image sequence 42 (corresponding to) without any brightness adjustment applied to any image is shown. Figure 9 (e.g., showing the raw signal level of each image). Therefore, many images are blurry and difficult to evaluate.

[0136] For group display, the objective of this invention is to adjust all images in a time-lapse sequence as a group to display as much detail as possible while preserving the relative differences in overall brightness between the individual images due to dye transitions (e.g., visual variations). One embodiment of the invention does not base the display on the corresponding white levels of all images in the sequence (e.g.,...). Figure 9 Instead of individually brightening all images in the sequence, the algorithm examines the white levels of all images in the sequence (e.g., the white levels stored in the corresponding metadata for each image) and finds the maximum white level, referred to here as "wmax". An overlay scaling factor (e.g., a scaling factor spanning all images in the sequence) is defined as the maximum number of available image intensity levels divided by wmax (e.g., 255 / wmax). This same overlay scaling factor is then applied to all images in the sequence, such that the relative brightness differences between images are preserved. Figure 11 The FA image sequence 44 with overlay adjustment is shown (corresponding to) Figure 10 Each image is brightened by the same overlay scaling factor (e.g., 255 / wmax). As shown, these images retain the relative intensity variations that represent the true fluorescence signal, but... Figure 10The original FA image sequence 42 is different. Figure 11 The image is brightened to optimize visible details.

[0137] In an alternative embodiment, instead of applying the same coverage scaling factor to all images in the sequence, each image can be individually adjusted based on its corresponding stored white level, but with its brightness setting adjusted taking into account the group's wmax. In this way, the relative brightness of images in the sequence (e.g., the group) is compressed, making them generally brighter and easier to interpret, while still preserving some similarity in fluorescence signal changes during angiography. This compression brightening technique can be achieved by applying a custom brightness scaling factor to individual images in the sequence, which can be calculated as:

[0138]

[0139] Where B(T) is a customized brightness scaling applied to the image acquired at time T during the time-lapse sequence of the angiography examination, w(T) is the white level of the image acquired at time T, ∝ is a compression parameter selected within a predetermined range (e.g., within the range [0, 1]), and wmax is the coverage scaling factor, for example, as described above. Setting the compression parameter ∝ = 0 is equivalent to applying as follows: Figure 11 The overlay scaling factor wmax shown is unchanged, while setting ∝=1 is equivalent to automatically brightening the image based on its individually optimized display settings (unmodified), such as... Figure 9 As shown.

[0140] The compression parameter ∝ can be set to a fixed value during a separate angiography examination (e.g., preferably, the compression parameter ∝ does not change during image sequence acquisition), and ideally, it can be set to a value selected by a clinician (e.g., a system operator). That is, input signals from the system operator can modify the relative brightness compression between images in the sequence by adjusting the value of the compression parameter ∝. For example, the system operator can adjust the compression parameter ∝ by using a brightness slider, by using increment / decrement buttons, and / or by directly inputting a value for ∝.

[0141] Figures 12A to 12F This provides some examples of how this compression brightening technique affects the brightness of individual images in a time-lapse sequence for different values ​​of the compression parameter ∝. Figures 12A to 12F In each image, the left side shows the resulting sequence of bright images, and the right side shows a curve of the angiographic image brightness versus time. Figure 12A In this case, the compression parameter ∝ is set to zero, which is equivalent to brightening all images using the same overlay scaling factor (e.g., "wmax" in the current case), as referenced above. Figure 11As shown. For comparison purposes, since the original capture data was also stored, multiple curves are displayed. Curve 50 shows the relative brightness between the original images in the captured image series over time (e.g., from zero to ten minutes). Curve 52 shows the relative brightness between images when each image is automatically brightness-adjusted according to its stored white level, which generally results in similar brightness between the images. Curve 54 corresponds to the relative brightness between images when all images are brightness-adjusted according to the same wmax value. As shown, the relative differences in wmax curve 54 follow the relative differences in original image curve 50.

[0142] exist Figure 12B In this case, the compression parameter ∝ is set to 1, which is equivalent to automatically brightening each image individually based on its stored white levels, as shown in the reference above. Figure 9 As discussed above, curve 50 shows the relative brightness between the original images, curve 54 shows the relative brightness between images using the same brightness factor wmax, and curve 52 shows the relative brightness between images over time, with each image brightened based on its corresponding white level. As shown in curve 52, automatic brightening (individually) based on the corresponding white level of each image hides the differences in relative brightness between images over time.

[0143] Figures 12C to 12F This illustrates how modifying the compression parameter ∝ affects the brightness of an image sequence for different values ​​of ∝. Figures 12C to 12F In this configuration, the compression parameter ∝ increases in increments of 0.2, ranging from... Figure 12C In the case of ∝=0.2 to Figure 12F In this case, ∝ = 0.8. Figures 12C to 12F Each of these provides a curve showing the resulting relative brightness between images in the time-lapse sequence (56C to 56F, respectively). For comparative purposes, Figures 12C to 12F Each of the following diagrams illustrates curve 50 (original image), curve 52 (automatically brightened to their maximum value based on their stored white levels), and curve 54 (image brightened based on wmax). As shown by curves 56C to 56F, increasing ∝ has the effect of increasing the brightness of the displayed image sequence compared to the original image curve 50, while maintaining the relative difference between images similar to the original image curve 50.

[0144] Graphical User Interface

[0145] As described above, this system provides extended focusing capabilities. In addition to allowing the user to select from multiple predetermined points / regions for focusing, or to freely select any random point for focusing, the invention also provides additional features for capturing multiple imaging types (imaging modes) and / or for modifying a specific imaging type to suit a specific patient. For illustration, Figure 13An example of a graphical user interface (GUI) for an ophthalmic imaging device is provided, highlighting some of these additional features. The GUI provides an acquisition (interface) screen 21 for acquiring (e.g., capturing) patient images / scans. Optionally, patient data 23 identifying the current patient can be displayed at the top of the acquisition screen 21. The acquisition screen 21 can provide multiple viewports (e.g., framed areas on the display for viewing information), display elements, and icons to guide the instrument operator in setting acquisition parameters, capturing and viewing images, and analyzing data. For example, a preview window 15 with multiple focus assist positions f1 to f7 overlaid on a preview (fundus) image (e.g., an infrared preview image) 11 can be provided within the viewport. As described above, the system operator can focus by selecting any point (e.g., a target focus area) on the preview image 11 using a user input device (e.g., an electronic indicator device, a touchscreen, etc.).

[0146] Additional focusing options, such as (step adjustment) focus button 33 or focus slider 35, may be provided if needed. In an exemplary embodiment, the target focus area of ​​the preview window 15 and focus button 33 or focus slider 35 may be mutually exclusive, such that using one negates the other. For example, using focus button 33 or focus slider 35 may negate any previously manually selected target focus area on the preview window 15, and conventional global focus adjustment may be applied to the entire fundus image 11 based on focus button 33 and / or focus slider 35. Similarly, assigning (e.g., selecting) a target focus area on the viewing screen 15 may override any focus setting of focus button 33 or focus slider 35. Alternatively, the target focus area of ​​the preview window 15 and focus button 33 or slider 35 may work collaboratively. For example, if a user selects a target focus area to focus on within the fundus image 11, the system may adjust the focus at the selected focus area accordingly, as described above. However, if the user wishes to further adjust the focus settings of the selected target focus area beyond the range automatically provided by the system, the user can use the focus button 33 and / or the slider 35 to further adjust the focus of the selected target focus area. Therefore, the system can provide manual focus for any selected target focus area within the fundus image 11 (e.g., within the preview screen 15).

[0147] The acquisition screen 21 can display one or more pupil streams 39 of real-time images (e.g., infrared images) of the eye's pupil from different viewpoints, which can be used to facilitate alignment between the ophthalmic imaging system and the patient. For example, overlay guide lines such as a translucent band 47 or a crosshair 49 can be superimposed on the live stream to convey a target location or range at which the patient's pupil should be positioned for good image acquisition. The overlay guide informs the system operator of the current offset between the target pupil location for imaging and the current location of the patient's pupil center, so that corrective actions can be taken.

[0148] Once the operator is satisfied with the capture settings, they can capture an image by using the capture start input, such as capture button 37.

[0149] As an image is captured, it can be displayed as a thumbnail in the capture bay section 41 of the acquisition screen 21. A display filter 43, which provides a drop-down list of filter options (e.g., lateral, scan (or image) type, etc.), can be used to refine (select or limit) the thumbnail displayed in the capture bay 41. If an imaging mode requiring the acquisition of multiple images is selected (e.g., an ultra-wide field of view (UWF) option that can synthesize two captured images, or an automatic synthesis option that can synthesize a predetermined number of captured images offset from each other), the capture bay 41 can be pre-filled with placeholder icons 45 (e.g., dashed circles) indicating placeholders for the images to be captured. As each desired image is captured, the placeholder icon 45 can be replaced with a thumbnail of the captured image to provide the user with an indication of the execution status of the imaging mode.

[0150] However, it should be understood that multiple options can be provided to the system operator before capturing images (e.g., initiating a scanning operation). For example, additional display elements and icons can be provided to allow the system operator (user) to select the type of image (scan) to be acquired and to ensure proper focus and alignment of the ophthalmic imaging system on the patient. For example, the side of the eye being examined can be displayed / selected, for example, via the side icon 25, highlighting which eye (right or left) is being examined / imaged (e.g., right eye (OD) or left eye (OS)).

[0151] Various user-selectable scanning (e.g., imaging) options can be displayed on the acquisition screen 21. Scanning options may include an FOV button 27 for the user to select the imaging FOV for one or more images to be acquired. The FOV option button may include a wide field (WF) option for standard single-image operation, an ultra-wide field (UWF) option for capturing and synthesizing two images, an "auto-synthesis" option that provides a preset synthesis sequence for collecting and synthesizing a predetermined number (e.g., four) of images, and / or a user-definable "synthesis" option that allows the user to submit a user-specified synthesis sequence. Additionally, checkboxes may be provided to indicate whether the user wishes to perform stereo imaging and / or use an external fixation target. Selecting an external fixation target disables the internal fixation target in the system during imaging.

[0152] Fundus imaging systems can support a variety of scan types (e.g., imaging modalities). Examples of scan types can include color imaging, which provides a full-color view of the fundus, but requires strong light that may be uncomfortable for the patient. Another type of image is infrared imaging, which is invisible to the patient and therefore more comfortable. Infrared images are typically monochrome and are more helpful in identifying details than color images. Some types of tissue have naturally occurring photosensitive molecules that cause them to fluoresce at specific wavelengths, such as 500 to 800 nm, and these tissues can be used to identify potential pathological areas / structures that might not be easily defined in color or infrared frequencies. Imaging the eye using selected specific wavelengths to make the target type of tissue fluoresce is called fundus autofluorescence imaging (FAF). FAF imaging at different frequencies can cause different types of tissue to fluoresce automatically, thus providing different diagnostic tools for different types of pathology. Other imaging modalities include FA and ICGA, which, as mentioned above, utilize injected dyes to track blood flow across a series of images.

[0153] In this example, the acquisition screen 21 provides a scan type section 29 for selecting from multiple imaging modalities. Examples of selectable scan types may include: color (e.g., true color imaging using visible light, such as red, green, and blue light components / channels); infrared (imaging using invisible infrared light, such as by using an infrared laser); FAF-green (green-excited fundus autofluorescence) and FAF-blue (blue-excited fundus autofluorescence); FAF-NIA (fundus autofluorescence using near-infrared light); FA (fluorescein angiography, such as by injecting a fluorescein dye into the patient's bloodstream); FA-NIA (fluorescein angiography using near-infrared light); and ICGA (indocyanine green angiography, such as by injecting an indocyanine green dye into the patient's bloodstream).

[0154] The acquisition screen 21 can also provide input for various image capture settings, such as for dilated (mydriatic) eyes, non-dilated (non-mydriatic) eyes, and / or potentially photosensitive eyes (e.g., patients with photophobia). Users can select between mydriatic (Myd) and non-mydriatic (non-Myd) imaging modes by selecting the appropriate mode via button 31. Optionally, the system can automatically select (e.g., auto-select) which mode is appropriate based on the condition of the eye.

[0155] Before initiating the image acquisition sequence, the operator can utilize several photophobia-related settings. Optionally, the system can identify / label the patient as a candidate previously diagnosed with photophobia or light sensitivity. In either case, the system can alert the operator to this fact and suggest adjusting the imaging sequence accordingly. For example, if the patient is a light sensitivity candidate, a warning icon / button 61 can be illuminated, changed color, and / or flashed to attract the operator's attention. The operator can then select the default light sensitivity settings by selecting a button and / or manually selecting from a list of light intensity modification options 63, each option adjusting the applied light intensity during image acquisition. Adjusting (adjusting) image acquisition settings based on the patient's medical condition (e.g., photophobia) is an example of patient-adjusted diagnosis.

[0156] Patient-regulated diagnosis

[0157] Ophthalmic imaging systems can use flash, for example, in scanning mode, to acquire images of the fundus. For example, the CLARUS 500 manufactured by Carl Zeiss Meditec... TM Flashes may be used to capture high-resolution, wide-field-of-view images. Certain types of flashes (e.g., flashes used for FAF-blue mode imaging) can be bright and have short-term visual effects on patients, including speckled, diminished, and / or color-diminished vision. For patients experiencing photophobia (e.g., eye discomfort or pain due to light exposure), the light source of an ophthalmic imaging system can be significantly more effective and trigger short-term effects, including pain, migraines, nausea, tearing, and other conditions.

[0158] Typically, when imaging a patient with known light sensitivity, the system operator can take steps to reduce the patient's exposure. For example, the system operator can diagnose a patient with light sensitivity (e.g., image the patient's eye) by avoiding dilation of the patient's eye and performing the test (e.g., imaging) in a dark room environment. If possible, the system operator can try to reduce the system's light intensity and use a smaller FOV for imaging, but this may result in reduced image quality.

[0159] This paper proposes an ophthalmic imaging system with diagnostic modalities (e.g., imaging modalities) for patients with photosensitivity, and / or a system capable of identifying patients who may be candidates for photosensitivity.

[0160] This system can provide imaging modes 'adjusted' for photosensitive patients (e.g., photophobia mode or photosensitivity mode), which reduces the light output of the ophthalmic imaging system but provides feasible image quality for diagnosis (e.g., minimal). Figure 14 An example procedure is provided to adjust (e.g., modify) the amount of light reduction based on the patient's relative photosensitivity. This may include an optional automatic mode for patients identified (e.g., in their medical records) as photosensitivity, and within a graphical user interface (GUI), such as within an image acquisition screen (e.g., Figure 13 Buttons 61 and / or 63 provide the system operator with a manual selection / deselection option. Therefore, the process can begin with step 60, which involves automatically or manually setting the photosensitivity mode, for example, by reviewing a patient record. Optionally, the assessment of the patient's photosensitivity level can be adjusted / determined by providing additional information / parameters (step 62), such as iris size, iris color, and / or bright / dark retinal detection.

[0161] Scan-based ophthalmic imaging systems can use scan tables that specify the frequency, intensity, and / or duration of light applied to a scan line at a given location on the fundus. Typically, one scan table is constructed / defined for each imaging type. However, implementation of this photophobia mode may include generating one or more scan tables using a portion of the current / standard light settings (e.g., default light settings) for a given imaging type. Scan tables may provide reduced light intensity and reduced duration (e.g., shorter flash duration). Scan tables may also include unique scan settings based on the patient's pupil size and / or iris color (e.g., additional intensity and duration adjustments). The minimum feasible image quality based on the minimum exposure in each imaging mode / type (color, FAF-B, FAF-G, IGC, FA) can be predetermined based on clinical testing. For example, a patient with photosensitivity can be examined (imagined) using a photosensitivity mode by allowing the system to automatically select one of the alternative scan tables based on the amount of the patient's photosensitivity (step 64). Alternatively, the user can select predefined alternative scan tables. The system then scans (images) the patient using the selected scan table (step 66) and then evaluates the acquired images (step 68). This evaluation can be performed automatically by the system, for example, by determining the image quality of the acquired images, or by the system operator. If the obtained image quality is insufficient, in step 70, if necessary, additional images can be acquired using a step function that increases the intensity to the normal mode intensity. For example, this can be achieved by using... Figure 13Enter 63 to manually select the step size increase.

[0162] This system may include optional components / parameters (e.g., as shown in optional step 62), such as automatic pupil size detection, iris color, and automatic bright / dark retinal detection (e.g., using an infrared preview image of the retina). Lighter retinas are more prone to photophobia than darker retinas. For a given image acquisition operation, a direct tabular correlation can be established between different combinations of photophobia indicators and corresponding light / duration settings (e.g., scan table selection). Alternatively, a mathematical equation can be created for the optimal (minimum) power setting based on a combination of image quality influence parameters. Furthermore, this system can be further integrated with DNA data to automatically adjust diagnostic device settings based on markers or sets of markers that can indicate potential photosensitivity. This can be further extended to a mathematical evaluation of the optimal set of markers.

[0163] Figure 15 This demonstrates how to select and / or modify the scan table (e.g., the intensity level used for image acquisition) based on pupil size and / or iris color. Optionally, the system's default intensity and duration settings can be set to the minimum settings associated with light-colored eyes, and the intensity (and / or duration) can be increased for less light-sensitive eyes. Further optionally, the system's default intensity and duration settings can be set to the minimum settings associated with light-colored eyes, and if the captured image quality is not greater than the minimum, the intensity and / or duration can be automatically increased in predetermined step increments to capture additional images within the current image capture sequence until an image of minimum quality is obtained.

[0164] This patient-modulated diagnosis can be based on known patient conditions or diseases, such as adding a dry eye mode for OCT imaging or utilizing multiple operating modes to enhance existing ophthalmic imaging systems. This method can also be applied to patients with mental health conditions where discomfort (e.g., stress or anxiety) can be triggered by the intensity of flashes or brightness. For example, the patient-modulated diagnosis of this invention can be applied to patients with post-traumatic stress disorder (PTSD) to minimize patient discomfort and adverse conditions.

[0165] return Figure 13 As shown in the scan type section 29, this system can also provide the ability to acquire multiple image types (different imaging modes) with a single button. As explained more fully below, this can be achieved through a scan table that supports multiple imaging modalities within a single image capture sequence. For example, a multispectral option / button 63 can be provided for capturing multiple images across wavelengths of the spectrum. Figure 16An example series of images across multiple wavelengths is shown, captured in response to a single capture command input when multispectral option 63 is selected. Another example could be the blood oxygenation measurement option / button 65, which determines a measurement of oxygen in the blood by capturing multiple images at different selected wavelengths, as will be explained more fully below. A color + infrared option / button 67 is also provided, which captures color and infrared images in a single image capture sequence in response to a single image capture button. Another option could be the FA + ICGA option / button 69, for capturing FA and ICGA images or image sequences in response to a single image capture command. This document provides a preferred method for implementing the FA + ICGA option.

[0166] In response to a single "capture" command, acquire FA and ICGA images sequentially.

[0167] Fluorescein angiography (FA) and indocyanine green angiography (ICGA) can be performed as part of a single examination. The ICG dye can be injected immediately before or after fluorescein, or a mixed pellet can be used. FA images can be acquired using a blue / green illumination source, and ICGA images can be acquired using a near-infrared illumination source. Combining the two modalities into a single examination reduces patient time spent in the chair. However, acquiring a sufficient number of FA and ICGA captures at different stages of angiography is a burden for both the patient (especially those who are more photosensitive) and the operator. This burden can be alleviated by using an imaging device capable of acquiring FA and ICGA images simultaneously within a single exposure sequence.

[0168] Acquiring both FA and ICGA images simultaneously presents challenges for system design because the light source and filter requirements differ for the corresponding modalities. Due to the risk of eye movement, exposure needs to be completed within a short time (~100ms) while still obtaining sufficient image quality for the physician to assess the condition of the eye.

[0169] For any instrument that delivers visible light to the retina, patient comfort is a crucial consideration for ensuring compliance; patients with photophobia may move excessively or blink during imaging, or even refuse to complete the examination. As explained by J.M. Stringham et al. in “Action spectrum for photophobia,” JOSAA, 2003, 20(10), 1852–1858, photophobia sensitivity increases with decreasing wavelength. The short-wavelength light (470–510 nm) commonly used in FA is far more uncomfortable than the near-infrared light used in ICGA. It has been further shown that the perceived flash duration increases with flash duration (Osaka, N., “Perceived brighmess as a function of flash duration in the peripheral visual field,” Perception & Psychophysics, 1977, Vol. 22(1), pp. 63–69). Therefore, to help patient comfort, it is desirable to minimize the duration of potentially painful light exposure (and the flash energy).

[0170] Figure 17 This invention is compared to two prior methods that provide FA+ICGA functionality in response to a single capture command. The first prior method 82 uses blue and infrared laser sources to simultaneously deliver light to the eye and collects the emitted fluorescence simultaneously using two independent photodetectors. The second prior method 84 uses staggered (or alternating) illumination and detection: alternating laser sources such that one row of the FA image is scanned, then one row of the ICGA image is scanned, then the second row of the FA image is scanned, and so on. The advantage of both methods is that the acquired FA and ICGA images will be accurately co-registered even with eye movement. This could be beneficial for image post-processing and visualization tools.

[0171] However, both of these previous methods also have their drawbacks. The truly simultaneous method 82 requires relatively complex hardware; for example, each FA and ICGA collection path requires a separate detector and filter. The interleaved method 84 may have disadvantages in terms of patient comfort, as the blue / green fluorescent illumination source (which is much more cumbersome to patients than the ICGA light source) is perceived by the patient as continuous throughout the entire exposure duration (i.e., prolonged flash). This can cause photophobia responses (e.g., blinking, pupillary constriction, eye movements, and Bell's phenomenon).

[0172] The currently preferred method 86 responds to a single "capture" command from the user by sequentially capturing FA and ICGA images. That is, within a single image capture sequence (e.g., within a single flash time), a complete ICGA image (or image sequence) is captured, followed immediately by a complete FA image (or image sequence).

[0173] In a preferred embodiment, the ICGA image is scanned first because the required infrared source will not cause discomfort to the patient. Once the ICGA image acquisition scan is complete, the FA scan begins. This differs from the previous two methods that perform FA+ICGA capture simultaneously, which use two light sources to illuminate the eye simultaneously (82) or alternate between multiple FA and ICGA acquisitions to establish images of both (84).

[0174] Because the current sequential FA+ICGA method 86 reduces the perceived time interval of visible light exposure (compared to the staggered FA+ICGA method 84), it can cause less patient photophobia and reduce the probability of blinking and / or eye movement artifacts. That is, the time between the start and end of visible light (e.g., for FA) is much shorter than that of staggered flashes delivering the same energy, thus minimizing the intervals at which patients may experience photophobia. This can potentially improve patient comfort and image quality in angiography. This method 86 also avoids the complex hardware requirements of the first method 82. In fact, this method 86 can be implemented using a properly defined scan table.

[0175] In addition to the current FA+ICGA, this system can provide additional multispectral fundus imaging modes 63-67, as referenced. Figure 13 As shown.

[0176] Single-shot multispectral imaging with slit scanning ophthalmoscope

[0177] Multispectral fundus imaging can provide enhanced visibility and discrimination of retinal structure and function by integrating information obtained from images using multiple light bands (e.g., visible and invisible reflectance and autofluorescence).

[0178] True-color fundus imaging is the standard for retinal examination. Near-infrared light can penetrate deeper into the retina because its longer wavelengths are less easily absorbed by blood and melanin. Therefore, infrared fundus imaging can sometimes provide supplementary information to true-color imaging, making subretinal features (e.g., small warts under the retinal pigment epithelium) more visible.

[0179] Multispectral imaging methods can be used to non-invasively detect blood oxygen in the retina. As explained by JMBeach et al. in "Oximetry of retinal vessels by dual-wavelength imaging: calibration and infuence of pigmentation" in Journal of Applied Physiology, 86, Vol. 2 (1999): 748-758, two or more excitation wavelengths (one of which is insensitive to oxygenation in blood absorption) can be used to capture reflective images, which can be used to infer blood oxygen saturation.

[0180] In fundus autofluorescence (FAF) imaging, the acquired image information varies depending on the spectral content of the excitation and collection bands. These differences can be used to extract clinical information about retinal conditions. For example, collecting FAF at individual bands can be used to distinguish fluorophores in the retina, as described in M. Hammer's "Color Autofluorescence Imaging in Age-Related Macular Degeneration and Diabetic Retinopathy," Investigative Ophthalmology & Visual Science, 49, Vol. 13 (2008): 4207-4207. Similarly, changing the FAF excitation band can be used to estimate the optical density of macular pigment, as in Delori... As described in C et al., “Macular pigment density measured by autofluorescence spectrometry: comparison with reflectionometry and heterochromatic flicker photometry”, JOSAA, 18, No. 6 (2001): 1212-1230.

[0181] The full text of all references mentioned herein is incorporated herein by reference.

[0182] In most commercial fundus imaging systems, acquiring multispectral data typically requires multiple acquisitions, which is time-consuming for the operator and uncomfortable for the patient. This system offers a pathway to extended wide-field slit-scan fundus imaging, providing multispectral fundus imaging using only a single-capture protocol.

[0183] Some existing fundus imaging systems offer imaging at multiple wavelengths, but these are typically provided as separate scan modes, each requiring a dedicated capture input. Therefore, for example, to acquire RGB color and infrared images, two separate scans are needed, and the resulting images may not be registered (due to unavoidable eye movement / realignment between scans). Existing methods for pulse oximetry employ a simultaneous dual-wavelength approach. A Bayer filter in the color sensor is used to separate the wavelength channels, and analysis software provides the calculation of vascular hemoglobin oxygen saturation. Previous systems have been used in academic research that employ two separate FAF imaging modalities to assess macular pigmentation. CLARUS TM The 500 also offers two different FAF excitation / detection bands. However, each mode requires a separate dedicated capture.

[0184] This paper presents a technique for simultaneous color and infrared imaging. In one application, sequential multispectral scanning eliminates the need to acquire infrared images as additional scans. Instead, each true-color scan automatically provides a perfectly registered infrared image. Additional image information can be provided to the user via a 4-band channel splitting function on the instrument software's viewing screen. For example, Figure 18 A color (e.g., true color) fundus image 88-C consisting of four channels is shown. That is, in addition to the typical red channel 88-R, green channel 88-G, and blue channel 88-B, the current color image 88-C also has an infrared channel 88-IR.

[0185] This ophthalmic imaging system uses sequential light sources and a monochrome camera to image the retina. This is achieved by using a scan table that provides instructions for the hardware to turn on a specific light source for each slit. This method offers great flexibility in scanning and sequencing different types of light.

[0186] The sequence of scan table commands is used to group the acquired slit images by individual color, allowing them to be assembled into individual images. These images are then combined in processing to form a true-color image. The scan table can be arranged in rows and parameter tables, used by the system to identify specific scan locations, excitation wavelengths (e.g., channels), etc. For example, in a typical scan table for true-color mode, each row commands slit acquisition, and the "Channel" column tracks the light color of that particular acquisition, which is then used to assemble the image color channels individually. In this case, the channels could be R, G, and B to represent red, green, and blue LED acquisitions. To extend the capture of the color image to include infrared image capture, the scan table can be modified to include additional infrared acquisitions (represented as "I" in the "Channel" column of the scan table), such that each R, G, and B sequence in the "Channel" column of the color image is expanded to an R, G, B, and I sequence. This effectively provides a 4-channel multispectral image. Therefore, as part of the scan, each true-color image will also provide a perfectly registered infrared image. It is important to note that infrared light will not cause discomfort to the patient and therefore will not significantly affect the flash brightness. For FAFs with multiple excitations, green and blue illumination will be similarly ordered in the scan table, with long-pass barrier filters (e.g., >650 nm for collecting the emitted FAF signal).

[0187] Because all color channel component images are acquired in a single flash sequence, they will be perfectly registered (any eye movement is prohibited during scanning). Each channel view can be selected in the Channel Segmentation viewing screen. For example, Figure 19 An example of a "channel segmentation" viewing screen 90 is shown, which provides option 92 for use by the system operator, allowing the operator to view the red, green, blue, and infrared channels separately. In another embodiment, infrared image information can be combined with a color image to form a composite image. For example, the red channel of the color image can be linearly combined with infrared image information to provide a "depth-enhanced" true-color fundus image.

[0188] Unlike previous systems that provided infrared imaging as a unique capture mode, this technology can automatically provide infrared images as part of every true-color capture. The advantage of this is that it eliminates the need for additional chair time, causes no significant additional discomfort to the patient, and the resulting infrared images are perfectly registered with the color images. This feature may help increase the interest of retinal specialists in infrared imaging, as the additional information provided by infrared is offered to them free of charge.

[0189] This paper also proposes a technique for measuring retinal vascular oxygenation. Oxygen saturation can be non-invasively detected in the retina using multispectral reflectance imaging. The basic method requires only two excitation wavelengths. One excitation wavelength is chosen such that it is absorbed by hemoglobin (which binds to oxygen) and is independent of oxygen saturation (isotonic) levels. The other excitation wavelength should exhibit a significant difference in absorption between oxyhemoglobin and deoxyhemoglobin.

[0190] Figure 20 The extinction spectra of oxygenated and deoxygenated hemoglobin are shown. Suitable isoextinction points are at 390, 422, 452, 500, 530, 545, 570, 584, and 797 nm. The wavelengths used should be as close as possible to minimize wavelength-dependent differences in light transmission within the retina (e.g., scattering). Currently preferred methods for pulse oximetry provide a wide field of view and utilize slit scanning techniques and sequential illumination (rather than separation using Bayer filters, for example, as in existing methods).

[0191] This system achieves blood oxygenation measurement using a dedicated excitation filter within a slit wheel. The filter can be a dual-bandpass or stacked filter with a 5-10 nm notch to allow light of two wavelengths to pass through. The wavelength notch center can be selected to suit dual-wavelength blood oxygenation measurement, and can be provided by independent LEDs in the lightbox (e.g., 570 nm and 615 nm). The method for performing blood oxygenation measurement can be as follows:

[0192] a) In single-scan mode, green and red light are sorted to collect two images of 570nm and 615nm in a single capture (without recording them).

[0193] b) Divide the blood vessels.

[0194] c) Apply the method used by Beach et al. [1] (to calibrate changes in vessel diameter and retinal pigmentation).

[0195] d) Display the oxygen saturation graph. Figure 21 An example of an oxygen saturation graph is provided.

[0196] This system can also be used to measure macular pigment optical density. In this case, by sequencing blue and green excitations, a dual-wavelength excited autofluorescence image can be obtained in a single scan. The obtained data allows for an objective measurement of macular pigment optical density (MPOD). MPOD may help in clinically determining a patient's risk of developing age-related macular degeneration (AMD). AMD is a leading cause of blindness in the Western world. It is a progressive and incurable disease affecting the macular region of the retina, leading to loss of central high-resolution color vision. This disease primarily affects people over 50 years of age. In Western countries, the incidence is generally higher due to older age, dietary habits, and milder average eye pigmentation. MPOD measurement can offer physicians several advantages, especially in screening settings. MPOD measurement will provide healthcare providers with the possibility of offering dietary supplements (many are available on the market) and expanding their reach to patients' dietary health. Currently, the preferred method uses sequential FAF capture utilizing multiple excitation bands, enabling objective measurement of macular pigment optical density using a single image capture.

[0197] In a preferred embodiment, autofluorescence images with dual-wavelength excitation are obtained in a single scan by sequencing blue and green excitations using a >650 nm barrier filter typically used for FAF-green. A non-mydriatic mode can be used to allow small pupil imaging in OD screening settings, and patient comfort regarding the flash can be optimized by limiting the scan to the macula and surrounding area. The obtained data should allow for objective measurement of macular pigment optical density.

[0198] Figure 22 The principle of the MPOD measurement (left) and the MPOD distribution (right) are illustrated. The FAF-green image overlays the normalized difference between FAF-green and FAF-blue (in a predefined color, such as yellow). This normalized difference can be used to infer the MPOD distribution.

[0199] This article presents some considerations for multispectral imaging. Additional light sources of different wavelengths can be incorporated into the lightbox design for single-exposure, sequential, multispectral imaging. This can provide multispectral imaging characteristics with single-exposure (rather than multiple exposures) and perfectly registered image output.

[0200] Figure 23 A lightbox design with eight light sources (two each of red, green, blue, and infrared) is shown. Typically, in non-mydriatic mode, only one of each light source is necessary for imaging, although using two of each color in a larger pupil offers some advantages (for a better image SNR and more effective suppression of reflections). However, the lightbox design can be modified to include additional sources. This can provide… Figure 13The multispectral capability 63, using various combinations of LED (light-emitting diode) sources and filters, provides imaging in the wavelength range of 500-940 nm, such as... Figure 16 As shown.

[0201] Return to Figure 13 The system provides multiple multi-function (combination) keys 63-69 that respond to a single control input from the system operator to capture more than one type of image. These combinations are predefined and presented as fixed multi-function keys 63-69, but optionally, the operator can define custom multi-function keys that respond to a single capture command to capture multiple images of different imaging modalities. For example, a settings window can be provided to the operator to link additional image types to a single image type selection. Note that different types of images may require different types of filters, and if the system only supports the use of one filter at a time, this may limit the number of image types that can be linked together (e.g., only image types that require similar filters can be linked together). The system can accordingly limit the available selections. Alternatively, if the system has multiple detectors, each with its own filter, a beam splitter can be used to transmit light of different wavelengths via their respective filters to different detectors. This will increase the number of image types that can be linked and captured in response to a single capture command.

[0202] The following provides a description of various hardware and architectures applicable to this invention.

[0203] Fundus imaging system

[0204] Two types of imaging systems used for fundus imaging are flood illumination imaging systems (or flood illumination imagers) and scanning illumination imaging systems (or scanning imagers). A flood illumination imager, for example, uses a flash lamp to simultaneously flood the entire field of interest (FOV) of the sample with light and captures a full-frame image of the sample (e.g., the fundus) with a full-frame camera (e.g., a camera with a sufficiently large two-dimensional (2D) light sensor array to capture the desired FOV as a whole). For example, a flood illumination fundus imager would flood the fundus and capture a full-frame image of the fundus in a single image capture sequence from the camera. A scanning imager provides a scanning beam that scans across an object (e.g., the eye), and as the scanning beam scans across the object, it images at different scanning locations, producing a series of image fragments that can be reconstructed (e.g., synthesized) to produce a composite image of the desired FOV. The scanning beam can be a point, a line, or a two-dimensional region, such as a slit or a wide line.

[0205] Figure 24An example of a slit-scanning ophthalmic system SLO-1 for imaging the fundus F is shown. The fundus F is the inner surface of the eye E opposite the lens (or optic disc) CL and may include the retina, optic disc, macula, fovea, and posterior pole. In this example, the imaging system is in a so-called “scan-to-de-scan” configuration, wherein a scan line beam SB passes through the optical components of the eye E (including the cornea Cm, iris Irs, pupil Ppl, and lens CL) to scan the fundus F. In the case of a floodlight fundus imager, a scanner is not required, and light is applied immediately across the entire desired field of view (FOV). Other scanning configurations are known in the art, and the specific scanning configuration is not important to the present invention. As shown, the imaging system includes one or more light sources LtSrc, preferably a multicolor LED system or a laser system, wherein the light collection rate has been appropriately adjusted. An optional slit Slt (adjustable or static) is located in front of the light source LtSrc and can be used to adjust the width of the scan line beam SB. Furthermore, the slit Slt can remain stationary during imaging or can be adjusted to different widths to allow for different levels of confocality and different applications, for specific scans, or to suppress reflections during scanning. An optional objective lens ObjL can be placed in front of the slit Slt. The objective lens ObjL can be any existing lens, including but not limited to refractive, diffractive, reflective, or hybrid lenses / systems. Light from the slit Slt passes through the pupil splitter SM and is directed to the scanner LnScn. It is desirable to bring the scanning plane and the pupil plane as close together as possible to reduce vignetting in the system. Optional optics DL can be included to manipulate the optical distance between the images of the two components. The pupil splitter SM transmits the illumination beam from the light source LtSrc to the scanner LnScn and reflects the detection beam from the scanner LnScn (e.g., reflected light returning from the eye E) toward the camera Cmr. The task of the pupil splitter SM is to split the illumination and detection beams and help suppress system reflections. The scanner LnScn can be a rotating galvanometer scanner or other types of scanners (e.g., piezoelectric or voice coil, microelectromechanical systems (MEMS) scanners, electro-optic deflectors, and / or rotating polygon scanners). Depending on whether pupil segmentation is performed before or after the scanner LnScn, the scanning can be divided into two steps, where one scanner is in the illumination path and the other is in the detection path. A specific pupil segmentation setup is described in detail in U.S. Patent No. 9,456,746, the entire contents of which are incorporated herein by reference.

[0206] From the scanner LnScn, an illumination beam passes through one or more optics, in this case a scanning lens SL and an ophthalmic or eyepiece OL, which allows the pupil of the eye E to image onto the system's image pupil. Typically, the scanning lens SL receives the scanning illumination beam from the scanner LnScn at any of a plurality of scanning angles (incident angles) and produces a scanning line beam SB with a substantially flat surface focal plane (e.g., a collimated optical path). The ophthalmic lens OL can focus the scanning line beam SB onto the fundus F (or retina) of the eye E and image the fundus. In this way, the scanning line beam SB produces a transverse scanning line across the fundus F. One possible configuration of these optics is a Keplerian telescope, in which the distance between two lenses is chosen to create an approximately telecentric intermediate fundus image (4-f configuration). The ophthalmic lens OL can be a single lens, an achromatic lens, or an arrangement of different lenses. As those skilled in the art will know, all lenses can be refractive, diffractive, reflective, or a combination of these. The focal lengths of the ophthalmic lens (OL), scanning lens (SL), and the dimensions and / or forms of the pupillary divider (SM) and scanner (LnScn) can vary depending on the desired field of view (FOV). Therefore, an arrangement can be envisioned where multiple components can switch in and out of the optical path, for example, by using triggers, motorized wheels, or detachable optical elements, depending on the FOV. Since changes in the FOV result in different beam sizes across the pupil, pupillary splitting can also change with the FOV. For example, a 45° to 60° field of view is typical or standard for fundus cameras. Higher fields of view (e.g., 60°–120° or higher wide field of view FOVs) are also feasible. Wide field of view FOVs may be ideal for combining wide-line fundus imaging (BLFI) with another imaging modality (e.g., optical coherence tomography (OCT)). The upper limit of the field of view can be determined by the accessible working distance combined with the physiological conditions surrounding the human eye. Because the typical human retina has a field of view (FOV) of 140° horizontally and 80°–100° vertically, it may be desirable to have an asymmetrical field of view for the highest possible FOV on the system.

[0207] The scanning line beam SB passes through the pupil Ppl of the eye E and is directed to the retina or fundus surface F. The scanner LnScn1 adjusts the position of the light on the retina or fundus F to illuminate a series of lateral positions on the eye E. Reflected or scattered light (or emitted light in the case of fluorescence imaging) is guided back along a similar path to the illumination to define the collection beam CB on the detection path to the camera Cmr.

[0208] In the "scan-de-scan" configuration of this exemplary slit-scan ophthalmic system SLO-1, the light returning from the eye E is "de-scanned" by the scanner LnScn on its way to the pupillary segmentation mirror SM. That is, the scanner LnScn scans the illumination beam from the pupillary segmentation mirror SM to define a scan illumination beam SB passing through the eye E; however, since the scanner LnScn also receives the returning light from the eye E at the same scanning position, it has the effect of de-scanning the returning light (e.g., canceling the scanning action) to define a non-scanning (e.g., stable or stationary) collected beam from the scanner LnScn to the pupillary segmentation mirror SM, which folds the collected beam toward the camera Cmr. At the pupillary segmentation mirror SM, reflected light (or, in the case of fluorescence imaging, emitted light) is separated from the illumination beam onto a detection path pointing toward the camera Cmr, which may be a digital camera with a light sensor to capture an image. An imaging (e.g., objective) lens ImgL may be located in the detection path to image the fundus onto the camera Cmr. Similar to the objective lens ObjL, the imaging lens ImgL can be any type of lens known in the art (e.g., refractive, diffractive, reflective, or hybrid lens). Additional operational details, particularly methods for reducing artifacts in images, are described in PCT Publication WO2016 / 124644, the entire contents of which are incorporated herein by reference. The camera Cmr captures the received images, for example, creating an image file, which can be processed by one or more (electronic) processors or computing devices (e.g., ...). Figure 31 The computer system shown further processes the data. Therefore, the collected beam (returning from all scan positions of the scan line beam SB) is collected by the camera Cmr, and the full-frame image Img can be composed of a synthesis of the individually captured collected beams, for example, through synthesis. However, other scanning configurations are also conceivable, including configurations where the illumination beam scans on the eye E and the collected beam scans on the camera's light sensor array. Several embodiments of a slit scanning ophthalmoscope, including various designs where the returned light sweeps across the camera's light sensor array and where the returned light does not sweep across the camera's light sensor array, are described by reference to PCT Publication WO2012 / 059236 and U.S. Patent Publication 2015 / 0131050, which are incorporated herein by reference.

[0209] In this example, the camera Cmr is connected to a processor (e.g., a processing module) Proc and a display (e.g., a display module, computer screen, electronic screen, etc.) Dspl. Both can be part of the imaging system itself, or they can be part of separate dedicated processing and / or display units, such as a computer system, where data is transmitted from the camera Cmr to the computer system via cable or a computer network including wireless networks. The display and processor can be integrated. The display can be a conventional electronic display / screen or a touchscreen type, and can include a user interface for displaying and receiving information to and from the instrument operator or user. The user can interact with the display using any type of user input device known in the art, including but not limited to a mouse, knob, button, pointer, and touchscreen.

[0210] When imaging, it may be desirable to keep the patient's gaze fixed. One way to achieve this is to provide a fixed target that the patient can be guided to look at. The fixed target can be inside or outside the instrument, depending on which area of ​​the eye is being imaged. Figure 24 An embodiment of an internally fixed target is illustrated. In addition to the main light source LtSrc used for imaging, a second optional light source FxLtSrc, such as one or more LEDs, can be positioned such that a light pattern is imaged onto the retina using a lens FxL, a scanning element FxScn, and a reflector / mirror FxM. The fixed scanner FxScn can move the position of the light pattern, and the reflector FxM guides the light pattern from the fixed scanner FxScn to the fundus F of the eye E. Preferably, the fixed scanner FxScn is positioned such that it lies in the pupillary plane of the system, allowing the light pattern on the retina / fundus to move according to the desired gaze position.

[0211] By selecting filtering elements based on the light source and wavelength used, the slit-lamp ophthalmoscope system can operate in different imaging modes. When imaging the eye with a series of colored LEDs (red, blue, and green), true-color reflective imaging (similar to the imaging observed by clinicians when examining the eye with a handheld or slit-lamp ophthalmoscope) can be achieved. The image for each color can be built progressively with each LED turned on at each scanning position, or each color image can be captured individually as a whole. These three color images can be combined to display a true-color image or displayed individually to highlight different features of the retina. The red channel best highlights the choroid, the green channel highlights the retina, and the blue channel highlights the anterior retina. Furthermore, light of specific frequencies (e.g., individual colored LEDs or lasers) can be used to excite different fluorophores in the eye (e.g., autofluorescence), and the resulting fluorescence can be detected by filtering out the excitation wavelength.

[0212] Fundus imaging systems can also provide infrared (IR) reflected images, for example, by using an infrared laser (or other infrared light source). The advantage of infrared mode is that the eye is not sensitive to infrared wavelengths. This allows users to continuously capture images without interfering with the eye (e.g., in preview / alignment mode), assisting the user during instrument alignment. Furthermore, infrared wavelengths increase penetration into tissues and can provide improved visibility of choroidal structures. Additionally, fluorescein angiography (FA) and indocyanine green angiography (ICG) imaging can be performed by collecting images after injecting a fluorescent dye into the subject's bloodstream.

[0213] Optical coherence tomography system

[0214] In addition to fundus photography, autofluorescence (FAF), and fluorescein angiography (FA), ophthalmic images can be created using other imaging modalities, such as optical coherence tomography (OCT), OCT angiography (OCTA), and / or ophthalmic ultrasound. This invention, or at least a portion thereof, as understood in the art, with minor modifications, can be applied to these other ophthalmic imaging modalities. More specifically, this invention can also be applied to ophthalmic images generated by OCT / OCTA systems that produce OCT and / or OCTA images. For example, this invention can be applied to frontal OCT / OCTA images. Examples of fundus imagers are provided in U.S. Patent Nos. 8,967,806 and 8,998,411, examples of OCT systems are provided in U.S. Patent Nos. 6,741,359 and 9,706,915, and examples of OCTA imaging systems are provided in U.S. Patent Nos. 9,700,206 and 9,759,544, all the entire contents of which are incorporated herein by reference. For completeness, this article provides an exemplary OCT / OCTA system.

[0215] Figure 25A generalized frequency-domain optical coherence tomography (FD-OCT) system suitable for collecting three-dimensional image data of the eye is illustrated. The FD-OCT system OCT_1 includes a light source LtSrc1. Typical light sources include, but are not limited to, broadband light sources or scanning laser sources with short time coherence lengths. The beam from the light source LtSrc1 is typically guided by an optical fiber Fbr1 to illuminate a sample, for example, the eye E; a typical sample is tissue within the human eye. The light source LtSrc1 can be a broadband light source with a short time coherence length in the case of spectral domain OCT (SD-OCT), or a wavelength-tunable laser source in the case of scanning source OCT (SS-OCT). The light can typically be scanned by a scanner Scnr1 located between the output of the optical fiber Fbr1 and the sample E, such that the beam (dashed line Bm) is scanned laterally (in the x and y directions) over the region of the sample to be imaged. In the case of full-field-of-view OCT, a scanner is not required, and the light travels through the entire desired field of view (FOV) at once. The light scattered from the sample is collected and typically fed into the same optical fiber Fbr1 used to guide the illumination. Reference light from the same light source LtSrc1 propagates along a separate path, in this case involving fiber Fbr2 and a retroreflector RR1 with an adjustable optical delay. Those skilled in the art will recognize that a transmission reference path can also be used, and the adjustable delay can be placed in the sample or reference arm of the interferometer. The collected sample light is typically combined with the reference light in a fiber coupler Cplr1 to form an optical interference in the OCT photodetector Dtctr1 (e.g., a photodetector array, digital camera, etc.). Although a single fiber port leading to detector Dtctr1 is shown, those skilled in the art will recognize that various designs of the interferometer can be used for balanced or unbalanced detection of the interference signal. The output from detector Dtctr1 is provided to processor Cmp1 (e.g., a computing device), which converts the observed interference into depth information of the sample. The depth information can be stored in a memory associated with processor Cmp1 and / or displayed on display Scn1 (e.g., a computer / electronic display / screen). The processing and storage functions can be located within the OCT instrument or in an external processing unit (e.g., Figure 31 The functions performed on the computer system shown are transmitted to the external processing unit. This unit can be dedicated to data processing or performing other very general tasks, rather than being dedicated to the OCT device. The processor Cmp1 may include, for example, a field-programmable gate array (FPGA), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a system-on-a-chip (SoC), a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), or a combination thereof, which performs some or all of the data processing steps before being transmitted to the main processor or in parallel.

[0216] The sample and reference arms in the interferometer can be composed of bulk optics, fiber optics, or hybrid bulk optics systems, and can have different architectures, such as Michelson, Mach-Zehnder, or designs based on common paths known to those skilled in the art. The beams used herein should be interpreted as any carefully guided optical path. Instead of a mechanical scanning beam, an optical field can illuminate a one-dimensional or two-dimensional region of the retina to generate OCT data (e.g., see U.S. Patent No. 9,332,902; D. Hillmann et al., “Holoscopy-holographic optical coherence tomography”, Optics Letters 36(13): 23902011; Y. Nakamura et al., “High-Speed ​​three-dimensional human retinal imaging by line field spectral domain optical coherence tomography”, Optics Express 15(12): 71032007; Blazkiewicz et al., “Signal-to-noise ratio study of full-field Fourier-domain optical coherence tomography”, Applied Optics 44(36): 7722(2005)). In time-domain systems, the reference arm needs to have an adjustable optical delay to generate interference. Balanced detection systems are commonly used in TD-OCT and SS-OCT systems, while spectrometers are used for the detection port of SD-OCT systems. The invention described herein can be applied to any type of OCT system. Various aspects of the invention can be applied to any type of OCT system or other types of ophthalmic diagnostic systems and / or multiple ophthalmic diagnostic systems, including but not limited to fundus imaging systems, visual field testing devices, and scanning laser polarimeters.

[0217] In Fourier domain optical coherence tomography (FD-OCT), each measurement is a real-valued spectral interferogram (Sj(k)). The real-valued spectral data typically undergoes several post-processing steps, including background subtraction and dispersion correction. The Fourier transform of the processed interferogram produces a complex-valued OCT signal output. The absolute value |Aj| of this complex-valued OCT signal reveals the scattering intensity distribution for different path lengths; therefore, scattering is a function of depth (z-direction) in the sample. Similarly, phase... It can also be extracted from complex-valued OCT signals. The scattering distribution as a function of depth is called an axial scan (A-scan). A set of A-scans measured at adjacent locations in a sample produces a cross-sectional image (tomogram or B-scan) of the sample. A set of B-scans acquired at different lateral locations on the sample constitutes a data volume or cube. For a given amount of data, the term fast axis refers to the scanning direction along a single B-scan, while slow axis refers to the axis along which multiple B-scans are collected. The term "cluster scan" can refer to a single cell or block of data generated by repeated acquisition at the same (or substantially the same) location (or region) for analyzing motion contrast, which can be used to identify blood flow. A cluster scan can consist of multiple A-scans or B-scans acquired at approximately the same location on the sample at relatively short time intervals. Because the scans in a cluster scan belong to the same region, the static structure remains relatively unchanged between scans, while the motion contrast between scans that meet predetermined criteria can be identified as blood flow. Various methods for generating B-scans are known in the art, including but not limited to: along the horizontal or x-direction, along the vertical or y-direction, along the diagonals of x and y, or in a circular or spiral pattern. A B-scan can be in the xz dimension, but can be any cross-sectional image including the z-dimensional dimension.

[0218] In OCT angiography or functional OCT, analytical algorithms can be applied to OCT data collected at the same or nearly the same sample locations on the sample at different times (e.g., cluster scans) to analyze motion or flow (see, for example, U.S. Patent Publications 2005 / 0171438, 2012 / 0307014, 2010 / 0027857, 2012 / 0277579 and U.S. Patent No. 6,549,801, the entire contents of which are incorporated herein by reference). OCT systems can use any of a variety of OCT angiography processing algorithms (e.g., motion contrast algorithms) to identify blood flow. For example, motion contrast algorithms can be applied to intensity information derived from image data (intensity-based algorithms), phase information derived from image data (phase-based algorithms), or complex image data (complexity-based algorithms). A frontal image is a 2D projection of 3D OCT data (e.g., by averaging the intensity of each individual A-scan, such that each A-scan defines pixels in the 2D projection). Similarly, a frontal vascular system image is an image displaying motion-contrast signals, where the data dimension corresponding to depth (e.g., along the z-direction of the A-scan) is displayed as a single representative value (e.g., a pixel in a 2D projected image), typically displayed by summing or integrating all or individual portions of the data (e.g., see U.S. Patent No. 7,301,644, the entire contents of which are incorporated herein by reference). An OCT system providing angiographic imaging capabilities may be referred to as an OCT angiography (OCTA) system.

[0219] Figure 26 An example of a frontal vascular system image is shown. After processing the data using any motion contrast technique known in the art to enhance motion contrast, a range of pixels corresponding to a given tissue depth from the surface of the internal limiting membrane (ILM) in the retina can be summed to generate a frontal (e.g., frontal view) image of the vascular system.

[0220] Neural Networks

[0221] As described above, the present invention can utilize neural network (NN) machine learning (ML) models. For completeness, a general discussion of neural networks is provided herein. The present invention can use any of the neural network architectures described below, either alone or in combination. A neural network or neural network is a network of (nodes) composed of interconnected neurons, where each neuron represents a node in the network. Groups of neurons can be arranged hierarchically, and in a multilayer perceptron (MLP) arrangement, the output of one layer is fed forward to the next layer. An MLP can be understood as a feedforward neural network model that maps a set of input data to a set of output data.

[0222] Figure 27 An example of a multilayer perceptron (MLP) neural network is shown. Its structure can include multiple hidden (e.g., inner) layers HL1 to HLn, which map an input layer InL (receiving a set of inputs (or vector inputs) in_1 to in_3) to an output layer OutL, which produces a set of outputs (or vector outputs), such as out_1 and out_2. Each layer can have any given number of nodes, which are schematically shown as circles within each layer in this document. In this example, the first hidden layer HL1 has two nodes, while hidden layers HL2, HL3, and HLn each have three nodes. Generally, the deeper the MLP (e.g., the more hidden layers an MLP has), the stronger its learning ability. The input layer InL receives vector input (schematically shown as a three-dimensional vector consisting of in_1, in_2, and in_3) and can apply the received vector input to the first hidden layer HL1 in the sequence of hidden layers. The output layer OutL receives the output from the last hidden layer (e.g., HLn) in the multi-layer model, processes its input, and produces a vector output (exemplarily shown as a two-dimensional vector consisting of out_1 and out_2).

[0223] Typically, each neuron (or node) produces a single output, which is fed forward to neurons in the immediately following layer. However, each neuron in a hidden layer can receive multiple inputs, either from the input layer or from the outputs of neurons in the immediately preceding hidden layer. Generally, each node can apply a function to its inputs to produce its output. Nodes in hidden layers (e.g., learning layers) can apply the same function to their respective inputs to produce their respective outputs. However, some nodes (e.g., nodes in the input layer InL) receive only one input and can be passive, meaning they simply relay the value of their single input to their output, for example, providing a copy of their input to their output, as indicated by the dashed arrow within the node in the input layer InL.

[0224] For illustrative purposes, Figure 28 A simplified neural network consisting of an input layer INL', a hidden layer HL1', and an output layer OutL' is shown. The input layer INL' is shown with two input nodes i1 and i2, which receive inputs Input_1 and Input_2 respectively (e.g., the input nodes of layer INL' receive a two-dimensional input vector). The input layer INL' is fed forward to a hidden layer HL1' with two nodes h1 and h2, and the hidden layer HL1' is in turn fed forward to an output layer OutL' with two nodes o1 and o2. The interconnections or links between neurons (shown as solid arrows in the diagram) have weights w1 to w8. Typically, in addition to the input layer, a node (neuron) can receive the output of the node in its immediately preceding layer as input. Each node can compute its output by summing the products of its inputs (multiplying each input by its corresponding interconnection weight, adding (or multiplying by) a constant defined by another weight or bias that may be associated with that particular node (e.g., node weights (or biases) w9, w10, w11, w12 corresponding to nodes h1, h2, o1, and o2, respectively), and then applying a nonlinear or logarithmic function to the result. The nonlinear function can be called an activation function or a transfer function. Various activation functions are known in the art, and the choice of a particular activation function is not important for this discussion. However, it is important to note that the operation of an ML model, or the behavior of a neural network, depends on the weight values, and these weight values ​​can be learned so that the neural network provides the desired output for a given input.

[0225] During the training or learning phase, the neural network learns (e.g., is trained to determine) appropriate weight values ​​to achieve the desired output for a given input. Before training the neural network, each weight can be individually assigned an initial (e.g., random and optionally non-zero) value, such as a random number seed. Various methods for assigning initial weights are known in the art. The weights are then trained (optimized) such that, for a given training vector input, the neural network produces an output close to the desired (predetermined) training vector output. For example, the weights can be incrementally adjusted over thousands of iterations using a technique called backpropagation. In each iteration of backpropagation, the training input (e.g., a vector input or training input image / sample) is fed forward through the neural network to determine its actual output (e.g., a vector output). The error of each output neuron or output node is then calculated based on the actual neuron output and the target training output of that neuron (e.g., the training output image / sample corresponding to the current training input image / sample). The weights are then updated based on the degree of influence each weight has on the total error, making the output of the neural network closer to the desired training output, propagating back through the neural network (in the direction from the output layer back to the input layer). This loop is then repeated until the actual output of the neural network falls within an acceptable error range of the expected training output for a given training input. It's understood that each training input may require multiple backpropagation iterations before reaching the expected error range. Typically, an epoch refers to one backpropagation iteration across all training samples (e.g., one forward pass and one backward pass), making training a neural network potentially require many epochs. Generally, the larger the training set, the better the performance of the trained ML model, so various data augmentation methods can be used to increase the size of the training set. For example, when the training set consists of pairs of corresponding training input and training output images, the training images can be divided into multiple corresponding image segments (or blocks). Corresponding blocks from the training input and training output images can be paired to define multiple training block pairs from one input / output image pair, which expands the training set. However, training on large training sets places high demands on computational resources (e.g., memory and data processing resources). The computational requirements can be reduced by dividing the large training set into multiple mini-batches, where the mini-batch size defines the number of training samples in one forward / backward pass. In this case, one epoch can include multiple mini-batches. Another problem is the possibility that neural networks (NNs) overfit the training set, thereby reducing their ability to generalize from a specific input to different inputs. The overfitting problem can be mitigated by creating an ensemble of neural networks or by randomly dropping nodes from the neural network during training, which effectively removes the dropped nodes from the network. Various dropout modulation methods are known in the art, such as reverse dropout.

[0226] It should be noted that the operation of a trained neural network (NN) is not a direct algorithmic step of the operation / analysis process. In fact, when a trained NN receives input, it does not analyze that input in the traditional sense. Instead, regardless of the subject or nature of the input (e.g., a vector defining a real-time image / scan or a vector defining some other entity, such as a demographic description or activity record), the input will undergo the same predefined architectural construction of the trained neural network (e.g., the same node / layer arrangement, training weights and biases, predefined convolution / deconvolution operations, activation functions, pooling operations, etc.), and it may be unclear how the architecture of the trained network produces its output. Furthermore, the values ​​of the training weights and biases are not deterministic and depend on many factors, such as the amount of time given to the neural network for training (e.g., the number of epochs in training), the random initial values ​​of the weights before training begins, the computer architecture of the machine training the NN, the selection of training samples, the distribution of training samples across multiple mini-batches, the choice of activation functions, and the choice of error functions to modify the weights, even if training is interrupted on one machine (e.g., with the first computer architecture) and completed on another machine (e.g., with a different computer architecture). The key point is that the reasons why trained ML models achieve certain outputs are still unclear, and extensive research is currently underway to try to determine the factors underlying ML model outputs. Therefore, the processing of real-time data by neural networks cannot be simplified to simple algorithmic steps. Instead, its operation depends on its training architecture, training sample set, training sequence, and various circumstances during ML model training.

[0227] In summary, the construction of a neural network (NN) machine learning model can include a learning (or training) phase and a classification (or operation) phase. In the learning phase, the neural network can be trained for a specific purpose, and a set of training examples, including training (sample) inputs and training (sample) outputs, can be provided to the neural network, and optionally a set of validation examples to test the progress of training. During this learning process, various weights associated with the nodes and node interconnections in the neural network are incrementally adjusted to reduce the error between the actual output of the neural network and the desired training output. In this way, a multi-layer feedforward neural network (e.g., as described above) can be made capable of approximating any measurable function to any desired accuracy. The result of the learning phase is a (neural network) machine learning (ML) model that has been learned (e.g., trained). In the operation phase, a set of test inputs (or real-time inputs) can be submitted to the learned (trained) ML model, which can apply what it has learned to produce output predictions based on the test inputs.

[0228] picture Figure 26 and Figure 27Like regular neural networks, convolutional neural networks (CNNs) consist of neurons with learnable weights and biases. Each neuron receives input, performs an operation (e.g., a dot product), and optionally follows a non-linear path. However, a CNN can take raw image pixels at one end (e.g., the input) and provide a classification (or category) score at the other end (e.g., the output). Because a CNN expects an image as input, it is optimized for processing volumes (e.g., the pixel height and width of the image, plus the image depth, such as color depth, e.g., RGB depth defined by three colors: red, green, and blue). For example, a CNN layer can be optimized for neurons arranged in three dimensions. Neurons in a CNN layer may also be connected to a small region preceding that layer, rather than all neurons in a fully connected NN. The final output layer of a CNN can reduce the entire image to a single vector (classification) arranged along the depth dimension.

[0229] Figure 29An example convolutional neural network architecture is provided. A convolutional neural network can be defined as a sequence of two or more layers (e.g., layer 1 to layer N), where each layer may include a (image) convolution step, a (result) weighted sum step, and a nonlinear function step. Convolution can be performed on the input data, for example, by applying filters (or kernels) over a moving window on the input data, to produce a feature map. Each layer and its components may have different predetermined filters (from a filter bank), weights (or weighting parameters), and / or function parameters. In this example, the input data is an image with a given pixel height and width, which may be the raw pixel values ​​of the image. In this example, the input image is shown as a depth image with three color channels RGB (red, green, and blue). Optionally, the input image may undergo various preprocessing steps, and the preprocessed results may be used in place of the original input image or be input in addition to the original input image. Some examples of image preprocessing may include: retinal angiography segmentation, color space transformation, adaptive histogram equalization, connected component generation, etc. Within a layer, the dot product between a given weight and the small regions connected to it in the input volume can be computed. Many ways to configure a CNN are known in the art, but as an example, layers can be configured to apply element-wise activation functions, such as a maximum (0, x) threshold at zero. Pooling functions (e.g., along the xy direction) can be performed to downsample the volume. Fully connected layers can be used to determine the classification output and produce a one-dimensional output vector, which has been found useful for image recognition and classification. However, for image segmentation, a CNN needs to classify each pixel. Since each CNN layer tends to degrade the resolution of the input image, another stage is needed to upsample the image back to its original resolution. This can be achieved by applying a transposed convolution (or deconvolution) stage TC, which typically does not use any predefined interpolation methods but instead has learnable parameters.

[0230] Convolutional neural networks have been successfully applied to many computer vision problems. As mentioned above, training CNNs typically requires large training datasets. The U-Net architecture, based on CNNs, can usually be trained on smaller training datasets than traditional CNNs.

[0231] Figure 30An exemplary U-Net architecture is illustrated. This exemplary U-Net includes an input module (or input layer or stage) that receives an input U-in (e.g., an input image or image patch) of any given size (e.g., 128×128 pixels). The input image can be a fundus image, an OCT / OCTA frontal image, a B-scan image, etc. However, it should be understood that the input can be of any size and dimension. For example, the input image can be an RGB color image, a monochrome image, a volumetric image, etc. The input image undergoes a series of processing layers, each shown at exemplary dimensions, but these dimensions are for illustrative purposes only and will depend on, for example, the image size, convolutional filters, and / or pooling stages. The architecture consists of a shrinking path (comprising four encoding modules), followed by an expanding path (comprising four decoding modules) and four copy and cut links (e.g., CC1 to CC4), located between corresponding modules / stages that copy the output of an encoding module in the shrinking path and cascade it to the input of the corresponding decoding module in the expanding path. This results in a distinctive U-shape, from which the architecture is named. The shrinking path is analogous to an encoder, whose basic function is to capture context via compact feature maps. In this example, each encoding module in the shrinking path consists of two convolutional neural network layers, followed by a max-pooling layer (e.g., a downsampling layer). For example, the input image U-in undergoes two convolutional layers, each with 32 feature maps. The number of feature maps may double with each pooling, starting with 32 feature maps in the first block, 64 in the second, and so on. The shrinking path thus forms a convolutional network consisting of multiple encoding modules (or stages), each providing a convolutional level, followed by an activation function (e.g., a rectified linear unit, ReLU, or sigmoid layer) and a max-pooling operation. The expanding path is analogous to a decoder, whose function is to provide localization and preserve spatial information, despite downsampling and any max-pooling during the shrinking phase. In the shrinking path, spatial information is reduced while feature information is increased. The expanding path consists of multiple decoding modules, where each decoding module concatenates its current value with the output of the corresponding encoding module. That is, features and spatial information are combined in the expansion path through a series of upconvolutions (e.g., upsampling, transposed convolutions, or deconvolutions) and concatenations with high-resolution features from the contraction path (e.g., via CC1 to CC4). Thus, the output of the deconvolutional layer is concatenated with the corresponding (optionally cropped) feature map from the contraction path, followed by two convolutional layers and an activation function (optionally batch normalized). The output from the last module in the expansion path can be fed into another processing / training block or layer, such as a classifier block, which can be trained together with the U-Net architecture.

[0232] The module / level (BN) between the shrinking and expanding paths can be referred to as the "bottleneck". A bottleneck BN can consist of two convolutional layers (with batch normalization and optional dropout).

[0233] Computing device / system

[0234] Figure 31 An example computer system (or computing device or computer apparatus) is illustrated. In some embodiments, one or more computer systems may provide the functionality described or illustrated herein and / or perform one or more steps of one or more methods described or illustrated herein. The computer system may take any suitable physical form. For example, the computer system may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, the computer system may reside in a cloud, which may include one or more cloud components within one or more networks.

[0235] In some embodiments, the computer system may include a processor Cpnt1, a memory Cpnt2, a storage device Cpnt3, an input / output (I / O) interface Cpnt4, a communication interface Cpnt5, and a bus Cpnt6. The computer system may also optionally include a display Cpnt7, such as a computer monitor or screen.

[0236] Processor Cpnt1 includes hardware for executing instructions, such as those that make up a computer program. For example, processor Cpnt1 may be general-purpose computing on a central processing unit (CPU) or a graphics processing unit (GPGPU). Processor Cpnt1 may retrieve (or fetch) instructions from internal registers, internal caches, memory Cpnt2, or storage device Cpnt3, decode and execute the instructions, and write one or more results to internal registers, internal caches, memory Cpnt2, or storage device Cpnt3. In a particular embodiment, processor Cpnt1 may include one or more internal caches for data, instructions, or addresses. Processor Cpnt1 may include one or more instruction caches and one or more data caches, for example, for storing data tables. Instructions in the instruction cache may be copies of instructions in memory Cpnt2 or storage device Cpnt3, and the instruction cache may accelerate the retrieval of those instructions by processor Cpnt1. Processor Cpnt1 may include any suitable number of internal registers and may include one or more arithmetic logic units (ALu). Processor Cpnt1 may be a multi-core processor; or may include one or more processors Cpnt1. Although this disclosure describes and illustrates a particular processor, this disclosure considers any suitable processor.

[0237] Memory Cpnt2 may include main memory for storing instructions that processor Cpnt1 executes or saves temporary data during processing. For example, a computer system may load instructions or data (e.g., a data table) from storage device Cpnt3 or from another source (e.g., another computer system) into memory Cpnt2. Processor Cpnt1 may load instructions and data from memory Cpnt2 into one or more internal registers or internal caches. To execute instructions, processor Cpnt1 may retrieve and decode instructions from internal registers or internal caches. During or after instruction execution, processor Cpnt1 may write one or more results (which may be intermediate or final results) to internal registers, internal caches, memory Cpnt2, or storage device Cpnt3. Bus Cpnt6 may include one or more memory buses (each bus may include an address bus and a data bus) and may couple processor Cpnt1 to memory Cpnt2 and / or storage device Cpnt3. Optionally, one or more memory management units (MMUs) facilitate data transfer between processor Cpnt1 and memory Cpnt2. Memory Cpnt2 (which may be fast volatile memory) may include random access memory (RAM), such as dynamic RAM (DRAM) or static RAM (SRAM). Storage device Cpnt3 may include a long-term or high-capacity storage device for data or instructions. Storage device Cpnt3 may be internal or external to a computer system and includes one or more of the following: disk drive (e.g., hard disk drive HDD or solid-state drive SSD), flash memory, ROM, EPROM, optical disk, magneto-optical disk, magnetic tape, universal serial bus (USB) accessible drive, or other types of non-volatile memory.

[0238] The I / O interface Cpnt4 can be software, hardware, or a combination of both, and includes one or more interfaces (e.g., serial or parallel communication ports) for communicating with I / O devices, enabling communication with a person (e.g., a user). For example, I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, camera, pen, tablet computer, touchscreen, trackball, camera, another suitable I / O device, or a combination of two or more of these devices.

[0239] The communication interface Cpnt5 provides a network interface for communication with other systems or networks. Communication interface Cpnt5 may include a Bluetooth interface or other types of packet-based communication. For example, communication interface Cpnt5 may include a network interface controller (NIC) and / or a wireless NIC or wireless adapter for communication with wireless networks. Communication interface Cpnt5 can provide communication with Wi-Fi networks, ad hoc networks, personal area networks (PANs), wireless PANs (e.g., Bluetooth WPANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular telephone networks (e.g., GSM networks), the Internet, or combinations of two or more of these networks.

[0240] The Cpnt6 bus can provide communication links between the aforementioned components of a computing system. For example, the Cpnt6 bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand bus, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Fast (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses or combinations of two or more of these buses.

[0241] Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0242] In this document, where appropriate, computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these media. Computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0243] Although the invention has been described in conjunction with several specific embodiments, it will be apparent to those skilled in the art that many further substitutions, modifications, and variations will be readily apparent from the foregoing description. Therefore, the invention described herein is intended to encompass all substitutions, modifications, applications, and variations that may fall within the spirit and scope of the appended claims.

Claims

1. A method for determining a defocus metric of an ophthalmic imaging system, the method comprising: The first area of ​​the retina that is illuminated; A first image of the light returning from the first region is collected on the detector; A second region of the retina of the eye is illuminated, the second region partially overlapping the first region; A second image of the light returning from the second region is collected on the detector; A third image is defined by obtaining the difference between the first image and the second image, the third image comprising a linear region; Determine the width measurement of the linear region; and The defocus metric is determined based on the width measurement.

2. The method according to claim 1, wherein, The width measurement is based on the second moment of the intensity distribution of the third image.

3. The method according to claim 1 or 2, further comprising: The third image is segmented into a foreground portion and a background portion, wherein the width measurement is determined from the foreground portion.

4. The method according to any one of claims 1 to 3, further comprising: The Gaussian shape is fitted to the linear region of the third image, and the width measurement is determined at least in part based on the fitted Gaussian shape.

5. The method according to any one of claims 1 to 4, wherein: The first area is illuminated by a first lighting line, which has a first width along the width dimension; The second area is illuminated by a second illumination line, the second illumination line having the first width along the width dimension; The second region is offset from the first region along the width dimension, wherein the offset is less than the first width; and The width of the linear region of the third image is defined by the offset of the second region and the first region along the width dimension.

6. The method according to any one of claims 1 to 5, wherein: The intensity difference between the first image and the second image is captured. A positive linear region is generated by selecting the maximum value of the intensity difference and setting the remaining values ​​of the intensity difference except the maximum value to zero, and a negative linear region is generated by selecting the minimum value of the intensity difference and setting the remaining values ​​of the intensity difference except the minimum value to zero. And select one of the positive linear region and the negative linear region as the linear region of the third image.

7. The method according to any one of claims 1 to 6, further comprising: The resolution of the third image is improved by applying the following process: in, It is the observed intensity. It is the point spread function of the illumination. It is the point spread function for detection. It is a rectangular strip of lighting, marked with an asterisk. "" represents convolution, and the point " " indicates multiplication, O represents the weight, and O represents the object.

8. The method according to any one of claims 1 to 6, further comprising: The resolution of the third image is improved by first approximating sinusoidal illumination using weights as follows: Where wi is the weight. O represents the rectangular strip of light, and O represents the object. Then define in It is the observed intensity, the letter It is a scaling factor. It is the point spread function of the detection, marked with an asterisk. "" represents convolution, and the point " " indicates multiplication; Letters are removed by intensity normalization. Scaling factor; and Reassemble the sine curve.

9. The method according to any one of claims 1 to 8, further comprising: The topological information of the retina is determined based on the displacement of the centroid of the linear region in the third image.

10. The method according to claim 9, wherein, The topological information for the retinal region is determined by following the positional change of the centroid of the linear region along the length of the third image.

11. The method according to claim 9 or 10, further comprising: The tumor volume can be determined, blood vessels can be segmented, or an optical disc can be segmented based on the topological information.

12. The method according to any one of claims 1 to 11, wherein, The ophthalmic imaging system is a fundus imaging system.

Images