Segmentation and classification of geographic atrophy patterns in patients with age-related macular degeneration from widefield autofluorescence images

Abstract

An automated segmentation and recognition system / method for identifying geographic atrophy (GA) phenotypic patterns in fundus autofluorescence images. The hybrid process combines a supervised pixel classifier with an active contour algorithm. A trained machine learning model (e.g., SVM or U-Net) provides an initial GA segmentation / classification, and is followed by a Chan-Vese active contour algorithm. The junction regions of the GA segmented regions are then analyzed for geometric regularity and light intensity regularity. The GA phenotype is determined, at least in part, from these parameters.

Description

Technical Field

[0001] The present invention generally relates to the field of ophthalmic autofluorescence images. More specifically, it relates to the classification of areas of geographic atrophy found in fundus autofluorescence images. Background Art

[0002] Age-related macular degeneration (AMD) is the most common cause of blindness in the elderly population in industrialized countries. Geographic atrophy (GA) is an advanced form of AMD characterized by the loss of photoreceptors, the retinal pigment epithelium (RPE), and the choriocapillaris layer. An estimated 5 million people worldwide are affected by geographic atrophy. GA can lead to irreversible loss of visual function and is the cause of severe visual impairment in approximately 20% of AMD cases. To date, there are no approved treatments to impede the progression of GA. However, in recent years, advances in the understanding of the pathogenesis of GA have led to multiple potential therapies in clinical trials. Nonetheless, early identification of GA progression is crucial to slow its effects.

[0003] It has been shown that different phenotypic patterns of abnormal fundus autofluorescence (FAF) images help identify GA progression. Therefore, identifying GA lesions and their phenotypes may be an important factor in determining AMD disease progression and clinical diagnosis. Visual inspection of FAF images of GA lesions and their phenotypes by medical staff is effective but time-consuming. Several segmentation algorithms have been developed to aid in the assessment of GA lesions, but many of these algorithms are semi-automatic and require manual input to segment GA lesions.

[0004] An object of the present invention is to provide a fully automatic method for quantification of GA lesions and their phenotypic patterns in medical images.

[0005] Another objective of the present invention is to enhance the GA lesion segmentation algorithm with the ability to recognize and classify different phenotypic patterns.

[0006] Another object of the present invention is to provide a framework for automatic segmentation and identification of GA phenotypic patterns in wide-field FAF images. Summary of the invention

[0007] The above objects are met in a system / method for classifying (e.g., identifying) geographic atrophy (GA) in an eye. For example, GA areas (e.g., GA segmentations in fundus or frontal images) can be identified as a "diffused" phenotype or a "banded" phenotype. Both phenotypes have been empirically determined to be indicative of high rates of progression of geographic atrophy. The system may include an ophthalmic diagnostic device, such as a fundus imager or optical coherence tomography (OCT) device for generating / capturing ophthalmic images. Alternatively, the ophthalmic diagnostic device may be embodied as a computer system that accesses pre-existing ophthalmic images from a data store using a computer network (e.g., the Internet). The ophthalmic diagnostic device is used to acquire a fundus image. Preferably, the image is a fundus autofluorescence (FAF) image, as GA lesions are generally more recognizable in such images.

[0008] The acquired image is then submitted to an automatic GA identification process to identify (e.g., segment) the GA region in the image. The present GA segmentation method is fully automated and can be based on a deep learning neural network. GA segmentation can also be based on a two-stage segmentation process, such as a hybrid process combining a supervised classifier (or pixel / image segmentation) with an active contour algorithm. The supervised classifier is preferably a machine learning (ML) model and can be implemented as a support vector machine (SVM) or a deep learning neural network, preferably a U-Net type convolutional neural network. This first-stage classifier / segmentation ML model identifies the initial GA region (lesion) in the image and feeds the result to an active contour algorithm for second-stage segmentation. The active contour algorithm can be implemented as an improved Chase-Vese segmentation algorithm, in which the initial GA region identified in the first stage is used as a starting point (e.g., initial contour) in the Chase-Vese segmentation. The result is a robust GA segmentation of the acquired image.

[0009] The identified GA regions are then submitted for analysis to identify their specific phenotype. For example, the system / method may use the contour non-uniformity measurement of the GA region to identify a 'diffuse' phenotype, and the intensity uniformity measurement to identify a 'banded' phenotype. Either of these two phenotypes is indicative of a high progression rate GA region. The analysis may be performed in a two-step process, wherein the first of the two measurements is calculated, and if the first measurement is above a first threshold, the GA region may be classified as having a high progression rate, and the second of the two measurements need not be calculated. However, if the first measurement is not above the first threshold, a second measurement may be calculated and compared to the second threshold to determine whether it indicates a high progression rate GA. For example, if the contour non-uniformity measurement is greater than a first predetermined threshold (e.g., the non-uniformity of the peripheral contour of the GA region is greater than a predetermined threshold), the GA region may be classified as a 'diffuse' phenotype, and if the intensity uniformity measurement is greater than a second threshold, the GA region may be classified as a 'banded' phenotype.

[0010] Other objects and attainments of the present invention as well as a more complete understanding of the present invention will become apparent and appreciated by referring to the following description and claims taken in conjunction with the accompanying drawings.

[0011] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Any embodiment feature mentioned in one claim category (e.g., system) may also be claimed in another claim category (e.g., method). Dependencies or back-references in the appended claims are selected for formal reasons only. However, any subject matter resulting from deliberate back-reference to any previous claim may also be claimed, thereby disclosing and claiming any combination of claims and their features, regardless of the dependencies selected in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the drawings, like reference symbols / characters refer to like parts:

[0013] Figure 1 Three grayscale images of the true color images are provided, alongside three corresponding autofluorescence images of similar areas of the eye.

[0014] Figure 2 Example FAF images of the GA region are provided.

[0015] Figure 3A Some examples of FAF images of 'diffuse' phenotype GA lesions are provided.

[0016] Figure 3B An example of a FAF with a 'ribbon' phenotype GA lesion is provided.

[0017] Figure 4 A summary architecture for automatic segmentation and identification of GA phenotypic patterns is provided.

[0018] Figure 5 A more detailed process for automatically segmenting and identifying GA phenotypic patterns according to the present invention is described.

[0019] Figure 6 Four GA regions delineated by human experts are shown.

[0020] Figure 7 shows the GA delineation provided by the support vector machine and corresponds to Figure 6 GA area.

[0021] Figure 8 The five GA regions delineated by human experts are shown.

[0022] Fig. 9 shows the GA delineation provided by deep learning,neural networks, and corresponds to Figure 8 GA area.

[0023] Fig.10 A 'diffuse' phenotype GA region is shown, with its centroid indicated by a circle.

[0024] Fig.11 The 'ribbon-like' phenotype GA region is shown, with the centroid and thirteen random point sets equally spaced along the GA segmentation perimeter.

[0025] Fig.12 An example method for automatically classifying geographic atrophy in the eye is shown.

[0026] Fig.13 An example of a slit-scanning ophthalmic system for imaging the fundus is shown.

[0027] Fig.14 A generalized frequency domain optical coherence tomography system suitable for use with the present invention for collecting 3-D image data of an eye is shown.

[0028] Fig.15 An example of a frontal vascular system image is shown.

[0029] Fig.16 An example of a multilayer perceptron (MLP) neural network is shown.

[0030] Fig.17 A simplified neural network consisting of an input layer, a hidden layer, and an output layer is shown.

[0031] Fig.18 An example convolutional neural network architecture is shown.

[0032] Fig.19 An example U-Net architecture is shown.

[0033] Fig. 20 An example computer system (or computing device or computing device) is shown. DETAILED DESCRIPTION

[0034] Early diagnosis is critical for the successful treatment of a wide range of eye diseases. Optical imaging is the method of choice for noninvasive examination of the retina. Although age-related macular degeneration (AMD) is known to be a leading cause of vision loss, diagnosis is often not made until damage is present. Therefore, the goal of advanced ophthalmic imaging devices is to provide diagnostic tools to help detect and monitor pathological changes in the preclinical stages of the disease.

[0035] A variety of ophthalmic imaging systems are known in the art, such as fundus imaging systems and optical coherence tomography systems, which can be used with the present invention. Examples of ophthalmic imaging modalities are provided below, for example, see Fig.13 and 14 Any of these devices can be used to provide an image of the fundus, which is the inner surface of the eye opposite the eye lens (or crystalline lens) and may include the retina, optic disc, macula, fovea, and posterior pole.

[0036] Ophthalmic imaging systems can generate full-color images. Other imaging techniques, such as fluorescein angiography or indocyanine green angiography (ICG), can be used to capture high-contrast images of specific ophthalmic features, such as blood vessels or lesions. High-contrast images are obtained by collecting images after injecting a fluorescent dye into the subject's bloodstream and capturing the image using a specific light frequency (e.g., color) selected to excite the fluorescent dye.

[0037] Alternatively, high contrast images can be obtained without the use of fluorescent dyes. For example, a single light source (such as a colored LED or laser) that provides light at a specific frequency can be used to excite different naturally occurring fluorophores in the eye in a technique called autofluorescence. The resulting fluorescence can be detected by filtering out the excitation wavelength. This can make certain features / tissues of the eye easier to discern than with true color images. For example, fundus autofluorescence (FAF) imaging can be performed with green or blue excitation, which excites the natural fluorescence of lipofuscin, producing a monochrome image. For illustration purposes, Figure 1 Three grayscale images 11A, 13A and 15A of true color images are provided, alongside three corresponding autofluorescence images 11B, 13B and 15B of the same region of the eye. It is apparent that some tissue features that are not well defined in the grayscale images 11A, 13A and 15A (e.g. in the central region of each image) are easily identified in the corresponding autofluorescence images 11B, 13B and 15B.

[0038] AMD typically begins as dry (atrophic) AMD and progresses to wet (neovascular) AMD. Dry AMD is characterized by the formation of drusen (e.g., small white or yellowish deposits) on the retina (e.g., beneath the macula), causing it to worsen or degenerate over time. Early dry AMD can be treated with nutritional therapy or supplements, and wet AMD can be treated with injections (e.g., injections of ranibizumab, Avastin, and Eylea), but there are currently no approved treatments for advanced dry AMD, although clinical trials are ongoing. Geographic atrophy (GA) is a late form of dry AMD characterized by patches of degenerated or dead cells in the retina. GA can be defined as any well-defined, circular or oval area of ​​hypopigmentation, or a marked loss of the retinal pigment epithelium (RPE), in which the choroidal vessels are more prominent than the surrounding area and have a minimum size, such as 175 μm in diameter. In the past, color fundus images were used to image and identify GA, but color fundus imaging was unable to visualize the features of lesions associated with GA progression. Due to the loss of RPE cells containing lipofuscin (an autofluorescent pigment), atrophic areas appear darker in fundus autofluorescence (FAF) imaging. This produces a high contrast between atrophic and non-atrophic areas, which can more easily define GA areas than color fundus images. The use of autofluorescence has been found to be a more effective imaging modality for evaluating GA lesions and monitoring structural changes in lesions. In particular, FAF using a 488nm excitation wavelength is currently the industry's preferred technique for morphological assessment of GA lesions. For illustrative purposes, Figure 2 Example FAF images of the GA region are provided.

[0039] FAF images of GA can be characterized by abnormal patterns of hyperautofluorescence around atrophic areas. This has led to the classification or identification of specific phenotypes of GA. Bindewald et al. first proposed the classification of this characteristic hyperautofluorescence into different patterns in "Classification of Fundus Autofluorescence Patterns in Early Age-Related Macular Disease" Invest Ophthalmol Vis Sci., 2005, 46: 3309-14. Subsequently, several studies were published documenting the impact of different phenotypic patterns on disease progression and their ability to serve as prognostic determinants, as discussed in: Holz FGet al., “Progression of Geographic Atrophy and Impact of Fundus Autofluorescence Patterns in Age-Related Macular Degeneration,” Am J. Ophthalmol, 2007; 143: 463-472; Fleckenstein M. et al., “The'Diffuse-Trickling' Fundus Autofluorescence Phenotype in Geographic Atrophy,” Invest Ophthalmol Vis Sci., 2014, 55: 2911-20; and Jeong, YJ et al., “Predictors for the Progressionof Geographic Atrophy in Patients with Age-Related Macular Degeneration: Fundus Autofluorescence Study with Modified Fundus Camera,” Eye, 2014 online, 28(2), 209-218, DOI: 10.1016 / 0300-944-8444. 10.1038 / eye.2013.275. The above references are incorporated herein by reference in their entirety.

[0040] Of particular interest are the 'diffuse' (or 'diffuse-trickling') GA phenotype and the 'banding' GA phenotype, both of which have been empirically determined to indicate a higher rate of GA progression. Therefore, in addition to identifying areas of GA, the correct phenotypic classification of identified GA areas is equally important. 'Diffuse' GA areas may be characterized by a degree of non-uniformity in their peripheral outline (e.g., a measure of outline non-uniformity above a predetermined non-uniformity threshold). Thus, it may be determined that Figure 2 The GA region is not a 'diffuse' phenotype, as its contours are relatively uniform. Figure 3A Several examples of FAF images of 'diffuse' phenotype GA lesions are provided, all of which exhibit a high degree of contour non-uniformity. Because GA regions may also have high concentrations of hyperautofluorescence surrounding atrophic regions, a 'band-like' GA region may be characterized by the amount of variation in hyperautofluorescence intensity along its perimeter. More specifically, a GA region may be characterized as a 'band-like' phenotype if the variation in contour hyperautofluorescence intensity of the GA region is relatively uniform (e.g., the intensity uniformity measure is above a predetermined intensity threshold). For example, it may be determined that Figure 2 The GA region is not a ‘ribbon’ phenotype because it lacks uniformity of hyperautofluorescence along its contour. By comparison, Figure 3B An example of FAF of a GA lesion of ‘ribbon-like’ phenotype, with uniform hyperautofluorescence along its contour, is provided.

[0041] Automated quantification of GA lesions and their phenotypic patterns can greatly aid in determining disease progression and clinical diagnosis of AMD. As mentioned above, there are currently several segmentation algorithms for GA lesion assessment, but none of these algorithms address the identification and classification of different phenotypic patterns. That is, all segmentation methods available for GA lesions to date lack the ability to automatically classify different GA phenotypic patterns. Most segmentation methods reported in the literature are semi-automatic and require manual input to segment GA lesions. In addition, all previously reported segmentation methods were developed for use with standard field of view (FOV) images (e.g., FOV of 45° to 60°), where the location of GA lesions is typically limited to predefined areas of standard FOV images. However, there has been a recent increase in interest in widefield images (e.g., with FOV 60° to 120° or greater), emphasizing the need for an automated GA assessment method that is applicable to widefield images. The use of widefield images complicates the use of typical GA segmentation algorithms, because the location of GA lesions cannot be confined to predefined areas of widefield images, and certain (physical) landmarks in diseased eyes may not be clearly defined.

[0042] Figure 4A summary architecture for automatic segmentation and identification of GA phenotypic patterns is provided. First, a measure of image quality (IQ) is determined (block B1). This may include applying an image quality algorithm to the FAF image to reject images with ungradable quality (IQ measure below a predefined value). If the image is rejected, the process can jump directly to B9. If the IQ measure is sufficient for scoring, the process continues to block B3.

[0043] Block B3 provides optional pre-processing of the FAF image. This may include optic disc and / or vessel detection, region of interest (ROI) selection, and histogram correction. Automatic ROI selection (which may be based on the detection of the optic disc) limits the amount of pixels that need to be processed (e.g., classified), thereby reducing the implementation time of the segmentation process.

[0044] Block B5 provides a GA lesion segmentation. This can combine pixel-by-pixel classification and modified Chan-Vese active contours. For example, the classification can be provided by a machine learning (ML) model (e.g., a support vector machine (SVM) and / or a deep learning neural network) that provides an initial GA segmentation. The GA segmentation is then submitted as a starting point to the Chan-Vese active contour algorithm, which further refines the GA segmentation. This hybrid process uses a novel combination of a supervised pixel classifier and active contours.

[0045] The GA segmentation of block B5 can then be submitted to a phenotype classification block B7, which identifies and classifies junction areas near the GA segmented regions (e.g., junctions along the perimeter of the GA segmentation). For example, a set of random points that are equidistant from each other and distributed along the perimeter of the GA segmentation can be selected. The distance from each selected point to the centroid of the GA segmentation can then be calculated. These distances can then be used to determine a measure of the smoothness of the perimeter contour of the GA segmentation (e.g., the smoothness of the contour delineated by the GA). Intensity ridges and valleys can be calculated along a direction perpendicular to the perimeter contour of the GA segmentation, for example, using a Hessian filter gradient derivative and / or a directional Gaussian filter. This can provide a measure of the regularity of light intensity along the perimeter of the GA segmentation. Both parameters can be used to classify the junction area phenotype. For example, a measure of the smoothness of the contour of the GA segmentation can be used to identify a 'diffuse' phenotype, while a measure of the regularity of the intensity of the GA segmentation can be used to identify a 'banded' phenotype.

[0046] Alternatively, a (e.g., deep learning) neural network may be implemented as block B7. In this case, the neural network may be trained to receive the GA segmentation of block B5 and classify a particular phenotype (e.g., diffuse or banded) for each GA segmentation. Still alternatively, the neural network may be trained to perform the functions of both the GA lesion segmentation block B5 and the phenotype classification block B7. For example, a set of images showing GA regions delineated by an expert and their respective phenotypes identified by the expert may be used as a training output set for the neural network, and the original undelineated images may be used as a training input set for the neural network. Thus, the neural network may be trained to not only segment (or delineate) GA lesion regions, but also to identify their respective phenotypes (e.g., 'diffuse' or 'banded').

[0047] Furthermore, it has been found that retinal vessels may cause certain fundus regions to be incorrectly identified as GA segmentations. This can be avoided by removing retinal vessels (e.g., retinal vessels identified in preprocessing block B3) from the FAF image before applying the GA segmentation, e.g., before block B5. In this case, the retinal vessels may optionally be removed from the training images used to train the learning model (SVM or neural network).

[0048] The final block B9 generates a report summarizing the results of the current architecture. For example, the report may state whether the image quality is too low to be graded, specify whether the identified GA lesion is of the 'high' progression rate type, specify the phenotype of the GA lesion, and / or recommend a follow-up examination date. Both the 'diffuse' and 'banded' phenotypes have been empirically determined to be indicative of a high progression rate GA; although a 'diffuse' phenotype may have a higher progression rate than a 'banded' phenotype. Follow-up examination date recommendations may be based on the type of GA lesion determined. For example, a more highly progressive GA lesion may require an earlier follow-up examination date than a less progressive or 'low' (or non-high) progressive GA lesion.

[0049] Figure 5A more detailed process for automatically segmenting and identifying GA phenotypic patterns according to the present invention is shown. For illustrative purposes, the present invention is described as being applied to widefield FAF images, but can also be applied to other types of ophthalmic imaging modalities, such as OCT / OCTA that can generate images that provide visual GA. For example, it can be applied to frontal OCT / OCTA images. The imaging system and / or method can begin by capturing or otherwise acquiring (e.g., by accessing a data store) a fundus autofluorescence image (step S1). The image quality of the acquired fundus autofluorescence image is then evaluated (e.g., measured values) using any suitable image quality (IQ) algorithm known in the art. For example, the measured value of image quality can be based on one or a combination of quantifiable factors such as sharpness, noise, dynamic range, contrast, vignetting, etc. If the measured value of image quality is less than a predefined image quality threshold (step S3 = yes), it is determined that the FAF image is not suitable for further analysis, and the process proceeds to step S29, in which a report is generated indicating that GA segmentation or quantification cannot be performed on the current FAF image. In essence, the image quality IQ algorithm is used to reject images with non-gradable quality. If the image quality is not less than the minimum threshold (step S3 = No), the process may proceed to the optional pre-processing step S5, or may optionally proceed directly to the GA image segmentation step S7.

[0050] The pre-processing step S5 may include a plurality of pre-processing sub-steps. For example, it may include optic disc and / or vascular detection (e.g., finding a mask), and identifying the image as coming from the patient's left or right eye. As explained above, physical landmarks may be difficult to identify in a diseased eye, but the optic disc may be identified by the concentration of vascular structures emanating therefrom. For example, a machine learning model (e.g., a deep learning neural network, as discussed below) may be trained to identify the optic disc in a wide-field image. Alternatively, another type of machine learning model (e.g., a support vector machine) may be trained to identify the optic disc, for example by associating the location of the optic disc with a location where the concentration of vascular structures originates. In either case, additional information external to the image may be used for training, such as information provided by an electronic medical record (EMR) or the Digital Imaging and Communications in Medicine (DICOM) standard.

[0051] As described above, it has been found that retinal vessels may cause false positives in GA segmentation. Therefore, preprocessing may also include identifying and removing retinal vessels before applying GA segmentation to reduce the number of false positives, such as reducing the identification of false GA areas. As described above, GA segmentation can be based on a machine learning model, and retinal vessels can be removed from its training image set. For example, a training output set of fundus images (such as FAF images) with manually delineated GA areas and manually identified phenotypes can be used to train a neural network, and a training input set of the same fundus images without manually delineated GA areas and without identified phenotypes can be used to train the neural network. If, during the operation phase (or testing phase), the neural network is to accept input test images from which retinal vessels have been removed, the neural network can be trained with training images from which retinal vessels have also been removed. That is, before training the neural network, retinal vessels can be removed from the fundus images in the training input set and the training output set.

[0052] The pre-processing step S5 may also include a region of interest (ROI) selection to limit processing (including GA segmentation) to ROIs identified within the FAF image. Preferably, the identified ROI will include the macula. ROI selection may be performed automatically based on landmark detection (optic disc and retinal vessels), image size, image entropy, and fundus scan information extracted from DICOM, etc. For example, depending on whether the image is of the left or right eye, the macula will be located to the right or left of the optic disc location within the image. It will be appreciated that ROI selection limits the amount of pixels that need to be classified (e.g., for GA segmentation), thereby reducing the implementation time of the present process / method.

[0053] The pre-processing step S5 may also include illumination correction and contrast adjustment. For example, uneven image illumination may be corrected by using background subtraction, and image contrast may be adjusted by histogram equalization.

[0054] The preprocessed images are then submitted to a two-stage lesion segmentation (e.g. Figure 4Block B5), including GA segmentation and active contour analysis. The first stage in block B5 is the GA segmentation / classifier stage (S7), which is preferably based on machine learning and identifies one or more GA regions, while the second stage in block B5 is the active contour algorithm applied to the identified GA regions. The GA segmentation stage S7 can provide GA classification on a sub-image region by sub-image region basis, where each sub-image region can be a pixel (e.g. on a pixel by pixel basis) or each sub-image region can be a group of multiple pixels (e.g. a window). In this way, each sub-image region can be individually classified as a GA sub-image or a non-GA sub-image. The active contour analysis stage S9 can be based on the known Chan-Vese algorithm. In this case, the Chan-Vese algorithm is modified to change the energy and direction of motion of contour growth. For example, region-based image features are used to indicate contour motion. In essence, the segmentation of GA lesions in block B5 uses a hybrid process that combines (e.g. pixel by pixel) classification and Chan-Vese active contours. Optionally, the proposed hybrid algorithm uses a novel combination of a supervised (e.g., pixel) classifier (S7) and a geometric active contour model (S9). As will be appreciated by those skilled in the art, a geometric active contour model typically starts with defining a contour in the image plane of the initial segmentation (e.g., a starting point), and then evolves the contour according to some evolution equations to stop at the boundary of the foreground region. In this case, the geometric active contour model is based on a modified Chan-Vese algorithm.

[0055] The GA segmentation / classifier stage (S7) is preferably based on machine learning and can be implemented, for example, by using a support vector machine or by a (deep learning) neural network. Each implementation is described separately herein.

[0056] In general, a support vector machine (SVM) is a machine learning, linear model used for classification and regression problems, and can be used to solve both linear and nonlinear problems. The idea of ​​an SVM is to create a line or hyperplane that separates data into several categories. More formally, an SVM defines one or more hyperplanes in a multidimensional space, where the hyperplanes are used for classification, regression, outlier detection, etc. Essentially, an SVM model represents labeled training examples as points in a multidimensional space, mapped so that labeled training examples of different categories are divided by hyperplanes, which can be thought of as decision boundaries that separate different categories. When a new test input example is submitted to the SVM model, the test input is mapped to the same space and predicted to which category it belongs based on which side of the decision boundary (hyperplane) the test input lies.

[0057] In a preferred embodiment, SVM is used for image segmentation. The purpose of image segmentation is to divide an image into different sub-images with different features and to extract a target object. More specifically, in the present invention, an SVM is trained to segment the GA region in a FAF image (e.g., trained to clearly define the outline of the GA region in the FAF image). Various SVM architectures for image segmentation are known in the art, and the specific SVM architecture used for this task is not critical to the present invention. For example, a least squares SVM can be used for image segmentation based on pixel-by-pixel (or sub-image region-by-sub-image region) classification. Both pixel-level features (e.g., color, intensity, etc.) and texture features can be used as inputs to the SVM. Optionally, a group of SVMs (each providing specialized classification) can be linked to obtain better results.

[0058] Thus, an SVM classifier (e.g., by segmentation based on an SVM model) may be used for initial contour selection. Preferably, Haralick texture features, mean intensity, and variance parameters obtained from a gray-level co-occurrence matrix (e.g., an 11x11 window moving within a region of interest) are used to train the SVM classifier. As explained above, retinal vessels may optionally be removed from the training images. In any case, feature extraction is limited to a specific ROI (e.g., the ROI selected in step S5), which results in better temporal performance than applying GA segmentation / classification to the entire image. In this way, the SVM provides an initial contour selection (e.g., providing an initial GA segmentation as a starting point) to be submitted to the active contour algorithm of step S9 for better performance. The evolution time of the active contour algorithm is heavily dependent on the initial contour selection, as explained, for example, by Chen, Q. et al. in "Semi-automatic geographic atrophy segmentation for SD-OCT images", Biomedical Optics Express, 4(12), 2729-2750, the entire contents of which are incorporated herein by reference.

[0059] The performance of the SVM classifiers was tested and compared. Expert raters manually delineated GA regions in FAF images not used in training, and the results were compared with the segmentation results of the SVM model. Figure 6 shows the four GA regions delineated by human experts, and Figure 7 The corresponding GA delineation (eg, perimeter of the GA segmentation) provided by the present SVM is shown. As shown, the SVM model agrees well with the manual GA segmentation provided by the expert raters.

[0060] The GA segmentation / classifier stage (S7) can also be implemented by using a neural network (NN) machine learning (LM) model. Figures 16 to 19Various examples of neural networks are discussed, any of which or a combination of which may be used with the present invention. An exemplary implementation of a GA segmentation / classifier using a deep learning neural network is constructed based on the U-Net architecture (see Fig.19 ). In this case, the NN is trained using manually segmented images (e.g., images with GA regions segmented by a human expert) as training outputs and the corresponding non-segmented images as training inputs.

[0061] In an example implementation, using CLARUS TM 500 fundus cameras (ZEISS, Dublin, CA) obtained 79 FAF-Green images from 62 GA patients. These 79 FAF images were divided into 55 FAF images for training and 24 FAF images for testing. Optionally, retinal vessels can be removed from training and / or test images. Data augmentation methods are used to increase the size of training data, and 880 (image) fragments of 128x128 pixels are generated for training. In this U-Net, the contraction path consists of four convolutional neural network (CNN) blocks, and each CNN block consists of 2 CNN layers and a maximum pooling layer. The bottleneck (e.g., the block between the contraction path and the expansion path) consists of 2 CNN layers with an optional 0.5 reverse random inactivation (inverse dropout) to avoid overfitting problems. The expansion path of U-net is usually symmetrical with the contraction path, and in this paper, it consists of four CNN blocks behind the bottleneck. Each CNN block in the expansion path consists of a deconvolution layer and a connection layer and subsequent 2 CNN layers. The last CNN block in this expansion path (e.g. the fourth CNN block) provides a segmentation output that can be fed to an optional classification layer for labeling. This final classifier forms the final layer. This implementation uses a custom Dice coefficient loss function to train the machine learning model, but other loss functions such as the cross entropy loss function can also be used. The segmentation performance of the DL ML model can be fine-tuned by using post-learning optimization. For example, a method called 'Icing on the Cake' can be used, which trains only the final classifier (i.e. the last layer) again after the initial (normal) training is completed.

[0062] The DL method was compared with manually segmented GA regions. Figure 8 Five GA regions manually delineated by expert raters are shown, and Fig. 9The corresponding GA delineation provided by the present DL machine model is shown. For evaluation purposes, the fractional area difference, overlap rate and Pearson correlation between the measured regions were determined. The fractional area difference between the GA regions generated by the DL machine model and the GA regions generated by manual segmentation was 4.40% ± 3.88%. The overlap rate of manual and DL automatic segmentation was 92.76 ± 5.62, and the GA regions generated by the DL algorithm correlated with the manually graded GA regions by 0.995 (p-value < 0.001). Therefore, quantitative and qualitative evaluations show that the proposed DL model for segmenting GA in FAF images shows very strong agreement with expert manual grading.

[0063] Therefore, from Figure 6-9 It can be clearly seen that both SVM-based and DL-based classification provide good initial GA delineation or segmentation.

[0064] The GA segmentation result from step S7 can be submitted to the active contour algorithm as the initial contour selection (step S9). The present embodiment uses an improved Chan-Vese (CV) active contour segmentation algorithm. Typically, active contour algorithms have been used for GA segmentation in optical coherence tomography (OCT) images, as described in Niu, S. et al., "Automated geographicatrophy segmentation for SD-OCT images using region-based CV model via localsimilarity factor", Biomedical Optics Express, 2016 Feb 1, 7 (2), 581-600, the entire contents of which are incorporated herein by reference. However, in this case, the Chan-Vese active contour segmentation is used as the second stage of the two-part segmentation process. The initial contour provided by the SVM or DL ​​learning model classifier is close to the GA boundary annotated by the expert. This reduces the execution time of the Chan-Vese active contour segmentation and improves its performance.

[0065] The final GA segmentation result provided by the two-stage segmentation block B5 can optionally be submitted to additional morphological operations (step S11) to further refine the segmentation. Morphological operations (such as erosion, dilation, opening and closing) are applied to the output of the Chan-Vese active contour to refine the contour boundaries and remove small isolated areas.

[0066] Next, the size of the segmented GA is determined. Since the GA segmentation comes from a 2D image, but the fundus is curved, the 2D GA segmentation can be mapped to a 3D space (step S13) and can be measured in the 3D space to account for distortion and obtain more accurate area measurements. Any suitable method for mapping from 2D pixels to 3D coordinates based on known imaging geometry can be used. For example, a method well known in the art for mapping pixels in a 2D image plane to points on a sphere (e.g., 3D space) is stereographic projection. Other examples of 3D reconstruction from 2D fundus images are provided in Cheriyan, J. et al., "3D Reconstruction of Human Retina from Fundus Image-A Survey", international Journal of Modem Engineering Research, Vol. 2, Issue. 5, Sep.-Oct. 2012 pp-3089-3092, the entire contents of which are incorporated herein by reference.

[0067] Determining GA size measurements in 3D space can utilize additional information provided by the ophthalmic imaging system, if available. For example, some ophthalmic imaging systems store 3D positions in an array along with the 2D coordinates of each pixel position in the 2D image. This method may be included in the Digital Imaging and Communications in Medicine (DICOM) standard wide-field ophthalmic photography image module. Alternatively or in combination, some systems may store a model that can be run on any set of 2D positions to generate the associated 3D position.

[0068] If the determined area in 3D space is less than a predefined threshold (step S15 = Yes), then the GA segmentation is determined to be too small and not a true GA lesion, and the process proceeds to step S29 to generate a report. If the determined area is not less than a predefined threshold, e.g., 96K μm 2 , (step S15 = No), the GA segmentation is accepted as a true GA lesion, and the process attempts to identify the phenotype of the identified GA segmentation.

[0069] The first step in identifying the phenotype may be to analyze the contour smoothness of the GA segmentation (step S17). This may include multiple sub-steps. In order to identify and classify the junction area near the GA segmentation region (e.g., along the edge of the GA segmentation region), it may be started by determining the centroid of the GA segmentation region. This centroid calculation may be applied to a 2D image representation of the GA segmentation region. Fig.10 A diffuse phenotype GA region 21 is shown, with its centroid represented by circle 23. The next step may be to identify a set of random points that are equally spaced along the perimeter of the GA segmentation. Fig.11A band phenotype GA region 25 is shown having a centroid 27 and a set of thirteen random points 29 (each identified as the center of a white crosshair) equally spaced along the perimeter of the GA segment. It should be understood that thirteen is an exemplary value and any number of points sufficient to surround the perimeter of the GA region may be used. The distance (e.g., linear distance 31) from each point 29 to the centroid 27 is then calculated. Variations in distance 31 are used to distinguish a 'diffuse' phenotype from other phenotypes, such as a 'band' phenotype.

[0070] In step S19, if the distance change is greater than the distance-change threshold (step S19=Yes), the GA region is identified as a 'diffuse phenotype' (step S21) and the process proceeds to step S29, where a report is generated. This distance change or contour non-uniformity measure can be determined as the average distance change of the current GA segmentation, and the distance change threshold can be defined as 20% of the normalized average change of the (e.g. diffuse) GA segments. The normalized average change can be determined from a library of GA regions, such as a (normalized) library of FAF images of GA lesions. However, if the distance change is not greater than the distance change threshold (step S19=No), the process proceeds to step S23).

[0071] In step S23, intensity ridges and valleys are determined for (at least a portion of) the set of random points 29 along a direction perpendicular to the GA contour. This can be done using Hessian filter gradient derivatives and / or directional Gaussian filters. If the presence of ridges and valleys (light intensity regularity - hypofluorescence) above a preset intensity variation threshold exceeds a predetermined percentage (e.g. 60%) of the selected points 29 (step S25 = yes), the image (e.g. GA segmentation) is classified as a 'band-like' phenotype (step 27) and the process proceeds to step S29, where a report is generated. The intensity ridges and valleys may define an intensity variation measure, and the intensity variation threshold may be defined as 33% of the normalized mean intensity variation of the (e.g. band-like) GA segment. Such a normalized mean intensity variation may be determined from a library of GA segments, such as a (normalized) library of FAF images of GA lesions. If the light intensity regularity - hypofluorescence is not greater than this intensity variation threshold (step S25 = no), the phenotype cannot be determined and the process proceeds to step S29.

[0072] As mentioned above, a neural network may be trained to provide a GA segmentation, as described above with reference to step S7. However, a neural network may also be trained, e.g. Fig.19A U-Net may be trained to provide phenotypic classification. For example, by providing expert labeling of the specific phenotype of each expert-delineated GA lesion segmentation in the training image set, the neural network may be trained to provide phenotypic classification as well as or in addition to GA lesion segmentation recognition. In this case, the phenotypic recognition / classification steps associated with steps S17 to S25 may be omitted and provided by the neural network. Alternatively, the phenotypic classification provided by the neural network may be combined with the phenotypic recognition results of steps S17 to S25, such as by a weighted average.

[0073] Step S29 generates a report summarizing the results of this process. For images with non-scalable IQ (step S3 = yes), only the image quality is reported, while for other images, the generated report will have one or more of the following:

[0074] a. Area measurement of the currently visited GA segment;

[0075] b. The junctional zone phenotype of the current GA (if available);

[0076] c. Risk of progression (e.g., the risk of progression is reported as high for the ‘banded’ and ‘diffuse’ phenotypes);

[0077] d. Recommended follow-up time.

[0078] Fig.12 Another example method for automatically classifying geographic atrophy (GA) in the eye is shown. In this example, instead of checking each specific GA phenotype in the phenotype library (e.g., 'diffuse' phenotype and 'band-like' phenotype), once the current GA area is identified as any phenotype associated with a high progression rate (e.g., empirically associated), the method can stop classifying the current GA area. Then, the current GA area can be simply classified as a "high progression rate" GA without specifying its specific GA phenotype. Optionally, if necessary, such as by using user input via a graphical user interface (GUI), the process can continue to classify the current GA segmentation into one or more specific phenotypes. In this case, the method can further identify the "high progression rate" GA as a 'diffuse' phenotype or a 'band-like' phenotype. Optionally, the method can sort the search for GA phenotypes according to which phenotypes are more prevalent in a specific population (e.g., which the patient may belong to), or sort the search according to which phenotypes have been determined (e.g., empirically) to be associated with a higher progression rate GA than other phenotypes. For example, based on empirical observations, 'diffuse' phenotype GA may be considered to have a higher progression rate than 'banded' phenotype GA, and therefore, the method may first check whether the current GA segmentation belongs to the 'diffuse' phenotype, and then check the 'banded' phenotype only if it is determined that the current GA segmentation is not the 'diffuse' phenotype.

[0079] The method may start with method step M1 by acquiring an image of the fundus using an ophthalmological diagnostic device. The image (such as an autofluorescence image or a frontal image) may be generated by a fundus imager or OCT. The ophthalmological diagnostic device may be a device that generates the image, or alternatively may be a computing device that accesses the image from a data store of such images via a computer network (such as the Internet).

[0080] In method step M3, the acquired image is submitted to an automatic GA identification process, which identifies the GA region in the image. The automatic GA identification process may include a GA segmentation process, as known in the art. However, a preferred segmentation process is a two-step segmentation that combines GA classification (e.g., pixel by pixel) with active contour segmentation. The first step in the two-step process may be a trained machine model, such as an SVM or (e.g., deep learning) neural network, which segments / classifies / identifies the GA region in the image. For example, a neural network based on a U-Net architecture may be used, wherein its training set may include a training output set of professionally demarcated fluorescence images and a training input set of corresponding non-demarcated fluorescence images. Regardless of the type of GA segmentation / classification used in this initial stage, the identified GA segmentation may be submitted as a starting point to an active contour algorithm (e.g., Chan-Vese segmentation) to further refine the GA segmentation. Optionally, the result may be submitted to a morphological operation (e.g., erosion, dilation, opening and closing) to further clean the segmentation before entering the next method step M5.

[0081] Optionally, before proceeding to step M5, the final GA segmentation from step M3 can be mapped to a three-dimensional space representing the fundus shape and the area of ​​the mapped GA determined. If the area is less than a predefined threshold, the GA segmentation can be reclassified as non-GA and removed from further processing. Assuming that the identified GA segmentation is large enough to qualify as a true GA area, processing can proceed to step M5.

[0082] Method step M5 analyzes the identified GA area by determining one or two different measurements, each designed to identify one of two different GA phenotypes associated with high progression rate GA. More specifically, the contour inhomogeneity measurement can be used to identify a 'diffuse' phenotype GA, and the intensity uniformity measurement can be used to identify a 'banded' phenotype GA. If the first determined measurement confirms a 'diffuse' phenotype GA or a 'banded' phenotype GA (e.g., the measurement is above a predefined threshold), method step M7 classifies the identified GA area as a 'high progression rate' GA. If the first determined measurement does not confirm one of the 'diffuse' or 'banded' phenotypes, a second measurement can be determined to check whether the other of the two phenotypes is present. If the other phenotype is present, the identified GA area can again be classified as a 'high progression rate' GA. Optionally or alternatively, method step M7 can specify the specific phenotypic classification ('diffuse' or 'banded') identified for the GA area.

[0083] Fundus imaging system

[0084] Two types of imaging systems used to image the fundus are flood illumination imaging systems (or flood illumination imagers) and scanning illumination imaging systems (or scanning imagers). Flood illumination imagers simultaneously flood light onto the entire target field of view (FOV) of the specimen, such as by using a flash, and capture a full-frame image of the specimen (e.g., the fundus) using a full-frame camera (e.g., a camera with a two-dimensional (2D) photo sensor array that is large enough to capture the desired FOV as a whole). For example, a flood illumination fundus imager floods light onto the fundus of the eye, capturing a full-frame image of the fundus in a single image acquisition sequence of the camera. Scanning imagers provide a scanning beam that is scanned over an object (e.g., an eye), and images are formed at different scanning locations as the scanning beam is scanned over the object, thereby creating a series of image segments that can be reconstructed (e.g., montaged) to create a composite image of the desired FOV. The scanning beam can be a point, a line, or a two-dimensional area, such as a slit or a wide line.

[0085] Fig.13An example of a slit scanning ophthalmic system SLO-1 for imaging the fundus F is shown, which is the inner surface of the eye E opposite the eye lens (or crystalline lens) CL and may include the retina, optic disc, macula, fovea, and posterior pole. In this example, the imaging system is in a so-called "scan-descan" configuration, in which a scan line beam SB passes through the fundus F through the optical components of the eye E to be scanned (including the cornea Crn, iris Irs, pupil Ppl and lens CL). In the case of a floodlight fundus imager, no scanner is required and light is applied to the entire desired field of view (FOV) at once. Other scanning configurations are known in the art, and the specific scanning configuration is not critical to the present invention. As shown, the imaging system includes one or more light sources LtSrc, preferably a multi-color LED system or a laser system, in which the spread has been appropriately adjusted. An optional slit Slt (adjustable or static) can be placed in front of the light source LtSrc and can be used to adjust the width of the scan line beam SB. In addition, the slit Slt can remain stationary during imaging, or can be adjusted to different widths to allow different confocal levels and for different applications for specific scanning or for suppressing reflections during scanning. An optional objective lens ObjL can be placed in front of the slit Slt. The objective lens ObjL can be any of the most advanced lenses, including but not limited to refraction, diffraction, reflection or hybrid lenses / systems. The light from the slit Slt passes through the pupil spectroscope SM and is directed to the scanner LnScn. It is desirable to bring the scanning plane and the pupil plane as close together as possible to reduce the vignetting in the system. An optional optical device DL can be included to manipulate the optical distance between the images of the two components. The pupil spectroscope SM can pass the illumination beam from the light source LtSrc to the scanner LnScn, and reflect the detection beam from the scanner LnScn (e.g., reflected light returned from the eye E) to the camera Cmr. The task of the pupil spectroscope SM is to separate the illumination beam and the detection beam and help suppress system reflections. The scanner LnScn may be a rotating galvanometer scanner or other type of scanner (e.g., a piezoelectric or voice coil, a microelectromechanical system (MEMS) scanner, an optoelectronic deflector, and / or a rotating polygon scanner). Depending on whether pupil splitting is done before or after the scanner LnScn, the scan may be divided into two steps, with one scanner in the illumination path and a separate scanner in the detection path. A specific pupil splitting arrangement is described in detail in U.S. Patent No. 9,456,746, which is incorporated herein by reference in its entirety.

[0086] The illumination beam from the scanner LnScn passes through one or more optical devices, in this case a scanning lens SL and an ophthalmic lens or eyepiece OL, which allows the pupil of the eye E to be imaged to the imaging pupil of the system. Typically, the scanning lens SL receives the scanning illumination beam from the scanner LnScn at any one of a plurality of scanning angles (incident angles) and generates a scanning line beam SB having a substantially flat surface focal plane (e.g., a collimated optical path). The ophthalmic lens OL can focus the scanning line beam SB onto the fundus F (or retina) of the eye E and image the fundus. In this way, the scanning line beam SB generates a traversing scan line that passes through the fundus F. One possible configuration of these optical devices is a Keplerian telescope, in which the distance between the two lenses is selected to create an approximately telecentric intermediate fundus image (4-f configuration). The ophthalmic lens OL can be a single lens, an achromatic lens, or an arrangement of different lenses. As known to those skilled in the art, all lenses can be refractive, diffractive, reflective, or hybrid. The focal length of the ophthalmic lens OL, the scanning lens SL, and the size and / or shape of the pupillary beam splitter SM and the scanner LnScn can be different, depending on the desired field of view (FOV), and therefore, depending on the field of view, an arrangement can be envisioned in which multiple components can be switched in and out of the beam path, for example by using optical elements, motorized wheels, or flipping of removable optical elements. Since the change in field of view results in different beam sizes on the pupil, the pupillary beam splitting can also change with the change in FOV. For example, a field of view of 45° to 60° is a typical or standard FOV for fundus cameras. Higher fields of view, such as wide field FOVs of 60°-120° or more may also be feasible. The combination of a wide line fundus imager (BLFI) with other imaging modalities such as optical coherence tomography (OCT) may require a wide field FOV. The upper limit of the field of view can be determined by the accessible working distance combined with the physiological conditions around the human eye. Since the FOV of a typical human retina is 140° horizontally and 80°-100° vertically, an asymmetric field of view may be required to provide the highest possible FOV on the system.

[0087] The scan line beam SB passes through the pupil Ppl of the eye E and is directed toward the retina and fundus surface F. The scanner LnScn1 adjusts the position of the light on the retina or fundus F to illuminate a series of lateral locations on the eye E. The reflected or scattered light (or emitted light in the case of fluorescence imaging) is directed back to the camera Cmr along a path similar to the illumination to define a collection beam CB on a detection path.

[0088] In the "scan-reverse scan" configuration of the present exemplary slit scanning ophthalmic system SLO-1, the light returning from the eye E is "reverse scanned" by the scanner LnScn on the way to the pupilloscope SM. That is, the scanner LnScn scans the illumination beam from the pupilloscope SM to define a scanned illumination beam SB that passes through the eye E, but because the scanner LnScn also receives the return light from the eye E at the same scanning position, the scanner LnScn has the effect of reverse scanning the return light (e.g., canceling the scanning action) to define a non-scanning (e.g., stable or stationary) collection beam from the scanner LnScn to the pupilloscope SM, which is folded to the camera Cmr. At the pupilloscope SM, the reflected light (or emitted light in the case of fluorescence imaging) is separated from the illumination light onto a detection path directed to the camera Cmr, which can be a digital camera with a light sensor to capture the image. An imaging (e.g., objective) lens ImgL can be positioned in the detection path to image the fundus to the camera Cmr. As in the case of objective lens ObjL, imaging lens ImgL may be any type of lens known in the art (e.g., refractive, diffractive, reflective, or hybrid). Additional operational details, in particular methods of reducing image artifacts, are described in PCT Publication No. WO2016 / 124644, the contents of which are incorporated herein by reference in their entirety. Camera Cmr captures the received image, e.g., it creates an image file, which may be processed by one or more (electronic) processors or computing devices (e.g., Fig. 20 ) is further processed by a computer system shown in . Thus, the collection beam (returned from all scanning positions of the scan line beam SB) is collected by the camera Cmr, and a full-frame image Img can be constructed (e.g., by montage) from a complex of individually captured collection beams. However, other scanning configurations are also contemplated, including configurations in which the illumination beam is scanned over the eye E and the collection beam is scanned over the light sensor array of the camera. PCT Publication WO2012 / 059236 and U.S. Patent Publication No. 2015 / 0131050, incorporated herein by reference, describe multiple embodiments of a slit scanning ophthalmoscope, including various designs in which the return light is scanned across the light sensor array of the camera and in which the return light is not scanned across the light sensor array of the camera.

[0089] In this example, the camera Cmr is connected to a processor (e.g., a processing module) Proc and a display (e.g., a display module, a computer screen, an electronic screen, etc.) Dspl, both of which can be part of the imaging system itself, or can be part of a separate, dedicated processing and / or display unit, such as a computer system, where data is transferred from the camera Cmr to the computer system via a cable or a computer network (including a wireless network). The display and the processor can be integrated. The display can be a conventional electronic display / screen or a touch screen type, and can include a user interface for displaying information to and receiving information from an instrument operator or user. The user can interact with the display using any type of user input device known in the art, including but not limited to a mouse, a knob, a button, a pointer, and a touch screen.

[0090] While imaging is taking place, it may be desirable for the patient's line of sight to remain fixed. One way to achieve this is to provide a fixation target that can guide the patient's gaze. Depending on the area of ​​the eye to be imaged, the fixation target can be internal or external to the instrument. Fig.13 One embodiment of an internal fixation target is shown in FIG. In addition to the main light source LtSrc used for imaging, a second optional light source FxLtSrc, such as one or more LEDs, may be placed to image a light pattern onto the retina using a lens FxL, a scanning element FxScn, and a reflector / mirror FxM. A fixed scanner FxScn may move the position of the light pattern, and a reflector FxM directs the light pattern from the fixed scanner FxScn to the fundus F of the eye E. Preferably, the fixed scanner FxScn is positioned so that it is located at the pupil plane of the system so that the light pattern on the retina / fundus can be moved according to the desired fixed position.

[0091] Depending on the light source and wavelength selective filter elements used, the slit scanning ophthalmoscope system can operate in different imaging modes. When a series of colored LEDs (red, blue and green) are used to image the eye, true color reflectance imaging can be achieved (imaging is similar to what a clinician observes when examining the eye using a handheld or slit lamp ophthalmoscope). The image of each color can be gradually established with each LED turned on at each scanning position, or the image of each color can be taken completely separately. The three-color images can be combined to display a true color image, or they can be displayed separately to highlight different features of the retina. The red channel highlights the choroid well, the green channel highlights the retina, and the blue channel highlights the anterior retinal layer. In addition, light at a specific frequency (such as a separate colored LED or laser) can be used to excite different fluorophores in the eye (such as autofluorescence), and the resulting fluorescence can be detected by filtering out the excitation wavelength.

[0092] Fundus imaging systems can also provide infrared (IR) reflectance images, such as by using an infrared laser (or other infrared light source). The advantage of infrared (IR) mode is that the eye is not sensitive to IR wavelengths. This can allow the user to continuously capture images (e.g., in preview / alignment mode) without disturbing the eye to assist the user during instrument alignment. In addition, IR wavelengths increase penetration into tissues and can provide improved visualization of choroidal structures. In addition, fluorescein angiography (FA) and indocyanine green angiography (ICG) imaging can be accomplished by collecting images after the fluorescent dye is injected into the subject's bloodstream.

[0093] Optical Coherence Tomography Imaging System

[0094] In addition to fundus photography, fundus autofluorescence (FAF), fluorescein angiography (FA), ophthalmic images may also be created by other imaging modalities, such as optical coherence tomography (OCT), OCT angiography (OCTA), and / or ocular ultrasound. As understood in the art, the present invention or at least a portion of the present invention with slight modifications may be applied to these other ophthalmic imaging modalities. More specifically, the present invention may also be applied to ophthalmic images produced by an OCT / OCTA system that produces OCT and / or OCTA images. For example, the present invention may be applied to frontal OCT / OCTA images. Examples of fundus imagers are provided in U.S. Patents 8,967,806 and 8,998,411, examples of OCT systems are provided in U.S. Patents 6,741,359 and 9,706,915, and examples of OCTA imaging systems can be found in U.S. Patents 9,700,206 and 9,759,544, all of which are incorporated herein by reference in their entirety. For completeness, an exemplary OCT / OCTA system is provided here.

[0095] Fig.14A generalized frequency domain optical coherence tomography (FD-OCT) system for collecting 3-D image data of an eye suitable for the present invention is shown. The FD-OCT system OCT_1 includes a light source LtSrc1. Typical light sources include, but are not limited to, broadband light sources or swept laser sources with short temporal coherence lengths. The light beam from the light source LtSrc1 is typically routed through an optical fiber Fbr1 to illuminate a sample, such as an eye E; a typical sample is tissue in a human eye. The light source LrSrc1 can be either a broadband light source with a short temporal coherence length in the case of spectral domain OCT (SD-OCT) or a wavelength tunable laser source in the case of swept source OCT (SS-OCT). The light can be scanned, typically using a scanner Scnr1 between the output of the optical fiber Fbr1 and the sample E, so that the light beam (dashed line Bm) scans laterally (in x and y) over the sample area to be imaged. In the case of full-field OCT, no scanner is required, and the light is applied to the entire desired field of view (FOV) at once. Scattered light from the sample is typically collected into the same optical fiber Fbr1 used to route the light used for illumination. Reference light from the same light source LtSrc1 travels along a separate path, in this case involving optical fiber Fbr2 and a retroreflector RR1 with an adjustable optical delay. Those skilled in the art will recognize that a transmission reference path can also be used and that the adjustable delay can be placed in the sample or reference arm of the interferometer. The collected sample light is typically combined with the reference light in a fiber coupler Cplr1 to form optical interference in an OCT light detector Dtctr1 (e.g., a photodetector array, a digital camera, etc.). Although a single fiber port leading to the detector Dtctr1 is shown, those skilled in the art will recognize that interferometers of various designs can be used for balanced or unbalanced detection of interference signals. The output from the detector Dtctr1 is provided to a processor Cmp1 (e.g., a computing device), which converts the observed interference into depth information for the sample. The depth information can be stored in a memory associated with the processor Cmp1 and / or displayed on a display (e.g., a computer / electronic display / screen) Scn1. The processing and storage functions may be located within the OCT instrument or may be located in an external processing unit (e.g. Fig. 20 The collected data will be transmitted to the processing unit. The unit can be dedicated to data processing or perform other very general tasks that are not dedicated to the OCT device. For example, the processor Cmp1 can contain a field programmable gate array (FPGA), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a system on a chip (SoC), a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), or a combination thereof, which performs some or all of the data processing steps before passing it to the host processor or in parallel.

[0096] The sample arm and reference arm in the interferometer can consist of bulk optics, fiber optics or hybrid bulk optics systems and can have different architectures such as Michelson, Mach-Zehnder or common path based designs known to those skilled in the art.Beam as used herein should be interpreted as any carefully directed optical path. Instead of mechanically scanning the beam, a light field can illuminate a one-dimensional or two-dimensional region of the retina to generate OCT data (see, e.g., U.S. Pat. No. 9,332,902; D. Hillmann et al, "Holoscopy-holographic optical coherence tomography" Optics Letters 36(13):2390 2011; Y. Nakamura, et al, "High-Speed ​​three dimensional human retinal imaging by line field spectral domain optical coherence tomography" Optics Express 15(12):7103 2007; Blazkiewicz et al, "Signal-to-noise ratio study of full-field Fourier-domain optical coherence tomography" Applied Optics 44(36):7722 (2005)). In a time-domain system, the reference arm needs to have an adjustable optical delay to generate interference. Balanced detection systems are typically used in TD-OCT and SS-OCT systems, while spectrometers are used in the detection port of SD-OCT systems. The invention described herein can be applied to any type of OCT system. Various aspects of the invention can be applied to any type of OCT system or other type of ophthalmic diagnostic system and / or multiple ophthalmic diagnostic systems, including but not limited to fundus imaging systems, visual field testing devices, and scanning laser polarimeters.

[0097] In Fourier domain optical coherence tomography (FD-OCT), each measurement is a real-valued spectral interferogram (Sj(k)). The real-valued spectral data usually undergoes multiple post-processing steps, including background subtraction, dispersion correction, etc. The Fourier transform of the processed interferogram produces a complex-valued OCT signal output The absolute value of this complex OCT signal |Aj| reveals the distribution of scattered intensity at different path lengths and therefore scattering according to the depth (z-direction) in the sample. Similarly, the phase The scattering profile according to depth is called an axial scan (A-scan). A set of A-scans measured at adjacent positions in the sample generates a cross-sectional image (tomographic scan or B-scan) of the sample. The collection of B-scans collected at different lateral positions of the sample constitutes a data volume or cube. For a specific data volume, the term fast axis refers to the scanning direction along a single B-scan, and the slow axis refers to the axis along which multiple B-scans are collected. The term "cluster scan" may refer to a single unit or data block generated by repeated acquisition at the same (or substantially the same) position (or area) for the purpose of analyzing motion contrast, which can be used to identify blood flow. A cluster scan may consist of multiple A-scans or B-scans collected at relatively short time intervals at approximately the same position on the sample. Since the scans in a cluster scan are in the same area, static structures remain relatively unchanged between scans within the cluster scan, and motion contrast between scans that meet predefined criteria can be identified as blood flow. Various ways of generating B-scans are known in the art, including but not limited to: along the horizontal or x direction, along the vertical or y direction, along the diagonal of x and y, or in a circular or spiral pattern. The B-scan may be in the xz dimension, but may be any cross-sectional image that includes the z-dimension.

[0098] In OCT angiography or functional OCT, analysis algorithms can be applied to OCT data (e.g., cluster scans) acquired at the same or approximately the same sample location on a sample at different times to analyze motion or flow (see, e.g., U.S. Patent Application Nos. 2005 / 0171438, 2012 / 0307014, 2010 / 0027857, 2012 / 0277579, and U.S. Patent No. 6,549,801, all of which are incorporated herein by reference in their entirety). The OCT system can use any of a variety of OCT angiography processing algorithms (e.g., motion contrast algorithms) to identify blood flow. For example, the motion contrast algorithm can be applied to intensity information derived from the image data (intensity-based algorithms), phase information from the image data (phase-based algorithms), or complex image data (complex-based algorithms). The frontal image is a 2D projection of the 3D OCT data (e.g., by averaging the intensity of each individual A-scan, so that each A-scan defines a pixel in the 2D projection). Similarly, an en face vasculature image is an image displaying a motion contrast signal in which the dimension of data corresponding to depth (e.g., along the z direction of an A-scan) is displayed as a single representative value (e.g., a pixel in a 2D projection image), typically by summing or integrating all or isolated portions of the data (see, e.g., U.S. Patent No. 7,301,644, which is incorporated herein by reference in its entirety). An OCT system that provides angiographic imaging functionality may be referred to as an OCT angiography (OCTA) system.

[0099] Fig.15An example of an en face vasculature image is shown. After processing the data using any motion contrast technique known in the art to highlight motion contrast, a series of pixels corresponding to a given tissue depth from the retinal inner limiting membrane (ILM) surface can be summed to generate an en face (e.g., frontal view) image of the vasculature.

[0100] Neural Networks

[0101] As discussed above, the present invention may use a neural network (NN) machine learning (ML) model. For completeness, a general discussion of neural networks is provided herein. The present invention may use any of the neural network architectures described below, alone or in combination. A neural network or neural net is a (node) network of interconnected neurons, where each neuron represents a node in the network. Groups of neurons may be arranged in layers, with the output of one layer being fed forward to the next layer in a multilayer perceptron (MLP) arrangement. MLP may be understood as a feedforward neural network model that maps a set of input data to a set of output data.

[0102] Fig.16 An example of a multilayer perceptron (MLP) neural network is shown. Its structure may include multiple hidden (e.g., internal) layers HL1 to HLn, which map an input layer InL (receiving a set of inputs (or vector inputs) input_1 to input_3) to an output layer OutL, which produces a set of outputs (or vector outputs), such as output_1 and output_2. Each layer may have any given number of nodes, exemplarily shown herein as circles within each layer. In this example, the first hidden layer HL1 has two nodes, while the hidden layers HL2, HL3, and HLn each have three nodes. In general, the deeper the MLP (e.g., the larger the number of hidden layers in the MLP), the greater its learning ability. The input layer InL receives a vector input (exemplarily shown as a three-dimensional vector composed of input_1, input_2, and input_3), and may apply the received vector input to the first hidden layer HL1 in the hidden layer sequence. The output layer OutL receives the output from the last hidden layer (e.g., HLn) in the multilayer model, processes its input, and produces a vector output result (exemplarily shown as a two-dimensional vector composed of output_1 and output_2).

[0103] Typically, each neuron (or node) produces a single output that is fed to the neurons in the layer immediately following it. But each neuron in the hidden layer can receive multiple inputs, either from the input layer or from the output of the neuron in the immediately preceding hidden layer. Usually, each node can apply a function to its input to produce an output for the node. The nodes in the hidden layer (e.g., learning layer) can apply the same function to their respective inputs to produce their respective outputs. However, some nodes, such as the nodes in the input layer InL, receive only one input and may be passive, meaning that they simply forward the value of their single input to their output, such as they provide a copy of their input to their output, as shown by the dotted arrows in the input layer InL nodes.

[0104] For illustration purposes, Fig.17 A simplified neural network consisting of an input layer InL', a hidden layer HL1', and an output layer OutL' is shown. The input layer InL' is shown as having two input nodes i1 and i2, which receive inputs input_1 and input_2, respectively (e.g., the input nodes of layer InL' receive a two-dimensional input vector). The input layer InL' feeds forward to a hidden layer HL1' having two nodes h1 and h2, which in turn feeds forward to an output layer OutL' of two nodes o1 and o2. The interconnections or links between neurons (illustratively shown as solid arrows) have weights w1 to w8. Typically, in addition to the input layer, a node (neuron) may receive as input the output of its nearest previous layer node. Each node can calculate its output by multiplying each of its inputs by the corresponding interconnection weight for each input, summing the products of its inputs, adding (or multiplying) a constant defined by another weight or bias that may be associated with that particular node (e.g., node weights w9, w10, w11, w12 for nodes h1, h2, o1, and o2, respectively), and then applying a nonlinear or logarithmic function to the result. Nonlinear functions may be referred to as activation functions or transfer functions. A variety of activation functions are known in the art, and the choice of a particular activation function is not critical to this discussion. However, it is important to note that the operation of an ML model or the behavior of a neural net depends on the weight values, which can be learned so that the neural network provides the desired output for a given input.

[0105] The neural network learns (e.g., is trained to determine) appropriate weight values ​​to achieve the desired output for a given input during a training or learning phase. Before training the neural network, each weight can be individually assigned an initial (e.g., random and optionally non-zero) value, such as a random number seed. Various methods for assigning initial weights are known in the art. The weights are then trained (optimized) so that for a given training vector input, the neural network produces an output close to the expected (predetermined) training vector output. For example, the weights can be incrementally adjusted over thousands of iterations by a technique called backpropagation. In each backpropagation cycle, the training input (e.g., vector input or training input image / sample) is fed forward through the neural network to determine its actual output (e.g., vector output). The error of each output neuron or output node is then calculated based on the actual neuron output and the target training output of the neuron (e.g., the training output image / sample corresponding to the current training input image / sample). The weights are then propagated back through the neural network (in the direction from the output layer back to the input layer), and the weights are updated based on the degree of influence of each weight on the overall error so that the output of the neural network is closer to the desired training output. This cycle is then repeated until the actual output of the neural network is within an acceptable error range of the expected training output for the given training input. It is understood that multiple back-propagation iterations may be required for each training input to achieve the desired error range. Typically, an epoch refers to one back-propagation iteration (e.g., one forward propagation and one backward propagation) for all training samples, so training a neural network may require many epochs. In general, the larger the training set, the better the performance of the trained ML model, so various data augmentation methods can be used to increase the size of the training set. For example, when the training set includes paired corresponding training input images and training output images, the training images can be divided into multiple corresponding image segments (or fragments). The corresponding fragments from the training input images and the training output images can be paired to define multiple training fragment pairs from one input / output image pair, which expands the training set. However, training on large training sets places high demands on computing resources, such as storage and data processing resources. The computing requirements can be reduced by dividing the large training set into multiple small fragments, where the mini-batch size defines the number of training samples in one forward / backward pass. In this case, an epoch may include multiple mini-batches. Another problem is the possibility of the NN overfitting the training set, thereby reducing the ability to generalize from a specific input to a different input. The problem of overfitting can be alleviated by creating an ensemble of neural networks or randomly deleting nodes in the neural network during training, which effectively removes the dropped nodes from the neural network. Various dropout regulation methods are known in the art, such as inverse dropout.

[0106] It should be noted that the operation of the trained NN machine model is not a simple algorithm of operational / analytical steps. In fact, when the trained NN machine model receives an input, the input is not analyzed in the traditional sense. Instead, regardless of the subject matter or nature of the input (e.g., a vector defining a real-time image / scan or a vector defining some other entity, such as a demographic description or activity record), the input will be subjected to the same predefined architectural construction as the trained neural network (e.g., the same node / layer arrangement, trained weights and bias values, predefined convolution / deconvolution operations, activation functions, pooling operations, etc.), and it may not be clear how the architectural construction of the trained network produces its output. In addition, the values ​​of the trained weights and biases are not deterministic and depend on many factors, such as the time given to the neural network to train (e.g., the number of epochs in training), the random starting values ​​of the weights before training begins, the computer architecture of the machine training the NN, the choice of training samples, the distribution of training samples in multiple mini-batches, the choice of activation function, the choice of error function that modifies the weights, and even training interrupted on one machine (e.g., having a first computer architecture) and completed on another machine (e.g., having a different computer architecture). The key is that the reasons why a trained ML model reaches certain outputs are not clear, and a lot of research is currently underway to try to determine the factors that ML models base their outputs on. Therefore, the processing of real-time data by a neural network cannot be reduced to a simple step-by-step algorithm. Instead, its operation depends on its training architecture, training sample set, training sequence, and various circumstances in the training of the ML model.

[0107] In summary, the construction of a NN machine learning model can include a learning (or training) phase and a classification (or operation) phase. In the learning phase, a neural network can be trained for a specific purpose, and a set of training examples can be provided, including training (sample) inputs and training (sample) outputs, and optionally a set of validation examples to test the progress of the training. In this learning process, various weights associated with the nodes and node interconnections in the neural network are incrementally adjusted to reduce the error between the actual output of the neural network and the expected training output. In this way, a multi-layer feedforward neural network (such as discussed above) can be enabled to approximate any measurable function to any desired accuracy. The result of the learning phase is a (neural network) machine learning (ML) model that has been learned (e.g., trained). In the operation phase, a set of test inputs (or real-time inputs) can be submitted to the learned (trained) ML model, which can apply what it has learned to generate output predictions based on the test inputs.

[0108] and Fig.15 and 16Like conventional neural networks, convolutional neural networks (CNNs) are composed of neurons with learnable weights and biases. Each neuron receives an input, performs an operation (e.g., a dot product), and is optionally followed by a nonlinearity. However, CNNs can receive raw image pixels on one end (e.g., the input end) and provide classification (or category) scores on the other end (e.g., the output end). Because CNNs expect images as input, they are optimized for processing volumes (e.g., the pixel height and width of the image, plus the depth of the image, e.g., the color depth, such as RGB depth defined by three colors: red, green, and blue). For example, a layer of a CNN can be optimized for neurons arranged in 3 dimensions. Neurons in a CNN layer can also be connected to small regions of the layer before it, rather than to all neurons in a fully connected NN. The final output layer of a CNN can reduce the full image to a single vector (classification) arranged along the depth dimension.

[0109] Fig.18An example convolutional neural network architecture is provided. A convolutional neural network can be defined as a sequence of two or more layers (e.g., layer 1 to layer N), where a layer can include a (image) convolution step, a (result) weighted sum step, and a nonlinear function step. Convolution can be performed on its input data by applying a filter (or kernel), for example, on a moving window across the input data to generate a feature map. Each layer and component of a layer can have different predetermined filters (from a filter bank), weights (or weighting parameters), and / or function parameters. In this example, the input data is an image, which can be the raw pixel values ​​of the image, with a given pixel height and width. In this example, the input image is shown as having a depth of three color channels RGB (red, green, and blue). Optionally, the input image can be subjected to various preprocessing, and the preprocessing results can replace the original input image or be input as a supplement to the original input image. Some examples of image preprocessing can include: retinal vascular map segmentation, color space conversion, adaptive histogram equalization, connected component generation, etc. Within a layer, a dot product can be calculated between given weights and the small areas to which they are connected in the input volume. Many ways of configuring a CNN are known in the art, but as an example, a layer can be configured to apply an element-wise activation function, such as max(0,x) with a threshold of zero. A pooling function (e.g., along the xy direction) can be performed to downsample the volume. Fully connected layers can be used to determine the classification output and produce a one-dimensional output vector, which has been found to be useful for image recognition and classification. However, for image segmentation, CNNs need to classify each pixel. Since each CNN layer tends to reduce the resolution of the input image, another stage is needed to upsample the image back to its original resolution. This can be achieved by applying a transposed convolution (or deconvolution) stage TC, which typically does not use any predefined interpolation method but has learnable parameters.

[0110] Convolutional neural networks have been successfully applied to many computer vision problems. As explained above, training CNNs usually requires large training datasets. The U-Net architecture is based on CNNs and can usually be trained on smaller training datasets than traditional CNNs.

[0111] Fig.19An example U-Net architecture is shown. This exemplary U-Net includes an input module (or input layer or stage) that receives an input U-input (e.g., an input image or image fragment) of any given size (e.g., 128x128 pixel size). The input image can be a fundus image, an OCT / OCTA front, a B-scan image, etc. However, it should be understood that the input can be of any size and dimension. For example, the input image can be an RGB color image, a monochrome image, a volume image, etc. The input image passes through a series of processing layers, each of which is illustrated with an exemplary size, but these sizes are only for illustrative purposes and will depend on, for example, the size of the image, the convolution filter, and / or the pooling stage. This architecture consists of a contraction path (consisting of four encoding modules) and an expansion path (consisting of four decoding modules), as well as four copy-and-crop links (e.g., CC1 to CC4) between the corresponding modules / stages, the copy-and-crop link copies the output of an encoding module in the contraction path and connects it to the input of the corresponding decoding module in the expansion path. This produces a typical U-shape, from which the architecture is named. The contracting path is similar to the encoder, and its basic function is to capture context through compact feature maps. In this example, each encoding module in the contracting path includes two convolutional neural network layers, which can be followed by a maximum pooling layer (e.g., a downsampling layer). For example, the input image U_input passes through two convolutional layers, each with 32 feature maps. The number of feature maps may double with each pooling, starting with 32 feature maps in the first block, 64 in the second block, and so on. Therefore, the contracting path forms a convolutional network composed of multiple encoding modules (or stages), each module providing a convolution stage followed by an activation function (e.g., a rectified linear unit, ReLU, or sigmoid function layer) and a maximum pooling operation. The expanding path is similar to the decoder, and its function is to provide localization and preserve spatial information despite the downsampling and any maximum pooling performed in the contracting stage. In the contracting path, spatial information is reduced, while feature information is increased. The expanding path includes multiple decoding modules, each of which connects its current value to the output of the corresponding encoding module. That is, the features and spatial information are combined in the expanding path through a series of upconvolutions (e.g., upsampling or transposed convolutions or deconvolutions) and connections with high-resolution features from the contracting path (e.g., through CC1 to CC4). Thus, the output of the deconvolution layer is concatenated with the corresponding (optionally cropped) feature map from the contracting path, followed by two convolutional layers and an activation function (optionally batch normalization). The output from the last module in the expanding path can be fed to another processing / training block or layer, such as a classifier block, which can be trained with the U-Net architecture.

[0112] The module / stage (BN) between the contracting path and the expanding path may be called a “bottleneck.” A bottleneck BN may consist of two convolutional layers (with batch normalization and optional dropout).

[0113] Computing devices / systems

[0114] Fig. 20 An example computer system (or computing device or computer device) is shown. In some embodiments, one or more computer systems may provide the functions described or exemplified herein and / or perform one or more steps of one or more methods described or exemplified herein. The computer system may take any suitable physical form. For example, the computer system may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a module computer (COM) or a module system (SOM)), a desktop computer system, a laptop or notebook computer system, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, the computer system may be present in the cloud, which may include one or more cloud components in one or more networks.

[0115] In some embodiments, the computer system may include a processor Cpnt1, a memory Cpnt2, a storage Cpnt3, an input / output (I / O) interface Cpnt4, a communication interface Cpnt5, and a bus Cpnt6. The computer system may also optionally include a display Cpnt7, such as a computer monitor or screen.

[0116] Processor Cpnt1 includes hardware for executing instructions (such as those constituting a computer program). For example, processor Cpnt1 may be a central processing unit (CPU) or a general purpose computing graphics processing unit (GPGPU). Processor Cpnt1 may retrieve (or obtain) instructions from internal registers, internal caches, memory Cpnt2, or memory Cpnt3, decode and execute instructions, and write one or more results to internal registers, internal caches, memory Cpnt2, or memory Cpnt3. In a particular embodiment, processor Cpnt1 may include one or more internal caches for data, instructions, or addresses. Processor Cpnt1 may include one or more instruction caches, one or more data caches, for example, to store data tables. The instructions in the instruction cache may be copies of instructions in memory Cpnt2 or memory Cpnt3, and the instruction cache may speed up the retrieval of these instructions by processor Cpnt1. Processor Cpnt1 may include any suitable number of internal registers, and may include one or more arithmetic logic units (ALUs). Processor Cpnt1 may be a multi-core processor; or may include one or more processors Cpnt1. Although the present disclosure describes and illustrates a specific processor, the present disclosure contemplates any suitable processor.

[0117] Memory Cpnt2 may include main memory for storing instructions for processor Cpnt1 to execute or to save temporary data during processing. For example, a computer system may load instructions or data (e.g., a data table) from memory Cpnt3 or from another source (e.g., another computer system) to memory Cpnt2. Processor Cpnt1 may load instructions and data in memory Cpnt2 to one or more internal registers or internal caches. In order to execute instructions, processor Cpnt1 may retrieve and decode instructions from internal registers or internal caches. During or after execution of instructions, processor Cpnt1 may write one or more results (which may be intermediate or final results) to internal registers, internal caches, memory Cpnt2, or memory Cpnt3. Bus Cpnt6 may include one or more memory buses (each of which may include an address bus and a data bus) and may couple processor Cpnt1 to memory Cpnt2 and / or memory Cpnt3. Optionally, one or more memory management units (MMUs) facilitate data transfer between processor Cpnt1 and memory Cpnt2. Memory Cpnt2 (which may be fast volatile memory) may include random access memory (RAM), such as dynamic RAM (DRAM) or static RAM (SRAM). Memory Cpnt3 may include long-term or mass storage of data or instructions. Memory Cpnt3 may be internal or external to the computer system and include one or more disk drives (such as hard disk drives HDD or solid-state drives SSD), flash memory, ROM, EPROM, optical disks, magneto-optical disks, magnetic tapes, universal serial bus (USB) accessible drives or other types of non-volatile memory.

[0118] The I / O interface Cpnt4 may be software, hardware, or a combination of both, and includes one or more interfaces (e.g., serial or parallel communication ports) for communicating with an I / O device, which may enable communication with a person (e.g., a user). For example, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more of these.

[0119] The communication interface Cpnt5 can provide a network interface for communicating with other systems or networks. The communication interface Cpnt5 may include a Bluetooth interface or other types of data packet-based communications. For example, the communication interface Cpnt5 may include a network interface controller (NIC) and / or a wireless NIC or a wireless adapter for communicating with a wireless network. The communication interface Cpnt5 may provide communication with a WI-FI network, an ad hoc network, a personal area network (PAN), a wireless PAN (e.g., a Bluetooth WPAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a cellular telephone network (e.g., a global system for mobile communications (GSM) network), the Internet, or a combination of two or more of these.

[0120] The bus Cpnt6 may provide a communication link between the above components of the computing system. For example, the bus Cpnt6 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth technology (InfiniBand) bus, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable buses or a combination of two or more of these.

[0121] Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0122] Herein, where appropriate, a computer-readable non-transitory storage medium or medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, a field programmable gate array (FPGA) or an application-specific IC (ASIC)), a hard disk drive (HDD), a hybrid hard disk drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy disk, a floppy disk drive (FDD), a magnetic tape, a solid-state drive (SSD), a RAM drive, a secure digital card or drive, any other suitable computer-readable non-transitory storage medium, or any suitable combination of two or more of these. Where appropriate, a computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0123] Although the present invention has been described in conjunction with a number of specific embodiments, it will be apparent to those skilled in the art that many further substitutions, modifications and variations will be apparent from the foregoing description. Therefore, the invention described herein is intended to encompass all substitutions, modifications, applications and variations that may fall within the spirit and scope of the appended claims.

Claims

1. A method for classifying geographic atrophy (GA) in an eye, comprising: acquiring a fundus image of the eye using an ophthalmological diagnostic device; Using a computer processor: submitting the acquired images to a phenotype classifier based on a machine learning model, wherein the phenotype classifier identifies GA regions and then classifies the identified GA regions as either a band phenotype or a diffuse phenotype; and displaying the results of the phenotypic classifier on an electronic display or storing the results for further processing; wherein for each of the identified GA regions, the phenotype classifier: determining a first measurement that is one of a profile non-uniformity measurement or an intensity uniformity measurement adjacent to and along a perimeter of a currently identified GA region; and In response to determining that the first measurement value is above a first threshold, the currently identified GA region is classified as a high progression rate GA.

2. The method of claim 1, wherein the phenotype classifier is a neural network trained using a training output set of fundus images with manually delineated GA areas and with their phenotypes identified and a training input set of the same fundus images without delineated GA areas and without identified phenotypes; and in, Prior to training the neural network, retinal blood vessels are removed from the fundus images of the training input set and the training output set.

3. The method according to claim 1 or 2, further comprising: determining a region of interest (ROI) within the acquired image, the ROI including the macula of the eye; Therein, the phenotype classifier limits processing to the ROI.

4. The method according to any one of claims 1 to 3, further comprising: determining an image quality (IQ) measurement of the acquired image; Therein, in response to the determined image quality measurement being above a predefined IQ threshold, the acquired image is submitted to the phenotype classifier.

5. The method according to any one of claims 1 to 4, wherein the automatic GA region identification process is a two-stage GA region identification process comprising: A first stage based on a supervised pixel classifier, which applies pixel-by-pixel classification to the acquired image to identify a first GA region, and a second stage, using the first GA region as a starting point in the active contour segmentation process of defining a second GA region, identifies the defined second GA region as a band phenotype or a diffuse phenotype.

6. The method of claim 5, wherein the supervised pixel classifier is one of a trained neural network or a support vector machine.

7. The method according to claim 5 or 6, wherein the active contour segmentation process applies Chan-Vese active contour segmentation.

8. The method according to any one of claims 1 to 7, wherein phenotypic classification is based on the junctional regions of the identified GA regions.

9. The method according to any one of claims 1 to 8, wherein for each of the identified GA regions: The phenotype classifier maps the currently identified GA region into 3D space and determines the surface area of ​​the 3D mapped GA region; and In response to a surface area of ​​the 3D mapped GA region being less than a predefined minimum area threshold, the currently identified GA region is reclassified as a non-GA region.

10. The method of claim 1, wherein the phenotypic classifier: In response to the first measurement not being above the first threshold, determining a second measurement, the second measurement being the other of the profile non-uniformity measurement or the intensity uniformity measurement; and In response to the second measurement being above a second threshold, the currently identified GA region is classified as a high progression rate GA.

11. The method according to any one of claims 1 to 10, wherein for the currently identified GA region, the phenotypic classifier: determining a profile non-uniformity measure adjacent to and along a perimeter of said currently identified GA region; and In response to the determined profile heterogeneity measure being greater than a predefined heterogeneity threshold, the currently identified GA region is classified as a 'diffuse' phenotype.

12. The method of claim 11, wherein the non-uniformity threshold is based on a normalized mean profile variation of a sample GA region library.

13. The method according to any one of claims 1 to 12, wherein for the currently identified GA region, the phenotypic classifier: determining an intensity uniformity measure adjacent to and along a perimeter of said currently identified GA region; and In response to the determined intensity uniformity measure being greater than a predefined intensity uniformity threshold, classifying the currently identified GA region as a 'band-like' phenotype.

14. The method according to claim 13, wherein: Determining the intensity uniformity measure includes selecting a set of random points along a perimeter of the currently identified GA region and determining ridges and valleys of pixel intensity along a direction perpendicular to a perimeter contour extending from the random points; and The intensity uniformity threshold is based on the normalized average of the ridges and valleys of pixel intensity along the interface of the sample GA region.

15. The method according to any one of claims 1 to 14, wherein retinal blood vessels are removed from the acquired image before identifying the GA region.

16. A method for classifying geographic atrophy (GA) in an eye, comprising: acquiring a fundus image of the eye using an ophthalmological diagnostic device; Using a computer processor: submitting the image to an automatic GA identification process, the process identifying GA regions in the image; determining a first measurement that is one of a profile non-uniformity measurement or an intensity uniformity measurement in an image region adjacent to and along a perimeter of the identified GA region; and In response to the determined first measurement value being above a first threshold, classifying the GA region as a high progression rate GA; and The determined classification of the GA region is displayed on an electronic display or stored for further processing.

17. The method of claim 16, wherein the computer processor: In response to the first measurement not being above a first threshold, determining a second measurement, the second measurement being the other of the profile non-uniformity measurement or the intensity uniformity measurement; and In response to the second measurement value being higher than a second threshold, the currently identified GA region is identified as a high progression rate GA.

18. The method of claim 16 or 17, wherein the computer processor: identifying a region of interest (ROI) within the fundus image using an automated region of interest (ROI) identification process based at least on predefined anatomical landmarks, the ROI comprising the macula of the eye; and The automatic GA identification process is restricted to the image region within the ROI.

19. The method of claim 18, wherein the automatic region of interest (ROI) identification process comprises: identifying locations within the image where blood vessels are primarily concentrated; identifying an optic disc based at least in part on the identified location; searching for the fovea using the identified optic disc and determining whether the image is for the left eye or the right eye as a reference; and The ROI is defined based on the location of the fovea.

20. The method of claim 18 or 19, wherein a ROI consists of a plurality of sub-image regions, and the automatic GA identification process classifies each of the sub-image regions of the ROI as a GA sub-image or a non-GA sub-image on a sub-image region by sub-image region basis.

21. The method according to claim 20, wherein each of the sub-image areas consists of one pixel.

22. A method according to any one of claims 16 to 21, wherein retinal vessels are removed from the image before submission to the automatic GA identification process.

23. The method of any one of claims 16 to 22, wherein the automatic GA identification process comprises submitting the image to a trained machine learning (ML) model, which provides an initial GA segmentation.

24. The method of claim 23, wherein the ML model is a support vector machine.

25. The method of claim 23, wherein the ML model is a trained neural network based on a U-Net architecture.

26. The method according to any one of claims 23 to 25, wherein the automatic GA identification process further comprises: The results of the ML model are submitted to an active contour algorithm using the initial GA segmentation as a starting point, the results of the active contour algorithm at least partially identifying GA lesions in the image.

27. The method of claim 26, wherein the active contour algorithm is based on a modified Chan-Vese segmentation to change the energy and direction of motion of contour growing.

28. The method according to any one of claims 16 to 27, wherein the image is a planar image composed of two-dimensional (2D) pixels, and the automatic GA identification process comprises: mapping the identified GA region into a three-dimensional (3D) space to define a 3D mapped GA region; Determining the area of ​​the GA region of the 3D map; In response to the determined area of ​​the 3D mapped GA region being smaller than a predefined area threshold, the GA region is re-identified as a non-GA region.

29. The method of any one of claims 16 to 28, wherein determining the profile non-uniformity measure comprises: selecting a set of random points along the perimeter of the identified GA region; determining the distance from each random point to the centroid of the identified GA region; and The profile non-uniformity measure is based on a change in the determined distance.

30. The method of claim 29, wherein the set of random points are equally spaced around the perimeter of the GA region.

31. The method of claim 29 or 30, wherein the identified GA region is classified as a 'diffuse' phenotype based on the profile heterogeneity measure.

32. The method of any one of claims 16 to 31, wherein determining the intensity uniformity measurement comprises: selecting a set of random points along the perimeter of the identified GA region; determining an intensity variation measure for each random point based on ridges and valleys of pixel intensity along a direction perpendicular to a perimeter contour extending from each random point; and The intensity uniformity measurement is based on the intensity variation measurement.

33. The method of claim 32, wherein the identified GA region is classified as a "band-like" phenotype in response to the intensity change measure being greater than at least a predefined percentage of a predefined change threshold.

34. The method of claim 32 or 33, wherein determining the intensity variation measure comprises using at least one of a Hessian filter gradient derivative or a directional Gaussian filter.

35. A system for classifying geographic atrophy (GA) in an eye, comprising: an ophthalmological diagnostic device for acquiring a fundus image of the eye; Computer Processor: Phenotypic classifiers based on machine learning models, Display and / or storage device; The system is configured to perform the method according to any one of claims 1 to 15.

36. A system for classifying geographic atrophy (GA) in an eye, comprising: an ophthalmological diagnostic device for acquiring a fundus image of the eye; Computer Processor: an automatic GA identification processor that identifies GA regions in the image, Display and / or storage device; The system is configured to perform the method according to any one of claims 16 to 34.

Images