Multimodal prediction of geographic atrophy growth rate

Abstract

A method and system for assessing geographic atrophy in the retina is provided, comprising: receiving a set of fundus autofluorescence (FAF) images of the retina at a machine learning system; receiving a set of optical coherence tomography (OCT) images of the retina at a machine learning system; and predicting lesion growth rates for geographic atrophy lesions in the retina using the set of FAF images and the set of OCT images via the machine learning system.

Description

Cross-reference of related applications

[0001] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 121,125 entitled “Multimodal Prediction Of Geographic Atrophy Growth Rate,” filed 3 December 2020, U.S. Provisional Patent Application No. 63 / 169,764 entitled “Multimodal Prediction Of Geographic Atrophy Growth Rate,” filed 1 April 2021, U.S. Provisional Patent Application No. 63 / 181,813 entitled “Multimodal Prediction Of Geographic Atrophy Growth Rate,” filed 29 April 2021, and U.S. Provisional Patent Application No. 63 / 218,905 entitled “Multimodal Prediction Of Geographic Atrophy Growth Rate,” filed 6 July 2021, which are incorporated herein by reference in their entirety as fully described below and for all applicable purposes. [Technical Field]

[0002] field This description relates in general to the evaluation of geographic atrophy in the retina. More specifically, this description provides a method and system for predicting the growth rate of geographic atrophy lesions using images from multiple modalities, such as fundus autofluorescence (FAF) images and optical coherence tomography (OCT) images. [Background technology]

[0003] background Introduction Age-related macular degeneration (AMD) is the leading cause of vision loss in patients over 50 years of age. Geographic atrophy (GA) is one of the two progressive stages of AMD and is characterized by the progressive and irreversible loss of choroidal capillaries, retinal pigment epithelium (RPE), and photoreceptors. The progression of GA varies from patient to patient, and currently there are no FDA-approved treatments to prevent or delay the progression of GA. Therefore, predicting the progression of GA in individual patients may be important for studying GA and developing effective treatments. Currently, the diagnosis and monitoring of GA lesion expansion can be performed using fundus autofluorescence (FAF) images obtained by confocal scanning laser ophthalmography (cSLO). This type of imaging technique, which shows topographic mapping of lipofuscin in the RPE, can be used to measure changes in GA lesions over time. Furthermore, FAF images can be used to predict the rate of GA growth. However, in at least some cases, FAF images may not be able to predict the rate of GA growth with the desired level of accuracy. [Overview of the Initiative]

[0004] overview In one or more embodiments, a method for evaluating geographic atrophy in the retina is provided. A set of retinal autofluorescence (FAF) images is received by a machine learning system. A set of retinal optical coherence tomography (OCT) images is also received by the machine learning system. The lesion growth rate is predicted for geographic atrophy lesions in the retina using the set of FAF images and the set of OCT images via the machine learning system.

[0005] In one or more embodiments, a method for evaluating geographic atrophy in the retina is provided. A set of retinal autofluorescence (FAF) images is received by a machine learning system. A set of retinal infrared (IR) images is also received by the machine learning system. The lesion growth rate is predicted for geographic atrophy lesions in the retina using the set of FAF images and the set of IR images via the machine learning system. [Brief explanation of the drawing]

[0006] For a more complete understanding of the principles and advantages disclosed herein, refer to the following description in conjunction with the accompanying drawings.

[0007] [Figure 1A] This is a block diagram of the lesion evaluation system 100 according to various embodiments.

[0008] [Figure 1B] This is a schematic diagram of a lesion area analysis system 114 according to various embodiments.

[0009] [Figure 1C] This document illustrates exemplary process flows for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments.

[0010] [Figure 1D] This shows another exemplary process flow for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments.

[0011] [Figure 1E] This shows another exemplary process flow for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments.

[0012] [Figure 1F] This shows another exemplary process flow for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments.

[0013] [Figure 1G] This shows another exemplary process flow for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments.

[0014] [Figure 2] A flowchart of a process for predicting geographic atrophy according to various embodiments.

[0015] [Figure 3] A flowchart of a process for predicting geographic atrophy according to various embodiments.

[0016] [Figure 4] A flowchart of an exemplary method for predicting the lesion growth rate of geographic atrophy lesions in the retina according to various embodiments.

[0017] [Figure 5] A flowchart of another exemplary method for predicting the lesion growth rate of geographic atrophy lesions in the retina according to various embodiments.

[0018] [Figure 6] A flowchart of another exemplary method for predicting the lesion growth rate of geographic atrophy lesions in the retina according to various embodiments.

[0019] [Figure 7] An exemplary neural network that can be used to implement a deep learning neural network according to various embodiments is shown.

[0020] [Figure 8A] An exemplary single-modality multitask model according to various embodiments is shown.

[0021] [Figure 8B] An exemplary multimodality multitask model according to various embodiments is shown.

[0022] [Figure 9]This shows exemplary preprocessing steps for optical coherence tomography (OCT) volumes according to various embodiments.

[0023] [Figure 10] This shows forest plots comparing the model performance of three models and a benchmark model on (A) the development dataset and (B) the holdout dataset, according to various embodiments.

[0024] [Figure 11] The following are scatter plots of predicted GA lesion area versus observed GA lesion area and GA growth rate on holdout datasets according to various embodiments.

[0025] [Figure 12] The image shows residual plots of predicted GA lesion area versus observed GA lesion area and GA growth rate on a holdout dataset according to various embodiments.

[0026] [Figure 13] This shows plots of GA growth rate predictions based on subgroup residual analysis for holdout datasets under various embodiments.

[0027] [Figure 14] This paper shows gradient activation maps (GradAM) of GA lesion area and GA growth rate predictions using FAF only, OCT only, and multimodal multitask models according to various embodiments.

[0028] [Figure 15] This is a block diagram of a computer system according to various embodiments.

[0029] It should be understood that the drawings are not necessarily drawn to a consistent scale, and the objects within the drawings are not necessarily drawn to a consistent scale with respect to each other. The drawings are intended to provide clarity and understanding of the various embodiments of the apparatus, systems, and methods disclosed herein. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Furthermore, it should be understood that the drawings are not in any way limited to the scope of this instruction. [Modes for carrying out the invention]

[0030] Detailed explanation I. Overview For example, the ability to accurately predict the progression of geographic atrophy (GA) based on baseline assessments or longitudinal data can be useful in many different scenarios. For instance, prediction of GA progression can be used to improve patient stratification in clinical trials where the goal is to slow GA progression, thereby enabling a more accurate assessment of treatment efficacy. Furthermore, in some cases, prediction of GA progression can be used to understand disease etiology through correlation with genotype or phenotypic signatures.

[0031] GA lesions can be imaged using various imaging modalities. For example, fundus autofluorescence (FAF) imaging is used to quantify GA lesion area. GA growth rate, when measured using FAF images, is the change in lesion area over a period of time and is widely accepted as an anatomical parameter of GA progression in clinical trials. Furthermore, GA growth rate can be predicted from baseline FAF images. However, in at least some cases, GA growth rate predicted using baseline FAF images may not have the desired level of accuracy.

[0032] Accordingly, the various embodiments described herein provide methods and systems for predicting GA growth rates using images from multiple modalities to improve the accuracy of these predictions. More specifically, these images from multiple modalities can be processed using a machine learning system to generate predicted GA growth rates. Using images from multiple modalities improves this prediction. For example, a retinal image from a first modality may provide more information, greater feature resolution, or both, compared to a retinal image from a second modality. However, an image from a second modality may provide some information that cannot be identified or is not easily identified using an image from a first modality. Processing the information provided from both types of modalities via a machine learning system (e.g., a neural network system, a deep learning system, etc.) can improve the understanding of this information and its use in predicting GA growth rates.

[0033] In one or more embodiments, FAF and optical coherence tomography (OCT) images are used to predict GA growth rate. FAF images are two-dimensional, while OCT images are three-dimensional (3D). Therefore, OCT images can provide additional structural information about anatomical structures of the retina, such as GA lesions, which can provide a greater understanding of the onset and progression of GA in patients. For example, pseudodrusen retina (RPD), hyperreflective foci, multilayer thickness reduction, photoreceptor atrophy, and subretinal depression (e.g., wedge-shaped subretinal depression) are attributes identifiable in OCT images that are linked as potential precursors or biomarkers of disease progression. Therefore, this type of OCT-derived information can enable improved prediction of GA growth rate.

[0034] Generally, OCT and FAF imaging provide fundamentally different signals or information. For example, FAF can capture lipofuscin autofluorescence after exposure to blue light. Lipofuscin is observable in retinal pigment epithelial (RPE) cells. Therefore, FAF images provide a single field of view of RPE cells. On the other hand, OCT captures tissue reflectivity to near-infrared light in the form of a 3D image. OCT images may provide lesion and structural information not available in FAF images, while FAF images may image parts of the retina that are not distinguishable or clearly captured by OCT imaging. As described herein, the differences in these imaging modalities provide complementary information for better visualization of the pathogenesis of GA. Furthermore, OCT and FAF are not the only imaging modalities that can be useful by the methods and systems described herein, and the description herein should not be considered to limit the application of the methods and systems described herein to only these two modalities. Not only can IR imaging provide additional values, but other imaging modalities can provide similar additional values ​​in determining GA progression.

[0035] In some cases, FAF images may provide more accurate information compared to OCT images. For example, in some cases, FAF images may provide a more accurate estimate of baseline lesion area compared to OCT images. Therefore, predicting GA growth rate using both the FAF modality and the OCT modality together may be more accurate than using either the FAF modality alone or the OCT modality alone. In one or more embodiments, using both the FAF modality and the OCT modality may help ensure that the baseline lesion area from which the GA growth rate is predicted is accurate enough to enable improved GA growth rate prediction. Thus, according to various embodiments herein, multimodal analysis of various image types (e.g., OCT and FAF imaging data) can provide both more accurate readings of GA lesion area and GA growth rate.

[0036] Accordingly, the methods and systems of this disclosure enable automated GA growth prediction using a multimodal approach and a machine learning system. This multimodal approach may use both FAF and OCT images. In other embodiments, any of the imaging modalities may be replaced or complemented by a third modality, such as, for example, an infrared (IR) modality, but not limited to these. For example, FAF and IR images may be processed via a machine learning system to generate predicted GA growth rates. IR images, particularly near-infrared (NIR) images, may provide a wider field of view than OCT in-plane images. In some cases, IR images, more specifically NIR images, combined with FAF images may provide greater clarity for GA lesions. This higher resolution and clarity may enable improved identification of lesion area for GA lesions, and therefore ultimately improved GA growth rate prediction. In yet another embodiment, the FAF, OCT, and IR modalities may be used in combination to predict GA growth rates, with each of these modalities contributing at least some information or some improvement to at least one of the other modalities.

[0037] In various embodiments, the received imaging data can be preprocessed to enable focusing on specific areas of interest within the imaging data. This can be done, for example, by reducing noise and / or artifacts in the imaging data that may impair the ability to properly evaluate specific areas of interest. Furthermore, imaging data from various modalities can be combined or fused in various ways to ensure effective multimodal analysis for determining GA growth rate and lesion area. For example, the data from these various modalities may be fused into an integrated multichannel input that can then undergo a feature extraction process and be used as the basis for determining growth rate and lesion area. In another example, features may be extracted from individual imaging modalities, and then the extracted features themselves may be fused together for determining growth rate and lesion area.

[0038] The applications of such multimodal systems and methods are wide. For example, such systems and methods can be used as predictive tools for GA growth rate. Such systems and methods can be used to determine GA lesion area. Furthermore, such systems and methods can be extremely useful in the clinical trial space, and the embodiments herein can improve the reliability of clinical trial development by providing information for the design, implementation, and analysis of clinical trials. In particular, the various embodiments herein can enable adjustments in trials, pre-screening of patients, patient enrichment, patient stratification, and post-hoc data analysis (e.g., after completion of clinical trials).

[0039] Recognizing and taking into account the importance and usefulness of methodologies and systems that can provide the improvements described above, this specification describes various embodiments for evaluating GA progression using images from multiple modalities (e.g., FAF images and OCT images). More specifically, this specification describes various embodiments of methods and systems for processing these multimodal images using machine learning systems (e.g., neural network systems) to accurately predict the growth rate corresponding to GA lesions.

[0040] II. Definition This disclosure is not limited to these exemplary embodiments and uses, or the ways in which these exemplary embodiments and uses operate or are described herein. Furthermore, figures may be simplified or partial, and the dimensions of elements in the figures may be exaggerated or disproportionate.

[0041] Furthermore, wherever the terms “on,” “attached to,” “connected to,” “coupled to,” or similar terms are used herein, one element (e.g., a component, material, layer, substrate, etc.) can be “on,” “attached to,” “connected to,” or “coupled to” another element, regardless of whether one element is directly on top of another element, directly attached to another element, connected to another element, or coupled to another element, or whether one or more intervening elements exist between one element and the other. Furthermore, wherever a list of elements (e.g., elements a, b, c) is referenced, such reference is intended to include any one of the enumerated elements, any combination of fewer elements than all of the enumerated elements, and / or all combinations of the enumerated elements. The division of sections herein is merely for the convenience of consideration and does not limit any combination of elements described.

[0042] The term “subject” may refer to a subject in a clinical trial, a person undergoing treatment, a person undergoing anti-cancer therapy, a person being monitored for remission or recovery, a person undergoing a preventive health analysis (e.g., due to their medical history), or any other person or patient for which the purpose is intended. In various cases, “subject” and “patient” may be used interchangeably herein.

[0043] Unless otherwise defined, scientific and technical terms used in connection with these instructions herein have meanings generally understood by those skilled in the art. Furthermore, unless specifically required by context, singular terms shall include plural forms and plural terms shall include singular forms. In general, nomenclature and techniques used in connection with chemistry, biochemistry, molecular biology, pharmacology, and toxicology are described herein, are well known and commonly used in the art.

[0044] As used herein, “substantially” means sufficient to function for the intended purpose. Thus, the term “substantially” allows for minor, slight variations from absolute or perfect conditions, dimensions, measurements, results, etc., which are expected by those skilled in the art but do not significantly affect the overall performance. When used in relation to numerical values, or parameters or characteristics that can be expressed numerically, “substantially” means within 10 percent.

[0045] As used herein, the term “about” when used with respect to a numerical value or a parameter or characteristic that can be expressed as a numerical value means within 10% of the numerical value. For example, “about 50” means a value in the range of 45 or more and 55 or less.

[0046] The term "plural" means two or more.

[0047] As used herein, the term “plural” may mean two, three, four, five, six, seven, eight, nine, ten or more.

[0048] As used herein, the term "set" means one or more items. For example, a set of items includes one or more items.

[0049] As used herein, the phrase “at least one of” means, when used with a list of items, that one or more different combinations of the enumerated items may be used, or only one of the items in the list may be used. An item can be a specific object, thing, step, action, process, or category. In other words, “at least one of” means that any combination or any number of items from the list may be used, but not all of the items in the list may be used. For example, but are not limited to, “at least one of item A, item B, or item C” means item A, item A and item B, item B, item A, item B, and item C, item B and item C, or item A and C. In some cases, “at least one of item A, item B, or item C” means, but are not limited to, two of item A, one of item B and ten of item C, four of item B and seven of item C, or several other suitable combinations.

[0050] As used herein, “Model” may include one or more algorithms, one or more mathematical techniques, one or more machine learning algorithms, or a combination thereof.

[0051] As used herein, “machine learning” includes the practice of using algorithms to analyze data, learn from it, and then make decisions or predictions about something in the world. Machine learning uses algorithms that can learn from data without relying on rule-based programming.

[0052] As used herein, “artificial neural network” or “neural network” (NN) may refer to a mathematical algorithm or computational model that mimics an interconnected group of artificial neurons that process information based on a connectivity-theoretic approach to computation. A neural network, sometimes called a neural net, can use one or more layers of nonlinear units to predict the output of an incoming input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer may be used as an input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network produces an output from an incoming input according to the current values ​​of each set of parameters. In various embodiments, a reference to “neural network” may refer to one or more neural networks.

[0053] A neural network can process information in two ways: it is in training mode when it is being trained, and it is in inference (or prediction) mode when it actually performs what it has learned. A neural network learns through a feedback process (e.g., backpropagation) that allows the network to adjust the weight coefficients of individual nodes in the intermediate hidden layers (correcting their behavior) so that the output matches the output of the training data. In other words, a neural network learns by being fed training data (learning examples) and eventually learns how to arrive at the correct output even when presented with a new range or set of inputs. A neural network may include, for example, at least one of the following types of neural networks: feedforward neural networks (FNNs), recurrent neural networks (RNNs), modular neural networks (MNNs), convolutional neural networks (CNNs), residual neural networks (ResNets), ordinary differential equation neural networks (neural-ODEs), or other types of neural networks.

[0054] As used herein, “lesion” may include an area of ​​an organ or tissue that has been damaged (through injury or disease). This area may be continuous or discontinuous. For example, as used herein, a lesion may include multiple areas. A geographic atrophy (GA) lesion is an area of ​​the retina that is suffering from chronic progressive degeneration. As used herein, a GA lesion may include one lesion (e.g., one continuous lesion area) or multiple lesions (e.g., a discontinuous lesion area consisting of multiple distinct lesions).

[0055] As used herein, “total lesion area” may refer to the area covered by the lesion (including the total area), whether the lesion is continuous or discontinuous.

[0056] As used herein, “chronological” means over a period of time. This period may be days, weeks, months, years, or any other measure of time.

[0057] As used herein, “growth rate” in relation to a GA lesion may refer to the change in the lesion area of ​​a GA lesion over time and / or the rate at which the lesion area changes over time. This growth rate may also be called the GA growth rate.

[0058] As used herein, “merging” means merging data, clinical data, feature inputs, or inputs. This merging may also be called “merging together,” for example, two or more datasets, two or more clinical data (e.g., clinical factor data or clinical trial data), two or more feature inputs, or two or more inputs.

[0059] As used herein, "flattening" an OCT image means minimizing the distortion characteristics of the OCT image and forming a more consistent dataset. Flattening may also be referred to as volume flattening or flattening of an OCT volume.

[0060] III. Multimodal prediction of geographic atrophy (GA) growth rate Figure 1A is a block diagram of a lesion assessment system 100 according to various embodiments. The lesion assessment system 100 is used to assess geographic atrophy (GA) lesions in the retina of a subject. The lesion assessment system 100 includes a computing platform 102, data storage 104, and a display system 106. The computing platform 102 can take various forms. In one or more embodiments, the computing platform 102 includes a single computer (or computer system) or multiple computers communicating with each other. In other examples, the computing platform 102 takes the form of a cloud computing platform.

[0061] The data storage 104 and the display system 106 each communicate with the computing platform 102. In some examples, the data storage 104, the display system 106, or both may be considered part of the computing platform 102, or otherwise integrated. Thus, in some examples, the computing platform 102, the data storage 104, and the display system 106 may be separate components that communicate with each other, while in other examples, some combination of these components may be integrated together.

[0062] The lesion evaluation system 100 includes an image processor 108, which may be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, the image processor 108 is implemented on a computing platform 102.

[0063] The image processor 108 receives an image input 109 for processing. The image input 109 may include an image generated at a baseline or reference time. In some embodiments, the image input 109 may be referred to as the baseline image input.

[0064] According to various embodiments of this specification, the image input 109 may include any or all of the following: fundus autofluorescence (FAF) imaging data, optical coherence tomography (OCT) imaging data, and / or infrared (IR) imaging data.

[0065] In one or more embodiments, fundus autofluorescence (FAF) imaging data may include a set of fundus autofluorescence (FAF) images 110, and optical coherence tomography (OCT) imaging data may include a set of optical coherence tomography (OCT) images 112. In one or more embodiments, the set of FAF images 110 and the set of OCT images 112 are unaligned images. However, in other embodiments, the set of FAF images 110 and the set of OCT images 112 may be aligned images.

[0066] In some embodiments, the image input 109 may include images from other combinations of modalities. For example, in some embodiments, the image input 109 may include a set of fundus autofluorescence (FAF) images 110 and a set of infrared (IR) images 113. In some embodiments, the infrared (IR) imaging data may include a set of IR images 113, which may be, for example, a set of near-infrared (NIR) images. In one or more embodiments, the set of FAF images 110 and the set of IR images 113 are unaligned images. However, in other embodiments, the set of FAF images 110 and the set of IR images 113 may be aligned images.

[0067] In some embodiments, the image input 109 may include images from other combinations of modalities. For example, in some embodiments, the image input 109 may include a set of OCT images 112 and a set of infrared (IR) images 113. The set of IR images 113 may be, for example, a set of near-infrared (NIR) images. In one or more embodiments, the set of OCT images 112 and the set of IR images 113 are unaligned images. However, in other embodiments, the set of OCT images 112 and the set of IR images 113 may be aligned images.

[0068] In yet another embodiment, the image input 109 may include a set of FAF images 110, a set of OCT images 112, and a set of IR images 113. In one or more embodiments, any or all of the sets of FAF images 110, OCT images 112, and IR images 113 are unaligned images. However, in another embodiment, any or all of the sets of FAF images 110, OCT images 112, and IR images 113 may be aligned images.

[0069] The image processor 108 processes the image input 109 (for example, any one, two, or all of a set of FAF images 110, a set of OCT images 112, and a set of IR images 113) using the lesion area analysis system 114 to predict the lesion growth rate 116 corresponding to the GA lesion and determine the lesion area 120. According to various embodiments, the lesion area analysis system 114 can simultaneously predict the GA lesion area 120 and the GA growth rate 116.

[0070] The lesion area analysis system 114 can be implemented in various ways. Figure 1B shows schematic diagrams of the lesion area analysis system 114 according to various embodiments. As shown in Figure 1B, the lesion area analysis system 114 can be implemented using a neural network system 118. The neural network system 118 may include any number or combination of neural networks. In one or more embodiments, the neural network system 118 may take the form of a convolutional neural network (CNN) system including one or more neural networks. Each of these one or more neural networks may be a convolutional neural network itself. In some cases, the neural network system 118 may be a deep learning neural network system. In some cases, the neural network system 118 includes multiple subsystems, each including one or more neural networks. As disclosed herein, one or more neural networks in the neural network system 118 may include, for example, at least one of the following: a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a convolutional neural network (CNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural-ODE), a squeeze and excitation embedding neural network, a MobileNet, or another type of neural network.

[0071] In various embodiments, the lesion area analysis system 114 may include a lesion area detection module 122 and a lesion area calculation module 124. The lesion area analysis system 114 can predict individual GA areas and growth rates using FAF images and / or OCT volumes via the lesion area detection module 122 and / or the lesion area calculation module 124. In various embodiments, the calculation module 124 may utilize a neural network system to predict individual GA areas and growth rates. Prediction may be performed via the lesion area analysis system 114 using screening images of image input 109, which may include, for example, FAF images 110, OCT images 112, and IR images 113. In various embodiments, the GA growth rate (e.g., mm per year) may be used. 2 The (year) can be derived from a linear model fitted using all available FAF measurements based on accumulated FAF and OCT imaging, which can also be done over time, for example, every 24 weeks over two years.

[0072] According to various embodiments, GA growth rate prediction can be formulated as a regression task. For example, a neural network system 118 having three multitask convolutional neural networks (CNNs) can be used to predict (e.g., simultaneously) the GA lesion area and GA growth rate (e.g., annualized) via a lesion area detection module 122 and a lesion area calculation module 124, trained on multimodal imaging data (e.g., a combination of FAF and OCT images). In various embodiments, a linear model based on baseline GA lesion features, lesion area, lesion distance to the fovea, lesion continuity (monofocal / multifocal), and low-luminosity defects (LLD) for deriving the GA growth rate prediction can serve as a reference model for benchmarking performance.

[0073] Resized and normalized FAF / OCT images can be used as input (e.g., fused input) in the lesion area analysis system 114. For example, as will be described in more detail below, FAF images may be resized to 512 × 512 pixels and normalized between 0 and 1. In the case of OCT volumes (e.g., 3D images), preprocessing can be performed before using the images. For example, histogram matching may be applied first to calibrate the difference in image intensity between B scans, and then each B scan may be flattened along Bruch's membrane (BM). As a non-limiting example, three in-plane maps averaged over the entire depth, depths beyond the BM, and depths below the BM may be combined as a 3-channel input (e.g., fused input). In various embodiments, OCT preprocessing may include general image contrast improvement (or adjustment), with or without volume flattening. Flattening can be performed along any layer, such as the internal limiting membrane (ILM) of the retina. Alternatively, or in addition, OCT cross-sectional images may be integrated into the image input channel. As will be provided in more detail below, both the depth above and below the BM can be set to 100 pixels (390 μm), while the in-plane map can be resized to 512 × 512 pixels and normalized between 0 and 1. Setting the pixel dimensions and normalization intensity between 0 and 1 for any two or more imaging data sources (e.g., any two of the FAF image 110, OCT image 112, and / or IR image 113) enables the fusion of the imaging data described above before supplying it to the lesion area analysis system 114 and / or the neural network system 118, or to either of the modules of the lesion area detection module 122 and the lesion area calculation module 124.

[0074] Furthermore, in various embodiments, further data augmentation (which may be referred to herein as “offline”) may be performed on the dataset (e.g., a 3-channel input or a fused input). Augmentation may include, but is not limited to, horizontal flipping, rotation [ranging from -5 degrees to 5 degrees], and random brightness and contrast [ranging from -0.2 to 0.2]. After augmentation, the dataset may include the original FAF / OCT in-plane image and four modified versions of each FAF / OCT in-plane image, thus increasing the size of the dataset.

[0075] In various embodiments, the lesion area analysis system 114 processes the image input 109 to predict the lesion growth rate 116 and / or the estimated lesion area 120 (also referred to herein as the GA lesion area, GA region, and baseline lesion area). The rate 116 and the lesion area 120 may be determined simultaneously. The lesion area 120 may be the baseline total lesion area for the GA lesion that the lesion area analysis system 114 uses to predict the lesion growth rate 116. The lesion area analysis system 114 predicts the lesion growth rate 116 with higher accuracy compared to evaluating the GA lesion and predicting GA progression using images from a single modality.

[0076] For example, a non-limiting workflow example performed by the lesion area analysis system 114 to predict GA lesion area and / or lesion growth rate for GA lesions is described below with respect to Figures 1C, 1D, 1E, 1F, and 1G.

[0077] Figure 1C shows a process flow 10 for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments. In various embodiments, the process flow 10 is carried out using a system such as a lesion assessment system 100 and is performed by a lesion area analysis system 114 described with respect to Figure 1. As illustrated, the process flow 10 begins with receiving an image input 109, which may be one or more of, for example, fundus autofluorescence (FAF) imaging data, optical coherence tomography (OCT) imaging data, and / or infrared (IR) imaging data. In various embodiments, the image input 109 may include a set of FAF images 110, a set of OCT images 112, and / or a set of IR images 113.

[0078] In various embodiments, a set of FAF images 110, a set of OCT images 112, and / or a set of IR images 113 may be preprocessed via preprocessing 130 as needed. Preprocessing 130 is any process step that can be performed on the image input 109 in the process flow 10. For example, preprocessing 130 for preprocessing FAF images 110 may include, but are not limited to, one or more methods including, automatic macular field FAF image selection, extraction of a desired region (e.g., automask disk area), image contrast improvement (e.g., histogram equalization), and combining multiple FAF images into a single multi-channel input. In some embodiments, preprocessing 130 may include resizing the FAF acquisition data or images to appropriate dimensions, or normalizing the FAF acquisition data or images between 0 and 1. In some embodiments, preprocessing of FAF acquisition data may include macular field FAF image selection, extraction of a desired region, image contrast adjustment, or combination of multi-field FAF images.

[0079] In various embodiments, preprocessing 130 for preprocessing the OCT image 112 may include, but are not limited to, general image contrast adjustment (or improvement), with or without volume flattening. In various embodiments, flattening of the OCT image 112 may be performed along any layer, such as the internal limiting membrane (ILM) of the retina. Alternatively, or in addition, OCT cross-sectional images may be integrated into the image input channel. In various embodiments, preprocessing 130 may include flattening the OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more depths of total depth, depth above Bruch's membrane, and depth below Bruch's membrane, and combining the set of in-plane maps to generate a multi-channel OCT input for predicting the lesion growth rate for GA lesions. In various aspects, preprocessing the OCT imaging data may include generating a set of in-plane maps above and below the retinal membrane, and using the generated set of in-plane maps to predict the lesion growth rate for GA lesions. In some embodiments, artifacts such as corneal curvature, eye movements, and camera positioning can be pre-filtered. For example, flattening the OCT image 112 can make visualization easier by giving the image a more consistent shape, which also allows for efficient truncation of the image input 109. Flattened OCT images can provide minimal distortion characteristics.

[0080] Process flow 10 continues the analysis of the image input 109 with or without preprocessing using a neural network such as a convolutional neural network (CNN) 140, as shown in Figure 1C. While CNN 140 is described here for demonstration purposes, any other suitable neural network system, such as an artificial neural network (ANN) or recurrent neural network (RNN), may be used to perform the analysis. In some embodiments, CNN 140 may include one or more neural networks. Each of these one or more neural networks in CNN 140 may be a deep learning neural network. In some cases, CNN 140 may include multiple subsystems, each containing one or more neural networks. As disclosed herein, CNN140 can be one or more neural networks of a neural network system 118, including, for example, at least one of the following types of neural networks: feedforward neural networks (FNNs), recurrent neural networks (RNNs), modular neural networks (MNNs), residual neural networks (ResNets), ordinary differential equation neural networks (neural-ODEs), squeeze and excitation embedding neural networks, MobileNets, or other types of neural networks.

[0081] After analysis by CNN140, as shown in Figure 1C, one or more features can be extracted from the analysis and processed via Global Mean Pooling (GAP)150. The use of GAP allows for reduction of image size and improvement of computation speed, making feature detection more robust. GAP150 is designed to generate one or more feature maps for desired features and is particularly useful in the case of multi-class classification, as opposed to binary classification where GAP can be similarly used. Instead of adding fully connected layers on top of feature maps, GAP takes the average of each feature map, and the resulting vector is feedforward to a softmax layer, for example, which can normalize the CNN output. One advantage of GAP over fully connected layers is that it is more native to convolutional structures by forcing a correspondence between feature maps and categories.

[0082] In various embodiments, global average pooling 150 is used to fuse or merge the features of the image input 109 to form a fused input. In various embodiments, the fused feature input may be formed using a model that includes an average pooling method (described herein), a squeeze and excitation method, or a combination thereof.

[0083] In various embodiments, one or more extracted features (e.g., image features) may be fed into one or more high-density-256 layers 160, as shown in Figure 1C. High-density-256 layers are used as an example, where 256 refers to the neuron count on the layer, but any other suitable high-density layer with a given neuron count can be used. Nevertheless, high-density layers are generally hidden layers preceding the output layer of a CNN, containing layers of neurons, each neuron receiving input from all neurons from the previous layer of the CNN. In some cases, the output is derived directly from the convolutional layer, in which case the output is multidimensional. Therefore, methods such as Flatten() can be used to convert the multidimensional output into a single-dimensional input and then into a high-density layer / output layer for image classification.

[0084] In various embodiments, the lesion area 120 (or baseline lesion area) is predicted via this high-density 256 layer 160. In various embodiments, the lesion growth rate 116 is predicted via this high-density 256 layer 160.

[0085] In various embodiments, clinical factor data may be included or generated during global mean pooling (e.g., feature fusion) and / or before the convolutional layer output is fed into high-density layers such as high-density layers 256-160. In various embodiments, clinical factor data may include any clinical features and / or the subject's age, sex, smoking status, observed GA lesion area, distance of observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low-luminosity defect (LLD) score, or a combination thereof. Some of this data, such as age, sex, and smoking status, may be information collected as part of a provided imaging dataset. Other data, referred to as retinal biomarkers, may be generated via CNN as described above.

[0086] Figure 1D shows a process flow 20 for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments. In various embodiments, process flow 20 is implemented using a system such as a lesion assessment system 100 and is performed by a lesion area analysis system 114 described with respect to Figure 1. As disclosed herein, process flow 20 is a different process from that of process flow 10 and can be used and implemented to predict baseline lesion area and / or lesion growth rate for GA lesions in the retina. However, each step of process flow 20, such as preprocessing 130, CNN 140, global mean pooling 150, and high density layer 256, may be the same or substantially the same as those described with respect to Figure 1C and will not be described in further detail. As shown in Figure 1D, according to various embodiments, preprocessing 130 and CNN 140 may be the same or different for image inputs 109-1 and 109-2.

[0087] As shown in Figure 1D, process flow 20 can receive multiple inputs of imaging data (e.g., image input 109-1 and image input 109-2) from fundus autofluorescence (FAF) imaging data, optical coherence tomography (OCT) imaging data, and / or infrared (IR) imaging data. As illustrated, each of image input 109-1 and image input 109-2 can undergo its respective preprocessing via preprocessing 130, similar to that of process flow 10, and be analyzed via CNN 140. Once features are extracted from CNN 140, the features extracted from each of image input 109-1 and image input 109-2 can be combined to form a fused input. The fused input of image input 109-1 and image input 109-2 can be averaged via overall average pooling 150, as shown in Figure 1D, to generate an output fed to a high-density -256 layer 160. As shown in process flow 20, the output via the high-density -256 layer 160 may be a baseline lesion area 120 and / or lesion growth rate 116, as shown in Figure 1D. In various embodiments, the lesion area 120 and / or lesion growth rate 116 may be predicted simultaneously.

[0088] Figure 1E shows a process flow 30 for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments. In various embodiments, process flow 30 is implemented using a system such as a lesion assessment system 100 and is performed by a lesion area analysis system 114 described with respect to Figure 1. As disclosed herein, process flow 30 is a process further distinct from the processes of process flow 10 and process flow 20 and can be used and implemented to predict baseline lesion area and / or lesion growth rate for GA lesions in the retina. In process flow 30, a single image input 109 may be used to feed the extracted features to a CNN 140 into a global mean pooling 150, where a first high-density -256 layer 160 may generate a baseline lesion area prediction 120. A second high-density -256 layer 160 may generate the lesion periphery 126. Further high-density -256 layers 160 may generate other retinal biomarkers 128. Once the baseline lesion area 120, the periphery of the lesion 126, and any other retinal biomarkers 128 are determined, they can be fed into another high-density layer, such as high-density-n*170, to generate the lesion growth rate 116 of the patient's or subject's retina. For this final high-density layer 170, "n*" can represent the number of predicted biomarkers output from one of the high-density layers 160.

[0089] Figure 1F shows a process flow 40 for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments. In various embodiments, process flow 40 is implemented using a system such as a lesion assessment system 100 and is performed by a lesion area analysis system 114 described with respect to Figure 1. As disclosed herein, process flow 40 is a process that is further distinct from the processes of process flow 10, process flow 20, or process flow 30 and can be implemented for use in predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina. Process flow 40 is similar to process flow 20 in that it can receive multiple inputs of imaging data (i.e., image inputs 109-1 and 109-2) from fundus autofluorescence (FAF) imaging data, optical coherence tomography (OCT) imaging data, and / or infrared (IR) imaging data. As shown in Figure 1F, each of the multiple image inputs 109-1 and 109-2 can be processed independently to generate features. For example, image input 109-1 may be processed via an optional preprocessing 130, and the CNN 140 may extract features and feed them into a global mean pooling 150, which is used to predict a first high-density 256 layer 160 for predicting a baseline lesion area 120. Similarly, image input 109-2 may be processed via an optional preprocessing 130, CNN 140 to extract features and feed them into a global mean pooling 150, which is used to predict a second high-density 256 layer 160 for predicting a lesion periphery 126. Likewise, any additional image input 109 may be processed via an additional high-density 256 layer 160 through a process flow 40 to predict additional / other retinal biomarkers 128. As shown in Figure 1F, the preprocessing 130 and CNN 140 may be the same or different for image inputs 109-1 and 109-2, or any additional image input 109. Once baseline lesion area 120, lesion periphery 126, and any other retinal biomarkers 128 are generated, they may be supplied to another high-density layer, such as high-density-n*170, to generate the lesion growth rate 116 of the patient's or subject's retina.

[0090] Figure 1G shows a process flow 50 for predicting baseline lesion area and / or lesion growth rate for GA lesions in the retina, according to various embodiments. In various embodiments, the process flow 50 is implemented using a system such as a lesion assessment system 100 and is performed by a lesion area analysis system 114 described with respect to Figure 1. As shown in Figure 1G, instead of applying a CNN 140, the process flow 50 can use a segmentation CNN / computer vision (CV) 155 to extract features or feature maps, which can then be input into one or more computer vision algorithms 165 to predict baseline lesion area 120, lesion periphery 126, and / or any other retinal biomarkers 128. Similar to process flows 30 and 40, the baseline lesion area 120, lesion periphery 126, and any other retinal biomarkers 128 can be fed into another high-density layer such as high-density-n*170 to generate the lesion growth rate 116 of the patient's or subject's retina.

[0091] Figure 2 is a flowchart of process 200 for evaluating geographic atrophic lesions according to various embodiments. In various embodiments, process 200 is implemented using the lesion evaluation system 100 described in Figure 1. In particular, process 200 can be used to predict GA progression.

[0092] Step 202 includes receiving a set of fundus autofluorescence (FAF) images of the retina. Step 204 includes receiving a set of optical coherence tomography (OCT) images of the retina. The set of FAF images and the set of OCT images are from the same retina of the subject. Each of the set of FAF images and the set of OCT images may contain one or more images. In various embodiments, the set of FAF images and the set of OCT images are baseline images containing corresponding images from the same or substantially the same one or more time points in time (e.g., within the same time, within the same day, within the same 1-3 days, etc.).

[0093] Step 206 involves predicting the lesion growth rate for geographic atrophic lesions in the retina using a set of FAF images and a set of OCT images via a machine learning system. This predicted lesion growth rate may be more accurate than the growth rate predicted using only the set of FAF images or only the set of OCT images. As mentioned above, the set of OCT images may provide greater structural information about GA lesions because OCT images are three-dimensional. Furthermore, in some cases, OCT images may reveal certain features (e.g., precursors or biomarkers of disease progression) that are not readily identifiable in FAF images.

[0094] Figure 3 is a flowchart of process 300 for evaluating geographic atrophic lesions according to various embodiments. In various embodiments, process 300 is implemented using the lesion evaluation system 100 described in Figure 1. In particular, process 300 can be used to predict GA progression.

[0095] Step 302 involves receiving a set of fundus autofluorescence (FAF) images of the retina. The retina may belong to a subject diagnosed with geographic atrophy, or possibly the prodromal stage of geographic atrophy.

[0096] Step 304 includes receiving a set of infrared (IR) images of the retina. The set of FAF images and the set of IR images are from the same retina of the subject. Each of the set of FAF images and the set of IR images may contain one or more images. In various embodiments, the set of FAF images and the set of IR images are baseline images containing corresponding images for the same or substantially the same one or more time points in time (e.g., within the same time, within the same day, within the same 1-3 days, etc.). The set of infrared images may be, for example, a set of near-infrared (NIR) images.

[0097] Step 306 involves predicting the lesion growth rate for geographic atrophic lesions in the retina using a set of FAF images and a set of IR images via a machine learning system. This predicted lesion growth rate may be more accurate than the growth rate predicted using only the set of FAF images or only the IR images.

[0098] Figure 4 is a flowchart of method 400 for predicting the lesion growth rate for geographic atrophy lesions in the retina, according to various embodiments. In various embodiments, method 400 is implemented using a system such as the lesion evaluation system 100 described in Figure 1. In particular, method 400 can be used to predict the lesion growth rate of GA.

[0099] As shown in Figure 4, step 402 includes receiving fundus autofluorescence (FAF) imaging data of the retina. The FAF imaging data may be the FAF imaging data of image input 109 in Figure 1, and may include the fundus autofluorescence (FAF) image 110 in Figure 1. The FAF imaging data may include one or more sets of FAF images. One or more sets of FAF images may be unaligned or aligned images. The retina may belong to a subject diagnosed with geographic atrophy, or possibly the prodromal stage of geographic atrophy.

[0100] Step 404 includes receiving optical coherence tomography (OCT) imaging data of the retina. The OCT imaging data may be the OCT imaging data of image input 109 in Figure 1 and may include the OCT image 112 in Figure 1. The OCT imaging data may include one or more sets of OCT images. The one or more sets of OCT images may be unaligned or aligned images.

[0101] In some embodiments of Method 400, an optional step 406 may include receiving infrared (IR) imaging data of the retina. The IR imaging data may be the IR imaging data of the image input 109 in Figure 1 and may include the IR image 113 in Figure 1. The IR imaging data may include one or more sets of IR images. The one or more sets of IR images may be unaligned or aligned images.

[0102] In various embodiments of Method 400, one or more sets of FAF images, one or more sets of OCT images, and / or any one or more sets of IR images are from the same retina of the subject. Each of the one or more sets of FAF images, one or more sets of OCT images, and / or any one or more sets of IR images may include one or more images. In various embodiments, the one or more sets of FAF images, one or more sets of OCT images, and / or any one or more sets of IR images are baseline images that include corresponding images from the same or substantially the same one or more points in time (e.g., within the same time, within the same day, within the same 1-3 days, etc.).

[0103] Step 410 includes predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data and OCT imaging data. In some embodiments of Method 400, Step 410 may include predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data, OCT imaging data, and / or IR imaging data.

[0104] According to various embodiments, predictions are performed via a machine learning system, such as the lesion area analysis system 114 described with respect to Figure 1. Predicted lesion growth rates using FAF and OCT imaging data may be more accurate than growth rates predicted using only a set of FAF images or only a set of OCT images. As mentioned above, a set of OCT images may provide greater structural information about GA lesions because OCT images are three-dimensional. Furthermore, in some cases, OCT images may reveal certain features (e.g., precursors or biomarkers of disease progression) that are not readily identifiable in FAF images.

[0105] In some embodiments of Method 400, any step 410 may be performed before step 408. Any step 410 may include predicting the baseline lesion area for geographic atrophy lesions in the retina using FAF imaging data and OCT imaging data. According to various embodiments, the prediction of the baseline lesion area may be performed via a machine learning system, such as the lesion area analysis system 114 described with respect to Figure 1. In various embodiments, the machine learning system processes the FAF imaging data and OCT imaging data to generate an estimated baseline lesion area, which may be the baseline total lesion area for GA lesions that the machine learning system uses to predict the lesion growth rate performed in step 408. In various embodiments, the machine learning system may predict the lesion growth rate with higher accuracy compared to evaluating GA lesions and predicting GA progression using images from a single modality.

[0106] The machine learning system used in step 408 and / or any step 410 may be implemented using a neural network system such as the neural network system 118 in Figure 1. The neural network system used to predict the lesion growth rate for geographic atrophic lesions may include any number or combination of neural networks and may take the form of a convolutional neural network (CNN) system including one or more neural networks. Each of these one or more neural networks may be a convolutional neural network itself. In some cases, the neural network system may be a deep learning neural network system. In some cases, the neural network system may include multiple subsystems, each including one or more neural networks, and may include, for example, at least one of the following: feedforward neural networks (FNNs), recurrent neural networks (RNNs), modular neural networks (MNNs), convolutional neural networks (CNNs), residual neural networks (ResNets), ordinary differential equation neural networks (neural-ODEs), squeeze and excitation embedding neural networks, MobileNets, or other types of neural networks.

[0107] In various embodiments of Method 400, predicting the lesion growth rate in step 408 may further include generating a first input using FAF imaging data, generating a second input using OCT imaging data, combining the first and second inputs to form a fused input, and using the fused input to generate a lesion growth rate prediction for a geographic atrophic lesion.

[0108] In various embodiments, Method 400 may include generating or predicting one or more biomarkers (or retinal biomarkers) from a fused input. According to various embodiments disclosed herein, biomarkers may include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflex reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, extraretinal fallopian tube formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof. It should be noted that some of these biomarkers may be provided as external inputs based on clinical data provided, for example, as part of an imaging input.

[0109] In various embodiments of Method 400, predicting the lesion growth rate in step 408 may further include generating a first input using a set of FAF images, generating a second input using a set of OCT images, extracting a first objective feature from the FAF imaging data, extracting a second objective feature from the OCT imaging data, fusing the first objective feature and the second objective feature to form a fused feature input, and generating a lesion growth rate for a geographic atrophic lesion using the fused feature input.

[0110] In various embodiments, the retina may be associated with a patient. In such cases, Method 400 may further include receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for geographic atrophic lesions. In some cases, the clinical factor data may include the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low-light-deficient-defect (LLD) score, or a combination thereof.

[0111] In various embodiments, the fused feature input may be formed using a model that includes mean pooling, squeeze and excitation methods, or a combination thereof.

[0112] In various embodiments, method 400 may further include receiving infrared (IR) imaging data of the retina and using the FAF imaging data, OCT imaging data, and IR imaging data to predict the lesion growth rate for geographic atrophic lesions in the retina.

[0113] In various embodiments, Method 400 may also include preprocessing FAF imaging data to form a first input. In various embodiments, preprocessing may include automated macular field FAF image selection, extraction of a desired region, image contrast improvement, or combination of multi-field FAF images into a single multi-channel input. In such cases, preprocessing may include resizing the FAF imaging data to 512 × 512 pixels and normalizing the FAF imaging data between 0s and 1s. In various embodiments, Method 400 may also include preprocessing OCT imaging data to form a second input. In various embodiments, the preprocessed OCT image may include, but is not limited to, general image contrast improvement, 3D to 2D volume flattening, etc. In various embodiments, the flattening of the OCT image may be performed along any layer such as an internal limiting membrane (ILM), and the OCT cross-sectional image may be integrated into the image input channel. In such cases, preprocessing may include flattening OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more depths of total depth, depth above Bruch's membrane, and depth below Bruch's membrane, and combining the set of in-plane maps to generate a multi-channel OCT input for predicting the lesion growth rate for GA lesions. In some embodiments, preprocessing may include any preprocessing method disclosed herein with respect to preprocessing 130, as described in process flows 10, 20, 30, 40, and 50 in Figures 1C, 1D, 1E, 1F, and 1G, respectively.

[0114] Figure 5 is a flowchart of a method 500 for predicting the lesion growth rate for geographic atrophy lesions in the retina, according to various embodiments. In various embodiments, method 500 is implemented using a system such as the lesion evaluation system 100 described in Figure 1. In particular, method 500 can be used to predict the lesion growth rate of GA.

[0115] As shown in Figure 5, step 502 includes receiving fundus autofluorescence (FAF) imaging data of the retina. The FAF imaging data may be the FAF imaging data of image input 109 in Figure 1 and may include the fundus autofluorescence (FAF) image 110 in Figure 1. The FAF imaging data may include one or more sets of FAF images. One or more sets of FAF images may be unaligned or aligned images. The retina may belong to a subject diagnosed with geographic atrophy, or possibly the prodromal stage of geographic atrophy.

[0116] In some embodiments of Method 500, an optional step 504 may include receiving optical coherence tomography (OCT) imaging data of the retina. The OCT imaging data may be the OCT imaging data of the image input 109 in Figure 1 and may include the OCT image 112 in Figure 1. The OCT imaging data may include one or more sets of OCT images. The one or more sets of OCT images may be unaligned or aligned images.

[0117] As shown in Figure 5, step 506 includes receiving infrared (IR) imaging data of the retina. The IR imaging data may be the IR imaging data of the image input 109 in Figure 1, and may include the IR image 113 in Figure 1. The IR imaging data may include one or more sets of IR images. The one or more sets of IR images may be unaligned or aligned images.

[0118] In various embodiments of Method 500, one or more sets of FAF images, one or more sets of IR images, and / or any one or more sets of OCT images are from the same retina of the subject. Each of the one or more sets of FAF images, one or more sets of IR images, and / or any one or more sets of OCT images may include one or more images. In various embodiments, the one or more sets of FAF images, one or more sets of IR images, and / or any one or more sets of OCT images are baseline images that include corresponding images for the same or substantially the same one or more time points in time (e.g., within the same time, within the same day, within the same 1-3 days, etc.).

[0119] Step 510 includes predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data and IR imaging data. In some embodiments of Method 500, Step 510 may include predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data, IR imaging data, and / or OCT imaging data.

[0120] According to various embodiments, predictions are performed via a machine learning system, such as the lesion area analysis system 114 described with respect to Figure 1. Predicted lesion growth rates using FAF imaging data and IR imaging data may be more accurate than growth rates predicted using only a set of FAF images or only a set of IR images. As mentioned above, a set of OCT images may provide greater structural information about GA lesions because OCT images are three-dimensional. Furthermore, in some cases, OCT images may reveal certain features (e.g., precursors or biomarkers of disease progression) that are not readily identifiable in FAF images.

[0121] In some embodiments of Method 500, any step 510 may be performed before step 508. Any step 510 may include predicting the baseline lesion area for geographic atrophy lesions in the retina using FAF imaging data and IR imaging data. According to various embodiments, the prediction of the baseline lesion area may be performed via a machine learning system, such as the lesion area analysis system 114 described with respect to Figure 1. In various embodiments, the machine learning system processes the FAF imaging data and IR imaging data to generate an estimated baseline lesion area, which may be the baseline total lesion area for GA lesions that the machine learning system uses to predict the lesion growth rate performed in step 508. In various embodiments, the machine learning system may predict the lesion growth rate with higher accuracy compared to evaluating GA lesions and predicting GA progression using images from a single modality.

[0122] The machine learning system used in step 508 and / or any step 510 may be implemented using a neural network system such as the neural network system 118 in Figure 1. The neural network system used to predict the lesion growth rate for geographic atrophic lesions may include any number or combination of neural networks and may take the form of a convolutional neural network (CNN) system including one or more neural networks. Each of these one or more neural networks may be a convolutional neural network itself. In some cases, the neural network system may be a deep learning neural network system. In some cases, the neural network system may include multiple subsystems, each including one or more neural networks, and may include, for example, at least one of the following: feedforward neural networks (FNNs), recurrent neural networks (RNNs), modular neural networks (MNNs), convolutional neural networks (CNNs), residual neural networks (ResNets), ordinary differential equation neural networks (neural-ODEs), squeeze and excitation embedding neural networks, MobileNets, or other types of neural networks.

[0123] In various embodiments of Method 500, predicting the lesion growth rate in step 508 may further include generating a first input using FAF imaging data, generating a second input using IR imaging data, fusing the first and second inputs together, and generating a lesion growth rate for a geographic atrophic lesion using the fused input.

[0124] In various embodiments, Method 500 may include generating or predicting one or more biomarkers (or retinal biomarkers) from a fused input. According to various embodiments disclosed herein, biomarkers may include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflex reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, extraretinal fallopian tube formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof. It should be noted that some of these biomarkers may be provided as external inputs based on clinical data provided, for example, as part of an imaging input.

[0125] In various embodiments of Method 500, predicting the lesion growth rate in step 508 may further include generating a first input using a set of FAF images, generating a second input using a set of IR images, extracting a first objective feature from the FAF imaging data, extracting a second objective feature from the IR imaging data, fusing the first objective feature and the second objective feature to form a fused feature input, and generating a lesion growth rate for a geographic atrophic lesion using the fused feature input.

[0126] In various embodiments, the retina may be associated with a patient. In such cases, Method 500 may further include receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion. In some cases, the clinical factor data may include the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low-light-deficient-defect (LLD) score, or a combination thereof.

[0127] In various embodiments, the fused feature input may be formed using a model that includes mean pooling, squeeze and excitation methods, or a combination thereof.

[0128] In various embodiments, method 500 may further include predicting the lesion growth rate for geographic atrophic lesions in the retina using OCT imaging data of the retina, FAF imaging data, IR imaging data, and OCT imaging data.

[0129] In various embodiments, Method 500 may also include preprocessing FAF imaging data to form a first input. In various embodiments, preprocessing may include automated macular field FAF image selection, extraction of a desired region, image contrast improvement, or combination of multi-field FAF images into a single multi-channel input. In such cases, preprocessing may include resizing the FAF imaging data to 512 × 512 pixels and normalizing the FAF imaging data between 0s and 1s. In various embodiments, Method 500 may also include preprocessing OCT imaging data to form a second input. In various embodiments, the preprocessed OCT image may include, but is not limited to, general image contrast improvement, 3D to 2D volume flattening, etc. In various embodiments, the flattening of the OCT image may be performed along any layer such as an internal limiting membrane (ILM), and the OCT cross-sectional image may be integrated into the image input channel. In various embodiments, preprocessing may include flattening OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more depths of total depth, depth above Bruch's membrane, and depth below Bruch's membrane, and combining a set of in-plane maps to generate a multi-channel OCT input for predicting lesion growth rate for GA lesions. In some embodiments, preprocessing may include any preprocessing method disclosed herein with respect to preprocessing 130, as described in process flows 10, 20, 30, 40, and 50 in Figures 1C, 1D, 1E, 1F, and 1G, respectively.

[0130] Herein, we refer to Figure 6, which is a flowchart of a method 600 for predicting the lesion growth rate for geographic atrophy lesions in the retina according to various embodiments. In various embodiments, method 600 is implemented using a system such as the lesion evaluation system 100 described in Figure 1. In particular, method 600 can be used to predict the lesion growth rate of GA.

[0131] As shown in Figure 6, step 604 includes receiving optical coherence tomography (OCT) imaging data of the retina. The retina may belong to a subject diagnosed with geographic atrophy, or possibly the prodromal stage of geographic atrophy. The OCT imaging data may be the OCT imaging data of image input 109 in Figure 1, and may include the OCT image 112 in Figure 1. The OCT imaging data may include one or more sets of OCT images. The one or more sets of OCT images may be unaligned or aligned images.

[0132] In some embodiments of Method 600, an optional step 602 may include receiving fundus autofluorescence (FAF) imaging data of the retina. The FAF imaging data may be the FAF imaging data of image input 109 in Figure 1, and may include the fundus autofluorescence (FAF) image 110 in Figure 1. Any FAF imaging data may include one or more sets of FAF images. Any one or more sets of FAF images may be unaligned or aligned images.

[0133] Method 600 further includes step 606 which may include receiving infrared (IR) imaging data of the retina. The IR imaging data may be the IR imaging data of the image input 109 in Figure 1 and may include the IR image 113 in Figure 1. The IR imaging data may include one or more sets of IR images. The one or more sets of IR images may be unaligned or aligned images.

[0134] In various embodiments of Method 600, one or more sets of arbitrary FAF images, one or more sets of OCT images, and / or one or more sets of IR images are from the same retina of the subject. Each of the one or more sets of arbitrary FAF images, one or more sets of OCT images, and / or one or more sets of IR images may include one or more images. In various embodiments, one or more sets of arbitrary FAF images, one or more sets of OCT images, and / or one or more sets of IR images are baseline images that include corresponding images for the same or substantially the same one or more time points in time (e.g., within the same time, within the same day, within the same 1-3 days, etc.).

[0135] Step 610 includes predicting the lesion growth rate for geographic atrophic lesions in the retina using IR imaging data and OCT imaging data. In some embodiments of Method 600, Step 610 may include predicting the lesion growth rate for geographic atrophic lesions in the retina using IR imaging data, OCT imaging data, and / or FAF imaging data.

[0136] According to various embodiments, predictions are performed via machine learning systems, such as the lesion area analysis system 114 described with respect to Figure 1. Predicted lesion growth rates using IR and OCT imaging data may be more accurate than growth rates predicted using only a set of IR images or only a set of OCT images. As mentioned above, a set of OCT images may provide greater structural information about GA lesions because OCT images are three-dimensional. Furthermore, in some cases, OCT images may reveal certain features (e.g., precursors or biomarkers of disease progression) that are not readily identifiable in FAF images.

[0137] In some embodiments of Method 600, any step 610 may be performed before step 608. Any step 610 may include predicting the baseline lesion area for geographic atrophy lesions in the retina using IR and OCT imaging data. According to various embodiments, the prediction of the baseline lesion area may be performed via a machine learning system, such as the lesion area analysis system 114 described with respect to Figure 1. In various embodiments, the machine learning system processes the IR and OCT imaging data to generate an estimated baseline lesion area, which may be the baseline total lesion area for GA lesions that the machine learning system uses to predict the lesion growth rate performed in step 608. In various embodiments, the machine learning system may predict the lesion growth rate with higher accuracy compared to evaluating GA lesions and predicting GA progression using images from a single modality.

[0138] The machine learning system used in step 608 and / or any step 610 may be implemented using a neural network system such as the neural network system 118 in Figure 1. The neural network system used to predict the lesion growth rate for geographic atrophic lesions may include any number or combination of neural networks and may take the form of a convolutional neural network (CNN) system including one or more neural networks. Each of these one or more neural networks may be a convolutional neural network itself. In some cases, the neural network system may be a deep learning neural network system. In some cases, the neural network system may include multiple subsystems, each including one or more neural networks, and may include, for example, at least one of the following: feedforward neural networks (FNNs), recurrent neural networks (RNNs), modular neural networks (MNNs), convolutional neural networks (CNNs), residual neural networks (ResNets), ordinary differential equation neural networks (neural-ODEs), squeeze and excitation embedding neural networks, MobileNets, or other types of neural networks.

[0139] In various embodiments of Method 600, predicting the lesion growth rate in step 608 may further include generating a first input using IR imaging data and a second input using OCT imaging data, fusing the first and second inputs together to form a fused input, and using the fused input to generate a lesion growth rate for a geographic atrophic lesion.

[0140] In various embodiments, Method 600 may include generating or predicting one or more biomarkers (or retinal biomarkers) from a fusion input. According to various embodiments disclosed herein, biomarkers may include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflex reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, extraretinal fallopian tube formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof. It should be noted that some of these biomarkers may be provided as external inputs based on clinical data provided, for example, as part of an imaging input.

[0141] In various embodiments of Method 600, predicting the lesion growth rate in step 608 may further include generating a first input using a set of IR images, generating a second input using a set of OCT images, extracting a first objective feature from the IR imaging data, extracting a second objective feature from the OCT imaging data, fusing the first objective feature and the second objective feature to form a fused feature input, and generating a lesion growth rate for a geographic atrophic lesion using the fused feature input.

[0142] In various embodiments, the retina may be associated with a patient. In such cases, Method 600 may further include receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion. In some cases, the clinical factor data may include the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low-light-deficient-defect (LLD) score, or a combination thereof.

[0143] In various embodiments, the fused feature input may be formed using a model that includes mean pooling, squeeze and excitation methods, or a combination thereof.

[0144] In various embodiments, method 600 may further include predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data of the retina, IR imaging data, OCT imaging data, and FAF imaging data.

[0145] In various embodiments, Method 600 may also include preprocessing FAF imaging data to form a first input. In various embodiments, preprocessing may include automated macular field FAF image selection, extraction of a desired region, image contrast improvement, or combination of multi-field FAF images into a single multi-channel input. In such cases, preprocessing may include resizing the FAF imaging data to 512 × 512 pixels and normalizing the FAF imaging data between 0s and 1s. In various embodiments, Method 600 may also include preprocessing OCT imaging data to form a second input. In various embodiments, the preprocessed OCT image may include, but is not limited to, general image contrast improvement, 3D to 2D volume flattening, etc. In various embodiments, the flattening of the OCT image may be performed along any layer such as an internal limiting membrane (ILM), and the OCT cross-sectional image may be integrated into the image input channel. In such cases, preprocessing may include flattening OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more depths of total depth, depth above Bruch's membrane, and depth below Bruch's membrane, and combining the set of in-plane maps to generate a multi-channel OCT input for predicting the lesion growth rate for GA lesions. In some embodiments, preprocessing may include any preprocessing method disclosed herein with respect to preprocessing 130, as described in process flows 10, 20, 30, 40, and 50 in Figures 1C, 1D, 1E, 1F, and 1G, respectively.

[0146] In various embodiments, a system for implementing Method 400, Method 500, and / or Method 600 may include non-temporary memory and a hardware processor coupled to the non-temporary memory and configured to read instructions from the non-temporary memory and cause the system to perform the operations of Method 400, Method 500, and / or Method 600. In various embodiments, the system may be implemented using the lesion assessment system 100 shown in Figure 1. The operations performed by the system may include receiving retinal autofluorescence (FAF) imaging data, receiving retinal optical coherence tomography (OCT) imaging data, and / or retinal infrared (IR) imaging data, and using two of the FAF, OCT, and / or IR imaging data (e.g., FAF and OCT imaging data, FAF and IR imaging data, or OCT and IR imaging data) to predict the lesion growth rate for geographic atrophy (GA) lesions in the retina.

[0147] In various embodiments, a non-temporary computer-readable medium (CRM) may store executable computer-readable instructions to cause a computer system to perform the operations of Method 400, Method 500, and / or Method 600. In various embodiments, the operations may be performed or implemented using a system such as the lesion assessment system 100 shown in Figure 1. The CRM may include computer-readable instructions for performing operations including receiving retinal autofluorescence (FAF) imaging data, receiving retinal optical coherence tomography (OCT) imaging data, and / or retinal infrared (IR) imaging data, and using two of the FAF, OCT, and / or IR imaging data (e.g., FAF and OCT imaging data, FAF and IR imaging data, or OCT and IR imaging data) to predict the lesion growth rate for geographic atrophy (GA) lesions in the retina.

[0148] IV. Artificial Neural Networks Figure 7 shows an exemplary neural network that may be used to implement a computer-based model according to various embodiments of the present disclosure. For example, neural network 700 may include neural network system 118 of lesion area analysis system 114. As shown, the artificial neural network 700 includes three layers: an input layer 702, a hidden layer 704, and an output layer 706. Each of layers 702, 704, and 706 may include one or more nodes. For example, input layer 702 includes nodes 708-714, hidden layer 704 includes nodes 716-718, and output layer 706 includes node 722. In this example, each node in a hierarchy is connected to all nodes in adjacent hierarchies. For example, node 708 in input layer 702 is connected to both nodes 716 and 718 in hidden layer 704. Similarly, node 716 of the hidden layer is connected to all nodes 708-714 of the input layer 702 and node 722 of the output layer 706. Although only one hidden layer is shown for the artificial neural network 700, it is thought that the artificial neural network 700 used to implement neural network systems such as the neural network system 118 of the lesion area analysis system 114 may contain as many hidden layers as needed or desired.

[0149] In this example, the artificial neural network 700 receives a set of input values ​​(inputs 1-4) and generates an output value (output 5). Each node in the input layer 702 may correspond to a distinct input value. For example, if the artificial neural network 700 is used to implement a neural network system such as the neural network system 118 of the lesion area analysis system 114, each node in the input layer 702 may correspond to a distinct attribute of the OCT imaging data 110.

[0150] In some embodiments, each of the nodes 716–718 in the hidden layer 704 generates a representation that may include a mathematical calculation (or algorithm) that generates a value based on the input values ​​received from nodes 708–714. The mathematical calculation may include assigning different weights to each of the data values ​​received from nodes 708–714. Nodes 716 and 718 may include different algorithms and / or different weights assigned to the data variables from nodes 708–714 so that each of nodes 716–718 may generate different values ​​based on the same input values ​​received from nodes 708–714. In some embodiments, the weights initially assigned to each feature (or input value) of nodes 716–718 may be generated randomly (e.g., using a computer randomizer). The values ​​generated by nodes 716 and 718 may be used by node 722 in the output layer 706 to generate the output value of the artificial neural network 700. If the artificial neural network 700 is used to implement a neural network system such as the neural network system 118 of the lesion area analysis system 114, the output values ​​generated by the artificial neural network 700 may include the baseline lesion area 120 and / or the lesion growth rate 116.

[0151] The artificial neural network 700 can be trained by using training data. For example, the training data herein may be a set of images from OCT imaging data 112 (see Figure 1A). By providing training data to the artificial neural network 700, nodes 716-718 in the hidden layer 704 can be trained (tuned) so that the output layer 706 produces the optimal output based on the training data. By sequentially providing different training datasets and penalizing the artificial neural network 700 when its output is incorrect (e.g., when it produces a segmentation mask containing an incorrect GA lesion segment), the artificial neural network 700 (specifically, the representation of the nodes in the hidden layer 704) can be trained (tuned) to improve its performance in data classification. Tuning the artificial neural network 700 may include adjusting the weights associated with each node in the hidden layer 704.

[0152] While the above description relates to artificial neural networks as an example of machine learning, it will be understood that other types of machine learning methods may also be suitable for implementing various aspects of this disclosure. For example, support vector machines (SVMs) may be used to implement machine learning. SVMs are a set of related supervised learning methods used for classification and regression. An SVM training algorithm, which may be a non-stochastic binary linear classifier, can build a model that predicts whether a new example will fall into one category or another. Another example is that Bayesian networks may be used to implement machine learning. A Bayesian network is an acyclic stochastic graphical model that represents a set of random variables and their conditional independence by a directed acyclic graph (DAG). A Bayesian network can present a stochastic relationship between one variable and another. Another example is a machine learning engine that performs a machine learning process using a decision tree learning model. In some cases, the decision tree learning model may include a classification tree model and a regression tree model. In some embodiments, the machine learning engine uses a gradient boosting machine (GBM) model (e.g., XGBoost) as the regression tree model. Other machine learning techniques may be used to implement a machine learning engine, for example, via a random forest or a deep neural network. Other types of machine learning algorithms are not described in detail herein for simplicity, and it should be understood that this disclosure is not limited to any particular type of machine learning.

[0153] V. Exemplary applications of the systems and methods disclosed herein The following describes an exemplary workflow that illustrates the present invention in more detail. The disclosed system and method may be used to predict GA area and growth rate using fundus autofluorescence (FAF) images, infrared (IR) images, and / or spectral-region optical coherence tomography (OCT) volumes from baseline visits via a multimodal, multitask deep learning (DL) approach. A retrospective analysis may be performed using baseline FAF images, IR images, and / or OCT volumes from test eyes of patients with bilateral GA aligned to a prospective lamparizumab clinical trial. A retrospective analysis of 1722 patients / eyes from a prospective lamparizumab clinical trial demonstrates the feasibility of predicting simultaneous GA lesion area and annual GA growth rate using a multitask deep learning approach with baseline visit FAF images and / or OCT volumes. Accurate prediction of GA growth rate using baseline visit images helps improve the design, implementation, and analysis of clinical trials.

[0154] GA growth rate (mm 2 ( / year) All available measurements of lesion area (mm) 2 It was estimated as the slope of the linear fit to (graded by an independent leading center). The dataset was split into a development set (1279 patients / eye) and a holdout set (443 patients / eye). Three multitask convolutional neural network models, FAF only, OCT only, and multimodal (FAF and OCT), were used to simultaneously predict simultaneous lesion area and annual growth rate. Performance was defined as the intra-sample coefficient of determination (R) as the square of the Pearson correlation coefficient (r) between the observed lesion area / growth rate and the predicted lesion area / growth rate. 2 The results were evaluated by calculating the confidence interval (CI). The confidence interval (CI) was calculated by bootstrap resampling (B=10000).

[0155] In the development set, the mean R for GA lesion area prediction was calculated using FAF only, OCT only, and multimodal models. 2The performance for each was 0.93, 0.91, and 0.93, respectively, and for GA growth rate prediction, it was 0.48, 0.42, and 0.52, respectively. In the holdout dataset, the performance for GA lesion area prediction was FAF only, OCT only, and the multimodal model R 2 The performance (95% CI) was 0.96 (0.95~0.97), 0.91 (0.87~0.95), and 0.94 (0.92~0.96), respectively, while the GA growth rate prediction was 0.48 (0.41~0.55), 0.36 (0.29~0.43), and 0.47 (0.40~0.54), respectively.

[0156] These findings demonstrate the feasibility of using a multi-task DL approach to predict individual GA area and growth rate using baseline FAF images and / or OCT volumes. Artificial intelligence-based predictions using only screening images could potentially inform and improve clinical trial design and patient management.

[0157] Geographic atrophy (GA) is an advanced stage of age-related macular degeneration (AMD) affecting approximately 5 million people worldwide. It is characterized by the progressive loss of photoreceptors, retinal pigment epithelium (RPE), and choroidal capillary plates, and there is currently no approved treatment for it.

[0158] GA lesions can be detected by several modalities, including color ophthalmoscopy (CFP), fluorescein angiography (FA), fundus autofluorescence (FAF), near-infrared reflectance (NIR), optical coherence tomography (OCT), and optical coherence tomography angiography (OCTA). FAF shows the topographic mapping of endogenous fluorophores within lipofuscin granules in RPE and has been used in clinical trials to quantify GA lesion area. Changes in FAF-derived GA lesion area over a defined time (i.e., GA growth rate) have been used as the primary endpoint of GA in clinical trials.

[0159] OCT is a standard technique in clinical ophthalmology, capturing cross-sectional three-dimensional images of tissue microstructure with micrometer resolution. As OCT technology advances, it is now accepted that OCT images provide structural information that can help characterize GA precursors, onset, and progression. OCT imaging is recommended by the Conference on Atrophy (CAM) classification as a reference method for defining different atrophy phenotypes. Several potential precursors or biomarkers for progression to moderate to advanced AMD, including conversion to GA, have been observed on OCT images, and these include wedge-shaped subretinal hyporeflection, RPE attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, hyporeflective cores in drusen, and high central drusen volume.

[0160] Typically, there is significant variability in GA growth rates among individuals. Therefore, accurate and individualized GA growth rate predictions can be used to address important clinical and research issues. They can assist in patient counseling or inform patient screening, clinical trial design through enrichment and stratification, or clinical trial analysis with prognostic covariate adjustments to increase output. Furthermore, they can be used to better understand the pathogenesis of the disease by correlating with genotype or phenotypic characteristics.

[0161] Previous studies have attempted to predict GA growth over time using imaging modalities such as CFP, FAF, NIR, OCT, and OCTA. Generally, GA growth rates have been found to be linear. Recent studies on CFP have shown that GA growth rate strongly correlates with lesion perimeter. Findings from studies on FAF suggest that morphological descriptive features of the lesion, surrounding abnormal autofluorescence patterns, and previous progression rates are prognostic factors for GA lesion expansion. Studies using FAF and NIR images have shown that RPD (retinal peridialysis) highly predicts GA lesion growth. Predictive models based on features extracted from OCT volumes have demonstrated the ability to predict where GA is likely to grow. Another study on OCT has shown that the presence of outer retinal canal formation may be associated with slower lesion growth. Furthermore, studies on OCTA have shown that voids in the choroidal capillary lamellar may be precursors to GA lesion growth. Studies have also identified genetic, environmental, and demographic factors associated with the development of GA, but their effects on GA progression are not yet clear.

[0162] Despite previous findings, the precise mechanisms underlying GA disease progression remain unclear, and therefore, extracting both image-based and clinical features that accurately predict individual GA progression remains challenging. However, this presents an opportunity to apply novel deep learning techniques that do not require prior feature extraction and / or selection. Deep learning algorithms can be used to predict individual GA growth rates from baseline retinal images with promising results. Recurrent neural network-based predictive models can be used to predict where GA is likely to grow.

[0163] This study aimed to leverage state-of-the-art deep learning techniques on datasets from previous lamparizumab Phase 3 trials and observational studies to accurately predict GA growth rates. Three multi-task models were trained end-to-end on baseline images: FAF only, OCT only, and multimodal (FAF and OCT). Model performance was compared, and each model simultaneously predicted simultaneous GA lesion area and annular GA growth rate. Gradient-activated heatmap visualization techniques were used to determine image regions contributing to model predictions.

[0164] This retrospective study used data from test eyes of patients with bilateral GA aligned to the lampalizumab Phase 3 clinical trials (Saturation [NCT02247479] and Spectrum [NCT02247531]) or observational trial (Proxima A [NCT02479386]). The selection criteria for test eyes in these three trials were the same and previously described. The trials were in compliance with the Declaration of Helsinki and the Act on Health Insurance Portability and Liability. The protocols were approved by the institutional review boards at each site prior to the commencement of the trials. All patients provided written informed consent for future medical research and analysis.

[0165] In this study, 30-degree FAF images (768×768 pixels) of the macula and macular OCT volumes (496×1024×49 voxels) captured using Spectralis HRA+OCT (Heidelberg Engineering, Inc., Heidelberg, Germany) were analyzed. The automated real-time function (ART) value indicating the number of averaged images to obtain a single FAF image or B-scan OCT image was 15 or greater. Only test eye images from the baseline visit were used. Since no treatment effect was observed in the phase 3 trial, all treatment arms were pooled for this analysis. The GA lesion area from all study visits was graded on FAF images at the Central Reading Center by two trained readers with an adjudicator as needed using RegionFinder software (Heidelberg Engineering, Inc., Heidelberg, Germany). The GA growth rate (mm 2 / year) was derived from a linear model fitted using all available FAF measurements for each patient who underwent FAF and OCT imaging every 24 weeks over 2 years. The image dataset was split into a development dataset (1279 patients / eyes) and a holdout dataset (443 patients / eyes). The development dataset was further split into five for nested cross-validation (CV). The baseline characteristics of the patients included in the analysis were balanced across the dataset splits (Table 1 below). Overall, the baseline GA lesion area ranged from 2.54 to 17.78 mm 2 and the GA growth rate ranged from 0.15 to 5.98 mm 2 / year.

Table 1

[0166] We formulated GA growth rate prediction as a regression task. Three multitask convolutional neural networks (CNNs) were trained using baseline FAF-only, OCT-only, and multimodal (combination of FAF and OCT images) as inputs to simultaneously predict baseline GA lesion area and annual GA growth rate. A linear model based on baseline GA lesion features, lesion area, lesion distance to the fovea, lesion continuity (monofocal / multifocal), and low-luminosity defects (LLD) was used as a reference model to benchmark performance for deriving GA growth rate predictions. Baseline arrival images provide more prognostic information for GA disease progression than linear models based on baseline GA lesion features and LLD alone. Furthermore, the multimodal approach (see Figure 8B) provides more insight into disease progression and is superior to the single-modality approach (see Figure 8A).

[0167] Three CNNs were designed to simultaneously predict baseline GA lesion area and annual GA growth rate. The multi-task models were expected to potentially improve performance because they are less likely to overfit on the growth rate prediction task alone, find representations that capture information from both tasks, and provide additional information to lower feature extraction CNN layers. The multi-task approach has previously demonstrated good performance. All three models used the CNN-based deep learning network architecture Inception V3. The networks were pre-trained on ImageNet, an image dataset containing 14 million images in 1000 classes, before being fully retrained on the development dataset.

[0168] Figures 8A and 8B show the overall neural network architectures 800a and 800b. Figure 8A is the same as the process flow 10 shown in Figure 1C and shows a single-modality multitask model, included herein for comparison with Figure 8B. Figure 8B shows a multi-modality multitask model relating to various embodiments. Figure 8B is substantially similar to the process flow 20 described with respect to Figure 1D, except that neural network architecture 800b is an example where image input 109-1 is FAF imaging data / image and image input 109-2 is OCT imaging data / image. The results shown below are based on unprocessed FAF imaging data / image and OCT imaging data / image processed during analysis.

[0169] The model takes resized and normalized baseline FAF / OCT images as input. The FAF images were resized to 512×512 pixels and normalized between 0 and 1. For OCT volumes, histogram matching was first applied to calibrate the difference in image intensity between B scans, and then each B scan was flattened along the Bruch membrane (BM). Three in-plane maps, averaged over the entire depth, the BM upper depth and the BM lower depth were combined as three channel inputs. Both the BM upper depth and BM lower depth were 100 pixels (390 μm). The in-plane maps were resized to 512×512 pixels and normalized between 0 and 1.

[0170] Figure 9 shows an example of a workflow 900 including preprocessing steps for optical coherence tomography (OCT) volumes according to various embodiments. As shown in Figure 9, OCT volumes can be processed to perform histogram matching, followed by flattening the volume along Bruch's membrane (BM), and then in-plane mapping. Figure 9 also shows an example of scans before and after histogram matching, and an example of in-plane mapping as a 3-channel input. For example, the retinal surface obtained from an OCT scanner shows different forms of distortion in B-scan and C-scan slices. These artifacts are thought to be the result of several factors such as corneal curvature, eye movement, and camera position. Flattening the dataset (i.e., flattening the OCT images) facilitates visualization by making the dataset more consistent in shape and also allows for efficient trimming of the dataset. Flattened OCT images may have minimal distortion characteristic of optical coherence tomography images.

[0171] Offline data augmentation was performed on the development dataset, including horizontal flipping, rotation [ranging from -5 degrees to 5 degrees], and random brightness and contrast [ranging from -0.2 to 0.2]. After augmentation, the development dataset included the original FAF / OCT plane images and four modified versions within each FAF / OCT plane, increasing the size of the development dataset by five times. No augmentation was performed on the holdout dataset.

[0172] The performance of each model is defined as the within-sample coefficient of determination (R)², which is the square of the Pearson correlation coefficient (r) between the observed and predicted values. 2 The results were evaluated by calculating the 95% confidence interval (CI). The 95% confidence interval (CI) was derived by bootstrap resampling (B=10000).

[0173] In our initial efforts to characterize the image features on which the models could rely, we applied state-of-the-art gradient-weighted class activation mapping methods (GradCAM or GradAM for regression tasks) to derive and qualitatively examine heatmaps of three modeling approaches on the holdout dataset.

[0174] In the nested CV setup, tuning the model hyperparameters for each outer fold was performed using five inner CV folds. At each outer fold level, the best hyperparameter setting was selected and the model was retrained with these hyperparameters on the inner fold development dataset and used to predict on the outer fold dataset. For the holdout dataset, the five models with the best hyperparameters for each outer fold were retrained on the development dataset and averaged to obtain the final predictions for the holdout dataset.

[0175] The performance of the three multitasking models was first evaluated using inward folding on the development dataset (Table 2). [Table 2] FAF stands for fundus autofluorescence. GA stands for geographic atrophy. OCT stands for optical coherence tomography. SD stands for standard deviation.

[0176] For inward CV folding, the performance is the average R of 5 inward folds randomly divided, excluding the outward fold each time. 2 It is given as follows. [Table 3] The benchmark model is a linear model based on baseline GA lesion area, lesion distance to the fovea, lesion continuity (monofocal / multifocal), and low-intensity defects (LLD). CI represents the confidence interval. FAF represents fundus autofluorescence. GA represents geographic atrophy. OCT represents optical coherence tomography. SD represents the standard deviation.

[0177] Table 3 (above) shows the performance of benchmark models and three multi-task models in predicting baseline GA lesion area and annular GA growth rate in outer folding of the development and holdout datasets. In the development dataset, the multimodal models showed mean R values ​​of 0.42 (0.04) and 0.52 (0.05) for GA lesion area and GA growth rate prediction, respectively, compared to the FAF-only model [0.93 (0.03) and 0.48 (0.05)] and the OCT-only model [0.91 (0.03) and 0.93 (0.02)]. 2 The CV performance was observed in (standard deviation [SD]) (Table 3). In the holdout dataset, the R of GA lesion area and GA growth rate was observed. 2 The bootstrap 95% confidence intervals (CIs) were similar for FAF-only and multimodal models [0.96 (0.95-0.97) and 0.48 (0.41-0.55) vs. 0.94 (0.92-0.96) and 0.47 (0.40-0.54)], and lower for OCT-only models [0.91 (0.87-0.95) and 0.36 (0.29-0.43)] (Table 3). In comparison, a benchmark model previously developed using a linear regression model based on baseline arrival GA lesion area, lesion distance to the fovea, lesion continuity, and LLD showed an R of 0.16 (0.10-0.23) for predicting GA growth rate on the same holdout dataset. 2 The values ​​are shown (Table 3).

[0178] Figure 10 shows forest plots comparing the model performance of three models and a benchmark model on (A) a development dataset and (B) a holdout dataset, according to various embodiments. Figure 10 shows corresponding forest plots comparing the benchmark model with the three multitask models in the outer folding of the development and holdout datasets. On the development dataset, the sample size of the clinical benchmark model is 1485, and the sample size of the imaging model is 1279. The 95% CI is derived by B=10000 bootstrap resampling.

[0179] Figure 11 shows scatter plots 1100 of predicted GA lesion area versus observed GA lesion area and GA growth rate on a holdout dataset according to various embodiments. As shown in plot 1100, scatter plots of (A) predicted GA lesion area versus observed GA lesion area and (B) predicted GA growth rate versus observed GA growth rate for three models on the holdout dataset.

[0180] Figure 12 shows residual plots of predicted GA lesion area versus observed GA lesion area and GA growth rate on a holdout dataset according to various embodiments. The residual plots of (A) predicted GA lesion area-observed GA lesion area versus observed GA lesion area and (B) predicted GA growth rate-observed GA growth rate versus observed GA growth rate for three models on the holdout dataset are shown.

[0181] Figure 13 shows plots 1300 of GA growth rate predictions based on subgroup residual analysis for holdout datasets according to various embodiments. Plots 1300 in Figure 13 show subgroup analyses performed on the holdout dataset based on subsets of lesion continuity (monofocal / multifocal) and lesion location (subfoveal / exfoveal). No prediction bias was observed, and all three models showed similar performance across both subsets.

[0182] Figure 14 shows gradient activation maps (GradAM)1400 of GA lesion area and GA growth rate predictions using FAF only, OCT only, and multimodal multitask models according to various embodiments. The GradAM heatmaps1400 shown in Figure 14 are for GA lesion area and growth rate predictions for each model. The GA lesion area prediction heatmap highlights the lesion itself, while the GA growth rate heatmap highlights the area surrounding the lesion.

[0183] In the nested CV setup, the model hyperparameters for each outer fold were tuned using five inner CV folds. The tuned hyperparameters were the learning rate (0.0001, 0.0002, 0.0005), optimizer (Adams, SGD), loss function (mean squared error, mean absolute error), and dropout (0.1, 0.5, 0.9). The combined loss of the mean squared error or mean absolute error of GA lesion area prediction and GA growth rate prediction was used for training. The weight of each term was fixed at 0.5. The batch size was also kept constant at 16. The weight settings for both batch size and loss function were experimented with using one inner fold CV data set, and then the optimal settings for all folds were adopted to reduce computational cost. The selection of the CNN transfer learning model was determined similarly. Inception V3 performed best compared to VGG16, ResNet50, DenseNet121, and EfficientNets with one inner folding CV dataset. At each outer folding level, after selecting the best hyperparameter settings, the model was retrained with these hyperparameters on the complete inner folding development dataset and used to predict on the outer folding dataset. Next, for the holdout dataset, five models with the best individual folding hyperparameters for each outer folding from nested CV experiments of the model type were retrained across the entire development dataset. The results from these five models were averaged to obtain the final predictions for the holdout data trial. The three approaches—FAF only, OCT only, and multimodal—all underwent the nested CV and holdout processes described above. For the holdout data, 95% confidence intervals (CIs) were calculated by bootstrap resampling, and predictions for each model type on the holdout dataset were resampled 10,000 times and R 2 The R of these samples was calculated. 2 The range provided an estimate of the CI.

[0184] The backend used to design the CNN and load pre-trained weights was Keras 2.2.4 (2018; Google (Mountain View, California)) with Tensorflow 1.8.0 (2018; Google). The program was run on on-premises internal high-performance computing with Python 3.6.3.

[0185] This study demonstrated the feasibility of predicting individual simultaneous GA lesion area and annual GA growth rate using a multi-task deep learning approach with standardized baseline visit FAF and / or OCT images. Models using only FAF consistently performed well on both the development and holdout datasets. The multimodal approach showed a slight performance improvement on the development dataset but not on the holdout dataset.

[0186] All three deep learning models developed in this study performed significantly better than the reference model using baseline clinical features, suggesting that both FAF and OCT provide additional prognostic information beyond anatomical features quantified as GA lesion size, GA distance to the fovea, and lesion continuity. This finding is consistent with data from other studies: shape descriptive factors derived from GA lesions in FAF images had prognostic values ​​for GA progression in one study, and several other studies quantified autofluorescence patterns surrounding GA lesions that were strongly associated with GA progression rate. OCT image features, such as thinning of the outer granular layer, were also presumptively associated with GA progression.

[0187] One unique design element of the CNN model architecture used here was its ability to simultaneously predict both GA lesion area and GA growth rate. This was based on the prior knowledge that baseline GA lesion area is associated with GA progression over time. A multi-task model can find representations that capture information about both tasks in a shared CNN layer. Thus, the model is not entirely a "black box" compared to directly predicting only GA growth rate, as it was uniquely guided to look at clinically relevant regions of the image during the initial feature extraction phase. Furthermore, there is still no consensus or simple mathematical model regarding the relationship between GA area and growth rate. For example, one study found that GA area is approximately 12 mm². 2 We found that lesions grow secondarily up to a certain point, after which the growth rate stabilizes or decreases. In contrast, another study that used square root transformation to determine the relationship between baseline GA area and its progression over time found a negative correlation, suggesting that larger lesions may grow more slowly. The disclosed multi-task CNN model offers a unique nonlinear ability to link lesion area to progression. Finally, the multi-task model is less likely to overfit and demonstrates potential performance improvements. The performance of all three models in predicting GA lesion area is very good, and R was used for both nested CV and holdout trial results. 2 The ratio exceeded 0.90. Potentially, such models could be used in clinical trials for efficient pre-screening during patient recruitment, or in clinical practice for patient counseling.

[0188] GradCAM is one of the known post-attribution techniques for deep learning models that can provide insights into the image regions that contribute most to prediction. Here, we used a variant of GradCAM that is better suited to the regression task: GradAM. Interestingly, the GradAM heatmap reveals that the CNN network looked at the lesion itself for GA lesion area prediction and the region surrounding the lesion for GA growth rate prediction, and these are consistent with other findings.

[0189] This study utilized broadly and well-characterized lampalizumab clinical trial data, including patients with bilateral GA and widespread clinical symptoms. 2 Individual GA growth rate performance using FAF and / or OCT images, evaluated by numerical values, is the best reported in the literature to date, even with models using multiple past visits. However, model development, validation, and testing were performed entirely on the lamparizumab dataset alone. Any use cases other than population distribution in the lamparizumab trial data require further evaluation. The FAF and OCT images included in this trial were captured using equipment from the same vendor (Heidelberg Engineering, Inc., Germany), had similar ART values, and were of the high quality required for eligibility screening by the central leading center. In various embodiments, more interpretive CNN models or additional explainable techniques can be used to gain insights into the decision-making process, which can help to further understand the pathophysiology of GA lesion expansion and, in some cases, identify novel imaging biomarkers. For example, lesion shape features as well as other image extraction features can be explicitly incorporated into statistical models of GA growth rate prediction to enhance interpretability and ease of implementation in clinical trial analysis. Furthermore, images from other modalities (e.g., OCTA, dark microperipheral sensitivity) may provide additional predictive values ​​and can also be added to the input data for model training.

[0190] In summary, the feasibility of predicting individual GA lesion area and growth rate using a multi-task deep learning approach with baseline FAF and / or OCT images is demonstrated. On the holdout dataset, the performance of the multimodal approach was comparable to that of a simpler FAF-only model. This study can improve the reliability of clinical development by informing the design, implementation, and analysis of clinical trials, specifically prognostic covariate adjustment, pre-screening, enrichment, and stratification of patients, and / or post-hoc data analysis. This technique may also be considered for clinical practice and could potentially assist in patient counseling in the future. Further validation on additional datasets could confirm the robust performance and any potential advantages of the multimodal approach over the FAF-only model.

[0191] VI. Computer-Implemented Systems Figure 15 is a block diagram of a computer system according to various embodiments. Computer system 1500 may be an example of one implementation of the computing platform 102 described in Figure 1. In one or more examples, computer system 1500 may include a bus 1502 or other communication mechanism for communicating information and a processor 1504 coupled to the bus 1502 for processing information. In various embodiments, computer system 1500 may also include memory, which may be random access memory (RAM) 1506 or other dynamic storage device, coupled to the bus 1502 for determining instructions to be executed by the processor 1504. Memory may also be used to store temporary variables or other intermediate information during the execution of instructions executed by the processor 1504. In various embodiments, computer system 1500 may further include read-only memory (ROM) 1508 or other static storage device coupled to the bus 1502 for storing static information and instructions for the processor 1504. A storage device 1510, such as a magnetic disk or optical disk, may be provided and coupled to the bus 1502 for storing information and instructions.

[0192] In various embodiments, the computer system 1500 may be coupled via a bus 1502 to a display 1512, such as a cathode ray tube (CRT) or liquid crystal display (LCD), to display information to the computer user. An input device 1514, including alphanumeric and other keys, may be coupled to the bus 1502 to communicate information and command selections to the processor 1504. Another type of user input device is a cursor control device 1516, such as a mouse, joystick, trackball, gesture input device, gaze-based input device, or cursor direction keys, for communicating directional information and command selections to the processor 1504 and controlling cursor movement on the display 1512. This input device 1514 typically has two degrees of freedom with two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane. However, it should be understood that input devices 1514 enabling three-dimensional (e.g., x, y, and z) cursor movement are also contemplated herein.

[0193] In accordance with a particular implementation of this teaching, the results may be provided by the computer system 1500 in response to a processor 1504 executing one or more sequences of one or more instructions contained in RAM 1506. Such instructions may be read into RAM 1506 from another computer-readable medium or computer-readable storage medium, such as a storage device 1510. The execution of the instruction sequence contained in RAM 1506 causes the processor 1504 to execute the process described herein. Alternatively, hardwired circuits may be used instead of or in combination with software instructions to implement this teaching. Thus, the implementation of this teaching is not limited to a particular combination of hardware circuits and software.

[0194] As used herein, the terms “computer-readable medium” (e.g., datastore, memory device, data storage device, etc.) or “computer-readable storage medium” refer to any medium involved in providing instructions to the processor 1504 for execution. Such mediums can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Examples of non-volatile media include, but are not limited to, optical, solid-state, and magnetic disks, such as memory device 1510. Examples of volatile media include, but are not limited to, dynamic memory, such as RAM 1506. Examples of transmission media include, but are not limited to, coaxial cables, copper wires, and optical fibers, including wires with bus 1502.

[0195] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, and EPROMs, flash EPROMs, any other memory chips or cartridges, or any other tangible media that a computer can read.

[0196] In addition to computer-readable media, instructions or data may be provided as signals on a transmission medium included in a communication device or system to provide a sequence of one or more instructions to the processor 1504 of the computer system 1500 for execution. For example, a communication device may include a transceiver having signals indicating instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in this disclosure. Typical examples of data communication transmission connections include, but are not limited to, telephone modem connections, wide area networks (WANs), local area networks (LANs), infrared data connections, NFC connections, and optical communication connections.

[0197] It should be understood that the flowcharts, diagrams, and accompanying disclosures described herein may be implemented using the computer system 1500 as a standalone device or on a distributed network of shared computing resources, such as a cloud computing network.

[0198] The methods described herein may be implemented by various means depending on the application. For example, these methods may be implemented in hardware, firmware, software, or any combination thereof. In the case of hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, and / or combinations thereof.

[0199] In various embodiments, the methods described herein may be implemented as firmware and / or software programs and applications written in conventional programming languages ​​such as C, C++, and Python. When implemented as firmware and / or software, the embodiments described herein may be implemented on a non-temporary computer-readable medium on which a program causing a computer to perform the methods described herein is stored. It should be understood that the various engines described herein may be provided on a computer system such as computer system 1500, thereby causing the processor 1504 to perform analyses and decisions provided by these engines in accordance with instructions provided by one or a combination thereof of the memory components RAM 1506, ROM 1508, or storage devices 1510, and user input provided via input device 1514.

[0200] VII. Conclusion While this instruction is described in relation to various embodiments, it is not intended to be limited to such embodiments. On the contrary, this instruction includes various substitutions, modifications, and equivalents, as will be understood by those skilled in the art.

[0201] For example, the flowcharts and block diagrams described above illustrate the architecture, functionality, and / or operation of possible implementations of various methods and system embodiments. Each block in a flowchart or block diagram may represent a module, segment, function, operation or step, or a combination thereof. In some alternative implementations of an embodiment, one or more functions described in a block may be performed in an order different from the order shown in the diagram. For example, in some cases, two blocks shown consecutively may be executed substantially simultaneously. In other cases, blocks may be executed in reverse order. Furthermore, in some cases, one or more blocks may be added to replace or supplement one or more other blocks in the flowchart or block diagram.

[0202] Accordingly, when describing various embodiments, this specification may present methods and / or processes as a specific set of steps. However, unless a method or process relies on a specific sequence of steps described herein, the method or process should not be limited to the specific sequence of steps described herein, and those skilled in the art will readily understand that the order may vary and still remain within the spirit and scope of various embodiments.

[0203] VIII. Description of Embodiments Embodiment 1: A method comprising receiving retinal autofluorescence (FAF) imaging data, receiving retinal optical coherence tomography (OCT) imaging data, and using the FAF and OCT imaging data to predict the lesion growth rate for geographic atrophy (GA) lesions in the retina.

[0204] Embodiment 2: The method according to Embodiment 1, further comprising predicting the baseline lesion area for a GA lesion using FAF and OCT imaging data.

[0205] Embodiment 3: The method according to Embodiment 1 or 2, further comprising predicting the lesion growth rate by generating a first input using FAF imaging data, generating a second input using OCT imaging data, fusing the first input and the second input to form a fused input, and using the fused input to generate a lesion growth rate for a geographic atrophic lesion.

[0206] Embodiment 4: The method according to any one of Embodiments 1 to 3, further comprising extracting a biomarker from the fusion input.

[0207] Embodiment 5: The method according to Embodiment 4, wherein the biomarkers include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflection reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, outer retinal canal formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof.

[0208] Embodiment 6: The method according to any one of Embodiments 1 to 5, further comprising: predicting the lesion growth rate; generating a first input using a set of FAF images; generating a second input using a set of OCT images; extracting a first objective feature from the FAF imaging data; extracting a second objective feature from the OCT imaging data; fusing the first objective feature and the second objective feature to form a fused feature input; and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0209] Embodiment 7: The method according to Embodiment 6, wherein the retina is associated with a patient, and the method further comprises receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0210] Embodiment 8: The method according to Embodiment 7, wherein the clinical factor data includes the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low light-emitting defects (LLD) score, or a combination thereof.

[0211] Embodiment 9: The method according to any one of Embodiments 6 to 8, wherein the fused feature input is formed using a model that includes an average pooling method, a squeeze and excitation method, or a combination thereof.

[0212] Embodiment 10: The method according to any one of Embodiments 1 to 9, further comprising receiving infrared (IR) imaging data of the retina and predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data, OCT imaging data, and IR imaging data.

[0213] Embodiment 11: The method according to any one of Embodiments 3 to 10, further comprising preprocessing FAF imaging data to form a first input, wherein the preprocessing includes macular field FAF image selection, target region extraction, image contrast adjustment, or combination of multi-field FAF images.

[0214] Embodiment 12: The method according to any one of Embodiments 3 to 11, further comprising preprocessing OCT imaging data to form a second input, wherein the preprocessing includes generating a set of in-plane maps above and below the retinal membrane, and using the generated set of in-plane maps to predict the lesion growth rate for a GA lesion.

[0215] Embodiment 13: A system comprising: a non-temporary memory; and a hardware processor coupled to the non-temporary memory and configured to read instructions from the non-temporary memory and cause the system to perform operations including receiving retinal autofluorescence (FAF) imaging data, receiving retinal optical coherence tomography (OCT) imaging data, and using the FAF and OCT imaging data to predict the lesion growth rate for geographic atrophy (GA) lesions in the retina.

[0216] Embodiment 14: The system according to Embodiment 13, wherein the processor is configured to perform operations that further include predicting a baseline lesion area for a GA lesion using FAF and OCT imaging data.

[0217] Embodiment 15: The system according to Embodiment 13 or 14, further comprising predicting the lesion growth rate by generating a first input using FAF imaging data, generating a second input using OCT imaging data, fusing the first input and the second input to form a fused input, and using the fused input to generate a lesion growth rate for a geographic atrophic lesion.

[0218] Embodiment 16: The system according to Embodiment 15, wherein the processor is configured to perform operations that further include extracting biomarkers from fusion data.

[0219] Embodiment 17: A system according to any one of Embodiments 13 to 16, wherein predicting lesion growth rate includes generating a first input using a set of FAF images and generating a second input using a set of OCT images; extracting a first objective feature from the FAF imaging data and extracting a second objective feature from the OCT imaging data; fusing the first objective feature and the second objective feature to form a fused feature input; and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0220] Embodiment 18: The system according to Embodiment 17, wherein the retina is associated with a patient, and the processor is configured to perform operations further including receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0221] Embodiment 19: The system according to Embodiment 18, wherein clinical factor data includes age, sex, smoking status, observed GA lesion area, distance of observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low light-emitting defects (LLD) score, and combinations thereof.

[0222] Embodiment 20: The system according to any one of Embodiments 13 to 19, wherein the processor is configured to perform operations further including receiving infrared (IR) imaging data of the retina and using the FAF imaging data, OCT imaging data, and IR imaging data to predict the growth rate of a geographic atrophic lesion in the retina.

[0223] Embodiment 21: The system according to any one of Embodiments 13 to 20, wherein the processor is further configured to preprocess OCT imaging data, the preprocessing comprising flattening the OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more of the total depth, depths above Bruch's membrane, and depths below Bruch's membrane, and combining the set of in-plane maps to generate a multi-channel input for predicting the lesion growth rate for GA lesions.

[0224] Embodiment 22: A non-temporary computer-readable medium (CRM) that stores executable computer-readable instructions causing a computer system to perform operations including receiving retinal autofluorescence (FAF) imaging data, receiving retinal optical coherence tomography (OCT) imaging data, and using the FAF and OCT imaging data to predict the disease growth rate for geographic atrophy (GA) lesions in the retina.

[0225] Embodiment 23: The CRM according to Embodiment 22, wherein the operation further includes predicting the baseline lesion area for GA lesions using FAF and OCT imaging data.

[0226] Embodiment 24: The CRM according to Embodiment 22 or 23, further comprising predicting the lesion growth rate by generating a first input using FAF imaging data, generating a second input using OCT imaging data, fusing the first input and the second input to form a fused input, and using the fused input to generate a lesion growth rate for a geographic atrophic lesion.

[0227] Embodiment 25: The CRM according to any one of Embodiments 22 to 24, wherein the operation further includes extracting a biomarker from a fusion input.

[0228] Embodiment 26: The CRM according to Embodiment 25, wherein the biomarkers include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflection reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, outer retinal canal formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof.

[0229] Embodiment 27: A CRM according to any one of Embodiments 22 to 26, further comprising predicting lesion growth rate by generating a first input using a set of FAF images and a second input using a set of OCT images; extracting a first objective feature from the FAF imaging data and a second objective feature from the OCT imaging data; fusing the first objective feature and the second objective feature to form a fused feature input; and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0230] Embodiment 28: The CRM according to Embodiment 27, wherein the retina is associated with a patient, and the method further comprises receiving patient-associated clinical factor data, fusing the clinical factor data with a first-objective feature and a second-objective feature to form a fused feature input, and using the fused feature input to generate a lesion growth rate for a geographic atrophic lesion.

[0231] Embodiment 29: The CRM according to Embodiment 28, wherein the clinical factor data includes the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low light-resistance defect (LLD) score, or a combination thereof.

[0232] Embodiment 30: The CRM according to any one of Embodiments 27 to 29, wherein the fused feature input is formed using a model that includes the mean pooling method, squeeze and excitation method, or a combination thereof.

[0233] Embodiment 31: The CRM according to any one of Embodiments 22 to 30, further comprising receiving infrared (IR) imaging data of the retina and predicting the lesion growth rate for geographic atrophic lesions in the retina using FAF imaging data, OCT imaging data, and IR imaging data.

[0234] Embodiment 32: A CRM according to any one of Embodiments 24 to 31, wherein the operation is to preprocess FAF imaging data to form a first input, and the preprocessing further includes resizing the FAF imaging data to 512 × 512 pixels and normalizing the FAF imaging data between 0 and 1.

[0235] Embodiment 33: A CRM according to any one of Embodiments 24 to 32, wherein the operation is to preprocess OCT imaging data to form a second input, and the preprocessing further includes flattening the OCT imaging data along Bruch's membrane, averaging a set of in-plane maps over one or more depths of total depth, depth above Bruch's membrane, and depth below Bruch's membrane, and combining the set of in-plane maps to generate a multi-channel OCT input for predicting the lesion growth rate for GA lesions.

[0236] Embodiment 34: A method for evaluating geographic atrophy in the retina, comprising: receiving a set of fundus autofluorescence (FAF) images of the retina; receiving a set of optical coherence tomography (OCT) images of the retina; and using the set of FAF images and the set of OCT images to predict the lesion growth rate for geographic atrophy lesions in the retina via a machine learning system.

[0237] Embodiment 35: The method according to Embodiment 34, wherein predicting the lesion growth rate via a machine learning system includes predicting the baseline lesion area for a geographic atrophic lesion.

[0238] Embodiment 36: The method according to Embodiment 34 or 35, wherein predicting lesion growth rate via a machine learning system includes generating a first input using a set of FAF images and a second input using a set of OCT images, fusing the first input and the second input to form a fused input, and using the fused input to generate a ringing lesion growth rate for a geographic atrophic lesion.

[0239] Embodiment 37: A method for evaluating geographic atrophy in the retina, comprising: receiving a set of fundus autofluorescence (FAF) images of the retina; receiving a set of infrared (IR) images of the retina; and predicting the lesion growth rate for geographic atrophy lesions in the retina using the set of FAF images and the set of IR images via a machine learning system.

[0240] Embodiment 38: The method according to Embodiment 37, wherein predicting the lesion growth rate via a machine learning system includes predicting the baseline lesion area for a geographic atrophic lesion.

Claims

1. A method performed by a hardware processor, Receiving retinal autofluorescence (FAF) imaging data, The optical coherence tomography (OCT) imaging data of the retina is received, Using the FAF imaging data and the OCT imaging data, predict the growth rate of the geographic atrophy (GA) lesion in the retina. Methods that include...

2. The method according to claim 1, further comprising predicting the baseline lesion area for the GA lesion using the FAF imaging data and the OCT imaging data.

3. Predicting the growth rate of the aforementioned lesion is possible. The first input is generated using the FAF imaging data, and the second input is generated using the OCT imaging data. The first input and the second input are fused to form a fused input, Using the aforementioned fusion input, the lesion growth rate for the geographic atrophic lesion is generated. The method according to claim 1, further comprising:

4. The method according to claim 3, further comprising extracting a biomarker from the fusion input.

5. The method according to claim 4, wherein the biomarkers include lesion perimeter, lesion shape descriptive features, wedge-shaped subretinal reflection reduction, retinal pigment epithelium (RPE) attenuation and destruction, highly reflective lesions, reticular pseudodrusen (RPD), multilayer thickness reduction, photoreceptor atrophy, low-reflectivity core in drusen, large central drusen, surrounding abnormal autofluorescence pattern, previous GA progression rate, outer retinal canal formation, choroidal capillary laminae, GA lesion size, GA distance to the fovea, lesion continuity, or a combination thereof.

6. The FAF imaging data comprises a set of FAF images, and the OCT imaging data comprises a set of OCT images. Predicting the growth rate of the aforementioned lesion is possible. A first input is generated using the set of FAF images, and a second input is generated using the set of OCT images. Extracting the first objective features from the first input, Extracting the second objective features from the second input, Forming a fused feature input by fusing the features of the first objective described above with the features of the second objective described above, Using the aforementioned fused feature input, the lesion growth rate for the geographic atrophic lesion is generated. The method according to claim 1, further comprising:

7. The retina is associated with the patient, and the method is Receiving clinical factor data associated with the aforementioned patient, The clinical factor data is fused with the first objective feature and the second objective feature to form the fused feature input, Using the aforementioned fused feature input, the lesion growth rate of the geographic atrophic lesion is generated. The method according to claim 6, further comprising:

8. The method according to claim 7, wherein the clinical factor data includes the subject's age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low light-depletion (LLD) score, or a combination thereof.

9. The method according to claim 6, wherein the fused feature input is formed using a model that includes an average pooling method, a squeeze and excitation method, or a combination thereof.

10. Receiving infrared (IR) imaging data of the retina, Using the FAF imaging data, the OCT imaging data, and the IR imaging data, predict the growth rate of the geographic atrophic lesions in the retina. The method according to claim 1, further comprising:

11. The method according to claim 3, further comprising preprocessing the FAF imaging data to form the first input, wherein the preprocessing includes macular field FAF image selection, target region extraction, image contrast adjustment, or combination of multi-field FAF images.

12. The OCT imaging data is preprocessed to form the second input, wherein the preprocessing is To generate a set of in-plane maps above and below the retinal membrane, The method according to claim 3, further comprising preprocessing the OCT imaging data to form the second input, including predicting the lesion growth rate for the GA lesion using the generated set of in-plane maps.

13. It is a system, Non-temporary memory and The non-temporary memory is connected, and instructions are read from the non-temporary memory and the system is configured to Receiving retinal autofluorescence (FAF) imaging data, The optical coherence tomography (OCT) imaging data of the retina is received, Using the FAF imaging data and the OCT imaging data, predict the growth rate of the geographic atrophy (GA) lesion in the retina. A hardware processor configured to perform operations including A system that includes these features.

14. The system according to claim 13, wherein the processor is configured to perform an operation that further includes predicting a baseline lesion area for the GA lesion using the FAF imaging data and the OCT imaging data.

15. Predicting the growth rate of the aforementioned lesion is possible. The first input is generated using the FAF imaging data, and the second input is generated using the OCT imaging data. The first input and the second input are fused to form a fused input, Using the aforementioned fusion input, the lesion growth rate for the geographic atrophic lesion is generated. The system according to claim 13, further comprising:

16. The system according to claim 15, wherein the processor is configured to perform an operation that further includes extracting a biomarker from the fusion input.

17. The FAF imaging data comprises a set of FAF images, and the OCT imaging data comprises a set of OCT images. Predicting the growth rate of the aforementioned lesion is possible. A first input is generated using the set of FAF images, and a second input is generated using the set of OCT images. Extracting the first objective features from the first input and extracting the second objective features from the second input, Forming a fused feature input by fusing the features of the first objective described above with the features of the second objective described above, Using the aforementioned fused feature input, the lesion growth rate for the geographic atrophic lesion is generated. The system according to claim 13, including the system described in claim 13.

18. The retina is associated with the patient, and the processor, Receiving clinical factor data associated with the aforementioned patient, The clinical factor data is fused with the first objective feature and the second objective feature to form the fused feature input, Using the aforementioned fused feature input, the lesion growth rate of the geographic atrophic lesion is generated. The system according to claim 17, configured to perform an operation that further includes the operation.

19. The system according to claim 18, wherein the clinical factor data includes age, sex, smoking status, observed GA lesion area, distance of the observed GA lesion to the fovea of ​​the retina, image continuity, best corrected visual acuity (BCVA) score, low light-depletion (LLD) score, and combinations thereof.

20. The aforementioned processor, Receiving infrared (IR) imaging data of the retina, Using the FAF imaging data, the OCT imaging data, and the IR imaging data, predict the growth rate of the geographic atrophic lesions in the retina. The system according to claim 13, configured to perform an operation that further includes the operation.

21. The processor is further configured to preprocess the OCT imaging data, and the preprocessing is To flatten the OCT imaging data along the Bruch membrane, Averaging a set of in-plane maps over one or more of the total depth, depths above the Bruch film, and depths below the Bruch film, To generate a multi-channel input for predicting the lesion growth rate for the GA lesion, the set of in-plane maps is combined. The system according to claim 13, including the system described in claim 13.

22. A non-temporary computer-readable medium (CRM) that, in a computer system, Receiving retinal autofluorescence (FAF) imaging data, The optical coherence tomography (OCT) imaging data of the retina is received, Using the FAF imaging data and the OCT imaging data, predict the growth rate of the geographic atrophy (GA) lesion in the retina. A non-temporary computer-readable medium (CRM) that stores executable computer-readable instructions to perform an action including [a specific action].

Images