Neural network processing of OCT data to generate predictions of map-like atrophy growth rates

Abstract

Embodiments disclosed herein generally relate to predicting geographic atrophy lesion growth and / or geographic atrophy lesion size in an eye. The predictions can be generated by processing data objects using a neural network. The data objects can include three-dimensional data objects representing a representation of at least a portion of the eye, or multi-channel data objects representing one or more representations of at least a portion of the eye. The neural network can include a convolutional multitask neural network trained to learn features that predict both lesion growth and lesion size output.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims the benefit and priority of U.S. Provisional Patent Applications 62 / 944,201 (filed December 5, 2019) and 62 / 988,797 (filed March 12, 2020). Each of these applications is incorporated herein by reference in whole for all purposes. [Background technology]

[0002] background Geographic atrophy (GA) is a type of advanced age-related macular degeneration (AMD) that results in degeneration of photoreceptors and supporting cells, as well as progressive vision loss. This condition affects millions of people worldwide. In developed countries, approximately 1 in 29 people over the age of 75 have geographic atrophy. Geographic atrophy is characterized by progressive structural loss of the retinal pigment epithelium (RPE), adjacent photoreceptors, and choroidal capillary branches. The progression of geographic atrophy shows great inter-patient variability. Currently, there are no approved treatments to prevent or slow the progression of geographic atrophy.

[0003] Geographic atrophy lesions can be imaged using a variety of imaging modalities. Traditionally, two-dimensional fundus autofluorescence (FAF) images have been used to quantify the area of ​​geographic atrophy lesions. The change in FAF-derived lesion area over a period of time (geographic atrophy growth rate) is accepted as an anatomical outcome parameter indicating whether and / or the extent of progression of geographic atrophy in a subject. Nevertheless, the two-dimensional nature of FAF images can limit their ability to provide precise structural information regarding lesion area. Therefore, in addition to the quantification of GA lesion area generated by FAF images, there is a need for improved assessment of lesion area that can enhance the understanding of the onset and progression of GA. [Overview of the project] [Means for solving the problem]

[0004] overview In some embodiments, a method is provided. A three-dimensional data object corresponding to at least a partial depiction of a subject's eye is accessed. The three-dimensional data object may include a three-dimensional depiction of the volume of the subject's eye. The three-dimensional data object is processed using a convolutional neural network (e.g., an inception neural network) to generate a prediction of the subsequent growth of a geographic atrophic lesion in the eye or the subsequent size of a geographic atrophic lesion in the eye. The subsequent growth of a geographic atrophic lesion may include the growth of one or more geographic atrophic lesions in the eye, and / or the subsequent size of a geographic atrophic lesion may include the subsequent size of one or more geographic atrophic lesions in the eye. The convolutional neural network may include one or more three-dimensional convolutional modules, pooling layers, and / or one or more arbitrary units (e.g., attention units). The prediction is output.

[0005] In some embodiments, a convolutional neural network can be trained to simultaneously predict the current geographic atrophy lesion size and the subsequent geographic atrophy state. The subsequent geographic atrophy state may include the subsequent growth of the geographic atrophy lesion or the subsequent size of the geographic atrophy lesion.

[0006] In some embodiments, processing 3D data objects using convolutional neural networks can further yield another prediction of the current size of geographic atrophic lesions in the eye.

[0007] In some embodiments, the method may include generating a three-dimensional data object. For example, each image in a set of two-dimensional images may be segmented to identify segments bounded by the predicted positions of Bruch's membrane and / or the internal boundary membrane. Generating a three-dimensional data object may also include accessing a set of two-dimensional B-scan OCT images of the subject's eye. A set of feature maps may be generated from the set of two-dimensional B-scan OCT images. The three-dimensional data object may be generated to include the set of feature maps.

[0008] In some embodiments, generating a three-dimensional data object involves accessing a set of two-dimensional B-scan OCT images of a subject's eye. For each of the two-dimensional B-scan OCT images, a set of pixels depicting a specific structure of the eye (e.g., Bruch's membrane or internal limiting membrane) can be identified. Each of the B-scan OCT images can be flattened based on the set of pixels. The three-dimensional data object can be generated to include at least a portion of each flattened B-scan OCT image.

[0009] In some embodiments, the method may include training another convolutional neural network using a training dataset to generate a first set of learned parameter values. Transfer learning can be used to train a convolutional neural network by setting its set of parameter values ​​to the first set of learned parameter values, and then further training the convolutional neural network using another training dataset to generate a second set of learned parameter values. The convolutional neural network may then be comprised of the second set of learned parameter values ​​when processing 3D data objects.

[0010] In some embodiments, the method may further include determining the nature of a clinical trial based on predictions. For example, input data may be entered by a user including or identifying three-dimensional data objects. Predictions may be received, and based on the predictions, it may be determined that a subject is eligible to participate in a particular clinical trial. In another example, after receiving predictions, stratification may be determined for a particular clinical trial in which the subject is or will be involved. For example, stratification may involve assigning individual subjects to different treatment groups and / or control groups so that the groups have similar default predicted geographic atrophy assessments if no treatment is administered, and / or to analyze or normalize the results before making comparisons between different groups. Clinical trials may be generated based on the stratification. Clinical trial results may be output. In another example, after receiving predictions, adjustments may be determined for a particular clinical trial in which the subject is or will be involved. Adjustments may include changes to treatments, such as changes in drugs, changes in dosage, and / or changes in the time interval between treatments. The implementation of the adjusted clinical trial may be facilitated.

[0011] In some embodiments, a method is provided. The method includes a user device detecting input data that includes or identifies a three-dimensional data object corresponding to at least a partial depiction of a subject's eye. A request communication corresponding to a request to generate predicted subsequent geographic atrophy characteristics of the subject's eye is sent to a remote computing system. The request communication includes input data. In response to receiving the request communication, the remote computing system processes the three-dimensional data object using a convolutional neural network to generate a prediction of the subsequent growth of a geographic atrophy lesion in the eye or the subsequent size of a geographic atrophy lesion in the eye. The prediction is sent from the remote computing system to the user device that receives the prediction.

[0012] In some embodiments, the method may further include collecting a set of images of the subject's eyes. A three-dimensional data object can be generated using the set of images.

[0013] Some embodiments of this disclosure include the use of geographic atrophy prediction in the treatment of subjects. Geographic atrophy prediction is provided by a computing device that implements a computational model based on subject data to provide geographic atrophy prediction. The computational model includes a convolutional neural network configured to process three-dimensional data objects corresponding to at least a partial depiction of the subject's eye.

[0014] In some embodiments, a method is provided. The method comprises accessing a data object comprising at least three data channels. Each of the at least three data channels comprises a two-dimensional image corresponding to at least a partial depiction of a subject's eye. The method further comprises processing the data object using a convolutional neural network, generating predictions of the subsequent growth of a geographic atrophic lesion in the eye or the subsequent size of a geographic atrophic lesion in the eye, and outputting the predictions. The convolutional neural network may include one or more two-dimensional convolutional modules, pooling layers, and / or one or more arbitrary units. The subsequent growth of a geographic atrophic lesion may include the growth of one or more geographic atrophic lesions in the eye, and / or the subsequent size of a geographic atrophic lesion may include the subsequent size of one or more geographic atrophic lesions in the eye.

[0015] In some embodiments, the data object may include multiple different frontal OCT-based maps of the eye. For example, the data object may include at least two frontal OCT-based scans of the eye and at least one B-scan of the eye. In another example, the data object may include at least one OCT-based frontal scan of the eye and at least one image obtained using a different type of imaging modality than OCT (e.g., FAF).

[0016] In some embodiments, a convolutional neural network can be trained to simultaneously predict the current geographic atrophy lesion size and the subsequent geographic atrophy state. The subsequent geographic atrophy state may include the subsequent growth of the geographic atrophy lesion or the subsequent size of the geographic atrophy lesion.

[0017] In some embodiments, processing data objects using a convolutional neural network can further generate another prediction of the current size of geographic atrophic lesions in the eye.

[0018] In some embodiments, the method may further include determining the nature of a clinical trial based on predictions. For example, input data may be entered by a user including or identifying three-dimensional data objects. Predictions may be received, and based on the predictions, it may be determined that a subject is eligible to participate in a particular clinical trial. As another example, after receiving predictions, stratification may be determined for specific clinical trials in which the subject is or will be involved. For example, stratification may involve assigning individual subjects to different treatment and / or control groups so that the groups have similar default predicted geographic atrophy ratings if no treatment is administered, and / or so that the results can be analyzed or normalized before making comparisons between different groups. Clinical trials may be generated based on the stratification. Clinical trial results may be output.

[0019] In some embodiments, a method is provided. The method includes detecting, in a user device, input data including or identifying a data object corresponding to at least a partial depiction of a subject's eye. A request communication corresponding to a request to generate a predicted subsequent geographic atrophy characteristic of the subject's eye is transmitted to a remote computing system. The request communication includes the input data. In response to receiving the request communication, the remote computing system processes the data object using a convolutional neural network to generate a prediction of a subsequent growth of a geographic atrophy lesion in the eye or a subsequent size of a geographic atrophy lesion in the eye. The prediction is transmitted to the user device that receives the prediction from the remote computing system.

[0020] In some embodiments, the method can further include collecting a set of images of the subject's eye. The data object can be generated using the set of images.

[0021] Some embodiments of the present disclosure include the use of geographic atrophy prediction in the treatment of a subject. The geographic atrophy prediction is provided by a computing device that implements a computational model based on subject data to provide the geographic atrophy prediction. The computational model includes a convolutional neural network configured to process a data object corresponding to at least a partial depiction of the subject's eye.

[0022] Some embodiments of the present disclosure include a system including one or more data processors. The system can further include a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0023] In some embodiments, a computer program product is provided that is tangibly embodied on a non-transitory machine-readable storage medium. The computer program product can include a set of instructions configured to cause one or more data processors to execute some or all of one or more of the methods disclosed herein.

[0024] The present disclosure is described in conjunction with the following accompanying drawings:

Brief Description of the Drawings

[0025] [Figure 1-1] Shows the central B-scan of the OCT volume.

[0026] [Figure 1-2] Shows an exemplary FAF image used to quantify lesion area grading.

[0027] [Figure 1-3] Shows an exemplary identified lesion area.

[0028] [Figure 2] Shows an exemplary computing network for generating predictions of geographic atrophy lesion size and growth rate according to some embodiments.

[0029] [Figure 3] Shows an exemplary process for predicting the geographic atrophy lesion size and growth of a three-dimensional data object using a three-dimensional convolutional neural network according to some embodiments.

[0030] [Figure 4] Shows an exemplary process for predicting the geographic atrophy lesion size and growth of a data object using a two-dimensional convolutional neural network according to some embodiments.

[0031] [Figure 5]This document outlines a prediction framework using a multi-task 3D inception convolution module, according to several embodiments.

[0032] [Figure 6] This shows the architecture of a 3D inception convolution module according to several embodiments.

[0033] [Figure 7] This document outlines a prediction framework using a multi-task 2D convolutional module, according to several embodiments.

[0034] [Figure 8-1] The performance details for all holdout cases using the multitask SE model are shown. (A) shows a regression plot of the predicted geographic atrophy growth rate against the true growth rate, (B) shows a regression plot of the predicted geographic atrophy lesion area against the true geographic atrophy lesion area, (C) shows outlier OCT frontal images and central B scans in geographic atrophy growth rate prediction, and (D) shows outlier OCT frontal images and central B scans in geographic atrophy lesion area and geographic atrophy growth rate prediction. [Figure 8-2] The performance details for all holdout cases using the multitask SE model are shown. (A) shows a regression plot of the predicted geographic atrophy growth rate against the true growth rate, (B) shows a regression plot of the predicted geographic atrophy lesion area against the true geographic atrophy lesion area, (C) shows outlier OCT frontal images and central B scans in geographic atrophy growth rate prediction, and (D) shows outlier OCT frontal images and central B scans in geographic atrophy lesion area and geographic atrophy growth rate prediction. [Modes for carrying out the invention]

[0035] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label to distinguish similar components. Where only the first reference label is used herein, the description is applicable to any similar component having the same first reference label, regardless of the second reference label.

[0036] Detailed explanation I. Overview This disclosure describes a multi-task deep learning architecture that enables (e.g., simultaneous) prediction of current geographic atrophy (GA) lesion area and geographic atrophy lesion growth rate. GA progression is unique to each individual in terms of lesion area and growth rate. Currently, there are no approved treatments to prevent or delay GA progression.

[0037] Accurate and personalized prediction of geographic atrophy progression can be useful in addressing important trial and clinical issues. For example, geographic atrophy progression prediction can be used to provide subject stratification in clinical trials where slowing geographic atrophy progression is the endpoint, enabling targeted enrollment of clinical subjects and improved evaluation of treatment effects. Furthermore, personalized GA prognosis can be used to manage the disease more efficiently and to understand the pathogenesis of the disease by correlating it with genotype or phenotypic signatures.

[0038] Conventional fundus autofluorescence (FAF) images provide two-dimensional structural data regarding geographic atrophy lesion areas. Optical coherence tomography (OCT) images can provide structural information in addition to the lesion area. The additional structural information provided by high-resolution three-dimensional (3D) OCT images can provide insights into the onset and progression of geographic atrophy that are not possible with FAF images alone. For example, reticular pseudodrusen, high-reflection focus, multilayer thickness reduction, photoreceptor atrophy, and wedge-shaped subretinal reflection reduction detected on OCT images can be potential precursors or biomarkers associated with geographic atrophy transformation. However, OCT images are difficult to process due to their 3D nature. Current OCT-derived predictions rely on features extracted from OCT volumes rather than the complete OCT image, due to the complexity of processing 3D image volumes. In short, current OCT predictions involve additional feature extraction, feature engineering, or image processing steps, or some other abstraction of the baseline OCT image.

[0039] This embodiment provides a method for performing GA prediction directly from baseline OCT volumes without additional steps. In particular, this embodiment includes a multitask 3D neural network for GA lesion size detection and growth rate prediction using baseline 3D OCT images directly. In some embodiments, the deep learning architecture of the neural network may include a convolutional neural network (CNN). The deep learning architecture may be configured to receive (as input) 3D OCT image data and / or 3-channel image data representing data taken up at baseline time. In some cases, the deep learning architecture is configured to generate predictions corresponding to time after baseline time, even though it does not receive input collected at later time. Furthermore, the network within the deep learning architecture may be trained end-to-end. This deep learning technique can improve the accuracy of geographic atrophy statistics compared to state-of-the-art baseline models.

[0040] One or more OCT images collected from a subject may be preprocessed. Preprocessing may be performed to flatten one or more images to compensate for the curvature of the eye. For example, images may be flattened to a structure such as the internal limiting membrane (ILM). Preprocessing may include aligning and / or joining multiple (e.g., flattened) images (e.g., to generate a three-dimensional image). For example, preprocessing may include converting a set of A scans (e.g., showing a number of longitudinal scans) to B scans (e.g., to flatten B scans along a particular biological structure). In some cases, flattened images can be combined to generate a three-dimensional image (or flattening may be performed after combining two-dimensional images to form a three-dimensional image). Preprocessing may further include cropping and / or resizing one or more images to a predetermined size and / or by a predetermined amount.

[0041] Preprocessing may include generating each of the multiple data channels individually. The multiple channels may include two or more channels or three or more channels. At least one, at least two, or at least three of the channels may correspond to OCT channels associated with a particular type of scan (e.g., frontal C-scan or B-scan). In some cases, at least two of the at least three channels may correspond to different types of scans and / or different imaging modalities. For example, at least one channel may contain images acquired using OCT, and at least another channel may contain images acquired using fundus photography, infrared imaging, or scanning last fossa (SLO).

[0042] Preprocessed images can be processed by neural networks, including (for example) deep neural networks, 2D neural networks, 3D neural networks, and / or convolutional neural networks. A neural network may contain one or more convolutional layers (for example, each of two or more convolutional layers, or all of the convolutional layers, may have a convolutional filter of a different size than the other convolutional layer). A neural network may contain activation layers, normalization layers, and / or pooling layers.

[0043] The neural network may be trained and configured to predict one or more geographic atrophy statistic, such as the current volume of a geographic atrophy lesion, the future volume of a geographic atrophy lesion, and / or the future growth rate. The neural network may be a multi-task model trained and configured to simultaneously predict geographic atrophy lesion growth rate and geographic atrophy lesion size. A multi-task model may have a lower potential for overfitting to the growth rate prediction task than a non-multi-task model.

[0044] Figure 1A shows an exemplary B-scan OCT image that can be collected and processed in a deep learning workflow. Figure 1B shows an exemplary fundus autofluorescence (FAF) image. Figure 1C shows lesion grading for the FAF image in Figure 1B. The FAF image includes a geographic atrophic lesion region 105.

[0045] The predicted growth rate or volume of geographic atrophy lesions can be output (e.g., presented or transmitted) along with subject and / or eye identification. The predicted growth rate or volume can inform automated or human recommendations for treatment and / or clinical trial registration (e.g., subject stratification). For example, the predicted growth rate or volume can be used to determine (e.g., by a human or a computing device) whether a subject is eligible for a particular clinical trial. The determination can be made by evaluating rules indicating that the eligibility requirement is that the predicted growth rate or volume is above a predetermined lower threshold and / or below a predetermined upper threshold. As another example, the predicted growth rate or volume can be used for clinical trial stratification (e.g., assigning individual subjects to different treatment and / or control groups so that the groups have similar default predicted geographic atrophy assessments if no treatment is administered, and / or so that the results are analyzed or normalized before making comparisons between different groups).

[0046] II. Definition As used herein, the term “multitask model” refers to a model that includes one or more neural networks trained to simultaneously predict multiple statistical variables (e.g., current geographic atrophy lesion size and future geographic atrophy lesion growth rate). A multitask model may include one or more neural networks. Because a multitask model is trained to fit two sets of information simultaneously, it can avoid or reduce the likelihood of overfitting.

[0047] As used herein, the term "A-scan" refers to a one-dimensional image of the depth profile along the direction of incident light to the eye. Near-infrared light can be guided in a linear direction to generate data at multiple A-scan locations.

[0048] As used herein, the term “B-scan” refers to a cross-sectional view of the eye. In some cases, a B-scan is generated in response to a specific type of measurement. For example, near-infrared light may be transmitted to the eye, and a low-coherence interference signal may generate a two-dimensional B-scan image. In some cases, a B-scan is a frame composed of an array of A-scans. Depending on the orientation of the probe, a B-scan may have a transverse, longitudinal, or axial orientation.

[0049] As used herein, the term “frontal C-scan” refers to a lateral image of the eye layers at a specified depth. For example, a frontal C-scan may be a lateral image of the retinal or choroidal layer of the eye. A C-scan can be acquired by scanning at a constant depth in both high-speed lateral and low-speed axial directions. In some cases, a frontal C-scan is generated from a set of B-scans.

[0050] As used herein, the term “growth rate” refers to the change in size of a geographic atrophic lesion over time. The change in size may be absolute or relative. For example, the growth rate may be the change in volume or area of ​​geographic atrophy in an eye. In two dimensions (e.g., region), the growth rate is the change in volume or area per unit time (e.g., mm²). 2 It can be measured as a ratio or percentage (e.g., area at a later time point relative to area at a baseline time point) or as a percentage (e.g., area at a later time point relative to area at a baseline time point). In three dimensions (e.g., volume), the growth rate is measured as a unit volume per unit time (e.g., mm²). 3 It can be measured as a year, or as a percentage or ratio (e.g., volume at a later point in time relative to volume at a baseline point in time).

[0051] As used herein, the term “geographic atrophy size” refers to the measurable dimensions of one or more geographic atrophic lesions of the eye. For example, in two dimensions, geographic atrophy size is the geographic atrophic lesion (e.g., mm). 2 It can be defined as the area (measured by...).

[0052] As used herein, the term “current” refers to the baseline time. For example, the current geographic atrophic lesion size may be the geographic atrophic lesion size at the time the ocular image data was generated.

[0053] As used herein, the term “progression” refers to a measurable difference from a baseline time to a future time. For example, the progression of geographic atrophy may be the growth of geographic atrophy from a baseline time to a future time. In some cases, the progression of geographic atrophy may be determined by the growth rate.

[0054] III. Exemplary Computing Networks Figure 2 shows an exemplary personal GA prediction network 200 for generating predictions of the growth rate of geographic atrophy lesions, according to several embodiments. The GA progression controller 205 can be configured to train and run a machine learning model to generate predictions about the geographic atrophy state. More specifically, the GA progression controller 205 can be configured to receive one or more images of an eye and output a predicted growth rate or volume of geographic atrophy lesions.

[0055] Predictions can be generated using one or more machine learning models, such as one or more CNNs. The CNNs can include 3D CNNs and / or 2D CNNs. In some cases, the machine learning models can include ensemble machine learning models that process aggregated results from 3D and 2D neural networks.

[0056] The GA progress controller 205 may include an OCT input controller 220 that receives and / or acquires input data, which may include a set of input data objects. The input data objects may include data objects used in the training dataset to train a machine learning model and / or data objects processed by the trained machine learning model.

[0057] Each input data object may correspond to a specific subject, a specific eye, and / or a specific imaging date. An input data object may include images of at least a portion of an eye. An input data object may include images generated using imaging techniques disclosed herein, such as OCT and / or fundus photography. For example, an input data object may include a set of A-scan images (e.g., each depicting a different longitudinal scan of a specific eye), a set of B-scan images (e.g., each depicting a flattened image), or a set of C-scan images (e.g., each corresponding to a specific depth).

[0058] Input data objects may be received and / or acquired from one or more imaging systems 225, which may include an OCT device and / or a fundus camera. In some cases, the imaging system 225 may transmit one or more images to an image data store 230, which may be partially or fully accessible to the GA progression controller 205.

[0059] The input data objects can be preprocessed by the preprocessing controller 235. Preprocessing may include performing normalization and / or standardization. For example, preprocessing may include histogram matching across scans or channels within a data object (e.g., a set of B scans) to unify the intensity of B scans or multiple data objects within a dataset. The preprocessing controller 235 may select a B scan within a set of B scans to become a baseline B scan based on the average intensity level of the B scans. The preprocessing controller 235 may perform intensity normalization by performing histogram matching on the remaining B scans in the set of B scans.

[0060] The preprocessing controller 235 can also perform segmentation (e.g., BM segmentation or ILM segmentation) on each B-scan during preprocessing. After segmentation, each B-scan can be flattened along the depiction of a specific biological structure (e.g., Bruch's membrane (BM) boundary or internal boundary membrane (ILM) boundary). Segmentation can facilitate volume truncation (e.g., removing upper and lower background portions to reduce neural network limitations) and generate a frontal map. After flattening, the preprocessing controller 235 can trim each B-scan to a region of pixels around the boundary. For example, a flattened B-scan can be trimmed to a region 5 pixels above and 250 pixels below the ILM, or to a region 120 pixels above and 135 pixels below the BM. The trimmed region can be resized to a predetermined size (e.g., 512 pixels × 512 pixels) using (e.g.) resampling, interpolation, and / or bilinear interpolation.

[0061] The personalized GA prediction network 200 may include a lesion label detector 240 that can retrieve one or more labels associated with each input data object in the training dataset. Labels may be initially identified based on information provided by an annotator, a healthcare professional, and / or a validated database. Labels may include the size of the geographic atrophic lesion at subsequent time, the cumulative size of the geographic atrophic lesion at subsequent time, the difference between baseline time and subsequent time, the progression of the geographic atrophic lesion size, and / or the progression of the cumulative size of the geographic atrophic lesion.

[0062] The model training controller 245 can execute code to train one or more machine learning models using one or more training datasets. A machine learning model may include one or more preprocessing functions, one or more neural networks, and / or one or more postprocessing functions. It will be understood that with respect to each function, algorithm, and / or network in the model, one or more variables may be fixed during training. For example, the hyperparameters of a neural network (e.g., identifying the number of layers, the size of the input layers, the learning rate, etc.) may be predefined and unadjustable throughout the execution of the code. Furthermore, it will be understood that with respect to each function, algorithm, and / or network in the model, one or more variables may be learned through training. For example, the parameters of a neural network (e.g., identifying various inter-node weights) may be learned.

[0063] Each training dataset can contain a set of training data objects. Each data object can contain an input image (for example, an image showing a portion of an eye or a flattened version thereof). The input images can be collected at baseline time.

[0064] Each data object can be further associated with one or more labels (e.g., two or more labels). Each label may be at least partially based on information collected at a time after baseline (e.g., at least two weeks, at least one month, at least two months, at least six months, at least one year, at least two years, at least five years, or at least ten years from baseline). In some cases, one or more labels may include the size of a given geographic atrophy lesion at a subsequent time, the cumulative size of the geographic atrophy lesion at a subsequent time, the difference between baseline and a subsequent time, the progression of the size of a given geographic atrophy lesion, and / or the progression of the cumulative size of the lesion. Thus, a training dataset may include baseline images and label data indicating how and / or whether a subject's geographic atrophy progressed over a period of time.

[0065] The model training controller 245 can use training data to train a machine learning model. In addition to one or more conventional machine learning models (e.g., neural networks, bin-based classifiers, regression algorithms, etc.), the machine learning model may further include one or more pre-processing functions and / or one or more post-processing functions. For example, a pre-processing function may adjust the size, intensity distribution, and / or viewpoint of one or more input images, and / or a post-processing function may convert the model output into values ​​aligned with a given scale (or category list), recommendation, stratified identifier, etc.

[0066] The model training controller 245 can train various components of a machine learning model separately or together. For example, the training dataset can be used to train one or more preprocessing algorithms (e.g., performing segmentation) and neural networks simultaneously. As another example, the preprocessing algorithms may be trained separately from the neural networks (e.g., using the same or different parts of the training data).

[0067] The GA predictor generator 250 can process non-training data using its architecture and learned parameters to produce results. Results can be generated in response to request communications from a client device 210 (e.g., containing or identifying input data). For example, a request might identify a database identifier, subject, eye, and access password to utilize a set of images of a subject's eyes. The request communications could correspond to a request to generate predicted subsequent geographic atrophy characteristics (e.g., size or growth) of the subject's eyes. The subject to which the request relates may differ from each subject represented in the training data.

[0068] In some cases, the request includes data objects configured to be input into a machine learning model. In some cases, the request includes data objects configured to undergo preliminary processing (e.g., decoding, standardization, etc.) that is then configured to be input into a machine learning model. In some cases, the request includes information that can be used to retrieve data objects configured to be input into a machine learning model (e.g., one or more identifiers, verification codes and / or passcodes).

[0069] In some cases, post-processing functions handle predictions when using models, features, or rules. For example, a classifier may be used to assign severity prediction results to a subject eye input dataset based on preliminary results. Another example is when rules are used to determine, based on preliminary results, whether a change in treatment is recommended for consideration.

[0070] Therefore, the GA prediction generator 250 can access input data objects and feed the data objects to a machine learning model. The results of the machine learning model can predict the current geographic atrophy lesion size and / or subsequent geographic atrophy lesion state (e.g., subsequent growth or size of the geographic atrophy lesion) (e.g., at the time the data objects were generated). The GA progression controller 205 can output predictions characterizing the subsequent growth of geographic atrophy in the eye (e.g., growth of one or more geographic atrophy lesions in the eye) and / or subsequent size of geographic atrophy in the eye (e.g., subsequent size of one or more geographic atrophy lesions in the eye). For example, the predictions may include the subsequent growth or subsequent size of geographic atrophy in the subject's eye over a subsequent period, the predicted absolute or relative growth, and / or the probability or likelihood of the predicted relative or absolute size of the geographic atrophy lesion. The GA progression controller 205 may further generate predictions of the current size (e.g., area or volume) of one or more geographic atrophy lesions in the eye.

[0071] In some cases, the GA predictor generator 250 can run a 3D convolutional neural network (e.g., trained by the model training controller 245). The 3D convolutional neural network can be configured to receive and process 3D data objects (e.g., corresponding to a set of flattened B-scans). The 3D convolutional neural network can include one or more 3D convolutional modules (e.g., an inception convolutional module). Each 3D convolutional module can include a set of convolutional layers using at least two different sizes of convolutional filters (e.g., a 1×1×1 convolutional filter, a first 3×3×3 convolutional filter, and a second 3×3×3 convolutional filter). The 3D convolutional network can optionally or additionally include one or more 3D convolutional layers.

[0072] In some cases, the GA predictor generator 250 can run two-dimensional and / or multi-channel neural networks. For example, the data object may include multi-channel image data. Each channel within the image channel may include two-dimensional data corresponding to (e.g.) one or more imaging techniques and / or one or more imaging viewpoints. For example, the two-dimensional image data may include B-scans and / or one or more C-scans. Thus, in some cases, the multi-channel image data may include data corresponding to two or more different types of imaging acquisitions and / or two or more different types of viewpoints. The multi-channel image data may be processed by one or more neural networks to generate predictions of the growth rate and / or subsequent size of geographic atrophic lesions.

[0073] Each of the one or more channels in the data channel may include a two-dimensional image corresponding to at least a partial depiction of the subject's eye. In some examples, 3-channel data may include multiple frontal C-scan OCT images. In another example, 3-channel data may include at least two frontal C-scan OCT images and at least one B-scan image (e.g., a central B-scan or one or more B-scans corresponding to pseudo-randomly selected lateral positions). In yet another example, 3-channel data may include at least one frontal C-scan OCT image and at least one image obtained using a different type of imaging modality than OCT (e.g., infrared and / or fundus autofluorescence imaging or cSLO). Alternatively, 3-channel data may include a full-depth frontal OCT image, a sub-BM frontal OCT image (e.g., a depth of 100 pixels below the BM), and an upper BM frontal OCT image (e.g., a depth of 100 pixels above the BM). While some disclosures herein refer to “3-channel” data, object data, etc., it will be understood that data containing more than three channels may be used instead.

[0074] The neural network used by the GA predictor generator 250 may include a transfer learning neural network (Inception V3) and / or a dense layer. For example, the neural network may be initialized with parameters to predict the growth rate and / or subsequent size of different types of lesions in the eye (e.g., inguinal lesions, conjunctival lesions, or choroidal melanoma). The neural network can then be further trained to predict the growth rate and / or subsequent size of geographic atrophic lesions in the eye.

[0075] The network may include batch normalization, one or more activation layers (e.g., using ReLU activation), and / or one or more pooling layers (e.g., using max pooling). For example, batch normalization, ReLU activation, and / or pooling layers may follow each of one or more convolutional layers. The neural network may include global average pooling and / or dense layers (e.g., in the output layer and / or the layer following the last convolutional layer). The neural network may include an activation function (e.g., linear activation). The neural network may include attention units, such as standard squeeze and excitation (SE) attention units. An SE attention unit may include global average pooling of feature maps from the convolutional layers and applying one or more linear transformations to the result of the pooling. The attention unit may be configured to facilitate adaptive tuning of the weights of each feature map.

[0076] In some cases, the GA predictor generator 250 can process data objects using an ensemble model. The ensemble model may include multiple neural networks, such as one or more 3D convolutional neural networks and one or more 2D convolutional neural networks. The ensemble model can process data objects and aggregate the results from each neural network. For example, the ensemble model may average the predicted subsequent size and / or predicted growth rate of geographic atrophy lesions from each neural network. The ensemble model may output the average of the predicted subsequent size and / or subsequent growth of geographic atrophy in the subject's eye.

[0077] The GA analysis controller 255 can process predictions and communicate the results (or their processed versions) to a client device 210 or other system (e.g., associated with a lab technician or caregiver). For example, the GA analysis controller 255 can generate a prediction output that identifies the size of subsequent geographic atrophy or the subsequent growth of geographic atrophy in the subject's eye. In some cases, the GA analysis controller 255 can further post-process the results. For example, the GA analysis controller 255 may evaluate one or more rules to determine whether any of the conditions of one or more rules are met. For example, a rule may be configured to be met if the predicted lesion size and / or predicted progression is large enough to recommend a change in treatment. As another example, a rule may be configured to recommend a particular type of treatment approach if the predicted lesion size and / or predicted progression is within a predetermined open or closed range. As yet another example, a rule may be configured to indicate whether a subject is eligible for a clinical trial based on the predicted lesion size and / or predicted progression.

[0078] The output can then be presented and / or transmitted, thereby facilitating the display of the output data on, for example, a computing device (e.g., client device 210).

[0079] IV. Techniques for predicting the area and growth rate of geographic atrophic lesions Figure 3 illustrates an exemplary process, according to several embodiments, of using a convolutional neural network to predict the map-like atrophy size and growth of a 3D data object. In block 305, a 3D data object corresponding to at least a partial depiction of the subject's eye is generated. The 3D data object can be generated by segmenting each image of a set of 2D images (e.g., B-scans) to identify segments within the images (e.g., corresponding to Bruch's membrane and / or the internal limiting membrane). Alternatively, the 3D data object may be generated by accessing a 2D B-scan OCT image of the subject's eye, generating feature maps using the 2D B-scan OCT image, and generating a 3D data object containing a set of feature maps. As another example, the 3D data object may be generated by accessing a set of 2D B-scan OCT images of the subject's eye. A set of pixels depicting specific structures of the eye can be identified for each of the set of 2D B-scan OCT images. For example, a set of pixels depicting Bruch's membrane and / or the internal limiting membrane can be identified. Each of the 2D B-scan OCT images can be flattened based on the set of pixels. Next, a three-dimensional data object can be generated that includes at least a portion of each flattened B-scan OCT image.

[0080] In block 310, a three-dimensional data object is accessed. The three-dimensional data object can be accessed from local memory or storage. In some examples where the three-dimensional data object is generated in an imaging or remote system, the three-dimensional data object can be accessed from the imaging or remote system (e.g., imaging system 225 in Figure 2). The three-dimensional data object can be accessed in response to a remote computing system receiving a request communication (e.g., from a client system) to generate predicted subsequent geographic atrophy characteristics of the subject's eye.

[0081] In block 315, a 3D data object is processed using a convolutional neural network to generate predictions of the subsequent growth and / or subsequent size of geographic atrophic lesions in the eye. The convolutional neural network can be a 3D neural network having one or more convolutional modules (e.g., inception modules) and / or one or more pooling layers. Each convolutional module can include a 3D convolutional module containing a set of convolutional layers. At least two of the sets of convolutional layers can include convolutional filters of different sizes. For example, the first layer can use a 1×1×1 convolutional filter, and each of the one or more second layers can use a 3×3×3 convolutional filter. The convolutional neural network may further include attention units, such as standard SE attention units, which can add parameters to each channel of the convolutional block to facilitate adaptive tuning of the weights of each feature map.

[0082] In block 320, predictions are output. Predictions can be used to facilitate the determination of a subject's eligibility to participate in a particular clinical trial. For example, a prediction of the size of geographic atrophy above a threshold may indicate that a subject is eligible for a particular clinical trial. Additionally or alternatively, predictions can be used to facilitate the determination of stratification for a particular clinical trial in which a subject is or will be involved. For example, predictions can be used to assign subjects to different treatment and / or control groups so that the groups have similar default predicted geographic atrophy assessments if no treatment is administered, and / or to analyze or normalize the results before making comparisons between different groups. Clinical trial results can be generated based on the stratification. Clinical trial results may be further output.

[0083] In some cases, predictions can be used to facilitate a subject's choice of treatment and / or decision on whether to change their treatment. For example, a machine learning model can be trained using a training dataset corresponding to a set of subjects who have received or have received a particular treatment. Baseline images collected for each subject can correspond to the period when the treatment was just started or about to start. The trained model can then be used to predict the extent to which geographic atrophy will progress and / or the size of subsequent geographic atrophic lesions if the subject uses a particular treatment, and thus inform whether a caregiver should recommend a particular treatment (e.g., against another treatment).

[0084] Figure 4 illustrates an exemplary process, according to several embodiments, of using a convolutional neural network to predict the size and growth of a mapped atrophic lesion in a data object. In block 405, a data object containing at least three data channels is accessed. Each of the at least three data channels may include a two-dimensional image corresponding to at least a partial depiction of the subject's eye. The data object may include multiple different frontal OCT-based maps of the eye. In some cases, the data object may include at least two frontal OCT-based scans of the eye and at least one B-scan of the eye (e.g., a central B-scan or one or more B-scans corresponding to a pseudo-randomly selected lateral position). Alternatively, the data object may include at least one OCT-based frontal scan of the eye and at least one image obtained using a different type of imaging modality than OCT.

[0085] In block 410, data objects are processed using a convolutional neural network to generate predictions of the subsequent growth and / or subsequent size of geographic atrophic lesions in the eye. The convolutional neural network can further or alternatively generate another prediction of the current size of geographic atrophic lesions in the eye. The convolutional neural network can be a 2D neural network trained to simultaneously predict the current geographic atrophic lesion size and the subsequent geographic atrophic lesion state, which may include subsequent geographic atrophic lesion growth or subsequent geographic atrophic lesion size. The convolutional neural network can be trained using transfer learning. For example, the convolutional neural network can be initialized with parameters to predict the growth rate and / or subsequent size of different types of lesions in the eye (e.g., inguinal lesions, conjunctival lesions, or choroidal melanoma). The neural network can then be further trained to predict the growth rate and / or subsequent size of geographic atrophic lesions in the eye. The convolutional neural network may include one or more 2D convolutional modules and / or pooling layers. A convolutional neural network can further include one or more arbitrary units (e.g., SE attention units). A convolutional neural network can also include batch normalization, one or more activation layers (e.g., using ReLU activation) and / or one or more pooling layers (e.g., using max pooling).

[0086] In block 415, predictions (e.g., subsequent growth, current size, and / or future size) are output. These predictions can be used to facilitate the determination of a subject's eligibility to participate in a particular clinical trial. For example, a prediction of the size of a geographic atrophic lesion above a threshold may indicate that a subject is eligible for a particular clinical trial. Additionally or alternatively, predictions can be used to facilitate the determination of stratification for specific clinical trials in which a subject is or will be involved. Clinical trial results can be generated based on the stratification. Clinical trial results may be further output.

[0087] In some cases, predictions can be used to facilitate a subject's choice of treatment and / or decision on whether to change their treatment. For example, a machine learning model can be trained using a training dataset corresponding to a set of subjects who have received or have received a particular treatment. Baseline images collected for each subject can correspond to the period when the treatment was just started or about to start. The trained model can then be used to predict the extent to which geographic atrophy will progress and / or the size of subsequent geographic atrophic lesions if the subject uses a particular treatment, and thus inform whether a caregiver should recommend a particular treatment (e.g., against another treatment). [Examples]

[0088] V. Examples VA Example 1: Prediction of map-like atrophy growth rate by processing 3-dimensional OCT images using a neural network. VA1. Method The study was retrospectively conducted on the eyes of subjects with bilateral geographic atrophy enrolled in the lamparizumab Phase 3 trial SPECTRI (NCT 02247531). 1.92 × 6 × 6 mm 3A 496×1024×49-voxel macular Spectralis SD-OCT volume (Heidelberg Engineering, Inc., Heidelberg, Germany) of the area was used to predict the geographic atrophy lesion area and lesion growth. The geographic atrophy lesion area (mm 2 , measured from FAF images graded by two readers or adjudicators as needed) and the geographic atrophy lesion growth rate (mm 2 / year) were derived using all available visit measurements and fit to a linear model. The image dataset from 522 subjects was divided into five for each subject. Five-fold cross-validation was performed for all folds balanced for baseline factors and treatment groups. Each B-scan was flattened along the ILM and resized to a size of 128×128 pixels. Then, a multi-task three-dimensional convolutional neural network (CNN) model was trained to predict the geographic atrophy growth rate from the baseline OCT volume with initialized weights from the same model trained to predict the geographic atrophy area. The performance was evaluated by calculating the within-sample coefficient of determination (R 2 ) defined as the square of the Pearson correlation coefficient (r) between the true geographic atrophy growth rate and the predicted geographic atrophy growth rate.

[0089] V.A.2. Results The geographic atrophy lesion area prediction showed an average cross-validation R 2 of 0.88 (range: 0.77 to 0.92), and the geographic atrophy growth rate prediction showed an average cross-validation R 2 of 0.30 (range: 0.21 to 0.43). The growth rate prediction results were comparable to previous studies on predicting the geographic atrophy growth rate (R2 = 0.35) by deep learning using baseline multimodal retinal images (Normand G et al., IOVS 2019;60:ARVO E-Abstract 1452).

[0090] This embodiment demonstrates the feasibility of using OCT images to predict individual map-like atrophy growth rates using a three-dimensional deep learning approach. Predictions can be improved by combining them with other imaging modalities (e.g., infrared and fundus autofluorescence) and / or relevant clinical variables.

[0091] VB Example 2: Prediction of geographic atrophy growth rate by processing OCT images using a neural network. VB1 3D Modeling Method The prediction was formulated as a multi-task regression problem. The training dataset was

number

[0092] An overview of this approach is shown in Figure 5. The 3D OCT volume was first preprocessed and then input into a multi-task 3D inception CNN-based model. Furthermore, a self-attention mechanism using squeeze and excitation (SE) units was added to enhance the feature map. Geographic atrophy lesion area and growth rate were simultaneously predicted. Geographic atrophy growth rate was shown to have a weak correlation with geographic atrophy lesion area. The multi-task model is expected to find a representation that incorporates information from both tasks, reducing the likelihood of overfitting to only the growth rate prediction task.

[0093] VB1.a.OCT Image Preprocessing The OCT volume contained 1024 A-scans and 49 B-scans (496 × 1024 × 49 voxels) with a depth of 496 pixels. The volume size was reduced for the model input and focus on the retina (region of interest). For each B-scan, the image was flattened along the internal limiting membrane (ILM). The ILM is the boundary between the retina and the vitreous humor, pre-segmented by the OCT device software (Heidelberg Engineering, Inc., Heidelberg, Germany). Next, for each B-scan, the region 5 pixels above and 250 pixels below the ILM was trimmed and defined as the region of interest. Then, each trimmed region (256 pixels × 1024 A-scans) was resized to 128 × 128 pixels by bilinear interpolation (Figure 5). There was no reduction in the number of B-scans. Therefore, the original volume was reduced to 128 × 128 × 49 voxels for the model input.

[0094] VB1.b 3D Inception Convolution Module Figure 6 shows the details of the 3D inception convolution module. The network consisted of one normal convolutional layer and four 3D inception convolutional blocks. Of these, the normal convolutional layer and the first three 3D inception convolutional blocks shared area and growth rate prediction. Batch normalization, ReLU activation, and max pooling were performed after each convolutional layer. After the last convolutional block, a dense layer with global average pooling and linear activation was used as the output layer. The anisotropy of the 3D volume was considered in the selection of filter size, stride, and pooling size. Hyperparameters were optimized using random search.

[0095] VB1.c.SE Attention Unit and Joint Loss The 2D SE attention unit was generalized to 3D and added to three 3D inception modules. As in the 2D case, feature maps from the inception modules are reweighted by the outputs from the SE unit to account for interdependencies between channels. More specifically, the output feature maps from the 3D inception modules with dimensions H×W×D×C are transformed into a global tensor of dimension 1×1×1×C by 3D spatial averaging. This tensor is essentially a vector in the channel direction, a vector with the same dimension as the number of channels. A fully connected layer with ReLU activation is applied in the channel direction to reduce the output tensor to a dimension of 1×1×1×C / r, where r is called the reduction ratio and is a divisor of C, which is the number of channels. In this test, the reduction ratio was set to 4. A second fully connected layer and subsequent sigmoid activation with output dimension 1×1×1×C give each channel smooth gating. Finally, each feature map in the convolutional block is weighted by multiplying it by the outputs from the SE unit. Finally, the joint loss of the mean squared error in the prediction of geographic atrophic lesion area and growth rate was used for training. Each term was weighted at 0.5.

[0096] VB1.d. Baseline Model To compare performance and gain insights into the model, three additional baseline models were tested. The first baseline model (i.e., the base model) was a simple 3D Inception convolutional neural network for predicting only the geographic atrophy growth rate. The model was identical to the proposed model without the SE attention unit and multitask setting. For model parameter initialization, He initialization was employed. The second baseline model (i.e., the cascade model) used a cascade approach, first training the model to predict geographic atrophy lesion area. Then, the model was initialized using weights learned during area prediction training, and then trained to predict geographic atrophy growth rate. The cascade models trained for each prediction type shared the same 3D convolutional architecture as the first baseline model. The third baseline model (i.e., the multitask model) was a multitask model without the SE attention unit to see if the SE attention block provided additional performance improvements. The second and third baseline models were trained to predict current and future geographic atrophy lesion area.

[0097] VB2 2D Modeling Method Figure 7 shows an overview of the multi-task 2D convolutional network 700. The model includes a transfer learning Inception V3 neural network 705. A dense layer 710 is included after the transfer learning Inception V3 network 705. The geographic atrophic lesion area 715 and growth rate 720 are predicted simultaneously. The 2D multi-task model is expected to find a representation that incorporates information from both tasks.

[0098] VB3.Result The models were run on a complete training dataset of patients with bilateral geographic atrophy enrolled in the lamparizumab phase 3 trial SPECTRI (NCT 02247531) and evaluated five times against a holdout dataset. The training dataset consisted of 1934 visits from 560 subjects, and the holdout dataset consisted of 114 baseline visits from 114 subjects. The mean performance and standard deviation of the holdout dataset are shown in Table 1. The results indicate that the proposed models perform best for predicting geographic atrophy growth rate. The multitask framework contributed to the majority of the improvement compared to the first baseline model. SE attention block did not appear to significantly improve performance. The cascade model also performed better than the first baseline model. In contrast, performance for predicting geographic atrophy lesion area was comparable across all models.

[0099] Examples of regression plots of predicted geographic atrophy growth rate against true growth rate, and predicted geographic atrophy lesion area against true geographic atrophy lesion area, for all holdout cases from additional model runs are shown in Figures 8A and 8B. True and predicted values ​​of lesion area (R 2 The squared Pearson correlation coefficient between ) and was 0.92. Performance was better when the area of ​​geographic atrophy lesions was relatively small than when the area of ​​lesions was larger. Performance of growth rate prediction was better than deep learning-based prediction of geographic atrophy growth rate using both baseline FAF and infrared retinal images from a much larger population (2000+ eyes) (R 2This was comparable to previous studies regarding =0.43). Figure 8C shows outlier OCT frontal images and central B-scans in geographic atrophy growth rate prediction. The OCT volume can be seen to have motion artifacts and pixel intensity mismatches across the B-scans. Furthermore, the OCT volume size in this case was 496 × 512 × 49 voxels, and only about 5% of the training data had this resolution. However, the prediction of geographic atrophy lesion area was consistent with ground truth. Figure 8D shows another outlier OCT frontal images and central B-scans in both geographic atrophy lesion area and growth rate prediction. In this case, some B-scans had significantly lower image intensity and dark bands in the OCT frontal images, so image quality is likely the root cause as well. [Table 1]

[0100] VB4.Interpretation Therefore, the deep learning model was successfully trained to process OCT volumes and predict the progression of geographic atrophy disease without any prior feature manipulation. The novel, customized 3D multitask deep learning pipeline simultaneously predicted the current geographic atrophy lesion area and the future geographic atrophy lesion growth rate with high accuracy, based solely on baseline OCT volumes. The proposed model outperformed other baseline models.

[0101] VI. Further Considerations Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-temporary computer-readable storage medium containing instructions, such that when the instructions are executed on one or more data processors, the one or more data processors cause one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more processes. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to execute some or all of the methods disclosed herein and / or some or all of one or more processes.

[0102] The terms and expressions used are for illustrative purposes only, not limitation, and in using such terms and expressions there is no intention to exclude equivalents or parts of the features shown and described, however it should be recognized that various modifications are possible within the scope of the invention as described in the claims. Accordingly, while the invention as described in the claims is specifically disclosed by embodiments and optional features, modifications and variations of the concepts disclosed herein may be relied upon by those skilled in the art, and it should be understood that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0103] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, this description of preferred exemplary embodiments provides a possible description for implementing various embodiments for those skilled in the art. It will be understood that various modifications can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0104] This description provides specific details to offer a complete understanding of the embodiments. However, it will be understood that embodiments can be carried out without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid unnecessarily obscuring the embodiments with excessive detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. In certain embodiments, for example, the following are provided: (Item 1) Accessing a 3D data object that corresponds to at least a partial depiction of the subject's eyes, Processing the three-dimensional data object using a convolutional neural network to generate predictions of the subsequent growth or subsequent size of the geographic atrophic lesion in the eye, Outputting the aforementioned prediction Methods that include... (Item 2) The method according to item 1, wherein the convolutional neural network includes one or more three-dimensional convolutional modules. (Item 3) The method according to item 1 or 2, wherein the convolutional neural network includes a pooling layer. (Item 4) The method according to any one of items 1 to 3, wherein the convolutional neural network is trained to simultaneously predict the current geographic atrophy lesion size and the subsequent geographic atrophy state, the subsequent geographic atrophy state including subsequent geographic atrophy lesion growth or the subsequent geographic atrophy lesion size. (Item 5) The method according to any one of items 1 to 4, wherein processing the three-dimensional data object using the convolutional neural network further generates another prediction of the current size of the geographic atrophic lesion in the eye. (Item 6) The method according to any one of items 1 to 5, wherein the subsequent growth of the geographic atrophic lesion includes the growth of one or more geographic atrophic lesions in the eye, and / or the subsequent size of the geographic atrophic lesion includes the subsequent size of one or more geographic atrophic lesions in the eye. (Item 7) The method according to any one of items 1 to 6, further comprising generating the three-dimensional data object by performing a series of operations including segmenting each image in a set of two-dimensional images in order to identify segments bounded by predicted positions of Bruch's membrane and / or internal boundary membranes. (Item 8) Accessing a set of two-dimensional B-scan OCT images of the subject's eye, The process involves generating a set of feature maps using the aforementioned set of 2D B-scan OCT images, The method according to any one of items 1 to 6, further comprising generating the three-dimensional data object by performing a series of operations including generating the three-dimensional data object to include the set of feature maps. (Item 9) Accessing a set of two-dimensional B-scan OCT images of the subject's eye, For each of the aforementioned two-dimensional B-scan OCT images, the set of pixels that depicts a specific structure of the eye is identified. For each of the sets of two-dimensional B-scan OCT images, the two-dimensional B-scan OCT image is flattened based on the set of pixels. The three-dimensional data object is generated to include at least a portion of each flattened B-scan OCT image. The 3D data object is generated by performing a series of operations including the following: The method described in any one of items 1 to 6, further including the method described in any one of items 1 to 6. (Item 10) The first set of learned parameter values ​​is generated by training another convolutional neural network using the training dataset; Setting the set of parameter values ​​of the convolutional neural network to the first set of learned parameter values, Further training the convolutional neural network using a different training dataset to generate a second set of learned parameter values, wherein the convolutional neural network generates a second set of learned parameter values ​​that is comprised of the second set of learned parameter values ​​when processing the 3D data object. The method according to any one of items 1 to 9, further comprising training the convolutional neural network using transfer learning; (Item 11) The method according to any one of items 1 to 10, wherein the three-dimensional data object includes a three-dimensional depiction of the volume of the subject's eye. (Item 12) The user inputs input data that includes or identifies the three-dimensional data object, Receiving the aforementioned prediction, Based on the above prediction, it is determined that the subject is eligible to participate in a particular clinical trial. The method described in any one of items 1 through 11, further including the method described in any one of items 1 through 11. (Item 13) The user inputs input data that includes or identifies the three-dimensional data object, Receiving the aforementioned prediction, Based on the above prediction, determine the stratification for specific clinical trials in which the subject is or will be involved. The method described in any one of items 1 through 11, further including the method described in any one of items 1 through 11. (Item 14) To generate clinical trial results based on the stratification described above, Outputting the aforementioned clinical trial results The method described in item 13, further including the method described in item 13. (Item 15) The user inputs input data that includes or identifies the three-dimensional data object, Receiving the aforementioned prediction, Based on the aforementioned predictions, to determine adjustments to specific clinical trials in which the subject is or will be involved, To facilitate the implementation of the aforementioned adjusted specific clinical trials and The method described in any one of items 1 through 11, further including the method described in any one of items 1 through 11. (Item 16) Receiving input data that includes or identifies a 3D data object corresponding to at least a partial depiction of the subject's eye, Receiving a request communication corresponding to a request to generate predicted subsequent geographic atrophy characteristics of the subject's eye, wherein the request communication includes the input data. Using a convolutional neural network to process the three-dimensional data object to generate predictions of the subsequent growth or subsequent size of the geographic atrophic lesion in the eye, Outputting the aforementioned prediction Methods that include... (Item 17) The process involves collecting a set of images of the subject's eyes, wherein the three-dimensional data object is generated using the set of images. The method described in item 16, further including the method described in item 16. (Item 18) Use of geographic atrophy prediction in the treatment of a subject, wherein the geographic atrophy prediction is provided by a computing device that implements a computational model based on subject data to provide the geographic atrophy prediction, the computational model including a convolutional neural network configured to process three-dimensional data objects corresponding to at least a partial depiction of the subject's eye. (Item 19) Accessing a data object that includes at least three data channels, each of which includes a two-dimensional image corresponding to at least a partial depiction of the subject's eye, Using a convolutional neural network to process the data objects, predict the subsequent growth of the geographic atrophic lesion in the eye, or generate the subsequent size of the geographic atrophic lesion in the eye, Outputting the aforementioned prediction Methods that include... (Item 20) The method according to item 19, wherein the data object includes multiple different frontal OCT-based maps of the eye. (Item 21) The method according to item 19 or 20, wherein the data object includes at least two frontal OCT-based scans of the eye and at least one B-scan of the eye. (Item 22) The method according to any one of items 19 to 21, wherein the data object includes at least one OCT-based frontal scan of the eye and at least one image obtained using an imaging modality of a different type than OCT. (Item 23) One or more data processors, A non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the non-temporary computer-readable storage medium causes the one or more data processors to execute some or all of the methods disclosed herein. A system equipped with these features. (Item 24) A computer program product tangibly embodied in a non-temporary machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

Claims

1. One or more data processors; A non-temporary computer-readable storage medium containing instructions, wherein when an instruction is executed on one or more data processors, the non-temporary computer-readable storage medium causes the one or more data processors to perform a plurality of operations. A system comprising the above operation, The one or more data processors access a set of two-dimensional images that include at least a partial depiction of the subject's eyes, The one or more data processors access a three-dimensional data object corresponding to at least a partial depiction of the subject's eye, wherein the three-dimensional data object generated from the set of two-dimensional images is segmented based on the predicted position of the membrane in the subject's eye. Processing the three-dimensional data object using a convolutional neural network with one or more data processors, generating predictions of the subsequent growth or subsequent size of the geographic atrophic lesion in the eye, predicting the future geographic atrophic lesion area, and predicting the geographic atrophic lesion rate. The prediction is output by one or more of the aforementioned data processors. A system comprising a convolutional neural network trained to simultaneously predict the current geographic atrophy lesion size and the subsequent geographic atrophy state, wherein the subsequent geographic atrophy state includes subsequent geographic atrophy lesion growth or the subsequent geographic atrophy lesion size.

2. The system according to claim 1, wherein the convolutional neural network includes one or more three-dimensional convolutional modules.

3. The system according to claim 1 or 2, wherein the convolutional neural network includes a pooling layer.

4. The system according to any one of claims 1 to 3, wherein processing the three-dimensional data object using the convolutional neural network by one or more data processors further generates another prediction of the current size of the geographic atrophic lesion in the eye as a prediction result different from the prediction of the subsequent size of the geographic atrophic lesion in the eye.

5. The system according to any one of claims 1 to 4, wherein the subsequent growth of the geographic atrophic lesion includes the growth of one or more geographic atrophic lesions in the eye, and / or the subsequent size of the geographic atrophic lesion includes the subsequent size of one or more geographic atrophic lesions in the eye.

6. The system according to any one of claims 1 to 5, wherein the operation further comprises generating the three-dimensional data object by having one or more data processors perform a series of operations including segmenting each image in the set of two-dimensional images in order to identify segments bounded by predicted positions of the Bruch film and / or internal boundary film.

7. The operation described above is: The one or more data processors access a set of two-dimensional B-scan OCT images of the subject's eye, The one or more data processors generate a set of feature maps using the set of two-dimensional B-scan OCT images, The system according to any one of claims 1 to 5, further comprising generating the three-dimensional data object by having one or more data processors perform a series of operations including generating the three-dimensional data object to include the set of feature maps.

8. The operation described above is: The one or more data processors access a set of two-dimensional B-scan OCT images of the subject's eye, The one or more data processors identify, for each set of the two-dimensional B-scan OCT images, a set of pixels that depict a specific structure of the eye. The one or more data processors flatten each of the sets of two-dimensional B-scan OCT images based on the set of pixels, The one or more data processors generate the three-dimensional data object so as to include at least a portion of each flattened B-scan OCT image. The system according to any one of claims 1 to 5, further comprising generating the three-dimensional data object by one or more data processors by performing a series of operations including the above.

9. The operation described above is: The above-mentioned data processors train another convolutional neural network using the training dataset to generate a first set of learned parameter values; The one or more data processors set the set of parameter values ​​of the convolutional neural network to the first set of learned parameter values, The process involves further training the convolutional neural network using another training dataset with one or more data processors to generate a second set of learned parameter values, wherein the convolutional neural network is composed of the second set of learned parameter values ​​when processing the three-dimensional data objects. The system according to any one of claims 1 to 8, further comprising: training the convolutional neural network using transfer learning with the one or more data processors;

10. The system according to any one of claims 1 to 9, wherein the three-dimensional data object includes a three-dimensional depiction of the volume of the subject's eye.

11. The operation described above is: Input data is entered by the user that includes the three-dimensional data object, or identifies the three-dimensional data object with a database identifier, subject, eye, and access password for using the three-dimensional data object. The prediction is received by one or more data processors, The one or more data processors determine, based on the prediction, that the subject is eligible to participate in a particular clinical trial. The system according to any one of claims 1 to 10, further comprising:

12. The operation described above is: Input data is entered by the user that includes the three-dimensional data object, or identifies the three-dimensional data object with a database identifier, subject, eye, and access password for using the three-dimensional data object. The prediction is received by one or more data processors, The one or more data processors determine, based on the predictions, stratification for specific clinical trials in which the subject is or will be involved, the stratification including assigning individual subjects to different treatment groups and / or control groups such that the groups have similar default predicted geographic atrophy assessments if no treatment is administered, and / or normalize the results before making comparisons between different groups. The system according to any one of claims 1 to 10, further comprising:

13. The operation described above is: The one or more data processors generate clinical trial results based on the stratification, The one or more data processors output the clinical trial results. The system according to claim 12, further comprising:

14. The operation described above is: Input data is entered by the user that includes the three-dimensional data object, or identifies the three-dimensional data object with a database identifier, subject, eye, and access password for using the three-dimensional data object. The prediction is received by one or more data processors, The one or more data processors determine, based on the predictions, adjustments to a specific clinical trial in which the subject is or will be involved. The one or more data processors facilitate the execution of the coordinated specific clinical trial. The system according to any one of claims 1 to 10, further comprising:

15. One or more data processors; A non-temporary computer-readable storage medium containing instructions, wherein when an instruction is executed on one or more data processors, the non-temporary computer-readable storage medium causes the one or more data processors to perform a plurality of operations. A system comprising the above operation, The one or more data processors receive input data that includes a three-dimensional data object corresponding to at least a partial depiction of the subject's eye, or identifies the three-dimensional data object with a database identifier, subject, eye, and access password for using the three-dimensional data object corresponding to at least a partial depiction of the subject's eye, wherein the three-dimensional data object is generated from a set of two-dimensional images segmented based on the predicted position of the membrane in the subject's eye. The one or more data processors receive a request communication corresponding to a request to generate predicted subsequent geographic atrophy characteristics of the subject's eye, wherein the request communication includes the input data. The one or more data processors process the three-dimensional data objects using a convolutional neural network to generate predictions of the subsequent growth or subsequent size of the geographic atrophic lesions in the eye, predict the future geographic atrophic lesion area, and predict the geographic atrophic lesion rate. The prediction is output by one or more of the aforementioned data processors. A system comprising a convolutional neural network trained to simultaneously predict the current geographic atrophy lesion size and the subsequent geographic atrophy state, wherein the subsequent geographic atrophy state includes subsequent geographic atrophy lesion growth or the subsequent geographic atrophy lesion size.

16. The operation described above is: The process involves collecting a set of images of the subject's eyes using one or more data processors, wherein a three-dimensional data object is generated using the set of images. The system according to claim 15, further comprising:

17. A system for providing geographic atrophy prediction for the treatment of a subject, wherein the system is With one or more data processors; A non-temporary computer-readable storage medium containing instructions, wherein when an instruction is executed on one or more data processors, the non-temporary computer-readable storage medium causes the one or more data processors to perform a plurality of operations. The system is a computing device, The operation includes a computational model implemented by one or more data processors based on subject data to provide the geomorphic atrophy prediction, the computational model includes a convolutional neural network configured by the one or more data processors to process three-dimensional data objects corresponding to at least a partial depiction of the subject's eye, predict future geomorphic atrophy lesion area, and predict geomorphic atrophy lesion velocity, the three-dimensional data objects being generated by the one or more data processors from a set of two-dimensional images segmented based on predicted positions of membranes in the subject's eye, the convolutional neural network being trained to simultaneously predict current geomorphic atrophy lesion size and subsequent geomorphic atrophy state, the subsequent geomorphic atrophy state including subsequent geomorphic atrophy lesion growth or subsequent geomorphic atrophy lesion size.

18. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform the operation of the system described in any one of claims 1 to 17.

Images