Detection of geographic atrophy-related biomarkers from medical images using machine learning

Abstract

A processing system automatically detects regions of GA-related biomarkers from medical images. The processing system obtains a volumetric image of an eye comprising a set of image slices. For each of the image slices, the processing system applies a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers. The machine learning model is trained to learn relationships between training image slices from training volumetric images and labels identifying regions of GA-related biomarkers in the training volumetric images. The processing system generates from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers. The processing system furthermore outputs a visual representation of the inference map, such as a two dimensional enface map or a three-dimensional map, to a display device.

Description

DETECTION OF GEOGRAPHIC ATROPHY-RELATED BIOMARKERS FROM MEDICAL IMAGES USING MACHINE LEARNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 642,411 Filed on May 3, 2024 which is incorporated by reference herein.BACKGROUND

[0002] Geographic atrophy (GA) is a chronic progressive degeneration of the macula of the eye, which can lead to permanent loss of visual acuity. Example GA-related biomarkers include complete retinal pigment epithelium and outer retina atrophy (cRORA) and photoreceptor loss (which can be a precursor to cRORA or other types of GA-related biomarkers). These conditions are conventionally identified by medical experts through manual analysis of spectral- domain optical coherence tomography (SD-OCT) images of the eye. However, identification and detection using current methods are time consuming, prone to errors, and subject to disagreement or inconsistency between experts.SUMMARY

[0003] A computer-implemented method automatically detects GA-related biomarkers from medical images. A processing system obtains a volumetric image of an eye comprising a set of image slices. The processing system applies, for each of the image slices, a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers. The machine learning model is trained to learn relationships between training image slices from training volumetric images and labels identifying regions of GA-related biomarkers in the training volumetric images. The processing system generates from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers. The processing system outputs a visual representation of the inference map to a display device.

[0004] In an example, the likelihood values correspond to a three-dimensional array of values. Generating the inference map may include projecting the three-dimensional array of values to a two-dimensional enface map.

[0005] In an example, generating the inference map may comprise quantizing the likelihood values into binary values.

[0006] In an example, the volumetric image comprises a SD-OCT image.

[0007] In an example, the inferred regions of GA-related biomarkers correspond to at least oneof: regions of complete RPE and outer retina atrophy (cRORA) and regions of photoreceptor loss.

[0008] In an example, the machine learning model comprises an instance segmentation model.

[0009] In an example, the machine learning model comprises a mask region-based convolutional neural network (M R-CNN).

[0010] In an example, the machine learning model is trained according to a process comprising obtaining the training image slices, obtaining the labels identifying the regions of GA-related biomarkers in the training volumetric images, and applying a machine learning algorithm to the training image slices and the labels to learn parameters of the machine learning model.

[0011] In an example, applying the machine learning model to generate the likelihood values comprises concurrently identifying first likelihood values characterizing inferred regions complete RPE and outer retina atrophy (cRORA) and second likelihood values characterizing inferred regions of photoreceptor loss.

[0012] In an example, generating the likelihood values comprises generating the likelihood values on a per-pixel basis.

[0013] In further examples, the above-described methods may be implemented using a non- transitory computer-readable storage medium that stores instructions executable by one or more processors. Furthermore, a computer system may include one or more processors and a non- transitory computer-readable storage medium that stores instructions executable by the one or more processors.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure (FIG.) 1 is an example of a computing environment for automatically detecting GA-related biomarkers from medical images.

[0015] FIG. 2 is an example of a processing system for automatically detecting GA-related biomarkers from medical images using machine learning.

[0016] FIG. 3 is an example of a training process for training a machine learning model that can infer regions of GA-related biomarkers from medical images.

[0017] FIG. 4 is an example of an inference process for inferring regions of GA-related biomarkers in a volumetric medical image.

[0018] FIG. 5A is a first example of an inference map representing an inferred region of GA- related biomarkers derived from a two-dimensional image slice.

[0019] FIG. 5B is an example of an enface map representing an inferred region of GA-relatedbiomarkers derived from a three-dimensional volumetric image.

[0020] FIG. 6A illustrates an example series of enface maps representing changes in inferred regions of GA-related biomarkers over time.

[0021] FIG. 6B illustrates an example graph plotting the growth in inferred regions of GA- related biomarkers over time.DETAILED DESCRIPTION

[0022] The Figures (FIGS.) and the following description describe certain embodiments by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. Reference will now be made to several embodiments, examples of which are illustrated in the accompanying figures. Wherever practicable, similar or like reference numbers may be used in the figures and may indicate similar or like functionality.

[0023] A processing system automatically detects regions of GA-related biomarkers from medical images. The processing system obtains a volumetric image of an eye comprising a set of image slices. For each of the image slices, the processing system applies a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers. The machine learning model is trained to learn relationships between training image slices from training volumetric images and labels identifying regions of GA-related biomarkers in the training volumetric images. The processing system generates from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers. The processing system furthermore outputs a visual representation of the inference map (such as a two dimensional enface map or a three-dimensional map) to a display device.

[0024] FIG. 1 illustrates an example of a computing environment 100 for automatically detecting GA-related biomarkers from medical images. The computing environment 100 includes one or more medical imaging devices 102, a processing system 104, one or more client devices 106, and an image store 110 coupled by a network 108. In alternative embodiments, the imaging device 102, processing system 104, client devices 106, and / or image store 110 may be directly coupled to each other and are not necessarily all directly coupled to the network 108. Furthermore, in some embodiments, one or more components may be isolated from the network 108. In such embodiments, data transfers may nevertheless occur via physical storage devices that can be manually transported. For example, images from the medical imaging device 102 could be transported and loaded to the image store 110 (or a client device 106) via a physicalstorage medium such as a disk drive, solid state drive, or other storage medium. In some embodiments, two more components (such as the processing system 104 and image store 110) may be integrated into a single device.

[0025] The medical imaging device 102 captures medical images. In an embodiment, the medical imaging device 102 comprises a SD-OCT device that captures volumetric images of the eye as a set of image slices (which may also be understood as sections or layers) so as to capture the condition and / or morphology of the eye or a part of the eye. For example, in one implementation, each image slice is derived from a “B-scan” of the eye and represents a cross- sectional image slice along nasal-temporal axis (i.e., horizontal). Here, the volumetric image comprises a set of horizontal image slices captured at different vertical positions along the superior-inferior direction. Alternatively, the volumetric image may be derived from a set of “C scans” representing cross-sectional image slices along the superior-inferior axis (i.e., vertical) each focused at different depths. In other embodiments, the image slices may be captured at any arbitrary angle, including any angle relative to any anatomical position. Image slices may also comprise radial sections or other linearly scanned images. Furthermore, the volumetric images may alternatively be captured from other types of OCT or other medical imaging techniques (e.g., swept source OCT, time-domain OCT, radial OCT). While FIG. 1 shows only a single medical imaging device 102, the computing environment 100 may include multiple imaging devices 102 that may be operated by different institutions and may be physically distributed in different medical or research facilities. In some embodiments, the environment 100 may not itself comprise a medical imaging device 102, and may instead collect or receive imaging data from a remote or local data storage.

[0026] At least some subset of the image slices from the medical imaging device(s) 102 may be stored to the image store 110. The image store 110 can furthermore store historic images that may have originated from medical imaging devices 102 that are no longer in operation. The image store 110 may store the images themselves and may store various metadata associated with the images such as information about the patient, type of machine used to capture the images, scan direction, time of capture, etc. The metadata may furthermore include annotations (labels) associated with the images as described further below. The image store 110 may include any local storage (e.g., one or more on-site storage devices at one or more medical facilities) that may be co-located with one or more other components of the computing environment 100, cloud storage that may include remote and / or distributed network-based storage, or a combination thereof.

[0027] In an embodiment, the image store 110 stores images as individual two-dimensionalimages without necessarily preserving the associations between image slices from the same volumetric image capture. Alternatively, the image store 110 may store the image slices in a manner that preserves relationships between image slices within the same captured volume. For example, the image store 110 may store images in a volumetric image format (e.g., a DICOM (Digital Imaging and Communications in Medicine) format or other three-dimensional array of values). Alternatively, the image store 110 may store the images in their original two- dimensional image formats (e.g., as two-dimensional arrays of values) and may additionally store metadata describing relationships between image slices from the same original volumetric image.

[0028] The processing system 104 performs processing of medical images from the image store 110 and / or images that may be received directly from the medical imaging device 102 or client device 106. The processing system 104 may train one or more machine learning models by learning model parameters based on training images that are annotated with labels indicating regions of cRORA, photoreceptor loss, other types of GA-related biomarkers, or generalized GA-related biomarkers (without necessarily specifying the type). Once trained, the machine learning model may be applied to one or more input images to infer regions of the eye that exhibit one or more types of GA-related biomarkers. The processing system 104 may furthermore generate one or more outputs indicative of the inferences such as an enface (also referred to as “en face”) map, a three-dimensional inference map, various analytical metrics, and / or medical recommendations derived therefrom. In an example embodiment, the processing system 104 may track progression of GA-related biomarkers in a patient over time and may generate one or more metrics indicative of the progression.

[0029] The processing system 104 may be implemented using on-site computing or storage systems, cloud computing or storage systems, or a combination thereof and may be implemented utilizing local or cloud-based servers, which may include physical or virtual machines, or a combination thereof. Cloud-based servers may include private cloud systems, public cloud systems, hybrid public / private cloud systems, or a combination thereof. Accordingly, the processing system 104 may be local, remote, and / or distributed relative to the medical environments where the medical images are obtained and where one or more client devices operate. Furthermore, different portions of the processing system 104 may execute on different remote servers and various system elements of the processing system 104 may be communicatively coupled over a network 108.

[0030] The client device 106 may include any computing devices for inputting information and / or control commands, viewing various information, and / or interacting with various data.The client device 106 may execute one or more applications that include a user interface for performing various functions described herein. The application may comprise a web-based application accessible by a web browser or a locally installed application. The application may allow the client device 106 to facilitate functions such as controlling the medical imaging device 102, viewing, adding, removing, editing, or otherwise interacting with images in the image store 110, initiating processes performed by the processing system 104, inputting information to the processing system 104, viewing outputs from the processing system 104 (including various maps, analytical data, or recommendations), etc.

[0031] The client device 106 may comprise, for example, a mobile phone, a tablet, a laptop or desktop computer, or other computing device. The client device 104 may include conventional computer hardware such as a display, input device (e.g., touch screen), memory, a processor, and a non-transitory computer-readable storage medium that stores instructions for execution by the processor in order to carry out functions described herein.

[0032] The network 108 comprises communication pathways for communication between the medical imaging device 102, the processing system 104, and the client device 106. The network 108 may include one or more local area networks and / or one or more wide area networks (including the Internet). The network 108 may also include one or more direct wired or wireless connections (e.g., Ethernet, WiFi, cellular protocols, WiFi direct, Bluetooth, Universal Serial Bus (USB), or other communication link).

[0033] FIG. 2 is a block diagram illustrating an example of a processing system 104. The processing system 104 includes one or more processors 202 and one or more storage mediums 204. The one or more storage mediums 204 includes various functional modules (implemented as instructions executable by the one or more processors 202) including a user interface module 206, an administrative interface module 208, a data ingestion module 210, a training module 212, an inference module 214, a model store 216, and an analytics module 218. In alternative embodiments, the processing system 104 may include different or additional modules. The one or more processors 202 and one or more storage mediums 204 are not necessarily co-located and may be distributed (e.g., in a cloud architecture).

[0034] The user interface module 206 facilitates server-side functions of a user interface accessible on the client devices 106. In an example operation, the user may input various information via a user interface on the client device 106 that is communicated to the user interface module 106, such as medical information (e.g., patient data), control inputs (e.g., to operate the imaging device 102 or request analysis), or commands to request various analytics and / or visualizations.

[0035] The user interface module 206 may furthermore facilitate presentation of various outputs from the processing system 104. For example, the user interface module 206 may output enface maps indicative of regions of predicted GA-related biomarkers. The enface maps may be generalized or may be specific to one or types of GA-related biomarkers such as cRORA or photoreceptor loss. In further embodiments, outputs may comprise various analytics characterizing severity of GA-related biomarkers, progression of GA-related biomarkers, or recommendations relating to a diagnosis, treatment, follow up actions, etc.

[0036] Although the user interface module 206 is illustrated as a component of the backend server 104 in FIG. 2, all or a subset of the functions of the user interface module 206 may instead be executed on the client device 106. For example, the client device 106 may download an application from the processing system 104 that includes all or some of the functions of the user interface module 206. The client device 106 may locally execute instructions associated with these functions.

[0037] The administrative interface module 208 facilitates various administrative functions associated with operation of the processing system 104. For example, the administrative interface module 208 may present an interface that enables configuration of various machine learning parameters associated with the training module 212 and inference module 214. Furthermore, the administrative interface module 208 may provide interfaces for controlling versions and / or access to applications of the client devices 106 or may perform other administrative functions. The administrative interface module 208 may furthermore access, aggregate, and present various statistical data associated with operation of the training module 212 and / or inference module 214 to enable the machine learning models to be evaluated and improved.

[0038] The data ingestion module 210 facilitates ingestion of image data from the image store 110 and / or other data sources and may perform various pre-processing tasks. For example, the data ingestion module 210 may interpret and transform data from a native image format (e.g., as may be output by the imaging device 102) into a standardized format. The standardized format may comprise a standard image size, encoding format, resolution, etc. The data ingestion module 210 may furthermore encode metadata associated with the images into a standard format for use by the training module 212 and inference module 214.

[0039] The training module 212 trains one or more machine learning models based on a set of training images. For example, the training module 212 may apply a supervised machine learning algorithm to the training images to learn a set of model parameters (e.g., weights) for characterizing one or more input images based on their statistical similarities to the trainingimages. In some embodiments, the learning algorithm may comprise an algorithm fortraining a mask region-based convolutional neural network (M R-CNN). In other embodiments, the learning algorithm may train a different type of instance segmentation model (e.g., YOLO, YOLAACT++). In alternative embodiments, the learning process may employ different types of learning algorithms for generating other types of machine learning models such as other types of convolutional neural networks, other neural networks, or other supervised learning models. Further examples of processes applied by the training module 212 are described in further detail below with respect to FIG. 3.

[0040] In an embodiment, the training module 212 may operate directly on a dataset of SD-OCT image slices (or image slices derived from other types of scans) without necessarily performing segmentation to segment the retina from other anatomical features. The learning model can learn to directly identify areas of loss without necessarily first establishing the retinal boundaries. Furthermore, the training module 212 may operate to train the model to concurrently detect two or more different types of GA-related biomarkers (such as cRORA and photoreceptor loss). This approach may offer higher performance than approaches using independent models based on the correlations between different types of GA-related biomarkers.

[0041] The inference module 214 applies one or more trained models to an input volumetric image to generate inferences describing detected areas of GA-related biomarkers. The inferences may comprise pixel-by-pixel likelihoods that each indicate a probability of a pixel being within a region associated with one or more GA-related biomarkers. Likelihoods may furthermore relate to a specific type of GA-related biomarker. For example, the inference module 214 may generate multiple likelihood values per-pixel corresponding to cRORA and photoreceptor loss respectively. The inference module 214 may apply an inference algorithm corresponding to the same type of model(s) generated by the training module 212 (e.g., an instance segmentation model such as an M R-CNN, another type of instance segmentation model, or any other type of suitable model). Further examples of processes applied by the inference module 214 are described in further detail below with respect to FIG. 4.

[0042] Similar to the above, the inference module 214 may apply the machine learning model directly to SD-OCT image slices without necessarily performing explicit segmentation of the images to segment the retina from other anatomical features. Furthermore, the training module 212 may concurrently detect two or more different types of GA-related biomarkers (such as cRORA and photoreceptor loss) using a single trained model.

[0043] The analytics module 218 may generate various analytics associated with the inferences generated from the inference module 214 and / or other related input data (e.g., patient data,medical practitioner data, facility data, medical equipment data, etc.) For example, in an embodiment, the analytics module 218 may generate various metrics characterizing severity of GA-related biomarkers, distributions of different types of GA-related biomarkers, progression of GA-related biomarkers over time, etc. The analytics module 218 may further generate various medical recommendations such as a recommended diagnosis, treatment, follow up plan, qualifications for clinical trial participation, etc. The analytics module 218 may furthermore generate visual representations characterizing the inferences such as enface maps representing regions of GA-related biomarkers, a three-dimensional model indicating areas of GA-related biomarkers, movies showing progression of GA-related biomarkers overtime, and / or various charts, graphs, or other visual representations of the inferences and / or other metrics. The analytical module 218 may employ various rule-based models and / or machine learning models for generating the various types of analytical output described herein.

[0044] The model store 216 stores the one or more machine learning models generated by the training module 212. The model store 216 may be accessed by the inference module 214 to apply one or more models during an inference process. The model store 216 may furthermore store various analytical models that may be applied by the analytics module 218 to generate various analytical data as described herein.

[0045] FIG. 3 illustrates an example of a training process for training a machine learning model capable of inferring regions of GA-related biomarkers from input images. The processing system 104 obtains 302 training image slices. The training image slices may comprise two- dimensional images originating from SD-OCT or other volumetric image captures of eyes from a set of patients. The patients contributing to the training images may include patients exhibiting varying forms of GA-related biomarkers (e.g., cRORA and / or photoreceptor loss) and patients without GA-related biomarkers. Furthermore, the patients may have varying demographics, medical histories, or other characteristics to provide variations in parameters that may affect appearance of GA-related biomarkers in the training images.

[0046] In an embodiment, the training images may each be treated independently in the training process and thus do not necessarily embed information identifying which training images came from the same volumetric capture. In other embodiments, the spatial relationships between the image slices from the same volumetric capture may be preserved in the training dataset. For example, the training images may be encoded in a volumetric image format or metadata associated with the training images may identify the spatial relationships between images from the same volumetric image capture.

[0047] The processing system 104 furthermore obtains 304 labels identifying regions of GA-related biomarkers in the training images. The labels may specify a specific type of GA-related biomarkers such as cRORA and / or photoreceptor loss or may just generally characterize where GA-related biomarkers are present. These labels may be obtained from an expert reviewer that marks or otherwise identifies regions of GA-related biomarkers and / or the types of GA-related biomarkers present. In an embodiment, the labels may be specified on a pixel-by-pixel basis. For example, a two-dimensional label map may be obtained for each training image slice that indicates values on a per-pixel basis specifying whether a corresponding pixel is within a region where cRORA is present, whether a corresponding pixel is within a region where photoreceptor loss is present, or whether the corresponding pixel is in a region where no GA-related biomarkers are present. In alternative embodiments, labels may represent the regions of GA- related biomarkers in a different manner, such as, for example, by identifying pixels corresponding to a perimeter of a region of a GA-related biomarker.

[0048] In further embodiments, labels may identify other image features that may be informative of, but do not necessarily directly identify regions of GA-related biomarkers. For example, labels may identify locations of RPE attenuation and / or disruption, regions of choroidal hypertransmission, regions of discontinuity or disruption to the ellipsoid zone (EZ), or regions with evidence of complete or frank loss of the EZ layer. In these embodiments, the training process may learn correlations between these types of lower level image features and labels indicating the labelers conclusions with respect to regions of cRORA and / or photoreceptor loss, which may allow the resulting machine learning model to better infer the regions of GA-related biomarkers from the input images.

[0049] A learning algorithm is then applied 306 to learn model parameters for mapping between the image data of the image slices in the training set and their corresponding labels. In an example embodiment, the learning algorithm may comprise an algorithm for training an M R- CNN. In other embodiments, the learning algorithm may train a different type of instance segmentation model. In alternative embodiments, the learning process may employ different types of learning algorithms for generating other types of machine learning models such as other types of convolutional neural networks, other neural networks, or other supervised learning models.

[0050] In the above-described example, the learning algorithm operates on two-dimensional input images (and their corresponding labels), which are each treated as independent inputs. In other embodiments, the inputs may furthermore include information characterizing spatial relationships between the input images derived from the same volumetric capture. For example, instead of using two-dimensional input images, a learning algorithm may be employed thatoperates directly on three-dimensional volumetric images. In other embodiments, inputs to the learning algorithm may include metadata associated with two-dimensional input images that describes one or more relationships between images. For example, the input may indicate which images came from the same patient, which images came from the same eye, which images came from the same volumetric scan, and / or the specific spatial relationships between images originating from the same volumetric scan.

[0051] In yet further embodiments, the learning algorithm may operate on a time series of images captured from the same patient at different times (e.g., over a period of weeks, months, or years). Here, the time series may be indicative of the progression (or lack of progression) of GA-related biomarkers. In this example, other types of machine learning models (and associated learning and inference algorithms) may be employed that are configured for generating inferences from a sequence of input images. For example, the machine learning model may comprise a Long Short-Term Memory (LSTM) network that operates on sequential data or a Siamese network that can compare series images and analyze differences or changes.

[0052] In yet further embodiments, the learning algorithm may generate two or more independent machine learning models for characterizing different types of GA-related biomarkers. For example, a first machine learning model may be generated from the training images and labels indicating regions of cRORA, and a second machine learning model may be independently generated from the training images and labels indicating regions of photoreceptor loss. In this case, the first and second machine learning models may be trained to independently infer cRORA and photoreceptor loss respectively.

[0053] The processing system 104 outputs 308 the machine learning model. The machine learning model may be stored to the model store 216 to be accessed during an inference process described below.

[0054] FIG. 4 illustrates an example of an inference process for automatically identifying one or more regions of GA-related biomarkers (and optionally a specific type of GA-related biomarker) from a volumetric input image. An input volumetric image is obtained 402. The input volumetric image may comprise an SD-OCT scan of an eye or other volumetric image. The input volumetric image may be embodied as a set of two-dimensional image slices (e.g., B scans representing horizontal image slices, C scans representing vertical image slices, or other slices at arbitrary image planes).

[0055] The machine learning model is applied 404 to the input volumetric image to generate per- pixel likelihoods indicating a probability of respective pixels corresponding to a region in which a GA-related biomarker is present. In an embodiment, the machine learning model isindependently applied to each image slice. The machine learning model may output likelihood values between 0 and 1 (or other scoring range) where a higher value indicates a higher likelihood of the pixel belonging to a region corresponding to the GA-related biomarker. In an embodiment, the machine learning model may output multiple likelihood values per pixel corresponding to different types of GA-related biomarkers (e.g., a likelihood value per pixel for cRORA and a separate likelihood value per pixel for photoreceptor loss). Alternatively, a single likelihood value per pixel may be generated that characterizes presence of GA-related biomarkers generally.

[0056] The processing system 104 generates 406 an inference map from the per-pixel likelihoods. After processing all image slices from the input volumetric image, the inference map may comprise a three-dimensional map that corresponds to the original volumetric input image on a per-pixel basis. The inference map may be stored as a three-dimensional array of values or as a set of separate two-dimensional arrays corresponding to the same original volumetric image. In an embodiment, the likelihood values may be quantized in the inference map (e.g., to either 0 or 1 to form a binary map, or to another set of discrete values). For example, likelihood values of 0.5 or greater may be mapped to a 1 and likelihood values of less than 0.5 may be mapped to a 0. Alternatively, a different quantization threshold may be used.

[0057] In an embodiment, a set of two-dimensional inference maps derived from the input volumetric image may be projected to a single enface map. Here, a combining function may be applied to combine each set of likelihood values corresponding to the same pixel location. For example, if the image slices each represent an x-y image plane, the enface map may be generated by combining likelihood values in the z-direction. In one embodiment, the combining function may comprise a maximum function. In the case of a set of binary maps, the enface map thus includes a pixel value of 1 when any of the quantized likelihood values for that pixel location in any of the two-dimensional inference maps is 1. In this embodiment, the enface map is a two- dimensional binary map with values of 0 or 1. In other embodiments, a summation function may be applied to sum likelihood values corresponding to the same pixel location in the set of two- dimensional inference map slices that make up the three-dimensional inference map. In this case, higher values may be indicative of the strength of the predictions and / or the extent (e.g., in the z-direction) of the region. In further embodiments, one or more filtering functions may be applied to smooth the results. In yet further embodiments, a different type of combining function may be applied to transform the three-dimensional inference map to a two-dimensional enface map.

[0058] In an embodiment, separate enface maps may be generated in association with differenttypes of GA-related biomarkers inferred by the application of the one or more machine learning models. For example, a first enface map may be generated to represent regions of cRORA and a second enface map may be generated to represent regions of photoreceptor loss. In another embodiment, a single enface map may be generated that characterizes the different types of GA- related biomarkers using different visual indicators (e.g., intensity and / or color). For example, a first visual indicator may indicate regions where cRORA is present and a second visual indicator may represent regions where photoreceptor loss (without cRORA) is present.

[0059] A visual representation of the enface map is outputted 408. Here, the enface map may be outputted as an image, where different pixel values may be represented as different intensities or different colors. In further embodiments, a different visual representation may be outputted. For example, a visual representation of a three-dimensional map may be outputted without necessarily projecting the three-dimensional map to a two-dimensional enface map. Here, the three-dimensional map may be presented, for example, as a rotatable three-dimensional model, as a series of two-dimensional slices, or other three-dimensional representation.

[0060] In further embodiments, the visual representation may include one or more enface maps and / or one or more three-dimensional maps that derived from unquantized likelihood values. In this case, the higher values indicate pixel locations where the machine learning model infers presence of GA-related biomarkers with higher certainty.

[0061] In an embodiment, enface maps (or other representation) may be derived from volumetric images captured from the same patient at different times (e.g., over a period of weeks, months, and / or years). The enface maps may be combined (e.g., in a video) to illustrate progression (or lack thereof) of GA-related biomarkers (and / or specific types of GA-related biomarkers) over time.

[0062] In yet further embodiments, an analytical model may be applied to the likelihood maps and / or derived enface maps to generate one or more analytical outputs. Here, the analytical model may be developed to generate recommendations associated with a diagnosis, treatment, follow-up plan, or other medical analysis. The analytical model may comprise a rule-based model or a machine learning model trained on historical enface maps and / or likelihood maps together with patient data, medical outcomes, clinical decisions, or other data.

[0063] The above-described process relates to application of instance segmentation models the generate inferences based on individual volumetric image captured at a single time. In other embodiments, an inference process may be applied using models that generate predictions from a series of input volumetric images captured for the same patient at different times, thus potentially indicating progression (or lack of progression) of GA-related biomarkers. Here, the processingsystem may generate each enface map (or three-dimensional map) based on both current and historical images captured for the same patient. The processing system 104 may furthermore generate various analytics that may be indicative of progression of GA-related biomarkers (e.g., rate of progression). Progression may furthermore be visualized as a series of enface maps or other visualizations (e.g., as a movie).

[0064] FIG. 5A illustrates an example of an input image slice 510 (e.g., an OCT B-Scan) and a resulting inference map 520 generated from application of the machine learning model. The inference map 520 may be represented in an image that visually distinguishes region(s) 522 corresponding to GA-related biomarkers and regions 524 without GA-related biomarkers. For example, the inference map 520 may include relatively lighter regions (e.g., white) to indicate the region(s) of inferred GA-related biomarkers 522 and relatively darker regions (e.g., black) to indicate region(s) without the inferred GA-related biomarkers 524. Alternatively, the regions 522, 524 may be visually distinguished using color coding, dot or line patterns, labels, or other visual indicators. In further embodiments, the regions 522, 524 may be represented in the inference map 520 by depicting only their respective perimeter lines (e.g., using different line styles, color, or labels).

[0065] FIG. 5B illustrates an example of a set of input image slices 530 (e.g., a set of OCT B- Scans) and a resulting enface map 540 in which the set of likelihood maps are projected to a single two-dimensional map. Similar to FIG. 5A, the enface map may visually distinguish regions 542 corresponding to GA-related biomarkers and regions 544 without GA-related biomarkers using any of the techniques described above.

[0066] FIG. 6A-B illustrate additional examples of visual representations that can be generated by the above-described processing system 104. FIG. 6A shows a series of enface maps 610 representing progression of GA-related biomarkers over time. In the illustrated example, the regions 612 of cRORA are generally most predominate in the central areas while the regions 614 of photoreceptor loss are most predominate in areas adjacent to the perimeters of the cRORA regions 612. GA progression is evident in this example by the growing size of the cRORA region 612 (and corresponding growth of the photoreceptor loss regions 614 in the outermost areas). A color coding or greyscale coding of pixel values may be used in the enface maps 610 to distinguish between the regions 612 of cRORA, regions 614 of photoreceptor loss 614, and regions 616 where neither cRORA or photoreceptor loss are identified. For example, regions 612 cRORA may be represented by a relatively lighter grayscale value, regions of photoreceptor loss 614 may be represented by a middle grayscale value, and regions 616 without GA-related biomarkers may be represented by a relatively darker grayscale value. Alternatively, thedifferent regions 612, 614, 616, may be distinguished using varying fdl patterns, perimeter line styles, labels, or other visual indicators.

[0067] FIG. 6B is an example of a graph 650 that can be automatically generated by the analytical module 218 based on a series of inferred enface maps generated over time for a patient. Here, the area growth (in square millimeters) of cRORA 612 and photoreceptor loss 614 respectively is plotted over time (in days since a baseline). The plotted lines representing the different regions 612, 614 may be similarly greyscale-coded, color-coded, or otherwise visually distinguished.

[0068] The above-described system and methods may be utilized to automatically and rapidly generate an enface map comprising a two-dimensional representation of the retina characterizing observed regions of cRORA, photoreceptor loss, and / or other indicators of GA-related biomarkers. For example, the above-described system 104 may enable a medical practitioner to perform an SD-OCT scan of a patient and very quickly (e.g., in real-time or near real-time) view a resulting enface map or other representation characterizing GA-related biomarkers. The medical practitioner can therefore quickly intervene in a manner that may stop or reduce loss of visual acuity. Additionally, the above-described system enables enface maps or other representations to be tracked for patients over time for accurately monitoring and / or automatically characterizing progression of GA-related biomarkers. This progression-related analysis may further enable optimal diagnosis, treatment, and overall care for the patient. In further embodiments, the generated enface maps (e.g., individual maps or sequences of maps tracked over time) may enable computer systems to automatically generate recommended diagnoses and / or treatments. In yet further embodiment, the above-described system may output three-dimensional models of the eye (instead of or in addition to enface maps) that depict regions of cRORA, photoreceptor loss, or other GA-related biomarkers. The three-dimensional representations may beneficially enable a medical practitioner or an automated analysis system to generate diagnoses and / or recommended treatments that may not be apparent from traditional enface maps alone.

[0069] The machine learning techniques may beneficially be robust to variations in images resulting from different types of image capture devices 110 (which may generate images having differing visual appearances, encodings, or other differences). Furthermore, the machine learning techniques described herein may be robust to differences in the specific standards used to label the training images. For example, if guidelines for identifying GA-related biomarkers change, the same or similar learning techniques may be employed to train a new model (or retrain an existing model) using training images with labels reflecting these updated guidelines.The updated can then be directly deployed using the same general systems and methods described herein.

[0070] The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

[0071] Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

[0072] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible non- transitory computer readable storage medium or any type of media suitable for storing electronic instructions and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may include architectures employing multiple processor designs for increased computing capability.

[0073] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope is not limited by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

CLAIMS1. A computer-implemented method for detecting GA-related biomarkers from medical images, the method comprising: obtaining a volumetric image of an eye comprising a set of image slices; applying, for each of the image slices, a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers, the machine learning model trained to learn relationships between training image slices from training volumetric images and labels identifying regions of GA-related biomarkers in the training volumetric images; generating from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers; and outputting a visual representation of the inference map to a display device.

2. The computer-implemented method of claim 1, wherein the likelihood values correspond to a three-dimensional array of values, and wherein generating the inference map comprises: projecting the three-dimensional array of values to a two-dimensional enface map.

3. The computer-implemented method of claim 1, wherein generating the inference map comprises: quantizing the likelihood values into binary values.

4. The computer-implemented method of claim 1, wherein the volumetric image comprises a spectral domain optical coherent tomography image.

5. The computer-implemented method of claim 1, wherein the inferred regions of GA-related biomarkers correspond to at least one of: regions of complete RPE and outer retina atrophy (cRORA) and regions of photoreceptor loss.

6. The computer-implemented method of claim 1, wherein the machine learning model comprises an instance segmentation model.

7. The computer-implemented method of claim 1, wherein the machine learning model comprises a mask region-based convolutional neural network (M R-CNN).

8. The computer-implemented method of claim 1, wherein the machine learning model is trained according to a process comprising: obtaining the training image slices; obtaining the labels identifying the regions of GA-related biomarkers in the training volumetric images; andapplying a machine learning algorithm to the training image slices and the labels to learn parameters of the machine learning model.

9. The computer-implemented method of claim 1, wherein applying the machine learning model to generate the likelihood values comprises: concurrently identifying first likelihood values characterizing inferred regions complete RPE and outer retina atrophy (cRORA) and second likelihood values characterizing inferred regions of photoreceptor loss.

10. The computer-implemented method of claim 1, wherein the generating the likelihood values comprises generating the likelihood values on a per-pixel basis.

11. A non -transitory computer-readable storage medium storing instructions for detecting GA- related biomarkers from medical images, the instructions when executed by one or more processors causing the one or more processors to perform steps including: obtaining a volumetric image of an eye comprising a set of image slices; applying, for each of the image slices, a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers, the machine learning model trained to learn relationships between training image slices from training volumetric images and labels identifying the regions of the GA-related biomarkers in the training volumetric images; generating from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers; and outputting a visual representation of the inference map to a display device.

12. The non-transitory computer-readable storage medium of claim 11, wherein the likelihood values correspond to a three-dimensional array of values, and wherein generating the inference map comprises: projecting the three-dimensional array of values to a two-dimensional enface map.

13. The non-transitory computer-readable storage medium of claim 11, wherein generating the inference map comprises: quantizing the likelihood values into binary values.

14. The non-transitory computer-readable storage medium of claim 11, wherein the volumetric image comprises a spectral domain optical coherent tomography image.

15. The non-transitory computer-readable storage medium of claim 11, wherein the regions ofGA-related biomarkers correspond to at least one of: regions of complete RPE and outer retina atrophy (cRORA) and regions of photoreceptor loss.

16. The non-transitory computer-readable storage medium of claim 11, wherein the machine learning model comprises an instance segmentation model.

17. The non-transitory computer-readable storage medium of claim 11, wherein the machine learning model comprises a mask region-based convolutional neural network (M R- CNN).

18. The non-transitory computer-readable storage medium of claim 11, wherein the machine learning model is trained according to a process comprising: obtaining the training image slices; obtaining the labels identifying the regions of GA-related biomarkers in the training volumetric images; and applying a machine learning algorithm to the training image slices and the labels to learn parameters of the machine learning model.

19. The non-transitory computer-readable storage medium of claim 11, wherein applying the machine learning model to generate the likelihood values comprises: concurrently identifying first likelihood values characterizing inferred regions complete RPE and outer retina atrophy (cRORA) and second likelihood values characterizing inferred regions of photoreceptor loss.

20. A computing system comprising: one or more processors; and a non-transitory computer-readable storage medium storing instructions for detecting GA-related biomarkers from medical images, the instructions when executed by the one or more processors causing the one or more processors to perform steps including: obtaining a volumetric image of an eye comprising a set of image slices; applying, for each of the image slices, a machine learning model to generate likelihood values characterizing inferred regions of GA-related biomarkers, the machine learning model trained to learn relationships between training image slices from training volumetric images and labels identifying the regions of the GA-related biomarkers in the training volumetric images; generating from the likelihood values, an inference map that spatially maps the inferred regions of the GA-related biomarkers; and outputting a visual representation of the inference map to a display device.

Images