Machine learning to diagnose and localize nascent geographic atrophy in age-related macular degeneration

Abstract

A method and system for detecting nascent geographic atrophy includes receiving an optical coherence tomography (OCT) volumetric image of a subject's retina. Using a deep learning model, an output indicating whether nascent geographic atrophy is detected is generated using the OCT volumetric image. Using a saliency mapping algorithm, a map output of the deep learning model is generated, the map output indicating the contribution of a set of regions within the OCT volumetric image to the output generated by the deep learning model.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to and claims the benefit of the priority dates of U.S. Provisional Application No. 63 / 339,333, entitled "Machine Learning Enabled Diagnosis and Lesion Localization for Nascent Geographic Atrophy in Age-Related Macular Degeneration," filed on March 24, 2023, and U.S. Provisional Application No. 63 / 484,150, entitled "Machine Learning Enabled Diagnosis and Lesion Localization for Nascent Geographic Atrophy in Age-Related Macular Degeneration," filed on February 9, 2023, and is a continuation-in-part of International Application No. PCT / US22 / 47944, entitled "Methods and Systems for Biomarker Identification and Discovery," filed on October 26, 2022, each of which is incorporated by reference in its entirety into this specification.

[0002]

[0002] The subject matter described herein relates generally to machine learning, and more specifically to machine learning-based diagnosis and lesion localization techniques for newborn geographic atrophy (nGA) in age-related macular degeneration (AMD). [Background technology]

[0003]

[0003] Various imaging techniques have been developed to capture medical images of tissues, which can be analyzed to determine the presence or progression of disease. For example, optical coherence tomography (OCT) refers to a technique that uses light waves to capture two-dimensional slice images and three-dimensional volume images of tissues, such as a patient's retina, which can then be analyzed to diagnose, monitor, treat, etc., the patient. However, analysis of such images, which may contain large amounts of data, is performed manually, usually by subject matter experts, and thus can be cumbersome and very expensive. Therefore, it may be desirable to have methods and systems that facilitate consistent, accurate, and rapid analysis of large volumes of medical images, such as OCT images, for use in diagnosing, monitoring, and treating patients. Summary of the Invention

[0004]

[0004] The following summarizes some embodiments of the present disclosure to provide a basic understanding of the discussed technology. This summary is not an extensive overview of all contemplated features of the present disclosure, and is not intended to identify key or critical elements of all embodiments of the present disclosure, nor to delineate the scope of any or all embodiments of the present disclosure. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in summary form as a prelude to the more detailed description that is presented later.

[0005]

[0005] In one or more embodiments, the system includes at least one data processor and at least one memory storing instructions that, when executed by the at least one data processor, perform operations including applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on the patient's optical coherence tomography (OCT) volume, determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with the diagnosis of nascent geographic atrophy, and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0006]

[0006] In one or more embodiments, a computer-implemented method is provided that includes applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient, determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with the diagnosis of nascent geographic atrophy, and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0007]

[0007] In one or more embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one data processor, perform operations including applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on the patient's optical coherence tomography (OCT) volume, determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with a diagnosis of nascent geographic atrophy, and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0008] In one or more embodiments, a method is provided that includes receiving an optical coherence tomography (OCT) volumetric image of a subject's retina, generating, via a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image, and generating a map output of the deep learning model using a saliency mapping algorithm, the map output indicating a contribution of a set of regions in the OCT volumetric image to an output generated by the deep learning model.

[0009]

[0009] In one or more embodiments, the system includes a non-transient memory and a hardware processor coupled to the non-transient memory, and the hardware processor is configured to read instructions from the non-transient memory to cause the system to receive an optical coherence tomography (OCT) volumetric image of the subject's retina, generate, via a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image, generate a map output of the deep learning model using a saliency mapping algorithm, the map output indicating the contribution of a set of regions within the OCT volumetric image to the output generated by the deep learning model, and display the map output.

[0010]

[0010] In one or more embodiments, a system includes a non-transitory memory and a hardware processor coupled to the non-transitory memory, the hardware processor reading instructions from the non-transitory memory to cause the system to: train a deep learning model using a training dataset including training OCT images labeled as being evidence of nascent geographic atrophy or not being evidence of nascent geographic atrophy to form a trained deep learning model; receive optical coherence tomography (OCT) volumetric images of the subject's retina; and generate, via the trained deep learning model, a classification score using the OCT volumetric images, the classification score indicating whether nascent geographic atrophy has been detected. the deep learning model is configured to generate a classification score; generate a saliency volume map of the OCT volume image using a saliency mapping algorithm, the saliency volume map indicating the contribution of a set of regions in the OCT volume image to a diagnosis of geographic atrophy generated by the deep learning model; detect a set of potential biomarker regions in the OCT volume image using the saliency volume map; and generate a report confirming that nascent geographic atrophy has been detected when at least one potential biomarker region of the set of potential biomarker regions meets a set of criteria and the classification score meets a threshold.

[0011]

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed in this specification and, together with the description, help to explain some of the principles associated with the disclosed embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of a networked system 100 in accordance with one or more exemplary embodiments. [Diagram 2]1 is a flowchart of a process for processing an OCT volumetric image of a subject's retina to determine whether the OCT volumetric image is evidence of a selected retinal health category, in accordance with one or more exemplary embodiments. [Diagram 3] 1 is a flowchart of a process for identifying biomarkers in an OCT volumetric image of a subject's retina, in accordance with one or more exemplary embodiments. [Figure 4A] 4 is a flowchart of a process 400 for artificial intelligence assisted detection of newborn geographic atrophy (nGA), in accordance with one or more exemplary embodiments. [Figure 4B] 1 is a flowchart of a process for processing an OCT volumetric image of a subject's retina to determine whether the OCT volumetric image is evidence of newborn geographic atrophy (nGA), in accordance with one or more exemplary embodiments. [Diagram 5] 1 illustrates an annotated OCT slice image and a corresponding heat map of the annotated OCT slice image, according to one or more exemplary embodiments. [Figure 6] 1 is an illustration of a different map in accordance with one or more exemplary embodiments. [Figure 7] FIG. 1 is a system diagram illustrating an example of a neonatal geographic atrophy detection system according to some exemplary embodiments. [Figure 8A] FIG. 2 illustrates an example of a model for processing a 3D OCT volume, in accordance with one or more exemplary embodiments. [Figure 8B] 8 illustrates an example of an implementation of a classifier 802 that can be used to implement a classifier in accordance with one or more exemplary embodiments. [Figure 9] FIG. 1 illustrates an exemplary data flow diagram including data partitioning statistics in accordance with one or more exemplary embodiments. [Figure 10] FIG. 1 is a diagram of an output workflow for output generated from an OCT volume, in accordance with one or more exemplary embodiments. [Figure 11A] FIG. 11 is a diagram of a confusion matrix 1100 in accordance with one or more exemplary embodiments. [Figure 11B]1 is a graph of 5-fold cross-validation statistics in accordance with one or more exemplary embodiments. [Figure 12A] FIG. 12 is an illustration of an OCT image 1200 (eg, a B-scan) in which an nGA lesion is detected, according to one or more exemplary embodiments. [Figure 12B] 12 is a graph 1202 of a precision-recall curve for 5-fold cross-validation, in accordance with one or more exemplary embodiments. [Figure 12C] FIG. 12 is a diagram of a confusion matrix 1204 in accordance with one or more exemplary embodiments. [Figure 13] 13 is an illustration of an OCT image 1300 annotated with bounding boxes, in accordance with one or more exemplary embodiments. [Figure 14] 1 illustrates an example neural network that can be used to implement a computer-based model, according to various embodiments of the present disclosure. [Figure 15] FIG. 1 is a block diagram illustrating an example of a computing system in accordance with some illustrative embodiments.

[0013]

[0032] Wherever practical, like reference numerals refer to like structures, features or elements.

[0014]

[0033] It should be understood that the figures are not necessarily drawn to scale, and that objects in the figures are not necessarily drawn to scale relative to each other. The figures are intended to provide clarity and understanding to the various embodiments of the apparatus, systems, and methods disclosed herein. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or similar parts. Furthermore, it should be understood that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] I. Overview

[0034] Medical imaging technology is a powerful tool that can be used to generate medical images that allow medical practitioners to better visualize and understand a patient's medical problems, thus providing the same more accurate diagnostic and treatment options. For example, optical coherence tomography (OCT) is a non-invasive imaging technique that is particularly often used to capture images of the retina. OCT can be described as an ultrasound scanning technique that scatters light waves from tissue to generate OCT images in the form of two-dimensional (2D) and / or three-dimensional (3D) images of the tissue, similar to an ultrasound scan that uses sound waves to scan the tissue. 2D OCT images may also be referred to as OCT slices, OCT cross-sectional images, or OCT scans (e.g., OCT B-scans). 3D OCT images may also be referred to as OCT volume images and may be composed of many OCT slice images. The OCT images may then be used to diagnose, monitor, and / or treat the patient from whom the images were obtained. For example, OCT slice images and OCT volume images of the retina of an age-related macular degeneration (AMD) patient may be analyzed to provide AMD diagnosis and treatment options to the patient.

[0016]

[0035] While retinal OCT images may contain valuable information regarding a patient's ophthalmic condition, extracting information from OCT images is a resource-intensive and challenging task that may lead to drawing erroneous conclusions about the information contained in the OCT images. For example, when treating a patient with an ocular disease such as AMD, a large set of OCT slices of the patient's retina may be acquired and a set of trained human reviewers may be tasked with manually identifying biomarkers for AMD within the set of OCT slices. However, such a process may be cumbersome and difficult, resulting in time-consuming, inaccurate, and / or variability in identifying retinal disease biomarkers. Although trained subject matter experts who review OCT images may be used to improve the accuracy of biomarker identification, this process may still be cumbersome, may result in inherent undesirable variability between reviewers, and may be particularly costly. Thus, relying on such subject matter experts to review such large sets of OCT slices may not provide healthcare providers with the efficient, cost-effective, consistent, and accurate mechanism desired for identifying biomarkers for diseases such as AMD. Furthermore, manual review of OCT images may even be less successful in discovering new biomarkers that predict the future development of retinal disease.

[0017]

[0036] For example, geographic atrophy (GA) can be a sight-threatening late AMD complication. Color fundus photography (CFP) or fundus autofluorescence (FAF) can be used to identify GA, but by the time a subject matter expert can see evidence of GA on these types of images, there may already be significant loss of outer retinal tissue. To determine how to delay or prevent the onset of GA at the early stages of AMD, it may be beneficial to identify early signs or predictors of GA onset. For example, biomarkers for early identification and / or prediction of GA onset could be used to identify high-risk individuals to enrich clinical trial populations, serve as biomarkers for various stages of AMD progression, and / or serve as early endpoints in clinical trials aiming to prevent the onset of GA.

[0018]

[0037] OCT images have been used to identify nascent geographic atrophy (neoplastic GA or nGA), which may be a strong predictor of impending (e.g., within 6-30 months) onset of GA. Identifying optical coherence tomography (OCT) signs of nascent geographic atrophy (nGA) associated with the onset of geographic atrophy can enrich study inclusion criteria. For example, in some cases, retinas showing nascent GA in OCT images have a greater than 70-fold risk of developing GA compared to retinas not showing nascent GA. Thus, nascent GA may be a prognostic indicator of progression from early AMD to GA. Examples of anatomical biomarkers that define nascent GA in OCT images include, but are not limited to, subsidence of the inner nuclear layer (INL) and outer plexiform layer (OPL), hyporeflective wedge-shaped bands within the Henle fiber layer, or both.

[0019]

[0038] Accurate identification of nascent GA may improve the feasibility of evaluating preventive treatments for the development of GA. Manual grading of all B-scans in an optical coherence tomography volume can be a laborious and operationally costly task, especially as B-scan density increases to improve macular coverage. Increasing the speed of processing and reducing computational costs will improve the utility of optical coherence tomography biomarkers or endpoints.

[0020]

[0039] Thus, the embodiments described herein provide an artificial intelligence (AI)-based system and method for quickly, efficiently, and accurately detecting whether an OCT volumetric image of a retina is evidence of a selected health category for the retina. The selected health category may be, for example, a retinal disease (e.g., AMD) or a stage of a retinal disease. In one or more embodiments, the selected health category is nascent GA. In other embodiments, the selected health category may be another stage of AMD progression (e.g., early AMD, intermediate AMD, GA, etc.). A deep learning model may be trained to receive the OCT volumetric image and generate a health indication output that indicates whether the OCT volumetric image is evidence of a selected health category for the retina (e.g., nascent GA). For example, the health indication output may indicate a level of association between the OCT volumetric image and the selected health category. The level of association may be no association, some association, or complete association. The deep learning model may include, for example, a neural network model. As one non-limiting example, the deep learning model may generate a health indication output that is a probability (e.g., between 0.00 and 1.00) indicating the level of association between the OCT volumetric image and a selected health condition category.

[0021]

[0040] Additionally, the systems and methods described herein may be used to rapidly, efficiently, and accurately identify biomarkers for retinal disease and / or prognostic biomarkers for future retinal disease development. For example, the systems and methods described herein may be used to identify a set of biomarkers within an OCT volumetric image that are indicative of or otherwise correspond to a selected health state category. The systems and methods may also be used to identify a set of prognostic biomarkers within an OCT volumetric image that are predictive of a selected health state category (e.g., progression to a selected health state category within a selected time period).

[0022]

[0041] In one or more embodiments, a health condition identification system including a deep learning model is used to process the OCT volumetric images. The health identification system uses the deep learning model, which may include a neural network model, to generate a health indication output indicating whether the OCT volumetric images are evidence of a selected health condition category. In some cases, the selected health condition category may be one of a group of health condition categories of interest. In one or more embodiments, the selected health condition category is a selected stage of AMD. The selected stage of AMD may be, for example, nascent GA.

[0023]

[0042] In one or more embodiments, the health condition identification system uses a saliency mapping algorithm (also referred to as a saliency mapping technique) to generate a map output of the deep learning model that indicates whether a set of regions in the OCT volumetric image is associated with a selected health condition category. The saliency mapping algorithm may be used to identify the contribution (or importance) of various parts of the OCT volumetric image to the health indication output generated by the deep learning model for a given OCT volumetric image. The health condition identification system may use the map output to identify biomarkers in the OCT volumetric image. The biomarkers may indicate that the OCT volumetric image is currently probative of a selected health condition category for the retina. In some cases, the biomarkers may be predictive in that they indicate that the OCT volumetric image is predictive of the progression of the retina to a selected health condition category within a selected period of time (e.g., 6 months, 1 year, 2 years, 3 years, etc.).

[0024]

[0043] The saliency mapping algorithm described above may be implemented in a variety of ways. One example of a saliency mapping algorithm is Gradient Weighted Class Activation Mapping (Grad-CAM), a technique that provides a "visual explanation" in the form of a heat map for the decisions made by a deep learning model in performing predictions. That is, Grad-CAM may be implemented such that a trained deep learning model generates a saliency map or heat map of an OCT slice image, which indicates regions or locations of the OCT slice image (e.g., using color, contours, annotations, etc.) that the neural network model uses in making a decision and / or prediction regarding the stage of retinal disease shown in the OCT slice image. In one or more embodiments, Grad-CAM may determine the importance of each pixel in the OCT slice image to the health indication output generated by the deep learning model. Additional details regarding Grad-CAM can be found in RRSelvaraju et al., "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization," Arxiv:1610.02391 (2017), which is incorporated by reference in its entirety. Other non-limiting examples of saliency mapping techniques include Class Activation Mapping (CAM), SmoothGrad, VarGrad (Low-Variance Gradient Estimator for Variational Inference), etc.

[0025]

[0044] The saliency map generated by the saliency mapping algorithm may then be used to identify the location of one or more potential biomarkers on a given OCT slice image. For example, the saliency map may be used to generate a bounding box around each potential biomarker or potential biomarker region in the OCT slice image. Each bounding box may thus identify the location of a potential biomarker. In one or more embodiments, a scoring metric (e.g., a confidence score) may be used to determine which bounding boxes are or contain one or more biomarkers for a selected health condition category.

[0026]

[0045] Using a health status identification system having a deep learning model and a saliency mapping algorithm to classify retinal health status in OCT volume images and identify biomarkers for selected health status categories may reduce the time and cost associated with evaluating a subject's retina and improve the efficiency and accuracy with which diagnosis, monitoring, and / or treatment can be performed. Furthermore, the use of the embodiments described herein may allow subjects to be included in clinical trials at an earlier stage of AMD progression, potentially improving the benefit of such clinical trials. Furthermore, the use of the embodiments described herein may reduce the overall computing resources used and / or speed up computer performance with respect to classifying retinal health status, predicting future retinal health status, and / or identifying biomarkers for selected health status categories.

[0027]

[0046] In some exemplary embodiments, a deep learning model may be trained to detect nascent geographic atrophy based on optical coherence tomography images. The deep learning model may be trained based on information about the presence or absence of nascent geographic atrophy at the eye level to effectively localize these lesions. The ability to identify nascent geographic atrophy may be important when implementing such diagnostic tools in clinical trials, diagnosis, treatment, monitoring, research, etc. In some cases, the diagnostic output of the deep learning model may be subject to further validation or justification in the clinical setting. For example, instead of and / or in addition to presenting a diagnostic result, the deep learning model may suggest one or more areas that are likely to contain nascent geographic atrophy lesions. Thus, the clinician may make a final diagnosis by examining only a subset of B-scans or only areas of B-scans. The deep learning model in this case should exhibit a high recall in localizing nascent geographic atrophy lesions. In order to reduce the clinician's workload as much as possible, its precision rate should be much higher than the prevalence rate.

[0028] II. Examples of Health Status Identification II.A. Examples of Health Status Identification Systems

[0047] FIG. 1 is a block diagram of a networked system 100 according to one or more exemplary embodiments. The networked system 100 may include any number or combination of servers and / or software components that operate to perform various processes related to capturing OCT volumetric images of tissues such as the retina, processing the OCT volumetric images through deep learning models, processing the OCT volumetric images using a saliency mapping algorithm, identifying biomarkers indicative of current retinal health or predictive of retinal health, or combinations thereof. Exemplary servers may include standalone and enterprise-class servers operating a server OS, such as, for example, a MICROSOFT™ OS, a UNIX™ OS, a LINUX™ OS, or other suitable server-based OS. It may be understood that the servers used in the networked system 100 may be deployed in other manners, and that operations performed and / or services provided by such servers may be combined or separated for a given implementation, and may be performed by more or fewer servers. One or more servers may be operated and / or maintained by the same or different entities.

[0029]

[0048] The networked system 100 includes a health condition identification (HSI) system 101. The health condition identification system 101 can be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, the health condition identification system 101 can include a computing platform 102, a data storage 104 (e.g., a database, a server, a storage module, cloud storage, etc.), and a display system 106. The computing platform 102 can take a variety of forms. In one or more embodiments, the computing platform 102 includes a single computer (or computer system) or multiple computers in communication with each other. In other examples, the computing platform 102 takes the form of a cloud computing platform, a mobile computing platform (e.g., a smartphone, a tablet, etc.), or a combination thereof.

[0030]

[0049] The data storage 104 and the display system 106 each communicate with the computing platform 102. In some examples, the data storage 104, the display system 106, or both may be considered part of the computing platform 102 or may be otherwise integrated with the computing platform 102. Thus, in some examples, the computing platform 102, the data storage 104, and the display system 106 may be separate components that communicate with each other, while in other examples some combination of these components may be integrated together.

[0031]

[0050] The networked system 100 may further include an OCT imaging system 110, which may also be referred to as an OCT scanner. The OCT imaging system 110 may generate OCT imaging data 112. The OCT imaging data 112 may include an OCT volumetric image (i.e., a 3D OCT image) and / or an OCT slice image (i.e., a 2D OCT image). For example, the OCT imaging data 112 may include an OCT volumetric image 114. The OCT volumetric image 114 may be composed of multiple (e.g., 10, 100, 1000, etc.) OCT slice images. The OCT slice images may also be referred to as OCT B-scans or cross-sectional OCT images.

[0032]

[0051] In one or more embodiments, the OCT imaging system 110 includes an optical coherence tomography (OCT) system (e.g., an OCT scanner or machine) configured to generate OCT imaging data 112 of a patient's tissue. For example, the OCT imaging system 110 may be used to generate OCT imaging data 112 of a patient's retina. In some cases, the OCT system may be a large tabletop configuration used in a clinical setting, a portable or handheld dedicated system, or a "smart" OCT system integrated into a user's personal device such as a smartphone. The OCT imaging system 110 may include an image denoiser configured to remove noise and other artifacts from the raw OCT volumetric image to generate the OCT volumetric image 114.

[0033]

[0052] The health condition identification system 101 may communicate with the OCT imaging system 110 via a network 120. The network 120 may be implemented using a single network or a combination of multiple networks. The network 120 may be implemented using any number of wired, wireless, or optical communication links, or combinations thereof. For example, in various embodiments, the network 120 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. In another example, the network 120 may comprise a wireless telecommunications network (e.g., a cellular network) adapted to communicate with other communication networks, such as the Internet.

[0034]

[0053] The OCT imaging system 110 and the health condition identification system 101 may each include one or more electronic processors, electronic memory, and other suitable electronic components for executing instructions, such as program code and / or data stored on one or more computer-readable media, to perform the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as internal and / or external memory or data storage devices (e.g., data storage 104) of the various components of the networked system 100 and / or may be accessible via the network 120. Although only one of each of the OCT imaging system 110 and the health condition identification system 101 is shown, in other embodiments there may be more than one of each.

[0035]

[0054] In some embodiments, the OCT imaging system 110 may be maintained by an entity tasked with obtaining OCT imaging data 112 of a subject's tissue sample for purposes of diagnosis, monitoring, treatment, research, clinical trials, etc. For example, the entity may be a medical provider (e.g., an eye care provider) seeking to obtain OCT imaging data of a patient's retina for use in diagnosing an eye condition or disease (e.g., AMD) that the patient may have. As another example, the entity may be an administrator of a clinical trial tasked with collecting OCT imaging data of a subject's retina to monitor changes in the retina as a result of progression / regression of a disease affecting the retina and / or the effects of a drug administered to the subject to treat the disease. It should be noted that the above examples are non-limiting and the OCT imaging system 110 may be maintained by other entities and / or professionals who may use the OCT imaging system 110 to obtain OCT imaging data of the retina for the above-mentioned or any other medical purposes.

[0036]

[0055] In some embodiments, the health condition identification system 101 may be maintained by an entity tasked with identifying or discovering biomarkers for tissue diseases or conditions from OCT images of the tissue. For example, the health condition identification system 101 may be maintained by an eye care provider, a researcher, a clinical trial administrator, etc. tasked with identifying or discovering biomarkers for retinal diseases such as AMD. Although FIG. 1 illustrates the OCT imaging system 110 and the health condition identification system 101 as two separate components, in some embodiments, the OCT imaging system 110 and the health condition identification system 101 may be part of the same system or module (e.g., may be maintained by the same entity, such as a health care provider or a clinical trial administrator).

[0037]

[0056] The health condition identification system 101 may include an image processor 130 configured to receive the OCT imaging data 112 from the OCT imaging system 110. The image processor 130 may be implemented using hardware, firmware, software, or a combination thereof. In one or more embodiments, the image processor 130 may be implemented within the computing platform 102.

[0038]

[0057] The image processor 130 may include a model 132 (which may also be referred to as a health state model 132), a saliency mapping algorithm 134, and an output generator 136. The model 132 may include a machine learning model. For example, the model 132 may include a deep learning model. In one or more embodiments, the deep learning model includes a neural network model that includes one or more neural networks. The model 132 may be used to identify (or classify) the current and / or future health state of a subject's retina.

[0039]

[0058] For example, the model 132 may receive the OCT imaging data 112 as an input. In particular, the model 132 may receive an OCT volumetric image 114 of the subject's retina. The model 132 may process the OCT volumetric image 114 by processing at least a portion of the OCT slice images that make up the OCT volumetric image 114. In some embodiments, the model 132 processes all of the OCT slice images that make up the OCT volumetric image 114. The model 132 generates a health indication output 138 based on the OCT volumetric image 114, which indicates whether the OCT volumetric image 114 is evidence of a selected health category 140 for the subject's retina.

[0040]

[0059] For example, the health indication output 138 may indicate a level of association between the OCT volumetric image 114 and the selected health condition category 140. This level of association may be indicated via a probability. For example, in one or more embodiments, the health indication output 138 may be a level of association between the OCT volumetric image 114 and the selected health condition category 140, or a probability indicating how likely it is that the OCT volumetric image 114 is evidence of the selected health condition category 140. This level of association may be, for example, no association (e.g., a probability of 0.0), a weak association (e.g., a probability of 0.01 to 0.4), a medium association (e.g., a probability of 0.4 to 0.6), a strong association (e.g., a probability of 0.6 to 1.0), or some other type of association. These percentages are just some examples of probability ranges and levels of association. In other embodiments, other levels of association and / or other percentage ranges may be used. The process by which the model 132 generates the health indication output 138 is described in more detail with respect to FIG. 2 below.

[0041]

[0060] The selected health category 140 may be a retinal health state that refers to a current time or a future time (e.g., 6 months, 1 year, 2 years, etc. in the future). In other words, the selected health category 140 may represent a current health state or a future health state. The current time may be, for example, the time when the OCT volumetric image 114 was generated within a selected interval (e.g., 1 week, 2 weeks, 1 month, 2 months, etc.) of the time when the OCT volumetric image 114 was generated.

[0042]

[0061] In one or more embodiments, the selected health category 140 may be a selected stage of AMD. The selected health category 140 may be, for example, but not limited to, current nascent GA or future nascent GA. In some examples, the selected health category 140 represents a stage of AMD predicted to lead to nascent GA within a selected time period (e.g., 6 months, 1 year, 2 years, etc.). In other examples, the selected health category 140 represents a stage of AMD predicted to lead to the development of GA within a selected time period. In yet other examples, the selected health category 140 represents a stage of AMD predicted to lead to nascent GA within a selected time period. In this manner, the selected health category 140 may be of a current health state of the retina or may be of a prediction of a future health state of the retina. Other examples of health categories include, but are not limited to, early AMD, intermediate AMD, GA, etc.

[0043]

[0062] As described above, the model 132 may be implemented using a neural network model. The neural network model may include any number or combination of neural networks. The neural network may take the form of, but is not limited to, a convolutional neural network (CNN) (e.g., U-Net), a fully convolutional network (FCN), a stacked FCN, a stacked FCN with multi-channel learning, a feed-forward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural-ODE), a squeeze and excitation embedding neural network, MobileNet, or another type of neural network. In one or more embodiments, the neural network may itself be composed of at least one of a CNN (e.g., U-Net), an FCN, a stacked FCN, a stacked FCN with multi-channel learning, an FNN, an RNN, an MNN, a ResNet, a neural-ODE, a squeeze and excitation embedding neural network, MobileNet, or another type of neural network. In one or more embodiments, the neural network model takes the form of a convolutional neural network (CNN) system that includes one or more convolutional neural networks. For example, a CNN may include multiple neural networks, each of which may itself be a convolutional neural network.

[0044]

[0063] In one or more embodiments, the neural network model may include a set of encoders, each of which may be a single encoder or multiple encoders, and a decoder. In some embodiments, the one or more encoders and / or decoders may be implemented via a neural network, and the neural network may be composed of one or more neural networks. In some cases, the decoder and one or more encoders may be implemented using a CNN. The decoder and one or more encoders may be implemented as a Y-Net (Y-shaped neural network system) or a U-Net (U-shaped neural network system). Further details regarding neural networks are provided below with reference to FIG. 6.

[0045]

[0064] The health condition identification system 101 may also be used to identify (or detect) a set of biomarkers 142 for a selected health condition category 140. For example, the health condition identification system 101 may be used to identify a set of biomarkers 142 in the OCT volumetric image 114 that is probative of a selected health condition category 140 for the subject's retina. For example, if the selected health condition category 140 is currently nascent GA, the set of biomarkers 142 may include one or more anatomical biomarkers that indicate that the OCT volumetric image 114 is currently probative of the selected health condition category 140 of the retina. If the selected health condition category 140 represents a future health condition (e.g., predicted to progress to nascent GA within a selected time period), the set of biomarkers 142 may be prognostic of this future health condition.

[0046]

[0065] The health condition identification system 101 uses a saliency mapping algorithm 134 to identify the set of biomarkers 142. For example, the saliency mapping algorithm 134 may be used to identify the portions (or regions) of the OCT volumetric image 114 that most influenced or contributed most to the health indication output 138 of the model 132. For example, the saliency mapping algorithm 134 may indicate the importance of various portions (or regions) of the OCT volumetric image 114 for a selected health condition category 140.

[0047]

[0066] The saliency mapping algorithm 134 may include, but is not limited to, Grad-CAM, CAM, SmoothGrad, VarGrad, another type of saliency mapping algorithm or technique, or a combination thereof. The saliency mapping algorithm 134 may generate a saliency volume map 144 that indicates (e.g., via a heat map) the importance of various portions (or regions) of the OCT volumetric image 114 with respect to the selected health condition category 140. In other words, the saliency volume map 144 indicates the contribution of various portions of the OCT volumetric image 114 to the health display output 138 generated by the model 132. The saliency volume map 144 may be composed of multiple saliency maps, each of which corresponds to a different OCT slice image of the multiple OCT slice images in the OCT volumetric image 114. Each saliency map may visually indicate (e.g., via color, highlighting, shading, pattern, outline, text, annotation, etc.) the region of the corresponding OCT slice image that most influenced the model 132 with respect to the selected health condition category 140.

[0048]

[0067] The output generator 136 may receive and process the saliency volume map 144 to generate a map output 146. In one or more embodiments, the map output 146 is a filtered or modified form of the saliency volume map 144. In other embodiments, the map output 146 is the saliency volume map 144 overlaid on the OCT volumetric image 114 or a modified form of the saliency volume map 144. Just as the saliency volume map 144 may be composed of multiple saliency maps (2-dimensional), the map output 146 may be composed of multiple individual 2-dimensional maps. These maps may be heat maps or overlays of heat maps on the OCT slice images.

[0049]

[0068] In one or more embodiments, a filter (e.g., a threshold filter) may be applied to the saliency volume map 144 to identify a subset of saliency maps in the saliency volume map 144 to be modified. The threshold filter may be set, for example, to ensure that only those saliency maps that show a contribution of at least one region in the corresponding OCT slice image above a selected threshold are selected for the subset. This subset of saliency maps may then be modified such that the modified saliency volume map that is formed includes fewer maps than the saliency volume map 144. In this manner, when the map output 146 is generated, the map output 146 may be comprised of fewer maps than the saliency volume map 144. In other embodiments, other types of filtering steps and / or other pre-processing steps may be performed such that the generated map output 146 includes a smaller number of maps than the maps in the saliency volume map 144.

[0050]

[0069] The map output 146 may indicate whether a set of regions in the OCT volumetric image 114 is associated with a selected health condition category. For example, the map output 146 may indicate the contribution of a set of regions in the OCT volumetric image 114 to the health indication output 138 generated by the model 132. The regions may be pixel-level regions or regions formed by multiple pixels. The regions may be continuous or discontinuous regions. In some embodiments, the map output 146 visually locates the set of biomarkers 142. In other embodiments, the map output 146 may be further processed by the output generator 136 to identify which regions of the OCT volumetric image 114 are or contain biomarkers. The process of identifying the set of biomarkers 142 using the saliency mapping algorithm 134 and the output generator 136 is described in more detail with respect to FIGS. 2-3 below. In some embodiments, the saliency mapping algorithm 134 is integrated with or implemented as part of the output generator 136.

[0051]

[0070] In some embodiments, the model 132 may be trained with a training dataset 148, which may include OCT volumetric images of tissue, such that the model 132 may identify and / or discover biomarkers associated with tissue health categories (e.g., disease, condition, disease progression, etc.) from a test dataset of OCT volumetric images of tissue. In some cases, the tissue health categories may range from healthy to various stages of disease. For example, health categories related to the retina may range from healthy to various stages of AMD (including, but not limited to, early AMD, intermediate AMD, neo-GA, etc.). In some cases, different biomarkers may be associated with different health categories of disease.

[0052]

[0071] For example, AMD is the leading cause of vision loss in patients over 50 years of age. Initially, AMD appears as a dry form of AMD and progresses to wet form at later stages. In the dry form, small deposits called drusen form under the basement membrane of the retinal pigment epithelium (RPE) and the inner collagen layer of the Bruch's membrane (BM) of the retina, deteriorating the retina over time. In advanced stages, dry AMD can appear as geographic atrophy (GA), which is characterized by progressive and irreversible loss of choriocapillaris, RPE and photoreceptors. Wet AMD is manifested by abnormal blood vessels that originate in the choroid layer of the eye growing into the retina and leaking fluid from the blood into the retina. Thus, in some embodiments, drusen may be considered a biomarker for one type of AMD health category (e.g., dry AMD) and defective RPE may be considered a biomarker for another type of AMD health category (e.g., wet AMD). It should be noted that other health status categories (e.g., intermediate AMD, neonatal GA, etc.) may be defined for AMD (e.g., or other types of retinal disease) and at least one or more distinguishable biomarkers may be associated with these health status categories.

[0053]

[0072] As mentioned above, morphological changes and / or the appearance of new areas, boundaries, etc. in the retina or eye may be considered as biomarkers for retinal diseases such as AMD. Examples of such morphological changes may include distortions (e.g., shape, size, etc.), attenuation, abnormalities, missing or missing areas / boundaries, defects, lesions, etc. For example, as mentioned above, defects in the RPE may indicate retinal degenerative diseases such as AMD. As another example, the appearance of deposits (e.g., drusen), leakage, etc., areas not present in a healthy eye or retina, boundaries between them, etc. may also be considered as biomarkers for retinal diseases such as AMD. Other examples of retinal features that may be considered as biomarkers include retinal pseudodrusen (RPD), retinal hyperreflective lesions (e.g., lesions with reflectance equal to or greater than that of the RPE), hyporeflective wedge-shaped structures (e.g., appearing within the boundaries of the OPL), choroidal hyperpermeability defects, etc.

[0054]

[0073] The output generator 136 may generate other forms of output. For example, in one or more embodiments, the output generator may generate a report 150 that is displayed on the display system 106 or transmitted to a remote device (e.g., a cloud, a mobile device, a laptop, a tablet, etc.) via the network 120 or another network. The report 150 may include, for example, but not limited to, the OCT volumetric image 114, a saliency volumetric image, a map output of the OCT volumetric image, a list of identified biomarkers, treatment recommendations for the subject's retina, evaluation recommendations, monitoring recommendations, some other type of recommendation or instruction, or a combination thereof. The monitoring recommendations may include, for example, a plan for monitoring the subject's retina and a schedule of future OCT imaging appointments. The evaluation recommendations may include, for example, a recommendation to further review (e.g., manually review by a human reviewer) a subset of the plurality of OCT slice images that form the OCT volumetric image. The identified subset may include less than 5% of the plurality of OCT slice images. In some cases, the subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, 5%, 2%, or some other percentage of the multiple OCT slice images.

[0055]

[0074] In one or more embodiments, the health condition identification system 101 stores in the data storage 104 the OCT volumetric images 114 obtained from the OCT imaging system 110, the saliency map 144, the map output 146, the identification of the set of biomarkers 142, the report 150, other data generated during processing of the OCT volumetric images 114, or combinations thereof. In some embodiments, the portion of the data storage 104 that stores such information may be configured to comply with the security requirements of the Health Insurance Portability and Accountability Act (HIPAA), which mandates certain security procedures when handling patient data (e.g., OCT images of patient tissue, etc.), i.e., the data storage 104 may be HIPAA compliant. For example, the stored information may be encrypted and anonymized. For example, the OCT volumetric images 114 may be encrypted and processed to remove and / or obfuscate personally identifiable information (PII) of the subject from whom the OCT volumetric images 114 were obtained. In some cases, the communication link between the OCT imaging system 110 and the health condition identification system 101 utilizing the network 120 may also be HIPAA compliant. For example, the communication link may be a Virtual Private Network (VPN) that is end-to-end encrypted and configured to anonymize the PII data transmitted across it.

[0056]

[0075] In one or more embodiments, the health identification system 101 includes a system interface 160 that allows a human reviewer to interact with the images, maps, and / or other output generated by the health identification system 101. The system interface 160 may include, for example, but is not limited to, a web browser, an application interface, a web-based user interface, some other type of interface component, or a combination thereof.

[0057]

[0076] Although the discussion herein is generally directed to classification of OCT volumetric images (and OCT slice images) with respect to stages of AMD, and identification and discovery of AMD biomarkers from retinal OCT volumetric images (or OCT slice images), the discussion may be similarly applied to medical images of other tissues of a subject obtained using any other medical imaging technique. That is, the related discussion of the OCT volumetric images 114 and steps for classifying the OCT volumetric images 114 and for identification and / or discovery of AMD biomarkers via generation of saliency maps (e.g., heat maps) of retinal OCT slice images is intended as a non-limiting example, and the same or substantially similar method steps may be applied to identification and / or discovery of other tissue diseases from 3D images (e.g., OCT or otherwise) of the tissue.

[0058] II.B. Examples of Health Status Identification Methods

[0077] FIG. 2 is a flow chart of a process for processing an OCT volumetric image of a subject's retina to determine whether the OCT volumetric image is evidence of a selected retinal health category, according to one or more exemplary embodiments. The process 200 of FIG. 2 may be implemented using the health condition identification system 101 of FIG. 1. For example, at least some of the steps of the process 200 may be performed by a processor of a computer or server implemented as part of the health condition identification system 101. The process 200 may be implemented using the model 132, the saliency mapping algorithm 134, and / or the output generator 136 of FIG. 1. Furthermore, it is understood that additional steps may be performed before, between, or after the steps of the process 200 described below. Furthermore, in some embodiments, one or more of the steps may be omitted or performed in a different order.

[0059]

[0078] Process 200 may optionally include step 201 of training a deep learning model. The deep learning model may be an example of an implementation of model 132 of FIG. 1. The deep learning model may include, for example, but is not limited to, a neural network model. The deep learning model may be trained with a training dataset, for example, but is not limited to, training dataset 148 of FIG. 1. Examples of how a deep learning model may be trained are described in further detail below in Section II.D.

[0060]

[0079] Step 202 of process 200 includes receiving an optical coherence tomography (OCT) volumetric image of a subject's retina. The OCT volumetric image may be, for example, OCT volumetric image 114 of FIG. 1. The OCT volumetric image may be composed of multiple OCT slice images.

[0061]

[0080] Step 204 includes using the OCT volumetric image to generate, via the deep learning model, a health-indicating output, the health-indicating output indicating a level of association between the OCT volumetric image and a selected health category of the retina. The health-indicating output may be, for example, the health-indicating output 138 of FIG. 1. In one or more embodiments, the health-indicating output is a classification score. The classification score may be, for example, a probability that the OCT volumetric image, and therefore the retina captured in the OCT volumetric image, may be classified into a selected health category. In other words, the classification score may be a probability that the OCT volumetric image is evidence of a selected health category of the retina. In some embodiments, a probability score threshold (e.g., >0.5, >0.6, >0.7, >0.75, >0.8, etc.) is used to determine whether the OCT volumetric image is evidence of a selected health category.

[0062]

[0081] The selected health category may be, for example, selected health category 140 of FIG. 1. In one or more embodiments, the selected health category represents a current health condition of the retina (e.g., a current disease condition). In one or more other embodiments, the selected health category represents a future health condition (e.g., a future disease condition predicted to develop within a selected time period). For example, the selected health category may represent emerging GA that is currently present or predicted to develop within a selected time period (e.g., 3 months, 6 months, 1 year, 2 years, 3 years, or some other time period).

[0063]

[0082] The deep learning model may generate the health-indicating output in different ways. In one or more embodiments, the deep learning model generates an initial output for each OCT slice image in the OCT volumetric image to form a plurality of initial outputs. The initial output for the OCT slice image may be, for example, but not limited to, a probability that the OCT slice image is evidence of a selected health condition category of the retina. The deep learning model may use the plurality of initial outputs to generate the health-indicating output. For example, the deep learning model may average the plurality of initial outputs together to generate a health-indicating output that is a probability that the OCT volumetric images as a whole are evidence of a selected health condition category of the retina. In other words, the health-indicating output may be a probability that the retina may be classified into a selected health condition category. In other embodiments, a median value of the plurality of initial outputs may be used as the health-indicating output. In still other embodiments, the plurality of initial outputs may be combined or integrated in some other manner to generate the health-indicating output.

[0064]

[0083] Step 206 includes using a saliency mapping algorithm to generate a map output (e.g., map output 146) for the deep learning model, the map output indicating the contribution of a set of regions in the OCT volumetric image to the health-indicating output generated by the deep learning model. The contribution of a region in the OCT volume may be the importance or influence the region has to the health-indicating output generated by the deep learning model. The region may be defined as a single pixel or multiple pixels. The region may be continuous or discontinuous. In one or more embodiments, the saliency mapping algorithm receives data from the deep learning model. The data may include, for example, features, weights, or gradients used by the deep learning model to generate the health-indicating output in step 204. The saliency map algorithm may be used to generate a saliency map (or heat map) indicating the importance of various portions of the OCT volumetric image to a selected health condition category (which is a class of interest).

[0065]

[0084] For example, the saliency mapping algorithm may generate a saliency map for each OCT slice image of the OCT volumetric image. In one or more embodiments, the saliency mapping algorithm is implemented using Grad-CAM. The saliency map may be, for example, a heat map indicating the contribution (or importance) of each pixel in the corresponding OCT slice image to the health indication output generated by the deep learning model for the selected health condition category. The saliency maps for multiple OCT slice images in the OCT volumetric image may jointly form a saliency volume map. The saliency map may use color, annotation, text, highlighting, shading, patterns, or other types of visual indicators to indicate importance. In one example, different colors may be used to indicate different importance.

[0066]

[0085] The saliency volume map may be used to generate a map output in a variety of ways. In one or more embodiments, each saliency map of each OCT slice image may be filtered to generate a modified saliency map. For example, one or more filters (e.g., thresholds, processing filters, numerical filters, color filters, shading filters, etc.) may be applied to the saliency map to generate modified saliency maps that jointly form the modified saliency volume map. Each modified saliency map may visually signal the most important regions of the corresponding OCT slice image. In one or more embodiments, each modified saliency map is overlaid on its corresponding OCT slice image to generate the map output. For example, the modified saliency map may be overlaid on the corresponding OCT slice image such that the portions of the OCT slice image that were determined to be most important (or relevant) to the model of the selected health condition category are indicated. In one or more embodiments, the map output includes all of the overlaid OCT slice images. In one or more embodiments, the map output may provide a visual indication of the regions on each overlaid OCT slice image that have the most important or most influential contribution to the generation of the health indication output.

[0067]

[0086] In other embodiments, the modified saliency map is processed in another manner to generate a map output that indicates which regions of the OCT slice images most impact the model for the selected health condition category. For example, information from the modified saliency map may be used to annotate and / or otherwise graphically modify the corresponding OCT slice images to form the map output.

[0068]

[0087] Process 200 may optionally include step 208. Step 208 includes using the map output to identify a set of biomarkers in the OCT volumetric images for a selected health condition category (e.g., set of biomarkers 142 of FIG. 1). Step 208 may be performed in different ways. In one or more embodiments, a potential biomarker region may be identified in association with a selected region of the OCT slice images identified by the map output as being important or influential for the selected health condition category. The potential biomarker region may be identified as this selected region of the OCT slice images or may be defined based on this selected region of the OCT slice images. In one or more embodiments, a bounding box is created around the selected region of the OCT slice images to define the potential biomarker region.

[0069]

[0088] A scoring metric may be generated for the potential biomarker region. The scoring metric may include, for example, the size of the potential biomarker region, a confidence score of the potential biomarker region, some other metric, or a combination thereof. A potential biomarker region (e.g., a bounding box) may be identified as a biomarker for a selected health condition category if the scoring metric meets a selected threshold. For example, if the scoring metric includes a confidence score and a dimension, the selected threshold may include a confidence score threshold (e.g., a score minimum) and a minimum dimension. In some embodiments, a particular biomarker may be found on or span multiple OCT slice images. In one or more embodiments, bounding boxes that meet the threshold and are classified as biomarker regions may be identified on the corresponding OCT slice images to form a biomarker map.

[0070]

[0089] One or more of the identified biomarkers may be known biomarkers that have been previously confirmed by a human reviewer. In some embodiments, one or more of the biomarkers may be new biomarkers that were not previously known. In other words, identifying the set of biomarkers in step 208 may include the discovery of one or more new biomarkers associated with the selected health condition category. The discovery of one or more new biomarkers may be more likely, for example, when the selected health condition category represents a future health condition predicted to develop (e.g., future progression of AMD from early AMD or intermediate AMD to neonatal GA, future progression of AMD from neonatal GA to GA, future progression of AMD from intermediate AMD to GA, etc.).

[0071]

[0090] The process 200 may optionally include step 210. Step 210 includes generating a report. The report may include, for example, but not limited to, an OCT volumetric image, a saliency volumetric image, a map output of the OCT volumetric image, a list of identified biomarkers, a treatment recommendation for the subject's retina, an evaluation recommendation, a monitoring recommendation, any other type of recommendation or instruction, or a combination thereof. The monitoring recommendation may include, for example, a plan for monitoring the subject's retina and a schedule of future OCT imaging appointments. The evaluation recommendation may include, for example, a recommendation to further review (e.g., manually review by a human reviewer) a subset of the plurality of OCT slice images forming the OCT volumetric image. The identified subset may include less than 5% of the plurality of OCT slice images. In some cases, the subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, 5%, 2%, or some other percentage of the plurality of OCT slice images.

[0072]

[0091] In some embodiments, the health identification system 101 of FIG. 1 may prompt a user review (e.g., via the evaluation recommendations of the report 150 of FIG. 1 ) of a particular subset of OCT slice images in the OCT volume image to identify one or more features (or biomarkers) that are in the same or substantially similar location as the bounding boxes identified on the biomarker map. For example, in some cases, the health identification system 101 may include a system interface 160 that allows a reviewer (e.g., a medical professional, a trained reviewer, etc.) to access, review, and annotate the OCT slice images of the OCT volume image to identify and / or discover biomarkers. That is, for example, the system interface 160 may facilitate annotation by the reviewer of the OCT slice images with biomarkers. In some cases, instead of or in addition to allowing the reviewer to identify biomarkers on the OCT slice images shown on the biomarker map, the system interface 160 may be configured to allow the reviewer to correct or adjust the bounding boxes on the biomarker map (e.g., adjust the size, shape, or continuity of the bounding boxes). In some cases, the reviewer can annotate the bounding boxes to indicate adjustments to be made. In some cases, the annotated and / or adjusted biomarker map created by the reviewer may be fed back into the deep learning model (e.g., as part of the training dataset 148) for further training of the deep learning model.

[0073]

[0092] FIG. 3 is a flowchart of a process for identifying biomarkers in an OCT volumetric image of a subject's retina, according to one or more exemplary embodiments. The process 300 of FIG. 3 may be implemented using the health condition identification system 101 of FIG. 1. For example, at least some of the steps of the process 300 may be performed by a processor of a computer or server implemented as part of the health condition identification system 101. The process 300 may be implemented using the model 132, the saliency mapping algorithm 134, and / or the output generator 136 of FIG. 1. Furthermore, it is understood that additional steps may be performed before, between, or after the steps of the process 200 described below. Furthermore, in some embodiments, one or more of the steps may be omitted or performed in a different order.

[0074]

[0093] The process 300 may optionally include a step 302 of training a deep learning model. The deep learning model may be an example of an implementation of the model 132 of FIG. 1. The deep learning model may include, for example, but is not limited to, a neural network model. The deep learning model may be trained with a training dataset, for example, but is not limited to, the training dataset 148 of FIG. 1.

[0075]

[0094] Step 304 of process 300 includes receiving an optical coherence tomography (OCT) volumetric image of a subject's retina. The OCT volumetric image may be, for example, OCT volumetric image 114 of FIG. 1. The OCT volumetric image may be composed of multiple OCT slice images.

[0076]

[0095] Step 306 of process 300 includes using the OCT volumetric image to generate, via the deep learning model, a health indication output, the health indication output indicating a level of association between the OCT volumetric image and a selected retinal health category. For example, the health indication output may be a probability indicating the likelihood that a classification of the retina in the OCT volumetric image is of the selected health category. In other words, the health indication output may be a probability indicating how likely it is that the OCT volumetric image is evidence of a selected retinal health category.

[0077]

[0096] Step 308 of process 300 includes generating a saliency volume map of the OCT volumetric image using a saliency mapping algorithm, where the saliency volume map indicates the contribution of a set of regions in the OCT volumetric image to the health-indicating output generated by the deep learning model. Step 308 may be performed in a manner similar to the generation of the saliency volume map described with respect to step 206 of FIG. 2. The saliency mapping algorithm may include, for example, a Grad-CAM algorithm. The contribution may be determined based on features, gradients, or weights used in the deep learning model (e.g., features, gradients, or weights used in the final activation layer of the deep learning model).

[0078]

[0097] Step 310 of process 300 includes using the saliency volume map to detect a set of biomarkers for a selected health condition category. Step 310 may be performed in a manner similar to the identification of biomarkers described above with respect to step 208 of FIG. 2. For example, step 310 may include filtering the saliency volume map to generate a modified saliency volume map. The modified saliency volume map identifies a set of regions associated with the selected health condition category. Step 310 may further include identifying a potential biomarker region in association with a region of the set of regions. A scoring metric may be generated for the potential biomarker region. A potential biomarker region may be identified as including at least one biomarker if the scoring metric meets a selected threshold.

[0079] II.C. Example of a method for detecting newborn geographic atrophy (nGA)

[0098] FIG. 4A is a flow chart of a process 400 for artificial intelligence-assisted nascent geographic atrophy (nGA) detection, according to one or more exemplary embodiments. The detection of nGA described with respect to FIG. 4A may include detection of one or more nGA lesions, localization of one or more nGA lesions, or a combination thereof. Such detection may be considered a diagnosis of nGA. The process 400 may be implemented using the health condition identification system 101 of FIG. 1. For example, at least some of the steps of the process 400 may be performed by a processor of a computer or server implemented as part of the health condition identification system 101. The process 400 may be implemented using the model 132, the saliency mapping algorithm 134, and / or the output generator 136 of FIG. 1. Furthermore, it is understood that additional steps may be performed before, between, or after the steps of the process 400 described below. Furthermore, in some embodiments, one or more of the steps may be omitted or performed in a different order.

[0080]

[0099] Step 402 of process 400 includes training a machine learning model based on the OCT volumetric dataset. For example, one implementation of step 402 may include training a deep learning model using training data 148 of FIGURE 1 and OCT volumetric images 114 of FIGURE 1. The deep learning model may be an example of an implementation of model 132 of FIGURE 1.

[0081]

[0100] Step 404 of process 400 includes applying a machine learning model to determine a diagnosis of nascent geographic atrophy for the patient based on at least the patient's OCT volume. For example, an example implementation of step 404 may include processing the OCT imaging data 112 of FIG. 1 using the image processor 130 of FIG. 1 and diagnosing nGA using the health condition identification system 101. Diagnosing nGA based on the OCT volume may be an example implementation of step 204 of FIG. 2. The diagnosis of nGA may be made based on detection of one or more nGA lesions (e.g., detecting the onset of nGA based on detection of the presence of one or more nGA lesions).

[0082]

[0101] Step 406 of process 400 includes determining a location of one or more nGA lesions based on at least a saliency map that identifies one or more regions of the OCT volume associated with a supra-threshold contribution to a diagnosis of nGA. For example, one implementation of step 406 may include the health condition identification system 101 generating an output of the saliency map 134 of FIG. 1 to identify the location of the one or more nGA lesions.

[0083]

[0102] Step 408 of process 400 includes verifying a diagnosis of nGA and / or a location of one or more nGA lesions based on one or more inputs. An exemplary implementation of step 408 may include the health condition identification system 101 verifying a diagnosis of nascent geographic atrophy and / or a location of one or more nascent geographic atrophy lesions based on one or more user inputs.

[0084]

[0103] FIG. 4B is a flow chart of a process for processing an OCT volumetric image of a subject's retina to determine whether the OCT volumetric image is evidence of neonatal geographic atrophy (nGA), according to one or more exemplary embodiments. Process 450 of FIG. 4B may be implemented using health condition identification system 101 of FIG. 1. For example, at least some of the steps of process 450 may be performed by a processor of a computer or server implemented as part of health condition identification system 101. Process 450 may be implemented using model 132, saliency mapping algorithm 134, and / or output generator 136 of FIG. 1. Furthermore, it is understood that additional steps may be performed before, between, or after the steps of process 450 described below. Furthermore, in some embodiments, one or more of the steps may be omitted or performed in a different order.

[0085]

[0104] Process 450 may optionally include a step 452 of training a deep learning model. The deep learning model may be an example of an implementation of model 132 of FIG. 1. The deep learning model may include, for example, but is not limited to, a neural network model. The deep learning model may be trained with a training data set, such as, for example, but not limited to, training data set 148 of FIG. 1. Examples of how a deep learning model may be trained are described in further detail below in Section II.D.

[0086]

[0105] Step 454 of process 450 includes receiving an optical coherence tomography (OCT) volumetric image of a subject's retina. The OCT volumetric image may be, for example, OCT volumetric image 114 of FIG. 1. The OCT volumetric image may be composed of multiple OCT slice images.

[0087]

[0106] Step 456 includes generating an output indicating whether or not nascent geographic atrophy (nGA) is detected via the deep learning model. This output may be, for example, an example of an implementation of the health indicator output 138 of FIG. 1. In one or more embodiments, the output is a classification score for nGA. The classification score may be, for example, a probability that the OCT volumetric image, and thus the retina captured in the OCT volumetric image, may be classified as evidence of nGA (e.g., evidence of onset of nGA or another substage of nGA). In other words, the classification score may be a probability that the OCT volumetric image is evidence of retinal nGA. In some embodiments, a probability score threshold (e.g., >0.5, >0.6, >0.7, >0.75, >0.8, etc.) is used to determine whether the OCT volumetric image is evidence of nGA. Step 456 may be performed in a manner similar to the implementation of step 204 described with respect to FIG. 2.

[0088]

[0107] Step 458 includes using a saliency mapping algorithm to generate a map output (e.g., map output 146) for the deep learning model, where the map output indicates the contribution of a set of regions in the OCT volume image to the output generated by the deep learning model. The contribution of a region in the OCT volume may be the importance or influence that the region has to the output generated by the deep learning model.

[0089]

[0108] The region may be defined as a single pixel or multiple pixels. The region may be contiguous or discontinuous. In one or more embodiments, the saliency mapping algorithm receives data from a deep learning model. This data may include, for example, features, weights, or gradients used by the deep learning model to generate the output in step 456. The saliency map algorithm may be used to generate a saliency map (or heat map) that indicates the importance of various portions of the OCT volumetric image to a selected health condition category, which is a class of interest.

[0090]

[0109] For example, the saliency mapping algorithm may generate a saliency map for each OCT slice image of the OCT volumetric image. In one or more embodiments, the saliency mapping algorithm is implemented using Grad-CAM. The saliency map may be, for example, a heat map indicating the contribution (or importance) of each pixel in the corresponding OCT slice image to the health indication output generated by the deep learning model for the selected health condition category. The saliency maps for multiple OCT slice images in the OCT volumetric image may jointly form a saliency volume map. The saliency map may use color, annotation, text, highlighting, shading, patterns, or other types of visual indicators to indicate importance. In one example, different colors may be used to indicate different importance.

[0091]

[0110] The saliency volume map may be used to generate a map output in a variety of ways. In one or more embodiments, each saliency map of each OCT slice image may be filtered to generate a modified saliency map. For example, one or more filters (e.g., thresholds, processing filters, numerical filters, color filters, shading filters, etc.) may be applied to the saliency map to generate modified saliency maps that jointly form the modified saliency volume map. Each modified saliency map may visually signal the most important regions of the corresponding OCT slice image. In one or more embodiments, each modified saliency map is overlaid on its corresponding OCT slice image to generate the map output. For example, the modified saliency map may be overlaid on the corresponding OCT slice image such that the portions of the OCT slice image that were determined to be most important (or relevant) to the model of the selected health condition category are indicated. In one or more embodiments, the map output includes all of the overlaid OCT slice images. In one or more embodiments, the map output may provide a visual indication of the regions on each overlaid OCT slice image that have the most important or most influential contribution to the generation of the output.

[0092]

[0111] In other embodiments, the modified saliency map is processed in another manner to generate a map output that indicates which regions of the OCT slice images most impact the model for the selected health condition category. For example, information from the modified saliency map may be used to annotate and / or otherwise graphically modify the corresponding OCT slice images to form the map output.

[0093]

[0112] One or more regions identified by the map output can be shown to correspond directly with one or more nGA lesions, for example. For example, the regions identified in the map output can be considered the location of one or more nGA lesions.

[0094]

[0113] In one or more embodiments, other information may be annotated to the map output. For example, the map output may include bounding boxes created around selected regions of the OCT slice images identified as nGA lesions or as evidence of one or more nGA lesions. In some cases, the bounding boxes may be annotated with a scoring metric (e.g., confidence score, dimensions, etc.). In one or more embodiments, bounding boxes that meet a threshold dimension, a threshold confidence score, or both are classified as being evidence of nGA.

[0095]

[0114] Process 450 may optionally include step 460. Step 460 includes generating a report. The report may include, for example, but not limited to, an OCT volumetric image, a saliency volumetric image, a map output of the OCT volumetric image, a list of identified biomarkers, a treatment recommendation for the subject's retina, an evaluation recommendation, a monitoring recommendation, any other type of recommendation or instruction, or a combination thereof. The monitoring recommendation may include, for example, a plan for monitoring the subject's retina and a schedule of future OCT imaging appointments. The evaluation recommendation may include, for example, a recommendation to further review (e.g., manually review by a human reviewer) a subset of the plurality of OCT slice images forming the OCT volumetric image. The identified subset may include less than 5% of the plurality of OCT slice images. In some cases, the subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 45%, 15%, 10%, 5%, 2%, or some other percentage of the plurality of OCT slice images.

[0096]

[0115] In some embodiments, the health identification system 101 of FIG. 1 may prompt a user review (e.g., via the evaluation recommendations of the report 150 of FIG. 1 ) of a particular subset of OCT slice images in the OCT volume image to identify one or more features (or biomarkers) that are in the same or substantially similar location as the bounding boxes identified on the biomarker map. For example, in some cases, the health identification system 101 may include a system interface 160 that allows a reviewer (e.g., a medical professional, a trained reviewer, etc.) to access, review, and annotate the OCT slice images of the OCT volume image to identify and / or discover biomarkers. That is, for example, the system interface 160 may facilitate annotation by the reviewer of the OCT slice images with biomarkers. In some cases, instead of or in addition to allowing the reviewer to identify biomarkers on the OCT slice images shown on the biomarker map, the system interface 160 may be configured to allow the reviewer to correct or adjust the bounding boxes on the biomarker map (e.g., adjust the size, shape, or continuity of the bounding boxes). In some cases, the reviewer can annotate the bounding boxes to indicate adjustments to be made. In some cases, the annotated and / or adjusted biomarker map created by the reviewer may be fed back into the deep learning model (e.g., as part of the training dataset 148) for further training of the deep learning model.

[0097] II.D. Deep Learning Model Training Example

[0116] The deep learning models described above in Figures 1 (e.g., model 132), 2, 3, 4, and 5 may be trained in different ways. In one or more embodiments, the deep learning model is trained on a training dataset (e.g., training dataset 148 of Figure 1) that includes one or more training OCT volumetric images. Each of these training OCT volumetric images may be of a different retina that has been identified as displaying a disease or condition corresponding to a selected health category (e.g., neonatal GA, etc.). The retina may have displayed the disease or condition for a period of time at least substantially equal to the period of time after the training OCT volumetric images were taken or generated.

[0098]

[0117] In one or more embodiments, a deep learning model (e.g., model 132 of FIG. 1 , the deep learning model described in FIGS. 2-3 ) is trained to classify retinal health conditions based on a training dataset (e.g., training dataset 148 of FIG. 1 ) of OCT volumetric images of retinas of patients suffering from one or more health condition categories, such that the deep learning model can learn what retinal features and their locations within the OCT volumetric images are signatures of one or more health condition categories. When provided with a test dataset of OCT volumetric images, the trained deep learning model may be able to efficiently and accurately identify whether the OCT volumetric images are evidence of a selected health condition category.

[0099]

[0118] The training dataset of OCT volumetric images may include OCT images of the retina of a patient known to be afflicted with a given stage of AMD (i.e., the retinal health category may be said stage of AMD). In such a case, the deep learning model may be trained with the training dataset to learn what features in the OCT volumetric images correspond to, are associated with, or are indicative of that stage of AMD. For example, the patient may be afflicted with late stage AMD, and the deep learning model may identify or discover from the training dataset of OCT volumetric images of the patient's retina that anatomical features in the OCT volumetric images representing severely deformed RPE may be evidence of late stage AMD. In such a case, when provided with an OCT volumetric image of the patient's retina showing a severely deformed RPE, the trained deep learning model may identify the OCT volumetric image as belonging to a late stage AMD patient.

[0100]

[0119] In some embodiments, the deep learning model may be able to classify health states even based on unknown biomarkers of retinal disease. For example, the deep learning model may be provided with a training dataset of OCT volumetric images of the retina of patients suffering from some retinal disease (e.g., neonatal GA) for which all biomarkers may not be known. That is, biomarkers for that selected health category of retinal disease (e.g., neonatal GA) may not be known. In such a case, the deep learning model may process the dataset of OCT volumetric images and learn that features or patterns, e.g., lesions, in the OCT volumetric images are evidence of a selected health category.

[0101] III. Example OCT images and map output

[0120] 5 illustrates an annotated OCT slice image and a corresponding heat map of the annotated OCT slice image, according to one or more exemplary embodiments. The OCT slice image 502 may be an example of an implementation of an OCT slice image in the OCT volumetric image 114 of FIG. 1. The OCT slice image 502 may also be an example of an implementation from an OCT slice image in the training dataset 148 of FIG. 1. The OCT slice image 502 includes annotated regions 504 that have been marked by a human assessor as being biomarkers for emerging GA.

[0102]

[0121] Heat map 506 is an example of an implementation of at least a portion of map output 146 of Figure 1. Heat map 506 may be the result of overlaying a saliency map generated using a saliency mapping algorithm, such as saliency mapping algorithm 134 of Figure 1 (e.g., generated using Grad-CAM), onto OCT slice image 502. The saliency map was generated for a trained deep learning model that processed OCT slice image 502. Heat map 506 shows that region 508 had the most impact on the nascent GA model, indicating that the deep learning model, which may be model 132 of Figure 1, for example, accurately used the correct region of OCT slice image 502 for classification with respect to the nascent GA.

[0103]

[0122] The heat map 506 may be used to identify and locate biomarkers of nascent GA depicted within the region 508. For example, an output may be generated that identifies anatomical biomarkers located within the region 508. The biomarkers may be, for example, retinal pathology, defects in the retinal pigment epithelium (RPE), retinal detachment layers, or some other type of biomarker. In some cases, more than one biomarker may be present within the region 508. In some cases, filtering may be performed to identify specific pixels within the region 508 of the heat map 506 or within the region 504 of the OCT slice image 502 that are biomarkers. In some cases, the size of the region 508 may be used to determine whether the region 508 includes one or more biomarkers. In one or more embodiments, the size of the region 508 is greater than about 20 pixels.

[0104]

[0123] Such identification and location of biomarkers may enable medical professionals to diagnose, monitor, treat, etc., a patient whose retina is depicted in the OCT slice images 502. For example, an ophthalmologist reviewing the heat map 506 or information generated based on the heat map 506 may be able to recommend treatment or monitoring options prior to the onset of GA.

[0105]

[0124] 6 is an illustration of different maps according to one or more exemplary embodiments. Saliency map 602 is an example of an implementation of a saliency map that constitutes saliency volume map 144 of FIG. 1. Modified saliency map 604 is an example of an implementation of a modified saliency map after filtering (e.g., application of a threshold filter). Heat map 606 is an example of an implementation of components of map output 146 of FIG. 1. Heat map 606 includes a modified overlay of saliency map 602 on an OCT slice image.

[0106]

[0125] Biomarker map 608 is one example of an implementation of an output that may be generated by output generator 136 of Figure 1 using heat map 606. In biomarker map 608, a first bounding box identifies potential biomarker regions that do not have a high enough confidence score (e.g., >0.6) to be considered as biomarker regions that contain at least one biomarker. Additionally, biomarker map 608 includes a second bounding box that identifies potential biomarker regions that do have a high enough confidence score to be considered as biomarker regions that contain at least one biomarker.

[0107]

[0126] The images and map outputs (e.g., heat maps, saliency maps) shown in Figures 5-6 below are shown in one exemplary grayscale. In other embodiments, other grayscales may be used. For example, an OCT image such as that shown in Figure 5 may have an inverted or partially inverted grayscale relative to the grayscale shown in Figure 5. As a non-limiting example, the background shown in white in Figure 5 may be black in other exemplary embodiments. In some embodiments, various colors may be used to generate the map output. For example, the map output shown in grayscale in Figure 6 may be colored in other embodiments. In other embodiments, the biomarker map shown in Figure 6 may be annotated with color, may have potential biomarker regions identified by color, or both.

[0108] IV. Example of a Nascent Geographic Atrophy Detection System

[0127] 7-13 show a nascent geographic atrophy (nGA) detection system and various workflows using the system. The nascent geographic atrophy detection system 700 may be an example of an implementation of the health status display system 101 of FIG. 1. Training is described with respect to one or more different types of exemplary training datasets.

[0109]

[0128] 7 is a system diagram illustrating an example of an nascent geographic atrophy detection system 700 according to some exemplary embodiments. The nascent geographic atrophy detection system 700 may include a detection controller 710 including a diagnosis engine 712 and a localization engine 714, a data store 720, and a client device 730. The detection controller 710, the data store 720, and the client device 730 may be communicatively coupled via a network 740. The detection controller may be an example of a full or partial implementation of the image processor 130 of FIG. 1.

[0110]

[0129] The client device 730 may be a processor-based device including, for example, a mobile device, a wearable device, a personal computer, a workstation, an Internet of Things (IoT) device, etc. The data store 720 may be a database including, for example, a non-relational database, a relational database, an in-memory database, a graph database, a key-value store, a document store, etc. The data store 720 may be one example implementation of the data storage 104 of FIG. 1. The network 745 may be a wired and / or wireless network including, for example, a public land mobile network (PLMN), a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), the Internet, etc. The network 745 may be one example implementation of the network 120 of FIG. 1.

[0111]

[0130] In some exemplary embodiments, the diagnostic engine 712 may be implemented using a deep learning model, such as an artificial neural network (ANN)-based classifier. The diagnostic engine 712 may be an example of an implementation of the health status model 132 of FIG. 1. In some cases, the diagnostic engine 712 may be implemented as a residual neural network (ResNet)-based classifier. The diagnostic engine 712 may be configured to perform a nascent geographic atrophy (nGA) diagnosis, including determining whether a patient exhibits nascent geographic atrophy based on at least one or more optical coherence tomography (OCT) volumes of the patient. Lesion localization may be performed to locate one or more nascent geographic atrophy (nGA) lesions based on a visual description of the deep learning model applied by the diagnostic engine 712. For example, the localization engine 714 may be configured to perform nGA lesion localization, including determining the location of one or more lesions associated with nGA based on a saliency map that identifies regions of an OCT volume associated with a supra-threshold contribution to a diagnosis of nGA. The saliency map may be generated, for example, by applying gradient weighted class activation mapping (GradCAM), which outputs a heat map indicating how much each region in an image, such as an OCT volume, contributes to a class label ultimately assigned to the image.

[0112]

[0131] In some exemplary embodiments, the deep learning model implementing the diagnosis engine 712 may be trained on a dataset 725 stored, for example, in the data store 720. The dataset 725 may be one exemplary implementation of the training dataset 148 of FIG. 1. In one example, the dataset 725 includes, but is not limited to, a total of 1,884 optical coherence tomography volumes obtained from 280 eyes of 740 subjects with intermediate age-related macular degeneration (iAMD). Overall, 1,766 optical coherence tomography volumes were labeled as without neonatal geographic atrophy (nGA not detected) and 118 volumes were labeled as with neonatal geographic atrophy (nGA detected). Here, a diagnosis of neonatal geographic atrophy may also include neonatal geographic atrophy that has expanded to a size that meets the diagnostic criteria for complete retinal pigment epithelium and outer retinal atrophy (cRORA).

[0113]

[0132] The optical coherence tomography volume can be further labeled with the location of the nascent geographic atrophy lesions, for example, with a bounding box that covers the subsidence horizontally and starts at the inner boundary layer (ILL) layer and ends at the retinal pigment epithelium (RPE) layer vertically. The bounding box may be used in evaluating the weakly supervised lesion localization (e.g., performed by the localization engine 714) rather than training the model. Since the dataset 725 for training the deep learning model includes class labels for the 3D optical coherence tomography volume, the training of the deep learning model for performing the diagnosis and lesion localization of nascent geographic atrophy (nGA) can be considered weakly supervised.

[0114] IV.A. Example of a model for classifying OCT volumes with respect to nGA

[0133] 8A-8B show an example of a deep learning architecture for implementing the diagnosis engine 712 and the localization engine 714 of the detection controller 710 of the nascent geographic atrophy detection system 700 shown in Figure 7. The components shown in Figures 8A-8B may be examples of components used to implement the health condition identification system 101 of Figure 1.

[0115]

[0134] 8A illustrates an example of a model 800 for processing a 3D OCT volume, according to one or more exemplary embodiments. The model 800 can include a deep neural network-based OCT B-scan classifier, which is used to generate a classification score indicating whether nGA is detected in the OCT volume.

[0116]

[0135] 8B illustrates an example of an implementation of a classifier 802 that can be used to implement the classifier of the model 800 of FIG. 8A, according to one or more exemplary embodiments. The classifier 802 can be used to classify OCT B-scans. The classifier 802 can be implemented using a residual neural network (ResNet) backbone with outputs concatenated with rectified linear unit (ReLU) and fully connected (FC) layers. A late-fusion method using the residual neural network (ResNet) backbone can be applied to the 3D OCT volume.

[0117]

[0136] As an example, in FIG. 8A, B-scans are input to the B-scan classifier of model 800, and the output is a vector of classification logits for each B-scan. The B-scan logits are averaged to generate a classification score for each OCT volume. Considering B-scans as instances and OCT volumes as bags, this framework can be classified as an example of multi-instance learning, where model 800 is trained on weakly labeled data using the labels of the bags (OCT volumes). During the training process, given an OCT volume annotated as nascent geographic atrophy, model 800 may be forced to identify as many B-scans containing nascent geographic atrophy lesions as possible to improve the final prediction of nGA, and thus the trained model allows prediction of nGA labels on OCT volumes as well as individual B-scans.

[0118]

[0137] An exemplary B-scan classifier 802 is shown in detail in FIG. 8B. For example, individual B-scans of size 512×496 from a volume are passed through a residual neural network (e.g., ResNet-18) backbone, which outputs an activation map (e.g., a 512×16×16 activation map). Max pooling and average pooling layers can be applied to the output of the residual neural network, and then the respective outputs can be concatenated to generate a feature vector (e.g., a feature vector of length 1024). A fully connected layer can then be applied to the feature vector to generate a classification logit vector that corresponds to the categorical distribution of the B-scans.

[0119] IV.B. Model Training Example

[0138] FIG. 9 is an exemplary data flow diagram including data split statistics, according to one or more exemplary embodiments. The data flow diagram tracks the training of a model (such as model 800 of FIGS. 8A-8B) based on one example of data collected for various subjects as part of an experiment or study. Training data 900, which may be an example of dataset 725 of FIG. 7, was generated from 1,910 OCT volumes from 280 eyes of 140 participants with intermediate age-related macular degeneration (iAMD) (one volume per eye per semi-annual visit for up to three years). Volumes classified as neovascular age-related macular degeneration were excluded. Of the remaining 1,884 volumes, 118 volumes from 40 eyes of 28 participants were determined to be nGA positive. A five-fold cross-validation 902 was performed on the training data 900, and five models were trained in a "cross-validation" fashion in five different splits. In each split, the training data 900 was split into a training set, a validation set, and a test set for each patient. Early stopping was applied to monitor the F1 score of the validation set. Model performance evaluation was applied to the test set. Table 904 shows the fold statistics, number of volumes, and participants for the 5-fold cross-validation 902. Note that the number of eyes is twice the number of participants.

[0120]

[0139] If the training data 900 contains a small number of participants, the performance of the deep learning model may be tested on the entire dataset with five test sets from five different split folds. For each fold, the test set of optical coherence tomography volumes was obtained from approximately 20% of the participants classified based on whether the patient developed nGA or not. The OCT volumes from the remaining 80% of participants were further split into a training set (64%) and a validation set (16%), with volumes from one patient only being present in one set. Although the term cross-validation was used to describe the data split, the corresponding test set was not used in the training and validation process.

[0121]

[0140] In some exemplary embodiments, at least some pre-processing may be performed on the B-scans for standardization. For example, the B-scans may be resized (e.g., to 512x496 pixels) and then rescaled to the intensity range [0,1]. Data augmentations such as small angle rotations, horizontal flips, vertical flips, adding Gaussian noise, Gaussian blurring, etc. may be randomly applied to improve the invariance of the model to these transformations.

[0122]

[0141] Referring again to FIG. 8B, the residual neural network (ResNet) backbone of the classifier 802 may be pre-trained on the ImageNet dataset. In one or more embodiments, during training of the model, an Adam optimizer may be used to minimize focal loss and L2 weight decay regularization may be applied to improve the model's ability to generalize across the training data 900. In some cases, hyperparameter tuning may be performed using the training data 900 and a validation set to find optimal values ​​for learning rate and weight decay. The model 800 trained with optimal hyperparameters may be tested on a test set. Various metrics may be evaluated to indicate the performance of the model. Such metrics may include, but are not limited to, for example, the area under the curve (AUC), the area under the precision-recall curve (AUPRC), recall, precision, and F1 score. Additionally, a confusion matrix may be calculated.

[0123] IV.C. Example of Generating Map Output

[0142] FIG. 10 is a diagram of an output workflow for output generated from an OCT volume, according to one or more exemplary embodiments. In workflow 1000, the output of gradient-weighted class activation mapping can be overlaid on the input Oct image to easily visualize the saliency and original grayscale OCT image. Visually enhanced (e.g., specific coloring or highlighting) regions can indicate the location of nGA lesions. The saliency map can be used to reason about the decisions of a model (e.g., model 800), check the generalizability of the model, and explore and leverage the model's capabilities in detecting nascent geographic atrophy lesions.

[0124]

[0143] As shown in workflow 1000, B-scan logits are generated for each OCT B-scan of an OCT volume input to a model (e.g., model 800). These logits are used to classify the OCT volume as evidence of nGA or not. The GradCAM output of the model is displayed for each individual slice (e.g., slice 22). Adaptive thresholding is applied to the corresponding GradCAM output of gradient-weighted class activation mapping (in the viridis color map) before bounding boxes are generated by connected component analysis. A confidence score for the bounding box can be estimated based on the average saliency and the corresponding B-scan logits. The map output can be generated by overlaying the GradCAM output on the B-scan, overlaying the bounding box and its associated confidence score on the B-scan, or both. Bounding boxes with confidence scores below a threshold (e.g., <0.6) can be removed from further processing.

[0125]

[0144] The GradCAM output and the map output are useful for visually identifying nGA lesions. In one or more embodiments, each bounding box can be considered as potentially identifying an nGA lesion. For example, a bounding box with a confidence score above a threshold can be considered as the location of one or more nGA lesions. In one or more embodiments, the confidence score for each bounding box is calculated from the individual classification logits of the B-scan classifier. TIFF2025515049000002.tif10170, where S represents the sigmoid function, l represents the individual B-scan classification logit, n represents the amount of B-scans in the volume, h represents the average saliency of the detected regions, and Σh represents the total average saliency of all regions detected within the B-scans.

[0126]

[0145] A high confidence score may mean that there is a high probability that the detected area within the bounding box covers a nascent geographic atrophy lesion. Therefore, bounding boxes with confidence scores below a threshold (e.g., <0.6) may be removed by thresholding, and B-scans with one or more remaining bounding boxes (after thresholding) may be identified as B-scans exhibiting nascent geographic atrophy (nGA).

[0127]

[0146] In some exemplary embodiments, the aforementioned confidence score threshold may be determined based on B-scans with nascent geographic atrophy present in the validation set. For example, the number of classified nascent geographic atrophy B-scans and the recall of the diagnosis of nascent geographic atrophy B-scans may be plotted against different thresholds in the validation set, respectively. A lower threshold may result in the model generating fewer false negatives (e.g., true nascent geographic atrophy B-scans misclassified as non-nascent geographic atrophy) and a higher number of false positives (e.g., true non-nascent geographic atrophy B-scans misclassified as nascent geographic atrophy). When the detection controller 710 is introduced to patient screening, the B-scans classified as showing nascent geographic atrophy may be further reviewed and verified. Thus, the model may be adjusted to improve the recall while maintaining an acceptable number of B-scans classified as nascent geographic atrophy. In an exemplary implementation, the threshold may be increased from a small value with a step size of 0.02. A threshold may be selected such that further increasing the threshold reduces the recall by more than 0.2, but saves less than 1,000 additional B-scans for further review.

[0128]

[0147] In some exemplary embodiments, the detection controller 710 can generate an output of the B-scans indicative of nascent geographic atrophy that includes one or more major bounding boxes with a confidence score above a threshold. This confidence score can be considered as the confidence score of the B-scan. A diagnosis of nascent geographic atrophy B-scan that located the lesion was recorded as successful only if the bounding box output overlapped with the ground truth and / or expert-annotated bounding boxes.

[0129]

[0148] In some exemplary embodiments, the detection controller 710 may be deployed for AI-assisted diagnosis of nascent geographic atrophy (nGA). In exemplary embodiments, the detection controller 710 may suggest a diagnosis of nascent geographic atrophy and the location of one or more lesions. For example, the detection controller 710 may identify high-risk B-scans within a set of optical coherence tomography volumes that are determined by the underlying deep learning model to be indicative of nascent geographic atrophy. These high-risk B-scans may be displayed, for example, on the user interface 735 of the client device 730. The presence or absence of nascent geographic atrophy and the proposed location of the nascent geographic atrophy lesion may be confirmed based on one or more user inputs received at the client device 730.

[0130]

[0149] 11A is a diagram of a confusion matrix 1100 according to one or more exemplary embodiments. The confusion matrix 1100 may be an example of a confusion matrix generated for a 5-fold cross-validation of the performance of a model such as model 800 of FIGS. 8A-8B. N indicates negative normal volumes and P indicates positive nascent geographic atrophy volumes.

[0131]

[0150] 11B is a graph of statistics of 5-fold cross-validation according to one or more exemplary embodiments. The area under the curve (AUC), area under the precision-recall curve (AUPRC), recall, precision, and F1 score of the model on the test set from 5-fold cross-validation are shown in FIG. 11B. The average performance from the 5 folds is also shown, and the error bars indicate the 95% confidence interval (CI). The average precision and average recall are 0.76 (95% CI 0.60-0.91) and 0.74 (95% CI 0.56-0.93), respectively.

[0132]

[0151] 12A is an illustration of an OCT image 1200 (e.g., B-scan) in which nGA lesions are detected, according to one or more exemplary embodiments. The raw OCT image is displayed on the left, with boxes indicating where a human assessor annotated the presence of nGA and where a system (e.g., nascent geographic atrophy detection system 700) identified the presence of nGA. A true positive would be a B-scan in which the bounding box detected by the model overlaps with the bounding box annotated by the expert. A true negative would be a B-scan that contains neither the bounding box detected by the model nor the annotated bounding box.

[0133]

[0152] FIG. 12B is a graph 1202 of a 5-fold cross-validation precision-recall curve, in accordance with one or more exemplary embodiments.

[0134]

[0153] 12C is a diagram of a confusion matrix 1204 according to one or more exemplary embodiments, in which N indicates a negative B-scan without nGA lesions and P indicates a positive B-scan with nGA lesions.

[0135]

[0154] 12A-12C, the detection controller 710 can achieve robust B-scan diagnosis and lesion localization performance without utilizing any B-scan level grading or bounding box annotation. Overall, across the example dataset, the recall and precision of B-scan diagnosis with correctly localized lesions with bounding boxes are 0.93 and 0.27, respectively.

[0136]

[0155] Use of the type of nascent geographic atrophy detection system 700 described herein allows for more accurate and efficient detection of nGA while reducing the time required to process B-scans. As an example, instead of 92,316 individual B-scans, use of the nascent geographic atrophy detection system 700 may allow a clinician to review only 1,550 B-scans (or any other number of B-scans) in which nGA was detected (e.g., about 2%). Furthermore, use of the nascent geographic atrophy detection system 700 allows for detection of nGA in nGA lesions that would not be detectable by a human assessor.

[0137]

[0156] In some exemplary embodiments, the detection controller 710 including the above-mentioned deep learning model can diagnose nascent geographic atrophy on optical coherence tomography volumes. The detection controller 710 can perform a diagnosis of nascent geographic atrophy in a cohort that starts with intermediate age-related macular degeneration (iAMD) and has no obvious geographic atrophy lesions. Nascent geographic atrophy is believed to be a significant risk factor for progression to geographic atrophy. The detection controller 710 can provide a diagnosis for each B-scan based on a diagnostic label for each optical coherence tomography volume and identify the location of the lesions present therein.

[0138]

[0157] Datasets such as Dataset 725 and Training Data 900 can be highly imbalanced (e.g., a small proportion of cases, i.e., 6.26%, have neonatal geographic atrophy), and using a B-scan classifier with a pre-trained artificial neural network (ANN) backbone (e.g., 2D backbone pre-trained on Imagenet data) significantly improves model performance (F1 score increases from 0.25 to 0.74) for a training dataset of a limited number of OCT volumes.

[0139]

[0158] FIG. 13 is a diagram of an OCT image 1300 annotated with bounding boxes, according to one or more exemplary embodiments. The OCT image 1300 shows that bounding boxes can be used to localize lesions similar to nGA (e.g., drusen or hyperreflective lesions connected to the outer plexiform layer (OPL) containing the retinal pigment epithelium (RPE), drusen forming precipitate-like structures, cysts, or hyperreflective lesions). Such bounding boxes may be further analyzed by a human assessor. In some cases, a higher threshold of confidence score may be used to exclude lesions other than nGA.

[0140]

[0159] Nevertheless, our weakly supervised method for diagnosing neonatal geographic atrophy and localizing neonatal geographic atrophy lesions may be useful for screening patients when neonatal geographic atrophy is enriched and for diagnosing the stage of age-related macular degeneration when neonatal geographic atrophy is used as a biomarker or early endpoint for progression. In clinical trials with neonatal geographic atrophy as an inclusion / exclusion criterion or clinical biomarker or endpoint for progression, grading neonatal geographic atrophy on high-density B-scan optical coherence tomography volumes is laborious and operationally expensive, especially when screening large populations. The proposed AI-assisted diagnosis can significantly reduce the operational burden and improve the feasibility of such trials. Similar strategies can be applied to other trials where clinical enrichment is based on multiple anatomical biomarkers.

[0141] V. Neural Network Examples

[0160] FIG. 14 illustrates an exemplary neural network that can be used to implement a computer-based model according to various embodiments of the present disclosure. For example, a neural network 1400 may be used to implement the model 132 of the health condition identification system 101. As illustrated, the artificial neural network 1400 includes three layers: an input layer 1402, a hidden layer 1404, and an output layer 1407. Each layer 1402, 1404, 1407 may include one or more nodes. For example, the input layer 1402 includes nodes 1408-1414, the hidden layer 1404 includes nodes 1417 and 1418, and the output layer 1407 includes node 1422. In this example, each node in a layer is connected to all nodes in the adjacent layer. For example, node 1408 in the input layer 1402 is connected to both nodes 1417 and 1418 in the hidden layer 1404. Similarly, node 1417 in hidden layer 1404 is connected to all of nodes 1408-1414 in input layer 1402 and node 1422 in output layer 1407. Although only one hidden layer is shown in artificial neural network 1400, it is contemplated that an artificial neural network 1400 used to implement model 132 may include as many hidden layers as necessary or desired.

[0142]

[0161] In this example, the artificial neural network 1400 receives a set of input values ​​and generates an output value. Each node in the input layer 1402 may correspond to a distinct input value. For example, if the artificial neural network 1400 is used to implement the model 132, each node in the input layer 1402 may correspond to a distinct attribute of an OCT volumetric image of the retina (e.g., obtained from the OCT imaging system 110 of FIG. 1).

[0143]

[0162] In some embodiments, each of the nodes 1417 and 1418 in the hidden layer 1404 generates an expression that may include a mathematical calculation (or algorithm) that generates a value based on input values ​​received from the nodes 1408-1414. The mathematical calculation may include assigning different weights to each of the data values ​​received from the nodes 1408-1414. The nodes 1417 and 1418 may include different algorithms and / or different weights assigned to the data variables from the nodes 1408-1414 such that each of the nodes 1417 and 1418 may generate different values ​​based on the same input values ​​received from the nodes 1408-1414. In some embodiments, the weights initially assigned to the features (or input values) of each of the nodes 1417 and 1418 may be randomly generated (e.g., using a computer randomizer). The values ​​generated by the nodes 1417 and 1418 may be used by the node 1422 in the output layer 1407 to generate an output value for the artificial neural network 1400. When the artificial neural network 1400 is used to implement the model 132, the output values ​​generated by the artificial neural network 1400 may include a saliency map, such as, but not limited to, a heat map of a retinal OCT volumetric image (e.g., saliency map 144) that identifies biomarkers within the retina.

[0144]

[0163] The artificial neural network 1400 may be trained using training data. For example, the training data herein may be OCT volumetric images of the retina. The training data may be, for example, the training data set 148 of FIG. 1. By providing the artificial neural network 1400 with training data, the nodes 1417 and 1418 in the hidden layer 1404 may be trained (tuned) to generate optimal outputs in the output layer 1407 based on the training data. By successively providing different sets of training data and penalizing the artificial neural network 1400 when its output is incorrect (e.g., when it misidentifies a biomarker in an OCT volumetric image), the artificial neural network 1400 (specifically, the representation of the nodes in the hidden layer 1404) may be trained (tuned) to improve its performance in data classification. Tuning the artificial neural network 1400 may include adjusting the weights associated with each node in the hidden layer 1404.

[0145]

[0164] Although the above description concerns artificial neural networks as an example of machine learning, it is understood that other types of machine learning methods may also be suitable for implementing various aspects of the present disclosure. For example, machine learning may be implemented using a support vector machine (SVM). SVM is a set of related supervised learning methods used for classification and regression. The SVM training algorithm, which may be a non-probabilistic binary linear classifier, may build a model that predicts whether a new example falls into one category or another. As another example, machine learning may be implemented using a Bayesian network. A Bayesian network is an acyclic probabilistic graphical model that represents a set of random variables and their conditional independence in a directed acyclic graph (DAG). A Bayesian network may present a probabilistic relationship between one variable and another variable. Another example is a machine learning engine that performs a machine learning process using a decision tree learning model. In some cases, the decision tree learning model may include a classification tree model and a regression tree model. In some embodiments, the machine learning engine uses a gradient boosting machine (GBM) model (e.g., XGBoost) as the regression tree model. Other machine learning techniques may be used to implement the machine learning engine, for example, via random forests or deep neural networks. Other types of machine learning algorithms will not be described in detail herein for simplicity, and it will be understood that the present disclosure is not limited to any particular type of machine learning.

[0146] VI. Example Computing System

[0165] Fig. 15 is a block diagram illustrating an example of a computing system 1500 according to some exemplary embodiments. With reference to Figs. 4, 7-17, and 15, the computing system 1500 may be used to implement the detection controller 710 of Fig. 7, the client device 730 of Fig. 7, and / or any components therein.

[0147]

[0166] As shown in FIG. 15, the computing system 1500 may include a processor 1510, a memory 1520, a storage device 1530, and an input / output device 1540. The computing system 1500 may be one exemplary implementation of the health condition identification system 101 of FIG. 1. The processor 1510, the memory 1520, the storage device 1530, and the input / output device 1540 may be interconnected via a system bus 1550. The processor 1510 is capable of processing instructions for execution within the computing system 1500. Such executed instructions may implement one or more components, such as, for example, the detection controller 710, the client device 730, etc. In some exemplary embodiments, the processor 1510 may be a single-threaded processor. Alternatively, the processor 1510 may be a multi-threaded processor. The processor 1510 is capable of processing instructions stored in the memory 1520 and / or the storage device 1530 to display graphical information for a user interface, such as the display system 106 of FIG. 1 or the user interface 735 of FIG. 7, provided via the input / output device 1540.

[0148]

[0167] The memory 1520 is a computer-readable medium, such as volatile or non-volatile, that stores information within the computing system 1500. The memory 1520 may store, for example, data structures that represent a configuration object database. The storage device 1530 may provide persistent storage for the computing system 1500. The storage device 1530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, or other suitable persistent storage means. The storage device 1530 may be one exemplary implementation of the data storage 104 of FIG. 1. The input / output device 1540 provides input / output operations for the computing system 1500. In some exemplary embodiments, the input / output device 1540 includes a keyboard and / or a pointing device. In various implementations, the input / output device 1540 includes a display unit for displaying a graphical user interface.

[0149]

[0168] According to some demonstrative embodiments, the I / O devices 1540 may provide input / output operations for network devices. For example, the I / O devices 1540 may include an Ethernet port or other networking port for communicating with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0150]

[0169] In some exemplary embodiments, computing system 1500 can be used to execute various interactive computer software applications that can be used to organize, analyze, and / or store various forms of data. Alternatively, computing system 1500 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., creating, managing, editing spreadsheet documents, word processing documents, and / or any other objects, etc.), computing functions, communication functions, etc. Applications can include various add-in functions or can be stand-alone computing products or functions. When active within an application, the functions can be used to generate a user interface that is provided via input / output devices 1540. The user interface can be generated by computing system 1500 and presented to a user (e.g., on a computer screen monitor, etc.).

[0151]

[0170] One or more aspects or features of the subject matter described herein may be implemented in digital electronic circuitry, integrated circuits, specially designed ASICs, field programmable gate array (FPGA) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0152]

[0171] These computer programs, which may also be referred to as programs, software, software applications, applications, components, or codes, include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal, such as, for example, a magnetic disk, optical disk, memory, and programmable logic device (PLD). The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily, such as, for example, a non-transitory solid-state memory or a magnetic hard drive or any equivalent storage medium. A machine-readable medium may alternatively or additionally store such machine instructions in a transitory manner, such as, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0153]

[0172] To provide for user interaction, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as, for example, a cathode ray tube (CRT) or liquid crystal display (LCD) or light emitting diode (LED) monitor, for displaying information to a user, and a keyboard and a pointing device, such as, for example, a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, voice, or tactile input. Other possible input devices include touch screens or other touch sensitive devices (such as single or multi-point resistive or capacitive trackpads), voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices, and associated interpretation software, and the like.

[0154]

[0173] In the above description and claims, phrases such as "at least one of" or "one or more of" may be preceded by a conjunctive list of elements or features. The term "and / or" may also be used in a list of two or more elements or features. Unless otherwise stated, implicitly or explicitly, by the context in which it is used, such phrases are intended to mean any of the listed elements or features individually, or any of the listed elements or features in combination with any of the other listed elements or features. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are each intended to mean "A only, B only, or A and B together." A similar interpretation is intended for lists containing more than two items. For example, the phrases "at least one of A, B, C," "one or more of A, B, C," and "A, B, and / or C" are each intended to mean "A only, B only, C only, A and B together, A and C together, B and C together, or A, B and C together." Use of the term "based on" above and in the claims means "based at least in part on" and means that unrecited features or elements are permitted.

[0155]

[0174] The subject matter described herein may be embodied in a system, an apparatus, a method, and / or an article, depending on the desired configuration. The embodiments described in the above description do not represent all embodiments according to the subject matter described herein. Instead, the embodiments described in the above description are merely some examples according to aspects related to the described subject matter. Although several variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to those described herein. For example, the embodiments described above may be directed to various combinations and subcombinations of the disclosed features, and / or combinations and subcombinations of several additional features disclosed above. In addition, the logic flow depicted in the accompanying figures and / or described herein does not necessarily require the particular order shown or sequential order to achieve the desired results. Other embodiments may be within the scope of the following claims.

[0156] VII. Definitions and Context Examples

[0175] The present disclosure is not limited to these exemplary embodiments and applications or the manner in which the exemplary embodiments and applications operate or are described herein. Further, the figures may depict simplified or partial views, and dimensions of elements in the figures may be exaggerated or not to scale.

[0157]

[0176] When reference is made to a list of elements (e.g., elements a, b, c), such reference is intended to include any one of the listed elements alone, any combination of fewer than all of the listed elements, and / or all combinations of the listed elements. The section divisions herein are for ease of viewing only and do not limit the combinations of elements being described.

[0158]

[0177] The term "subject" may refer to a subject of a clinical trial, a person or animal undergoing treatment, a person or animal undergoing anti-cancer therapy, a person or animal being monitored for remission or recovery, a person or animal undergoing preventative health analysis (e.g., due to its medical history), or any other person or patient or animal of interest. In various instances, "subject" and "patient" may be used interchangeably herein.

[0159]

[0178] The term "OCT image" may refer to an image of a tissue, organ, etc., such as the retina, scanned or captured using optical coherence tomography (OCT) imaging technology. The term may refer to one or both of 2D "slice" images and 3D "volume" images. Unless expressly indicated, the term may be understood to include OCT volume images.

[0160]

[0179] Unless otherwise defined, scientific and technical terms used in connection with the present teachings described herein shall have the meanings commonly understood by those skilled in the art.Furthermore, unless otherwise required by context, singular terms shall include the plural and plural terms shall include the singular.In general, the nomenclature and techniques utilized in connection with chemistry, biochemistry, molecular biology, pharmacology and toxicology described herein are those well known and commonly used in the art.

[0161]

[0180] As used herein, "substantially" means sufficient to function for the intended purpose. Thus, the term "substantially" allows for minor, insignificant variations from an absolute or perfect condition, dimension, measurement, result, etc., that would be expected by one of ordinary skill in the art, but that do not significantly affect overall performance. When used in reference to a numerical value, or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within 10 percent.

[0162]

[0181] As used herein, the term "about" when used with respect to a numerical value or a parameter or characteristic that can be expressed as a numerical value means within 10% of the numerical value. For example, "about 50" means a value in the range of 45 to 55.

[0163]

[0182] The term "plural ones" means two or more.

[0164]

[0183] As used herein, the term "plurality" can be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.

[0165]

[0184] As used herein, the term "set" means one or more. For example, a set of items includes one or more items.

[0166]

[0185] As used herein, the phrase "at least one of," when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. An item may be a specific object, thing, step, operation, process, or category. In other words, "at least one of" means that any combination or number of items from the list may be used, but not all of the items in the list are required. For example, but not limited to, "at least one of item A, item B, or item C" means item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and item C. In some cases, "at least one of item A, item B, or item C" means, but is not limited to, two items A, one item B, and ten items C; four items B, and seven items C; or other suitable combinations.

[0167]

[0186] As used herein, a "model" may include one or more algorithms, one or more mathematical techniques, one or more machine learning (ML) algorithms, or a combination thereof.

[0168]

[0187] As used herein, "machine learning" can include the practice of using algorithms to analyze data, learn from it, and then make decisions or predictions about something in the world. Machine learning uses algorithms that can learn from data without relying on rule-based programming.

[0169]

[0188] As used herein, "artificial neural network" or "neural network" may refer to a mathematical algorithm or computational model that mimics an interconnected group of artificial neurons that process information based on a connectionist approach to computation. A neural network, also called a neural net, may use one or more layers of nonlinear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. In various embodiments, a reference to a "neural network" may be a reference to one or more neural networks.

[0170]

[0189] Neural networks, for example, can process information in two ways: they are in training mode when they are being trained (e.g., using a training data set) and they are in inference (or prediction) mode when they put into practice what they have learned (e.g., using a test data set). Neural networks can learn through a feedback process (e.g., backpropagation) that allows the network to adjust the weight coefficients of individual nodes in the intermediate hidden layers (modify their behavior) so that their outputs match those of the training data. In other words, neural networks can learn by being fed training data (training examples), and eventually learn how to arrive at the correct output even when presented with a new range or set of inputs.

[0171] VIII. Description of Example Embodiments

[0190] Embodiment 1: A system comprising at least one data processor and at least one memory storing instructions, which when executed by the at least one data processor, performs operations including applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient, determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with the diagnosis of nascent geographic atrophy, and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0172]

[0191] Embodiment 2: The system of embodiment 1, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a supra-threshold contribution to the diagnosis of nascent geographic atrophy.

[0173]

[0192] Embodiment 3: The system of embodiment 1 or embodiment 2, wherein the saliency map is generated by applying gradient weighted class activation mapping (GradCAM).

[0174]

[0193] Embodiment 4: The system of any one of embodiments 1-3, wherein the saliency map comprises a heat map.

[0175]

[0194] Embodiment 5: The system of any one of embodiments 1-4, wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

[0176]

[0195] Embodiment 6: The system of any one of embodiments 1 to 5, wherein the machine learning model includes a residual neural network (RNN) based classifier.

[0177]

[0196] Embodiment 7: The system of any one of embodiments 1-6, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-scans.

[0178]

[0197] Embodiment 8: The system of any one of embodiments 1 to 7, wherein the machine learning model is trained based on a dataset including a plurality of optical coherence tomography (OCT) volumes annotated with per-volume labels.

[0179]

[0198] Embodiment 9: The system of any one of embodiments 1 to 8, wherein the location of the one or more nascent geographic atrophy lesions is identified by one or more bounding boxes.

[0180]

[0199] Embodiment 10: The system of any one of embodiments 1 to 9, wherein the operations further include generating a user interface displaying an indication of the location of one or more nascent geographic atrophy lesions on the patient's optical coherence tomography volume, and verifying a diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions based on one or more user inputs received via the user interface.

[0181]

[0200] Embodiment 11: A computer-implemented method comprising: applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with the diagnosis of nascent geographic atrophy; and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0182]

[0201] Embodiment 12: The method of embodiment 11, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a supra-threshold contribution to the diagnosis of nascent geographic atrophy.

[0183]

[0202] Embodiment 13: The method of embodiment 11 or embodiment 12, wherein the saliency map is generated by applying Gradient Weighted Class Activation Mapping (GradCAM).

[0184]

[0203] Embodiment 14: The method of any one of embodiments 11 to 13, wherein the saliency map comprises a heat map.

[0185]

[0204] Embodiment 15: The method of any one of embodiments 11 to 14, wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

[0186]

[0205] Embodiment 16: The method of any one of embodiments 11 to 15, wherein the machine learning model includes a residual neural network (RNN) based classifier.

[0187]

[0206] Embodiment 17: The method of any one of embodiments 11 to 16, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-scans.

[0188]

[0207] Embodiment 18: The method of any one of embodiments 11 to 17, wherein the machine learning model is trained based on a dataset including a plurality of optical coherence tomography (OCT) volumes annotated with per-volume labels.

[0189]

[0208] Embodiment 19: The method of any one of embodiments 11 to 18, wherein the location of the one or more nascent geographic atrophy lesions is identified by one or more bounding boxes.

[0190]

[0209] Embodiment 20: The method of any one of embodiments 11-19, wherein the operations further include generating a user interface displaying a representation of the location of the one or more nascent geographic atrophy lesions on the patient's optical coherence tomography volume, and verifying a diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions based on one or more user inputs received via the user interface.

[0191]

[0210] Embodiment 21: A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, perform operations including applying a machine learning model trained to determine a diagnosis of nascent geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient, determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with a diagnosis of nascent geographic atrophy, and verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions.

[0192]

[0211] Embodiment 22. A method comprising: receiving an optical coherence tomography (OCT) volumetric image of a subject's retina; generating, via a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image; and generating a map output for the deep learning model using a saliency mapping algorithm, the map output indicating a contribution of a set of regions within the OCT volumetric image to an output generated by the deep learning model.

[0193]

[0212] Embodiment 23. The method of embodiment 22, wherein the saliency mapping algorithm includes a gradient weighted class activation mapping (GradCAM) algorithm, and the map output visually indicates the contribution of a set of regions within the OCT volumetric image to the output generated by the deep learning model.

[0194]

[0213] Embodiment 24. The method of embodiment 22 or embodiment 23, wherein the OCT volumetric image includes a plurality of OCT slice images that are two-dimensional, and the method further includes generating an evaluation recommendation based on at least one of the output or the map output, the evaluation recommendation identifying a subset of the plurality of OCT slice images for further review.

[0195]

[0214] Embodiment 25. The method of embodiment 24, wherein the subset comprises less than 5% of the plurality of OCT slice images.

[0196]

[0215] Embodiment 26. The method of any one of embodiments 22 to 25, further comprising displaying a map output, the map output including a saliency map overlaid on individual OCT slice images of the OCT volumetric image and a bounding box around at least one region of the set of regions.

[0197]

[0216] Embodiment 27. The method of embodiment 26, wherein the identifying comprises identifying a potential biomarker region in association with a region of the set of regions as being associated with nascent geographic atrophy, generating a scoring metric for the potential biomarker region, and identifying the biomarker region as containing at least one biomarker for a selected diagnosis of nascent geographic atrophy if the scoring metric meets a selected threshold.

[0198]

[0217] Embodiment 28. The method of embodiment 27, wherein the scoring metric comprises at least one of a size of the potential biomarker region or a confidence score of the potential biomarker region.

[0199]

[0218] Embodiment 29. The method of any one of embodiments 22 to 28, wherein generating the map output includes generating a saliency map for an OCT slice image of the OCT volumetric image using a saliency mapping algorithm, the saliency map indicating the importance of each pixel in the OCT slice image with respect to diagnosing nascent geographic atrophy; filtering the saliency map to generate a modified saliency map; and overlaying the modified saliency map on the OCT slice image to generate the map output.

[0200]

[0219] Embodiment 30. The method of any one of embodiments 22 to 28, wherein generating an output via a deep learning model includes generating an initial output for each OCT slice image of a plurality of OCT slice images forming the OCT volumetric image to form a plurality of initial outputs, and averaging the plurality of initial outputs to form a health indication output.

[0201]

[0220] EMBODIMENT 31. A non-transient memory; 1. A system comprising: a hardware processor coupled to a non-transitory memory, the hardware processor configured to read instructions from the non-transitory memory to cause the system to: receive an optical coherence tomography (OCT) volumetric image of a subject's retina; generate, via a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image; generate a map output for the deep learning model using a saliency mapping algorithm, the map output indicating a contribution of a set of regions within the OCT volumetric image to an output generated by the deep learning model; and display the map output.

[0202]

[0221] Embodiment 32. The method of embodiment 31, wherein the map output includes a saliency map overlaid on each individual OCT slice image of the OCT volumetric image.

[0203]

[0222] Embodiment 33. The system of embodiment 31 or embodiment 32, wherein the saliency mapping algorithm comprises a gradient weighted class activation mapping (GradCAM) algorithm.

[0204]

[0223] Embodiment 34. The system of any one of embodiments 31 to 33, wherein the deep learning model includes a residual neural network.

[0205]

[0224] EMBODIMENT 35. A non-transient memory; and a hardware processor coupled to a non-transitory memory, the hardware processor reading instructions from the non-transitory memory to cause the system to: train a deep learning model using a training dataset including training OCT images labeled as being evidence of nascent geographic atrophy or not evidence of nascent geographic atrophy to form a trained deep learning model; receive optical coherence tomography (OCT) volumetric images of a subject's retina; and generate, via the trained deep learning model, a classification score using the OCT volumetric images, the classification score indicating whether nascent geographic atrophy is detected. , generating a saliency volume map for the OCT volume image using a saliency mapping algorithm, the saliency volume map indicating the contribution of a set of regions in the OCT volume image to a diagnosis of geographic atrophy generated by the deep learning model; detecting a set of potential biomarker regions in the OCT volume image using the saliency volume map; and generating a report confirming that nascent geographic atrophy has been detected when at least one potential biomarker region of the set of potential biomarker regions meets a set of criteria and the classification score meets a threshold.

[0206]

[0225] Embodiment 36. The system of embodiment 35, wherein the saliency mapping algorithm comprises a gradient weighted class activation mapping (GradCAM) algorithm.

[0207]

[0226] Embodiment 37. The method of embodiment 35 or embodiment 36, wherein the classification score is the probability that the OCT volumetric image is evidence of nascent geographic atrophy, and the threshold is a value selected between 0.5 and 0.8.

[0208]

[0227] Embodiment 38. The system of any one of embodiments 35 to 37, wherein the OCT volumetric image includes a plurality of OCT slice images that are two-dimensional, and the hardware processor is further configured to read instructions from the non-transitory memory to cause the system to generate an evaluation recommendation based on at least one of the health indication output or the map output, and the evaluation recommendation identifies a subset of the plurality of OCT slice images for further review, the subset including less than 5% of the plurality of OCT slice images.

[0209] IX. Additional Considerations

[0228] While the present teachings have been described in conjunction with various embodiments, it is not intended that the present teachings be limited to such embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those skilled in the art.

[0210]

[0229] In describing various embodiments, the specification may present a method and / or process as a particular order of steps. However, to the extent that the method or process does not depend on the particular order of steps described herein, the method or process should not be limited to the particular order of steps described, and one of ordinary skill in the art can readily appreciate that the order may be altered and still be within the spirit and scope of the various embodiments.

Claims

1. at least one data processor; at least one memory storing instructions; wherein the instructions, when executed by the at least one data processor, applying a machine learning model trained to determine a diagnosis of newborn geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with said diagnosis of nascent geographic atrophy; verifying said diagnosis of neonatal geographic atrophy and / or said location of said one or more neonatal geographic atrophy lesions; The system on which operations, including

2. The system of claim 1 , wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a supra-threshold contribution to the diagnosis of nascent geographic atrophy.

3. The system of claim 1 , wherein the saliency map is generated by applying gradient weighted class activation mapping (GradCAM).

4. The system of claim 1 , wherein the saliency map comprises a heat map.

5. The system of claim 1 , wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

6. The system of claim 1 , wherein the machine learning model comprises a residual neural network (RNN) based classifier.

7. The system of claim 1 , wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-scans.

8. 10. The system of claim 1, wherein the machine learning model is trained based on a dataset including a plurality of optical coherence tomography (OCT) volumes annotated with per-volume labels.

9. The system of claim 1 , wherein the locations of the one or more nascent geographic atrophy lesions are identified by one or more bounding boxes.

10. The operation is generating a user interface displaying a representation of the location of the one or more nascent geographic atrophy lesions on the optical coherence tomography volume of the patient; verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions based on one or more user inputs received via the user interface; The system of claim 1 further comprising:

11. applying a machine learning model trained to determine a diagnosis of newborn geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with said diagnosis of nascent geographic atrophy; verifying said diagnosis of neonatal geographic atrophy and / or said location of said one or more neonatal geographic atrophy lesions; 23. A computer-implemented method comprising:

12. 12. The method of claim 11, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a supra-threshold contribution to the diagnosis of nascent geographic atrophy.

13. The method of claim 11 , wherein the saliency map is generated by applying gradient weighted class activation mapping (GradCAM).

14. The method of claim 11 , wherein the saliency map comprises a heat map.

15. The method of claim 11 , wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

16. The method of claim 11 , wherein the machine learning model comprises a residual neural network (RNN) based classifier.

17. The method of claim 11 , wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-scans.

18. 12. The method of claim 11, wherein the machine learning model is trained based on a dataset comprising a plurality of optical coherence tomography (OCT) volumes annotated with volumetric labels.

19. 12. The method of claim 11, wherein the locations of the one or more nascent geographic atrophy lesions are identified by one or more bounding boxes.

20. The operation is generating a user interface displaying a representation of the location of the one or more nascent geographic atrophy lesions on the optical coherence tomography volume of the patient; verifying the diagnosis of nascent geographic atrophy and / or the location of the one or more nascent geographic atrophy lesions based on one or more user inputs received via the user interface; The method of claim 11 further comprising:

21. A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, applying a machine learning model trained to determine a diagnosis of newborn geographic atrophy (nGA) in a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining a location of one or more nascent geographic atrophy lesions based at least on a saliency map associated with said diagnosis of nascent geographic atrophy; verifying said diagnosis of neonatal geographic atrophy and / or said location of said one or more neonatal geographic atrophy lesions; A non-transitory computer-readable medium on which operations including

22. Receiving an optical coherence tomography (OCT) volumetric image of a retina of a subject; generating, by a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image; generating a map output for the deep learning model using a saliency mapping algorithm, the map output indicating a contribution of a set of regions within the OCT volumetric image to the output generated by the deep learning model; The method includes:

23. 23. The method of claim 22, wherein the saliency mapping algorithm comprises a gradient weighted class activation mapping (GradCAM) algorithm, and the map output visually indicates the contribution of the set of regions within the OCT volumetric image to the output generated by the deep learning model.

24. the OCT volumetric image includes a plurality of OCT slice images that are two-dimensional; 24. The method of claim 22 or 23, wherein the method further comprises generating an evaluation recommendation based on at least one of the output or the map output, the evaluation recommendation identifying a subset of the plurality of OCT slice images for further review.

25. 25. The method of claim 24, wherein the subset comprises less than 5% of the plurality of OCT slice images.

26. 26. The method of claim 22, further comprising displaying the map output, the map output comprising a saliency map overlaid on individual OCT slice images of the OCT volumetric image and a bounding box around at least one region of the set of regions.

27. To identify, identifying potential biomarker regions associated with regions of said set of regions as being associated with said nascent geographic atrophy; generating a scoring metric for said potential biomarker regions; identifying a biomarker region as containing at least one biomarker for a selected diagnosis of newborn geographic atrophy if the scoring metric meets a selected threshold; 27. The method of claim 26, comprising:

28. 28. The method of claim 27, wherein the scoring metric comprises at least one of a size of the potential biomarker region or a confidence score of the potential biomarker region.

29. Generating the map output includes: generating a saliency map for an OCT slice image of the OCT volumetric image using the saliency mapping algorithm, the saliency map indicating the importance of each pixel in the OCT slice image with respect to diagnosing neo-geographic atrophy; filtering the saliency map to generate a modified saliency map; overlaying the modified saliency map onto the OCT slice images to generate the map output; 29. The method of any one of claims 22 to 28, comprising:

30. Generating the output by the deep learning model includes: generating an initial output for each OCT slice image of a plurality of OCT slice images forming the OCT volumetric image to form a plurality of initial outputs; averaging the initial outputs to form a health-indicating output; 29. The method of any one of claims 22 to 28, comprising:

31. A non-transient memory; a hardware processor coupled to the non-transitory memory; wherein the hardware processor reads instructions from the non-transitory memory to provide the system with: Receiving an optical coherence tomography (OCT) volumetric image of a retina of a subject; generating, by a deep learning model, an output indicating whether nascent geographic atrophy is detected using the OCT volumetric image; generating a map output for the deep learning model using a saliency mapping algorithm, the map output indicating a contribution of a set of regions within the OCT volumetric image to the output generated by the deep learning model; displaying the map output; and A system configured to:

32. The method of claim 31 , wherein the map output comprises a saliency map overlaid on each OCT slice image of the OCT volumetric image.

33. 33. The system of claim 31 or 32, wherein the saliency mapping algorithm comprises a gradient weighted class activation mapping (GradCAM) algorithm.

34. 34. The system of claim 31, wherein the deep learning model comprises a residual neural network.

35. A non-transient memory; a hardware processor coupled to the non-transitory memory; wherein the hardware processor reads instructions from the non-transitory memory to provide the system with: training a deep learning model using a training dataset that includes training OCT images labeled as either evidencing nascent geographic atrophy or not evidencing nascent geographic atrophy to form a trained deep learning model; Receiving an optical coherence tomography (OCT) volumetric image of a retina of a subject; generating, by the trained deep learning model, a classification score indicating whether or not nascent geographic atrophy is detected, using the OCT volumetric images; generating a saliency volume map for the OCT volumetric image using a saliency mapping algorithm, the saliency volume map indicating the contribution of a set of regions within the OCT volumetric image to a diagnosis of geographic atrophy generated by the deep learning model; using the saliency volume map to detect a set of potential biomarker regions within the OCT volumetric image; generating a report confirming that nascent geographic atrophy was detected when at least one potential biomarker region of said set of potential biomarker regions meets a set of criteria and said classification score meets a threshold; A system configured to:

36. 36. The system of claim 35, wherein the saliency mapping algorithm comprises a gradient weighted class activation mapping (GradCAM) algorithm.

37. 37. The method of claim 35 or 36, wherein the classification score is the probability that the OCT volumetric image evidences nascent geographic atrophy, and the threshold is a value selected between 0.5 and 0.

8.

38. 38. The system of claim 35, wherein the OCT volumetric image includes a plurality of OCT slice images that are two-dimensional, and the hardware processor is further configured to read instructions from the non-transitory memory to cause the system to generate an evaluation recommendation based on at least one of a health indication output or a map output, the evaluation recommendation identifying a subset of the plurality of OCT slice images for further review, the subset including less than 5% of the plurality of OCT slice images.

Images