Machine learning enabled diagnosis and lesion localization for neonatal map-like atrophy in age-related macular degeneration

Abstract

A method and system for detecting neonatal map atrophy. An optical coherence tomography (OCT) volumetric image of the retina of a subject is received. An output is generated using the OCT volume image using a deep learning model, where the output indicates whether a newborn map atrophy is detected. A map output for the deep learning model is generated using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volume image to an output generated by the deep learning model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to and claims the benefit of the priority date of U.S. Provisional Application No. 63 / 339,333, filed on March 24, 2023, entitled “Machine Learning-Enabled Diagnosis and Lesion Localization for Newly Degenerated Geographic Atrophy in Age-Related Macular Degeneration” and U.S. Provisional Application No. 63 / 484,150, filed on February 9, 2023, entitled “Machine Learning-Enabled Diagnosis and Lesion Localization for Newly Degenerated Geographic Atrophy in Age-Related Macular Degeneration”, which is a continuation-in-part application of International Application No. PCT / US22 / 47944, filed on October 26, 2022, entitled “Methods and Systems for Biomarker Identification and Discovery”, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter described herein relates generally to machine learning, and more specifically to machine learning-based diagnosis and lesion localization techniques for neonatal geographic atrophy (nGA) in age-related macular degeneration (AMD). Background Art

[0004] Various imaging techniques have been developed to capture medical images of tissue, which can then be analyzed to determine the presence or progression of disease. For example, optical coherence tomography (OCT) refers to a technique in which light waves are used to capture two-dimensional slice images and three-dimensional volumetric images of a patient's tissue (such as the retina), which can then be analyzed to diagnose, monitor, treat, etc. the patient. However, analysis of such images, which can contain large amounts of data, is typically performed manually by subject matter experts and, therefore, can be tedious and very expensive. Therefore, it may be desirable to have methods and systems that facilitate consistent, accurate, and rapid analysis of large numbers of medical images, such as OCT images, for use in diagnosing, monitoring, and treating patients. Summary of the invention

[0005] The following content summarizes some embodiments of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an extensive review of all expected features of the present disclosure, and is neither intended to identify the key or important elements of all embodiments of the present disclosure, nor is it intended to delineate the scope of any or all embodiments of the present disclosure. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in an overview form as a prelude to a more detailed description presented later.

[0006] In one or more embodiments, a system includes: at least one data processor; and at least one memory storing instructions that, when executed by the at least one data processor, cause operations including: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on a significance map associated with the diagnosis of neonatal geographic atrophy; and verifying the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0007] In one or more embodiments, a computer-implemented method is provided. The method includes: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining a location of one or more neonatal geographic atrophy lesions based at least on a saliency map associated with the diagnosis of neonatal geographic atrophy; and validating the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0008] In one or more embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one data processor, result in operations including: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on a significance map associated with the diagnosis of neonatal geographic atrophy; and verifying the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0009] In one or more embodiments, a method is provided. The method includes: receiving an optical coherence tomography (OCT) volume image of a subject's retina; generating an output using the OCT volume image via a deep learning model, wherein the output indicates whether new geographic atrophy is detected; and generating a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volume image to the output generated by the deep learning model.

[0010] In one or more embodiments, a system includes: a non-volatile memory; and a hardware processor coupled to the non-volatile memory and configured to read instructions from the non-volatile memory to cause the system to: receive an optical coherence tomography (OCT) volumetric image of a subject's retina; generate an output using the OCT volumetric image via a deep learning model, wherein the output indicates whether neoplastic geographic atrophy is detected; generate a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volumetric image to an output generated by the deep learning model; and display the map output.

[0011] In one or more embodiments, a system includes: a non-volatile memory; and a hardware processor coupled to the non-volatile memory and configured to read instructions from the non-volatile memory so that the system: trains a deep learning model using a training data set to form a trained deep learning model, the training data set including training OCT images that have been labeled as demonstrating neonatal geographic atrophy or not demonstrating neonatal geographic atrophy; receives an optical coherence tomography (OCT) volumetric image of a subject's retina; generates a classification score using the OCT volumetric image via the trained deep learning model, wherein the classification score indicates whether neonatal geographic atrophy is detected; generates a saliency volume map for the OCT volumetric image using a saliency mapping algorithm, wherein the saliency volume map indicates a level of contribution of a set of regions in the OCT volumetric image to a diagnosis of the geographic atrophy generated by the deep learning model; uses the saliency volume map to detect a set of potential biomarker regions in the OCT volumetric image; and generates a report, the report confirming detection of neonatal geographic atrophy when at least one potential biomarker region in the set of potential biomarker regions meets a set of criteria and when the classification score meets a threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some principles associated with the disclosed embodiments.

[0013] Figure 1 is a block diagram of a networked system 100 according to one or more exemplary embodiments.

[0014] Figure 2 Flowchart of a process for processing an OCT volume image of a subject's retina to determine whether the OCT volume image evidences a selected health condition category for the retina in accordance with one or more exemplary embodiments.

[0015] Figure 3 is a flow chart of a process for identifying biomarkers in an OCT volume image of a subject's retina according to one or more exemplary embodiments.

[0016] Figure 4A is a flow chart of a process 400 for artificial intelligence-assisted neonatal geographic atrophy (nGA) detection according to one or more exemplary embodiments.

[0017] Figure 4B Flowchart of a process for processing an OCT volume image of a subject's retina to determine whether the OCT volume image demonstrates neonatal geographic atrophy (nGA) according to one or more exemplary embodiments.

[0018] Figure 5 Annotated OCT slice images and corresponding heat maps of the annotated OCT slice images according to one or more exemplary embodiments are shown.

[0019] Figure 6 are illustrations of various diagrams according to one or more exemplary embodiments.

[0020] Figure 7 Depicted is a system diagram illustrating an example of a newborn geographic atrophy detection system in accordance with some exemplary embodiments.

[0021] Fig. 8A is an illustration of an example of a model for processing a 3D OCT volume according to one or more exemplary embodiments.

[0022] Figure 8B An example of an implementation of a classifier 802 is shown, which may be used to implement a classifier according to one or more exemplary embodiments.

[0023] Fig. 9 Depicted is an exemplary data flow diagram with data segmentation statistics in accordance with one or more exemplary embodiments.

[0024] Fig.10 is an illustration of an output workflow for output generated from an OCT volume according to one or more exemplary embodiments.

[0025] Fig.11A is a diagram of a confusion matrix 1100 according to one or more exemplary embodiments.

[0026] Fig. 11B Statistical graph of 5-fold cross validation according to one or more exemplary embodiments.

[0027] Fig. 12Ais an illustration of an OCT image 1200 (eg, a B-scan) in which an nGA lesion has been detected according to one or more exemplary embodiments.

[0028] Fig. 12B Graph 1202 is a precision-recall curve for 5-fold cross validation according to one or more exemplary embodiments.

[0029] Fig. 12C is a graphical representation of a confusion matrix 1204 according to one or more exemplary embodiments.

[0030] Fig.13 is an illustration of an OCT image 1300 that has been annotated with a bounding box according to one or more exemplary embodiments.

[0031] Fig.14 Exemplary neural networks that may be used to implement computer-based models according to various embodiments of the present disclosure are presented.

[0032] Fig.15 Depicted is a block diagram illustrating an example of a computing system in accordance with some example embodiments.

[0033] When applicable, like reference numerals refer to like structures, features, or elements.

[0034] It should be understood that the drawings are not necessarily drawn to scale, nor are the objects in the drawings necessarily drawn to scale relative to each other. The drawings are depictions of various embodiments intended for clarity and understanding of the devices, systems, and methods disclosed herein. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. In addition, it should be understood that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION

[0035] I. Overview

[0036] Medical imaging technology is a powerful tool that can be used to produce medical images, which enable health care practitioners to better visualize and understand the medical problems of their patients, and therefore provide the same more accurate diagnosis and treatment options. For example, optical coherence tomography (OCT) is a non-invasive imaging technology that is particularly popular for capturing images of the retina. OCT can be described as an ultrasonic scanning technology, and this technology scatters the light waves from tissue to generate OCT images in the form of two-dimensional (2D) images and / or three-dimensional (3D) images of tissue, similar to the ultrasonic scanning of tissue using sound waves to scan. 2D OCT images can also be referred to as OCT slices, OCT cross-sectional images or OCT scans (e.g., OCT B scans). 3D OCT images can be referred to as OCT volume images, and can be composed of many OCT slice images. OCT images can then be used to diagnose, monitor and / or treat the patient from which the image is obtained. For example, OCT slice images and OCT volume images of the retina of the patient suffering from age-related macular degeneration (AMD) can be analyzed to provide AMD diagnosis and treatment options to the patient.

[0037] Although the OCT images of the retina may contain valuable information about the ophthalmic condition of the patient, extracting information from the OCT images may be a resource-intensive and difficult task, resulting in erroneous conclusions about the information contained in the OCT images. For example, when treating a patient suffering from an ocular disease such as AMD, a large group of OCT slices of the patient's retina may be obtained, and a group of trained human examiners may be assigned to manually identify the biomarkers of AMD in a group of OCT slices. However, such a process may be cumbersome and challenging, resulting in slow, inaccurate and / or variable identification of biomarkers of retinal diseases. Although trained subject matter experts can be used to improve the accuracy of biomarker identification when checking OCT images, the process may still be laborious, may have inherent undesirable variability between examiners, and may be particularly expensive. Therefore, relying on such subject matter experts to check such large groups of OCT slices may not provide the required efficient, economical, consistent and accurate mechanisms for biomarkers of identification diseases (such as AMD) for health care providers. In addition, manually checking OCT images may be less successful when finding new biomarkers as a prognosis for future development of retinal diseases.

[0038] For example, geographic atrophy (GA) may be a late-stage visually threatening complication of AMD. Although color fundus photography (CFP) or fundus autofluorescence (FAF) can be used to identify GA, when subject matter experts can see evidence of GA on these types of images, there may be a large amount of loss in outer retinal tissue. Determining the mode of slowing down or preventing GA attack in the early stages of AMD may benefit from identifying early signs or predictive factors of GA attack. For example, the early identification and / or predicted biomarkers for GA attack can be used to identify high-risk individuals to enrich clinical trial populations, used as biomarkers in different stages of AMD progression, and / or potentially serve as the early endpoints of clinical trials intended to prevent GA attack.

[0039] OCT images have been used to identify newborn map atrophy (newborn GA or nGA), which may be a strong predictor of GA's imminent onset (e.g., within 6 to 30 months). The optical coherence tomography (OCT) signs of newborn map atrophy (nGA) associated with the onset of map atrophy help enrich the trial inclusion criteria. For example, in some cases, the risk of GA in the retina showing newborn GA in OCT images has increased by more than 70 times compared to those retinas that do not show newborn GA. Therefore, newborn GA may be a prognostic indicator for the progression of GA in early AMD. The examples of anatomical biomarkers defining newborn GA in OCT images include, but are not limited to, the sinking of the inner nuclear layer (INL) and the outer plexiform layer (OPL), the low-reflection wedge-shaped band in the Henle fiber layer, or both.

[0040] Accurate identification of de novo GA may improve the feasibility of evaluating preventive treatments for GA onset. Manual grading of all B-scans in an OCT volume can be a laborious and costly task, especially as B-scan density increases to improve macular coverage. Improving the speed and reducing the computational cost of the process would improve the utility of OCT biomarkers or endpoints.

[0041] Therefore, the embodiments described herein provide artificial intelligence (AI)-based systems and methods for quickly, efficiently and accurately detecting whether an OCT volume image of the retina demonstrates a selected health status category for the retina. The selected health status category may be, for example, a retinal disease (e.g., AMD) or a stage of a retinal disease. In one or more embodiments, the selected health status category is a new GA. In other embodiments, the selected health status category may be another stage of AMD progression (e.g., early AMD, intermediate AMD, GA, etc.). A deep learning model may be trained to receive an OCT volume image and generate a health indication output, which indicates whether the OCT volume image demonstrates a selected health status category for the retina (e.g., new GA). For example, the health indication output may indicate the level of association between the OCT volume image and the selected health status category. This level of association may be unassociated, partially associated, or fully associated. For example, a deep learning model may include a neural network model. As a non-limiting example, a deep learning model may generate a health indication output, which is a probability (e.g., between 0.00 and 1.00) indicating the level of association between the OCT volume image and the selected health status category.

[0042] In addition, the systems and methods described herein can be used to quickly, efficiently, and accurately identify biomarkers of retinal disease and / or prognostic biomarkers for future development of retinal disease. For example, the systems and methods described herein can be used to identify a set of biomarkers in an OCT volume image that indicate or otherwise correspond to a selected health condition category. The systems and methods can also be used to identify a set of prognostic biomarkers in an OCT volume image that are prognostic for a selected health condition category (e.g., progression to a selected health condition category within a selected time period).

[0043] In one or more embodiments, an OCT volume image is processed using a health status identification system including a deep learning model. The health identification system uses a deep learning model (which may include a neural network model) to generate a health indication output that indicates whether the OCT volume image demonstrates a selected health status category. In some cases, the selected health status category may be one of a group of health status categories of interest. In one or more embodiments, the selected health status category is a selected stage of AMD. The selected stage of AMD may be, for example, nascent GA.

[0044] In one or more embodiments, the health status identification system uses a saliency mapping algorithm (also referred to as a saliency mapping technique) to generate a graph output for a deep learning model, the graph output indicating whether a set of regions in an OCT volume image is associated with a selected health status category. The saliency mapping algorithm can be used to identify the level of contribution (or importance) of various parts of the OCT volume image to the health indication output generated by the deep learning model for a given OCT volume image. The health status identification system can use the graph output to identify biomarkers in the OCT volume image. The biomarker can indicate that the OCT volume image currently demonstrates a selected health status category for the retina. In some cases, the biomarker can be prognostic because it indicates that the OCT volume image is a prognosis for the retina to progress to a selected health status category within a selected time period (e.g., 6 months, 1 year, 2 years, 3 years, etc.).

[0045] The above-described saliency mapping algorithm can be implemented in various ways. An example of a saliency mapping algorithm is Gradient Weighted Class Activation Mapping (Grad-CAM), a technique that provides a "visual explanation" in the form of a heat map for the decisions made by a deep learning model when performing predictions. That is, Grad-CAM can be implemented for a trained deep learning model to generate a saliency map or heat map of an OCT slice image, where the heat map indicates (e.g., using color, outlines, annotations, etc.) areas or locations of the OCT slice image that the neural network model uses to determine and / or predict the disease stage of the retina shown in the OCT slice image. In one or more embodiments, Grad-CAM can determine the importance of each pixel in the OCT slice image to the health indication output generated by the deep learning model. More details about Grad-CAM can be found in RR Selvaraju et al., "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization," Arxiv:1610.02391 (2017), the entire contents of which are incorporated herein by reference. Other non-limiting examples of saliency mapping techniques include Class Activation Mapping (CAM), SmoothGrad, Low Variance Gradient Estimator for Variational Inference (VarGrad), and / or similar techniques.

[0046] Then, the saliency map generated by the saliency mapping algorithm can be used to locate one or more potential biomarkers on a given OCT slice image. For example, the saliency map can be used to generate a bounding box around each potential biomarker or potential biomarker region in the OCT slice image. Thus, each bounding box can locate a potential biomarker. In one or more embodiments, a scoring metric (e.g., a confidence score) can be used to determine which bounding boxes are or contain one or more biomarkers for a selected health condition category.

[0047] Using a health status identification system with a deep learning model and a saliency mapping algorithm to classify retinal health status and identify biomarkers for selected health status categories in OCT volume images can reduce the time and cost associated with evaluating a subject's retina, and can improve the efficiency and accuracy of achievable diagnosis, monitoring and / or treatment. In addition, using the embodiments described herein can allow subjects to join clinical trials at an earlier stage of their AMD progression and can increase the information potential of such clinical trials. Further, using the embodiments described herein can reduce the overall computing resources used and / or speed up the computer's performance in classifying retinal health status, predicting future retinal health status, and / or identifying biomarkers for selected health status categories.

[0048] In some exemplary embodiments, a deep learning model can be trained to detect newborn geographic atrophy based on optical coherence tomography imaging. The deep learning model can be trained based on information about the presence or absence of newborn geographic atrophy at eye level to effectively identify the location of these lesions. If such a diagnostic tool is used in clinical trials, diagnosis, treatment, monitoring, research, etc., the ability to locate newborn geographic atrophy may be crucial. In some cases, the diagnostic output of the deep learning model may be further validated or demonstrated in a clinical setting. For example, instead of presenting a diagnostic result and / or in addition to presenting a diagnostic result, the deep learning model can propose one or more areas that are highly likely to contain newborn geographic atrophy lesions. Therefore, the clinician can make a final diagnosis by examining only a subset of the B-scans, or even an area of ​​the B-scans. In this case, the deep learning model should have a high recall rate in locating newborn geographic atrophy lesions. Its accuracy should be much higher than the prevalence to minimize the workload of clinicians.

[0049] II. Exemplary Health Status Indicators

[0050] II.A. Exemplary System for Health Status Identification

[0051] Figure 11 is a block diagram of a networked system 100 according to one or more exemplary embodiments. The networked system 100 may include any number or combination of servers and / or software components that operate to perform various processes related to: capturing an OCT volume image of a tissue (such as a retina), processing the OCT volume image via a deep learning model, processing the OCT volume image using a saliency mapping algorithm, identifying a biomarker that indicates current retinal health or is prognostic of retinal health, or a combination thereof. Exemplary servers may include, for example, a server running an OS (such as MICROSOFT TM OS, UNIX TM OS、LINUX TM OS or other suitable server-based OS). It will be appreciated that the servers used in networked system 100 may be deployed in other ways, and for a given implementation, the operations performed and / or services provided by these servers may be combined or separated and may be performed by a greater or lesser number of servers. One or more servers may be operated and / or maintained by the same or different entities.

[0052] The networked system 100 includes a health status identification (HSI) system 101. The health status identification system 101 can be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, the health status identification system 101 may include a computing platform 102, a data storage 104 (e.g., a database, a server, a storage module, a cloud storage, etc.), and a display system 106. The computing platform 102 can take various forms. In one or more embodiments, the computing platform 102 includes a single computer (or computer system) or multiple computers that communicate with each other. In other examples, the computing platform 102 takes the form of a cloud computing platform, a mobile computing platform (e.g., a smart phone, a tablet computer, etc.), or a combination thereof.

[0053] Data storage 104 and display system 106 are each in communication with computing platform 102. In some examples, data storage 104, display system 106, or both may be considered part of or otherwise integral to computing platform 102. Thus, in some examples, computing platform 102, data storage 104, and display system 106 may be separate components that communicate with each other, but in other examples, some combination of these components may be integrated together.

[0054] The networked system 100 may further include an OCT imaging system 110, which may also be referred to as an OCT scanner. The OCT imaging system 110 may generate OCT imaging data 112. The OCT imaging data 112 may include an OCT volume image (i.e., a 3D OCT image) and / or an OCT slice image (i.e., a 2D OCT image). For example, the OCT imaging data 112 may include an OCT volume image 114. The OCT volume image 114 may be composed of a plurality of (e.g., 10s, 100s, 1000s, etc.) OCT slice images. The OCT slice image may also be referred to as an OCT B-scan or a cross-sectional OCT image.

[0055] In one or more embodiments, the OCT imaging system 110 includes an optical coherence tomography (OCT) system (e.g., an OCT scanner or machine) configured to generate OCT imaging data 112 of patient tissue. For example, the OCT imaging system 110 can be used to generate OCT imaging data 112 for a patient's retina. In some cases, the OCT system can be a large desktop configuration used in a clinical setting, a portable or handheld dedicated system, or a "smart" OCT system incorporated into a user's personal device (such as a smartphone). The OCT imaging system 110 can include an image denoiser configured to remove noise and other artifacts from a raw OCT volume image to generate an OCT volume image 114.

[0056] The health status identification system 101 can communicate with the OCT imaging system 110 via a network 120. The network 120 can be implemented using a single network or a combination of multiple networks. The network 120 can be implemented using any number of wired communication links, wireless communication links, optical communication links, or a combination thereof. For example, in various embodiments, the network 120 can include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. In another example, the network 120 can include a wireless telecommunications network (e.g., a cellular telephone network) suitable for communicating with other communication networks (such as the Internet).

[0057] The OCT imaging system 110 and the health status identification system 101 may each include one or more electronic processors, electronic memory, and other appropriate electronic components for executing instructions, such as program code and / or data stored on one or more computer-readable media, to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media such as memory or data storage devices (e.g., data storage 104) that are internal and / or external to various components of the networked system 100 and / or accessible via the network 120. Although only one of each of the OCT imaging system 110 and the health status identification system 101 is shown, more than one of each may be present in other embodiments.

[0058] In some embodiments, the OCT imaging system 110 may be maintained by an entity that is assigned responsibility for obtaining OCT imaging data 112 of a tissue sample of a subject for diagnosis, monitoring, treatment, research, clinical trials, and / or similar purposes. For example, the entity may be a healthcare provider (e.g., an ophthalmic healthcare provider) that seeks to obtain OCT imaging data of a patient's retina for diagnosis of an ocular condition or disease (e.g., AMD) that the patient may have. As another example, the entity may be an administrator of a clinical trial that is assigned responsibility for collecting OCT imaging data of a subject's retina to monitor retinal changes due to progression / regression of a disease affecting the retina and / or the effects of a drug administered to the subject to treat the disease. It should be noted that the above examples are non-limiting, and the OCT imaging system 110 may be maintained by other entities and / or professionals that are capable of using the OCT imaging system 110 to obtain OCT imaging data of the retina for the aforementioned or any other medical purposes.

[0059] In some embodiments, the health status identification system 101 may be maintained by an entity that is assigned responsibility for identifying or discovering biomarkers of a disease or condition in an OCT image. For example, the health status identification system 101 may be maintained by an ophthalmic healthcare provider, researcher, clinical trial manager, etc. that is assigned responsibility for identifying or discovering biomarkers of a retinal disease such as AMD. Figure 1 The OCT imaging system 110 and the health status identification system 101 are shown as two separate components, but in some embodiments, the OCT imaging system 110 and the health status identification system 101 can be part of the same system or module (e.g., and maintained by the same entity such as a healthcare provider or clinical trial manager).

[0060] The health status identification system 101 may include an image processor 130 configured to receive OCT imaging data 112 from the OCT imaging system 110. The image processor 130 may be implemented using hardware, firmware, software, or a combination thereof. In one or more embodiments, the image processor 130 may be implemented within the computing platform 102.

[0061] The image processor 130 may include a model 132 (which may also be referred to as a health condition model 132), a saliency mapping algorithm 134, and an output generator 136. The model 132 may include a machine learning model. For example, the model 132 may include a deep learning model. In one or more embodiments, the deep learning model includes a neural network model that includes one or more neural networks. The model 132 may be used to identify (or classify) current and / or future health conditions for the subject's retina.

[0062] For example, the model 132 may receive as input the OCT imaging data 112. Specifically, the model 132 may receive an OCT volume image 114 of the subject's retina. The model 132 may process the OCT volume image 114 by processing at least a portion of the OCT slice images that make up the OCT volume image 114. In some embodiments, the model 132 processes each of the OCT slice images that make up the OCT volume image 114. The model 132 generates a health indication output 138 based on the OCT volume image 114, wherein the health indication output 138 indicates whether the OCT volume image 114 demonstrates a selected health status category 140 for the subject's retina.

[0063] For example, the health indication output 138 may indicate a level of association between the OCT volume image 114 and a selected health condition category 140. This level of association may be represented via a probability. For example, in one or more embodiments, the health indication output 138 may be a probability indicating a level of association between the OCT volume image 114 and a selected health condition category 140 or how likely it is that the OCT volume image 114 demonstrates a selected health condition category 140. This level of association may be, for example, no association (e.g., a probability of 0.0), a weak association (e.g., a probability between 0.01 and 0.4), a moderate association (e.g., a probability between 0.4 and 0.6), a strong association (e.g., a probability between 0.6 and 1.0), or some other type of association. These percentages are only some examples of probability ranges and levels of association. Other levels of association and / or other percentage ranges may be used in other embodiments. This will be discussed below with respect to Figure 2 The process by which model 132 generates health indicator output 138 is described in greater detail.

[0064] The selected health status category 140 may refer to a health status for the retina at a current time point or a future time point (e.g., the next 6 months, 1 year, 2 years, etc.). In other words, the selected health status category 140 may represent a current health status or a future health status. The current time point may be, for example, the time at which the OCT volume image 114 was generated within a selected time interval (e.g., 1 week, 2 weeks, 1 month, 2 months, etc.) at which the OCT volume image 114 was generated.

[0065] In one or more embodiments, the selected health status category 140 can be the selected stage of AMD. The selected health status category 140 can be, for example, but not limited to, current nascent GA or future nascent GA. In some cases, the selected health status category 140 represents the stage of AMD's predicted GA in a selected time period (e.g., 6 months, 1 year, 2 years, etc.). In other cases, the selected health status category 140 represents the stage of AMD's predicted GA outbreak in a selected time period. In some other cases, the selected health status category 140 represents the stage of AMD's predicted GA in a selected time period. In this way, the selected health status category 140 can be used for the current health status of the retina or the prediction of the future health status of the retina. Other examples of health status categories include, but are not limited to, early AMD, mid-term AMD, GA, etc.

[0066] As described above, the model 132 can be implemented using a neural network model. The neural network model can include any number of neural networks or combinations of neural networks. The neural network can take the form of, but is not limited to, a convolutional neural network (CNN) (e.g., U-Net), a fully convolutional network (FCN), a stacked FCN, a stacked FCN with multi-channel learning, a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural ODE), a squeeze and excitation embedded neural network, a MobileNet, or another type of neural network. In one or more embodiments, the neural network itself can be composed of at least one of a CNN (e.g., U-Net), a stacked FCN, a stacked FCN with multi-channel learning, a FNN, a RNN, a MNN, a ResNet, a neural ODE, a squeeze and excitation embedded neural network, a MobileNet, or another type of neural network. In one or more embodiments, the neural network model takes the form of a convolutional neural network (CNN) system including one or more convolutional neural networks. For example, a CNN can include multiple neural networks, each of which can itself be a convolutional neural network.

[0067] In one or more embodiments, the neural network model may include a set of encoders, each of which may be a single encoder or multiple encoders; and a decoder. In some embodiments, the one or more encoders and / or the decoder may be implemented via a neural network, which in turn may be composed of one or more neural networks. In some cases, the decoder and the one or more encoders may be implemented using CNN. The decoder and the one or more encoders may also be implemented as a Y-net (Y-shaped neural network system) or a U-net (U-shaped neural network system). Figure 6 Provides more details related to neural networks.

[0068] The health condition identification system 101 may also be used to identify (or detect) a set of biomarkers 142 for a selected health condition category 140. For example, the health condition identification system 101 may be used to identify a set of biomarkers 142 in the OCT volume image 114 that demonstrate the selected health condition category 140 for the subject's retina. For example, when the selected health condition category 140 is current nascent GA, the set of biomarkers 142 may include one or more anatomical biomarkers that indicate that the OCT volume image 114 currently demonstrates the selected health condition category 140 for the retina. When the selected health condition category 140 indicates a future health condition (e.g., predicted to progress to nascent GA within a selected time period), the set of biomarkers 142 may be a prognosis for such future health condition.

[0069] The health condition identification system 101 uses a saliency mapping algorithm 134 to identify a set of biomarkers 142. For example, the saliency mapping algorithm 134 can be used to identify portions (or regions) of the OCT volume image 114 that have the greatest impact or contribution to the health indication output 138 of the model 132. For example, the saliency mapping algorithm 134 can indicate how important various portions (or regions) of the OCT volume image 114 are to the selected health condition category 140.

[0070] The saliency mapping algorithm 134 may include, but is not limited to, Grad-CAM, CAM, SmoothGrad, VarGrad, another type of saliency mapping algorithm or technique, or a combination thereof. The saliency mapping algorithm 134 may generate a saliency volume map 144 that indicates (e.g., via a heat map) how important the various portions (or regions) of the OCT volume image 114 are to the selected health condition category 140. In other words, the saliency volume map 144 indicates the contribution level of the various portions of the OCT volume image 114 to the health indication output 138 generated by the model 132. The saliency volume map 144 may be composed of a plurality of saliency maps, each saliency map corresponding to a different one of the plurality of OCT slice images in the OCT volume image 114. Each saliency map may visually indicate (e.g., via color, highlighting, shading, pattern, outline, text, annotation, etc.) the region of the corresponding OCT slice image that has the greatest impact on the model 132 for the selected health condition category 140.

[0071] The output generator 136 can receive and process the saliency volume map 144 to generate a map output 146. In one or more embodiments, the map output 146 takes the form of a filtered or modified form of the saliency volume map 144. In other embodiments, the map output 146 takes the form of a saliency volume map 144 or a modified form of the saliency volume map 144 superimposed on the OCT volume image 114. Similar to the way that the saliency volume map 144 can be composed of multiple saliency maps (two-dimensional), the map output 146 can be composed of multiple separate two-dimensional maps. These maps can be heat maps or overlays of heat maps on OCT slice images.

[0072] In one or more embodiments, a filter (e.g., a threshold filter) may be applied to the saliency volume map 144 to identify a subset of saliency maps in the saliency volume map 144 to be modified. The threshold filter may be set to ensure that only those saliency maps that indicate a contribution of, for example, at least one region in the corresponding OCT slice image above a selected threshold are selected for the subset. This subset of saliency maps may then be modified so that the modified saliency volume map formed includes fewer maps than the saliency volume map 144. In this way, when the map output 146 is generated, the map output 146 may be composed of fewer maps than the saliency volume map 144. In other embodiments, other types of filtering steps and / or other pre-processing steps may be performed so that the generated map output 146 includes a smaller number of maps than the maps in the saliency volume map 144.

[0073] The map output 146 may indicate whether a set of regions in the OCT volume image 114 are associated with a selected health condition category. For example, the map output 146 may indicate the level of contribution of a set of regions in the OCT volume image 114 to the health indication output 138 generated by the model 132. The regions may be pixel-level regions or regions formed by multiple pixels. The regions may be continuous or discontinuous regions. In some embodiments, the map output 146 visually locates a set of biomarkers 142. In other embodiments, the map output 146 may be further processed by the output generator 136 to identify which regions of the OCT volume image 114 are or include biomarkers. Figures 2 to 3 The process of identifying a set of biomarkers 142 using the significance mapping algorithm 134 and the output generator 136 is described in further detail. In some embodiments, the significance mapping algorithm 134 is integrated with the output generator 136 or implemented as part of the output generator.

[0074] In some embodiments, the model 132 can be trained with a training data set 148, which can include OCT volume images of tissues, so that the model 132 can identify and / or discover biomarkers associated with health status categories (e.g., diseases, conditions, disease progression, etc.) of the tissues from a test data set of OCT volume images of the tissues. In some cases, the health status categories of the tissues can range from healthy to various stages of disease. For example, the health status categories associated with the retina can range from healthy to various stages of AMD, including but not limited to early AMD, intermediate AMD, new GA, etc. In some cases, different biomarkers may be associated with different health status categories of the disease.

[0075] For example, AMD is the main cause of vision loss in 50 years old or above patients. Initially, AMD showed as dry AMD, and then progressed to wet type. For dry type, small deposits called drusen are formed under the inner collagen layer of the basement membrane of retinal pigment epithelium (RPE) and retinal Bruch's membrane (BM), thereby causing the retina to slowly deteriorate. In its late stage, dry AMD can show as map-like atrophy (GA), which is characterized by the progressive and irreversible loss of choroidal capillaries, RPE and photoreceptors. Wet AMD shows as abnormal blood vessels of the choroidal layer of eyes growing into the retina, and fluid leaks into the retina from the blood. Therefore, in some embodiments, drusen can be considered as a biomarker of a type of health status category (for example, dry AMD) of AMD, and the RPE of deletion can be considered as a biomarker of another type of health status category (for example, wet AMD) of AMD. It should be noted that other health status categories (for example, intermediate AMD, newborn GA, etc.) can be defined for AMD (for example, or other types of retinal diseases), and at least one or more distinguishable biomarkers can be associated with these health status categories.

[0076] As mentioned above, the morphological changes and / or the outward appearance of the new area, border etc. in the retina or eye can be considered as the biomarker of retinal diseases (such as AMD).The example of this morphological change can include deformation (for example, shape, size etc.), attenuation, abnormality, region / border missing or not existing, defect, damage and / or similar changes.For example, as mentioned above, the RPE of disappearance may indicate retinal degenerative diseases, such as AMD.As another example, the outward appearance (such as sediment (for example, drusen), leakage etc.) of the area not existing in healthy eye or retina, the border etc. between them may also be considered as the biomarker of retinal diseases such as AMD.Other examples that may be considered as the feature of biomarker in retina include reticular pseudodrusen (RPD), retinal high reflection lesions (for example, reflectivity is equal to or greater than the lesion of RPE), low reflection wedge-shaped structure (for example, appearing in the border of OPL), choroidal high transmission defect and / or the like.

[0077] The output generator 136 may generate other forms of output. For example, in one or more embodiments, the output generator may generate a report 150 to be displayed on the display system 106 or to be sent to a remote device (e.g., a cloud, a mobile device, a laptop, a tablet, etc.) via the network 120 or another network. The report 150 may include, for example, but not limited to, an OCT volume image 114, a saliency volume image, a map output for the OCT volume image, a list of any identified biomarkers, a treatment recommendation for the subject's retina, an evaluation recommendation, a monitoring recommendation, some other type of recommendation or instruction, or a combination thereof. For example, a monitoring recommendation may include a plan for monitoring the subject's retina and a schedule for future OCT imaging appointments. An evaluation recommendation may include, for example, a recommendation to further examine (e.g., manually by a human examiner) a subset of the plurality of OCT slice images that form the OCT volume image. The identified subset may include less than 5% of the plurality of OCT slice images. In some cases, a subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, 5%, 2%, or some other percentage of the plurality of OCT slice images.

[0078] In one or more embodiments, the health status identification system 101 stores in the data storage 104 an OCT volume image 114 obtained from the OCT imaging system 110, a saliency map 144, a map output 146, an identification of a set of biomarkers 142, a report 150, other data generated during processing of the OCT volume image 114, or a combination thereof. In some embodiments, the portion of the data storage 104 storing such information may be configured to comply with the security requirements of the Health Insurance Portability and Accountability Act (HIPAA), which mandates certain security procedures when processing patient data (e.g., such as OCT images of patient tissue), i.e., the data storage 104 may comply with HIPAA. For example, the stored information may be encrypted and anonymized. For example, the OCT volume image 114 may be encrypted and processed to remove and / or obfuscate personal identifying information (PII) of the subject from whom the OCT volume image 114 was obtained. In some cases, the communication link between the OCT imaging system 110 and the health status identification system 101 using the network 120 may also comply with HIPAA. For example, the communication link may be a virtual private network (VPN) that is end-to-end encrypted and configured to anonymize PII data transmitted therethrough.

[0079] In one or more embodiments, the health marker system 101 includes a system interface 160 that enables a human examiner to interact with images, graphs, and / or other output generated by the health marker system 101. The system interface 160 may include, for example, but not limited to, a web browser, an application program interface, a web-based user interface, some other type of interface component, or a combination thereof.

[0080] Although the discussion herein is generally directed to classifying OCT volume images (and OCT slice images) with respect to the stage of AMD and identifying and discovering biomarkers for AMD from OCT volume images (or OCT slice images) of the retina, such discussion may also be applicable to medical images of other tissues of a subject obtained using any other medical imaging technique. That is, the OCT volume image 114 and the related discussion of steps for classifying the OCT volume image 114 and for identifying and / or discovering AMD biomarkers via generating a saliency map (e.g., heat map) of the retinal OCT slice image are intended to be non-limiting illustrations, and the same or substantially similar method steps may be applicable to identifying and / or discovering other tissue diseases from 3D images of tissue (e.g., OCT or other modalities).

[0081] II.B. Exemplary Methods for Health Status Identification

[0082] Figure 2 Flowchart of a process for processing an OCT volume image of a subject's retina to determine whether the OCT volume image evidences a selected health condition category for the retina in accordance with one or more exemplary embodiments. Figure 2 In process 200 you can use Figure 1 The health status identification system 101 in the process 200 can be implemented. For example, at least some steps of the process 200 can be performed by a processor of a computer or server implemented as part of the health status identification system 101. The process 200 can be performed using Figure 1 132, saliency mapping algorithm 134, and / or output generator 136 in the process 200. In addition, it should be understood that additional steps may be performed before, during, or after the steps of the process 200 discussed below. In addition, in some embodiments, one or more of these steps may also be omitted or performed in a different order.

[0083] Process 200 may optionally include step 201 of training a deep learning model. The deep learning model may be Figure 1 An example of an implementation of model 132 in FIG. 132. For example, a deep learning model may include, but is not limited to, a neural network model. A deep learning model may be based on a training data set (e.g., but not limited to, Figure 1An example of how a deep learning model can be trained is described in further detail in Section II.D below.

[0084] Step 202 of process 200 includes receiving an optical coherence tomography (OCT) volumetric image of the subject's retina. The OCT volumetric image may be, for example, Figure 1 The OCT volume image 114 in FIG. 114. The OCT volume image may be composed of a plurality of OCT slice images.

[0085] Step 204 includes generating a health indicator output using the OCT volume image via a deep learning model, wherein the health indicator output indicates a level of association between the OCT volume image and a selected health condition category for the retina. The health indicator output may be, for example, Figure 1 The health indication output 138 in the image. In one or more embodiments, the health indication output is a classification score. The classification score can be, for example, a probability that the OCT volume image, and thereby the retina captured in the OCT volume image, can be classified as a selected health condition category. In other words, the classification score can be a probability that the OCT volume image demonstrates a selected health condition category for the retina. In some embodiments, a threshold value for the probability score (e.g., >0.5, >0.6, >0.7, >0.75, >0.8, etc.) is used to determine whether the OCT volume image demonstrates a selected health condition category.

[0086] The selected health status category may be e.g. Figure 1 140 in the selected health status category. In one or more embodiments, the selected health status category represents a current health status for the retina (e.g., a current disease state). In one or more other embodiments, the selected health status category represents a future health status (e.g., a future disease state predicted to develop within a selected time period). For example, the selected health status category may represent a nascent GA that is currently present or predicted to develop within a selected time period (e.g., 3 months, 6 months, 1 year, 2 years, 3 years, or some other time period).

[0087] The deep learning model can generate health indication outputs in different ways. In one or more embodiments, the deep learning model generates an initial output for each OCT slice image in the OCT volume image to form a plurality of initial outputs. The initial output for the OCT slice image can be, for example, but not limited to, the probability that the OCT slice image proves a selected health status category for the retina. The deep learning model can use the multiple initial outputs to generate a health indication output. For example, the deep learning model can average the multiple initial outputs together to generate a health indication output, which is the probability that the OCT volume image as a whole proves a selected health status category for the retina. In other words, the health indication output can be the probability that the retina can be classified as a selected health status category. In other embodiments, the median of the multiple initial outputs can be used as the health indication output. In yet other embodiments, the multiple initial outputs can be combined or integrated in some other way to generate a health indication output.

[0088] Step 206 includes generating an output (e.g., map output 146) for a deep learning model map using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volume image to a health indication output generated by the deep learning model. The level of contribution of a region in the OCT volume may be the importance or degree of influence of the region on the health indication output generated by the deep learning model. This region may be defined as a single pixel or multiple pixels. The region may be continuous or discontinuous. In one or more embodiments, the saliency mapping algorithm receives data from the deep learning model. Such data may include, for example, features, weights, or gradients used by the deep learning model to generate the health indication output in step 204. The saliency map algorithm may be used to generate a saliency map (or heat map) indicating the importance of the various portions of the OCT volume image relative to a selected health condition category (which is the category of interest).

[0089] For example, the saliency mapping algorithm can generate a saliency map for each OCT slice image of the OCT volume image. In one or more embodiments, the saliency mapping algorithm is implemented using Grad-CAM. The saliency map can be, for example, a heat map that indicates the contribution level (or importance) of each pixel in the corresponding OCT slice image to the health indication output generated by the deep learning model with respect to the selected health condition category. Figure 1 Together, a saliency volume map can be formed. The saliency map can use color, annotations, text, highlights, shading, patterns, or some other type of visual indicators to indicate importance. In one example, a range of colors can be used to indicate a range of importance.

[0090] The saliency volume map can be used to generate a map output in various ways. In one or more embodiments, each saliency map for each OCT slice image can be filtered to generate a modified saliency map. For example, one or more filters (e.g., threshold filters, processing filters, numerical filters, color filters, shading filters, etc.) can be applied to the saliency map to generate a modified saliency map that together forms a modified saliency volume map. Each modified saliency map can visually signal the most important area of ​​the corresponding OCT slice image. In one or more embodiments, each modified saliency map is superimposed on its corresponding OCT slice image to generate a map output. For example, the modified saliency map can be superimposed on the corresponding OCT slice image so that the indication is determined to be the portion of the OCT slice image that is most important (or most relevant) to the model for the selected health condition category. In one or more embodiments, the map output includes all superimposed OCT slice images. In one or more embodiments, the map output can provide a visual indication on each superimposed OCT slice image of the area that has the most important or greatest impact on the generation of the health indication output. .

[0091] In other embodiments, the modified saliency map is processed in another manner to generate a map output indicating which regions of the OCT slice images have the greatest impact on the model for the selected health condition category. For example, information from the modified saliency map can be used to annotate and / or otherwise graphically modify the corresponding OCT slice images to form a map output.

[0092] Process 200 may optionally include step 208. Step 208 includes identifying a set of biomarkers in the OCT volume image for a selected health condition category using the map output (e.g., Figure 1 A set of biomarkers 142 in the OCT slice image). Step 208 can be performed in different ways. In one or more embodiments, a potential biomarker region can be identified in association with a selected region of the OCT slice image, and the selected region is identified as being important or influential to the selected health condition category by the mapping output. The potential biomarker region can be identified as this selected region of the OCT slice image, or can be defined based on this selected region of the OCT slice image. In one or more embodiments, a bounding box is created around the selected region of the OCT slice image to define the potential biomarker region.

[0093] A scoring metric for a potential biomarker region can be generated. The scoring metric may include, for example, the size of the potential biomarker region, a confidence score for the potential biomarker region, some other metric, or a combination thereof. When the scoring metric meets a selected threshold, the potential biomarker region (e.g., a bounding box) can be identified as a biomarker for a selected health condition category. For example, if the scoring metric includes a confidence score and a dimension, the selected threshold may include a confidence score threshold (e.g., a minimum score) and a minimum dimension. In some embodiments, a particular biomarker may be visible in or across multiple OCT slice images. In one or more embodiments, a bounding box that meets the threshold and is classified as a biomarker region can be identified on the corresponding OCT slice image to form a biomarker map.

[0094] One or more of the identified biomarkers may be known biomarkers that have been previously seen by a human examiner. In some embodiments, one or more of the biomarkers may be new, previously unknown biomarkers. In other words, the identification of the set of biomarkers in step 208 may include the discovery of one or more new biomarkers associated with a selected health status category. The discovery of one or more new biomarkers may be more likely to occur, for example, when a selected health status category represents a future health status that is predicted to progress (e.g., AMD from early AMD or mid-term AMD to newborn GA; from newborn GA to GA; from mid-term AMD to future progression of GA, etc.).

[0095] Process 200 may optionally include step 210. Step 210 includes generating a report. The report may include, for example, but not limited to, an OCT volume image, a significance volume image, a map output for the OCT volume image, a list of any identified biomarkers, a treatment recommendation for the subject's retina, an evaluation recommendation, a monitoring recommendation, some other type of recommendation or instruction, or a combination thereof. For example, a monitoring recommendation may include a plan for monitoring the subject's retina and a schedule for future OCT imaging appointments. The evaluation recommendation may include, for example, a recommendation to further examine (e.g., manually by a human examiner) a subset of the multiple OCT slice images that form the OCT volume image. The identified subset may include less than 5% of the multiple OCT slice images. In some cases, the subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, 5%, 2%, or some other percentage of the multiple OCT slice images.

[0096] In some embodiments, Figure 1 The health identification system 101 in the Figure 1The assessment suggestions in the report 150 in the OCT volume image) allow the user to examine a specific subgroup of the OCT slice images within the OCT volume image to identify one or more features (or biomarkers) in the same or substantially similar location as the bounding box identified on the biomarker map. For example, in some cases, the health identification system 101 may include a system interface 160 that allows an inspector (e.g., a healthcare professional, a trained inspector, etc.) to access, inspect, and annotate the OCT slice images of the OCT volume image to identify and / or discover biomarkers. That is, for example, the system interface 160 can facilitate the inspector to annotate the OCT slice images with biomarkers. In some cases, instead of or in addition to allowing the inspector to identify the biomarkers on the OCT slice images shown in the biomarker map, the system interface 160 can be configured to allow the inspector to correct or adjust the bounding box on the biomarker map (e.g., adjust the size, shape, or continuity of the bounding box). In some cases, the inspector can annotate the bounding box to indicate the adjustment to be made. In some cases, the annotated and / or adjusted biomarker graph created by the examiner can be fed back to the deep learning model (e.g., as part of the training dataset 148) for additional training of the deep learning model.

[0097] Figure 3 is a flow chart of a process for identifying biomarkers in an OCT volume image of a subject's retina according to one or more exemplary embodiments. Figure 3 In process 300 you can use Figure 1 The health status identification system 101 in the process 300 can be implemented. For example, at least some steps of the process 300 can be performed by a processor of a computer or server implemented as part of the health status identification system 101. The process 300 can be performed using Figure 1 132, saliency mapping algorithm 134, and / or output generator 136 in the process 200. In addition, it should be understood that additional steps may be performed before, during, or after the steps of the process 200 discussed below. In addition, in some embodiments, one or more of these steps may also be omitted or performed in a different order.

[0098] Process 300 may optionally include step 302 of training a deep learning model. The deep learning model may be Figure 1 An example of an implementation of model 132 in FIG. 132. For example, a deep learning model may include, but is not limited to, a neural network model. A deep learning model may be based on a training data set (e.g., but not limited to, Figure 1 The training dataset 148 in is used for training.

[0099] Step 304 of process 300 includes receiving an optical coherence tomography (OCT) volumetric image of the subject's retina. The OCT volumetric image may be, for example, Figure 1 The OCT volume image 114 in FIG. 114. The OCT volume image may be composed of a plurality of OCT slice images.

[0100] Step 306 of process 300 includes generating a health indication output using the OCT volume image via the deep learning model, wherein the health indication output indicates a level of association between the OCT volume image and a selected health condition category for the retina. For example, the health indication output may be a probability indicating how likely it is that the classification of the retina in the OCT volume image is the selected health condition category. In other words, the health indication output may be a probability indicating how likely it is that the OCT volume image demonstrates the selected health condition category for the retina.

[0101] Step 308 of process 300 includes generating a saliency volume map of the OCT volume image using a saliency mapping algorithm, wherein the saliency volume map indicates a level of contribution of a set of regions in the OCT volume image to a health indication output generated by the deep learning model. Step 308 may be similar to the method described with respect to Figure 2 The contribution level may be determined based on features, gradients, or weights used in the deep learning model (e.g., features, gradients, or weights used in the last activation layer of the deep learning model).

[0102] Step 310 of process 300 includes using the saliency volume map to detect a set of biomarkers for a selected health condition category. Step 310 may be similar to the above description of Figure 2 The step 310 may be implemented in a manner that identifies the biomarkers described in step 208 of . For example, step 310 may include filtering the saliency volume map to generate a modified saliency volume map. The modified saliency volume map identifies a set of regions associated with a selected health condition category. Step 310 may further include identifying a potential biomarker region associated with one of the set of regions. A scoring metric may be generated for the potential biomarker region. When the scoring metric meets a selected threshold, the potential biomarker region may be identified as including at least one biomarker.

[0103] II.C. Exemplary Methods for Detecting Neonatal Geographic Atrophy (nGA)

[0104] Figure 4A 4 is a flow chart of a process 400 for artificial intelligence-assisted neonatal geographic atrophy (nGA) detection according to one or more exemplary embodiments. Figure 4A The detection of nGA described herein may include detecting one or more nGA lesions, locating one or more nGA lesions, or a combination thereof. Such detection may be considered a diagnosis of nGA. Process 400 may be performed using Figure 1 The health status identification system 101 in the embodiment of the present invention can be implemented. For example, at least some steps of the process 400 can be performed by a processor of a computer or server implemented as part of the health status identification system 101. The process 400 can be performed using Figure 1 400, and the like. The process 400 of FIG. 401 may be implemented by the model 132, the saliency mapping algorithm 134, and / or the output generator 136 in FIG. In addition, it should be understood that additional steps may be performed before, during, or after the steps of the process 400 discussed below. In addition, in some embodiments, one or more of these steps may also be omitted or performed in a different order.

[0105] Step 402 of process 400 includes training a machine learning model based on a dataset of OCT volumes. For example, one implementation of step 402 may include using Figure 1 The training data in 148 and Figure 1 The deep learning model can be trained using the OCT volume image 114 in Figure 1 An example of an implementation of model 132 in FIG.

[0106] Step 404 of process 400 includes applying a machine learning model to determine a diagnosis of neoplastic geographic atrophy for the patient based at least on the patient's OCT volume. For example, one example of an implementation of step 404 may include using a process Figure 1 OCT imaging data 112 Figure 1 The image processor 130 and the health status identification system 101 in the OCT volume can be used to diagnose nGA. The diagnosis of nGA based on the OCT volume can be Figure 2 The diagnosis of nGA can be based on the detection of one or more nGA lesions (eg, detecting the onset of nGA based on detecting the presence of one or more nGA lesions).

[0107] Step 406 of process 400 includes determining the location of one or more nGA lesions based at least on a saliency map identifying one or more regions of the OCT volume associated with a supra-threshold contribution to the diagnosis of nGA. For example, one implementation of step 406 may include the health status identification system 101 generating Figure 1 The output of the saliency map 134 in is used to identify the location of one or more nGA lesions.

[0108] Step 408 of process 400 includes verifying the diagnosis of nGA and / or the location of one or more nGA lesions based on the one or more inputs. An example implementation of step 408 may include the health status identification system 101 verifying the diagnosis of neonatal geographic atrophy and / or the location of one or more neonatal geographic atrophy lesions based on the one or more user inputs.

[0109] Figure 4B Flowchart of a process for processing an OCT volume image of a subject's retina to determine whether the OCT volume image demonstrates neonatal geographic atrophy (nGA) according to one or more exemplary embodiments. Figure 4B In process 450 you can use Figure 1 For example, at least some steps of process 450 may be performed by a processor of a computer or server implemented as part of health status identification system 101. Process 450 may be performed using Figure 1 450, and the like. In some embodiments, the present invention may be implemented by the model 132, the saliency mapping algorithm 134, and / or the output generator 136 in FIG. In addition, it should be understood that additional steps may be performed before, during, or after the steps of the process 450 discussed below. In addition, in some embodiments, one or more of these steps may also be omitted or performed in a different order.

[0110] Process 450 may optionally include a step 452 of training a deep learning model. The deep learning model may be Figure 1 An example of an implementation of model 132 in FIG. 132. For example, a deep learning model may include, but is not limited to, a neural network model. A deep learning model may be based on a training data set (e.g., but not limited to, Figure 1 An example of how a deep learning model can be trained is described in further detail in Section II.D below.

[0111] Step 454 of process 450 includes receiving an optical coherence tomography (OCT) volumetric image of the subject's retina. The OCT volumetric image may be, for example, Figure 1 The OCT volume image 114 in FIG. 114. The OCT volume image may be composed of a plurality of OCT slice images.

[0112] Step 456 includes generating an output via the deep learning model indicating whether neonatal geographic atrophy (nGA) is detected. The output may be, for example Figure 1138 in the health indication output 138. In one or more embodiments, the output is a classification score for nGA. The classification score can be, for example, a probability that the OCT volume image and thus the retina captured in the OCT volume image can be classified as demonstrating nGA (e.g., demonstrating the onset of nGA or another substage of nGA). In other words, the classification score can be the probability that the OCT volume image demonstrates nGA for the retina. In some embodiments, a threshold value for the probability score (e.g., >0.5, >0.6, >0.7, >0.75, >0.8, etc.) is used to determine whether the OCT volume image demonstrates nGA. The classification score can be similar to that for nGA. Figure 2 Step 456 is implemented in the manner described for implementing step 204 .

[0113] Step 458 includes generating an output for a deep learning model graph (e.g., graph output 146) using the saliency mapping algorithm, wherein the graph output indicates a contribution level of a set of regions in the OCT volume image to an output generated by the deep learning model. The contribution level of a region in the OCT volume can be the importance or influence of the region on the output generated by the deep learning model.

[0114] This area can be defined as a single pixel or multiple pixels. The area can be continuous or discontinuous. In one or more embodiments, the saliency mapping algorithm receives data from the deep learning model. Such data may include, for example, features, weights, or gradients used by the deep learning model to generate outputs in step 456. The saliency map algorithm can be used to generate a saliency map (or heat map) indicating the importance of the various portions of the OCT volume image relative to the selected health condition category (which is the category of interest).

[0115] For example, the saliency mapping algorithm can generate a saliency map for each OCT slice image of the OCT volume image. In one or more embodiments, the saliency mapping algorithm is implemented using Grad-CAM. The saliency map can be, for example, a heat map that indicates the contribution level (or importance) of each pixel in the corresponding OCT slice image to the health indication output generated by the deep learning model with respect to the selected health condition category. Figure 1 Together, a saliency volume map can be formed. The saliency map can use color, annotations, text, highlights, shading, patterns, or some other type of visual indicators to indicate importance. In one example, a range of colors can be used to indicate a range of importance.

[0116] The saliency volume map can be used to generate a map output in various ways. In one or more embodiments, each saliency map for each OCT slice image can be filtered to generate a modified saliency map. For example, one or more filters (e.g., threshold filters, processing filters, numerical filters, color filters, shading filters, etc.) can be applied to the saliency map to generate a modified saliency map that together forms a modified saliency volume map. Each modified saliency map can visually signal the most important area of ​​the corresponding OCT slice image. In one or more embodiments, each modified saliency map is superimposed on its corresponding OCT slice image to generate a map output. For example, the modified saliency map can be superimposed on the corresponding OCT slice image so as to indicate the portion of the OCT slice image that is determined to be the most important (or most relevant) to the model for the selected health condition category. In one or more embodiments, the map output includes all superimposed OCT slice images. In one or more embodiments, the map output can provide a visual indication on each superimposed OCT slice image of the area that has the most important or greatest impact on the generation of the output. .

[0117] In other embodiments, the modified saliency map is processed in another manner to generate a map output indicating which regions of the OCT slice images have the greatest impact on the model for the selected health condition category. For example, information from the modified saliency map can be used to annotate and / or otherwise graphically modify the corresponding OCT slice images to form a map output.

[0118] The one or more regions identified by the map output may indicate, for example, directly correspond to, one or more nGA lesions. For example, the regions identified in the map output may be considered to be the locations of the one or more nGA lesions.

[0119] In one or more embodiments, the map output may be annotated with other information. For example, the map output may include a bounding box created around a selected area of ​​the OCT slice image that is identified as an nGA lesion or demonstrates one or more nGA lesions. In some cases, the bounding box may be annotated with a scoring metric (e.g., a confidence score, size, etc.). In one or more embodiments, a bounding box that meets a threshold size, meets a threshold confidence score, or both is classified as demonstrating nGA.

[0120] Process 450 may optionally include step 460. Step 460 includes generating a report. The report may include, for example, but not limited to, an OCT volume image, a significance volume image, a map output for the OCT volume image, a list of any identified biomarkers, a treatment recommendation for the subject's retina, an evaluation recommendation, a monitoring recommendation, some other type of recommendation or instruction, or a combination thereof. For example, a monitoring recommendation may include a plan for monitoring the subject's retina and a schedule for future OCT imaging appointments. The evaluation recommendation may include, for example, a recommendation to further examine (e.g., manually by a human examiner) a subset of the multiple OCT slice images that form the OCT volume image. The identified subset may include less than 5% of the multiple OCT slice images. In some cases, the subset may include less than 50%, 45%, 40%, 35%, 30%, 25%, 45%, 15%, 10%, 5%, 2%, or some other percentage of the multiple OCT slice images.

[0121] In some embodiments, Figure 1 The health identification system 101 in the Figure 1 The assessment suggestions in the report 150 in the OCT volume image) allow the user to examine a specific subgroup of the OCT slice images within the OCT volume image to identify one or more features (or biomarkers) in the same or substantially similar location as the bounding box identified on the biomarker map. For example, in some cases, the health identification system 101 may include a system interface 160 that allows an inspector (e.g., a healthcare professional, a trained inspector, etc.) to access, inspect, and annotate the OCT slice images of the OCT volume image to identify and / or discover biomarkers. That is, for example, the system interface 160 can facilitate the inspector to annotate the OCT slice images with biomarkers. In some cases, instead of or in addition to allowing the inspector to identify the biomarkers on the OCT slice images shown in the biomarker map, the system interface 160 can be configured to allow the inspector to correct or adjust the bounding box on the biomarker map (e.g., adjust the size, shape, or continuity of the bounding box). In some cases, the inspector can annotate the bounding box to indicate the adjustment to be made. In some cases, the annotated and / or adjusted biomarker graph created by the examiner can be fed back to the deep learning model (e.g., as part of the training dataset 148) for additional training of the deep learning model.

[0122] II.D. Example Training of Deep Learning Models

[0123] Above Figure 1 (e.g., model 132), Figure 2 , Figure 3 , Figure 4 and Figure 5The deep learning model described in can be trained in different ways. In one or more embodiments, a training dataset (e.g., Figure 1 The deep learning model may be trained using a training dataset 148 in the training OCT volume images. Each of these training OCT volume images may have a different retina that has been identified as showing a disease or condition corresponding to a selected health condition category (e.g., newborn GA, etc.). The retina may continue to show the disease or condition for a length of time that is at least substantially equal to the duration after the training OCT volume images were taken or generated.

[0124] In one or more embodiments, a deep learning model (e.g., Figure 1 Model 132 in Figures 2 to 3 The deep learning model described in ) can be trained to be based on a training dataset of OCT volume images of the retina of patients with one or more health condition categories (e.g., Figure 1 The training dataset 148 in the OCT volume image is used to classify the health status of the retina, so that the deep learning model can learn what features of the retina and their locations in the OCT volume image are signals for one or more health status categories. When provided with a test dataset of OCT volume images, the trained deep learning model is then able to efficiently and accurately identify whether the OCT volume image demonstrates the selected health status category.

[0125] The training data set of OCT volume images can include the OCT images of the retina of the patient known to suffer from AMD in a given stage (that is, the health status category of the retina can be the AMD stage). In this case, the deep learning model can be trained with the training data set to learn which feature in the OCT volume image corresponds to, is associated with, or signals that stage of AMD. For example, the patient may be a late AMD patient, and the deep learning model can identify or find from the training data set of the OCT volume image of the patient's retina that the anatomical features in the OCT volume image representing a severely deformed RPE may be evidence of late AMD. In this case, when provided with the OCT volume image of the patient's retina showing a severely deformed RPE, the trained deep learning model can identify the OCT volume image as an image belonging to a late AMD patient.

[0126] In some embodiments, a deep learning model may be able to classify a health condition based on unknown biomarkers of a retinal disease. For example, a deep learning model may be provided with a training dataset of OCT volume images of the retina of patients with a certain retinal disease (e.g., neonatal GA) for which all biomarkers may be unknown. That is, the biomarkers for the selected health condition category (e.g., neonatal GA) for the retinal disease may be unknown. In this case, the deep learning model may process the dataset of OCT volume images and learn that features or patterns (e.g., lesions) in the OCT volume images are evidence of the selected health condition category.

[0127] III. Exemplary OCT Images and Exemplary Map Output

[0128] Figure 5 FIG. 5 shows an annotated OCT slice image and a corresponding heat map of the annotated OCT slice image according to one or more exemplary embodiments. Figure 1 The OCT slice image 502 may also be an example of an implementation of the OCT slice image in the OCT volume image 114. Figure 1 An example of an implementation of this is an OCT slice image in the training dataset 148. The OCT slice image 502 includes annotated regions 504 that have been labeled as biomarkers for de novo GA by a human grader.

[0129] Heat map 506 is for Figure 1 The heat map 506 may be an example of an implementation of at least a portion of the graph output 146. Figure 1 The result of superimposing a saliency map generated by the saliency mapping algorithm 134 in (e.g., generated using Grad-CAM) on the OCT slice image 502. A saliency map for the trained deep learning model processing the OCT slice image 502 is generated. Heat map 506 indicates that region 508 has the greatest impact on the model of the nascent GA and shows that the deep learning model (e.g., which may be Figure 1 The model 132 in FIG. 1 accurately uses the correct region of the OCT slice image 502 for its classification with respect to nascent GA.

[0130] Heat map 506 can be used to identify and locate biomarkers shown in region 508 for newborn GA. For example, an output identifying anatomical biomarkers located in region 508 can be generated. The biomarker can be, for example, a lesion in the retina, a missing retinal pigment epithelium (RPE), a detached layer of the retina, or some other type of biomarker. In some cases, there may be more than one biomarker in region 508. In some cases, filtering can be performed to identify certain pixels in region 508 of heat map 506 or in region 504 of OCT slice image 502 as biomarkers. In some cases, the size of region 508 can be used to determine whether region 508 contains one or more biomarkers. In one or more embodiments, the size of region 508 is greater than about 20 pixels.

[0131] Such identification and location of biomarkers may allow a healthcare practitioner to diagnose, monitor, treat, etc., a patient whose retina is depicted in OCT slice image 502. For example, an ophthalmologist examining heat map 506 or information generated based on heat map 506 may be able to recommend treatment options or monitoring options prior to the onset of GA.

[0132] Figure 6 602 is a diagram of different graphs according to one or more exemplary embodiments. Figure 1 6 is an example of an implementation of a saliency map for a saliency volume map 144 in FIG. Modified saliency map 604 is an example of an implementation of a saliency map that has been modified after filtering (e.g., applying a threshold filter). Heat map 606 is an example of an implementation of a saliency map for a saliency volume map 144 in FIG. Figure 1 One example of an implementation of components of Figure output 146. Heat map 606 comprises a modified overlay of saliency map 602 on an OCT slice image.

[0133] Biomarker graph 608 is for Figure 1 606. In the biomarker map 608, the first bounding box identifies a potential biomarker region that does not have a high enough confidence score (e.g., >0.6) to be considered a biomarker region that includes at least one biomarker. In addition, the biomarker map 608 includes a second bounding box that identifies a potential biomarker region that has a high enough confidence score to be considered a biomarker region that includes at least one biomarker.

[0134] An exemplary grayscale is shown below Figures 5 and 6 The images and graph outputs (e.g., heat maps, saliency maps) depicted in FIG. 10A and 10B may be used. In other embodiments, other grayscales may be used. For example, Figure 5The OCT images depicted in FIG. Figure 5 As a non-limiting example, Figure 5 The background shown in white in FIG. 1 may be black in other exemplary embodiments. In some embodiments, a range of colors may be used to generate the graph output. For example, Figure 6 The grayscale output of the graph shown in FIG. 1 may be colored in other embodiments. In other embodiments, Figure 6 The biomarker plots shown in can be annotated with colors, can have potential biomarker regions identified by colors, or both.

[0135] IV. Exemplary Systems for Detection of Neonatal Geographic Atrophy

[0136] Figures 7 to 13 A system for detecting neonatal geographic atrophy (nGA) and various workflows using the system are described. The neonatal geographic atrophy detection system 700 may be Figure 1 An example of an implementation of the health status indicator system 101 in FIG. Training is described for one or more different types of exemplary training data sets.

[0137] Figure 7 A system diagram illustrating an example of a new geographic atrophy detection system 700 according to some exemplary embodiments is depicted. The new geographic atrophy detection system 700 may include a detection controller 710 including a diagnosis engine 712 and a positioning engine 714, a data store 720, and a client device 730. The detection controller 710, the data store 720, and the client device 730 may be communicatively coupled via a network 740. The detection controller may be Figure 1 An example of an implementation of all or part of the image processor 130 in FIG.

[0138] The client device 730 may be a processor-based device, including, for example, a mobile device, a wearable device, a personal computer, a workstation, an Internet of Things (IoT) appliance, etc. The data store 720 may be a database, including, for example, a non-relational database, a relational database, an in-memory database, a graph database, a key-value store, a document store, etc. The data store 720 may be Figure 1 The network 745 may be a wired network and / or a wireless network, including, for example, a public land mobile network (PLMN), a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), the Internet, etc. The network 745 may be Figure 1 An example of an implementation of the training data 120 in FIG.

[0139] In some exemplary embodiments, the diagnosis engine 712 may be implemented using a deep learning model (e.g., an artificial neural network (ANN) based classifier). The diagnosis engine 712 may be Figure 1 An example of an implementation of a health status model 132 in . In some cases, the diagnostic engine 712 can be implemented as a residual neural network (ResNet) based classifier. The diagnostic engine 712 can be configured to perform a newborn geographic atrophy (nGA) diagnosis, which includes determining whether the patient exhibits newborn geographic atrophy based at least on one or more optical coherence tomography (OCT) volumes of the patient. Lesion localization can be performed based on a visual interpretation of a deep learning model applied by the diagnostic engine 712 to identify the location of one or more newborn geographic atrophy (nGA) lesions. For example, the positioning engine 714 can be configured to perform newborn geographic atrophy (nGA) lesion localization, which includes determining the location of one or more lesions associated with newborn geographic atrophy (nGA) based at least on a saliency map, wherein the saliency map identifies areas of the optical coherence tomography (OCT) volume associated with a supra-threshold contribution to the diagnosis of newborn geographic atrophy. A saliency map can be generated by applying, for example, Gradient Weighted Class Activation Mapping (GradCAM), which outputs a heat map of how much each region within an image (e.g., an Optical Coherence Tomography (OCT) volume) contributes to the class label ultimately assigned to the image.

[0140] In some exemplary embodiments, a deep learning model implementing the diagnostic engine 712 may be trained on a dataset 725, for example, stored in the data store 720. The dataset 725 may be Figure 1 An example of an implementation of the training data set 148 in . In one example, the data set 725 includes, but is not limited to, a total of 1,884 optical coherence tomography volumes from 280 eyes of 740 subjects with intermediate age-related macular degeneration (iAMD). Overall, 1,766 optical coherence tomography volumes were marked as without newborn geographic atrophy (i.e., no nGA was detected), and 118 volumes were marked as having newborn geographic atrophy (i.e., nGA was detected). As used herein, the diagnosis of newborn geographic atrophy can also include newborn geographic atrophy that is enlarged to a size that meets the diagnostic criteria for complete retinal pigment epithelial and outer retinal atrophy (cRORA).

[0141] The optical coherence tomography volume can be further labeled with the location of the neonatal geographic atrophy lesion, such as with a bounding box that covers the depression horizontally, starts at the inner limiting layer (ILL) vertically, and ends at the retinal pigment epithelium (RPE) layer. The bounding box can be used to evaluate weakly supervised lesion localization (e.g., performed by the localization engine 714), rather than for model training. Since the dataset 725 used to train the deep learning model includes class labels for the 3D optical coherence tomography volume, training the deep learning model to perform neonatal geographic atrophy (nGA) diagnosis and lesion localization can be considered weakly supervised.

[0142] IV.A. Exemplary Model for Classifying OCT Volumes for nGA

[0143] Fig. 8A and 8B Describes the realization Figure 7 An example of a deep learning architecture for the diagnosis engine 712 and the localization engine 714 of the detection controller 710 of the new geographic atrophy detection system 700 is shown. Fig. 8A and 8B The components shown in the figure can be used to implement Figure 1 An example of components of the health status identification system 101 in FIG.

[0144] Fig. 8A is an illustration of an example of a model 800 for processing a 3D OCT volume according to one or more exemplary embodiments. The model 800 is used to generate a classification score indicating whether nGA is detected in an OCT volume, which may include a deep neural network based OCT B-scan classifier.

[0145] Figure 8B A method for implementing the diagram according to one or more exemplary embodiments is shown. Fig. 8A An example of an implementation of the classifier 802 of the classifier in the model 800 in FIG. The classifier 802 can be used to classify OCT B-mode scans. The classifier 802 can be implemented using a residual neural network (ResNet) backbone, whose output is coupled with a rectifier linear unit (ReLU) and a fully connected (FC) layer. The late fusion method with the residual neural network (ResNet) backbone can be applied to the 3D OCT volume.

[0146] As an example, in Fig. 8A, B-scans are fed into the B-scan classifier of model 800 and the output is a vector of classification logical values ​​for each B-scan. The B-scan logical values ​​are averaged to generate a classification score for each OCT volume. Considering the B-scans as instances and the OCT volumes as bags, the framework can be classified as an example of multiple instance learning, where the model 800 is trained on weakly labeled data using labels on bags (OCT volumes). During training, given an OCT volume annotated as neo-geographic atrophy, the model 800 may be forced to identify as many B-scans as neo-geographic atrophy lesions to improve the final prediction of nGA, and thus the trained model allows prediction of nGA labels on OCT volumes as well as on individual B-scans.

[0147] Details of the exemplary B-scan classifier 802 are provided in Figure 8B . For example, a single B-scan from a volume of size 512×496 is passed through a residual neural network (e.g., ResNet-18) backbone, which outputs an activation map (e.g., a 512×16×16 activation map). Max pooling and average pooling layers can be applied to the output of the residual neural network before their respective outputs are concatenated to produce a feature vector (e.g., a 1024-long feature vector). A fully connected layer can then be applied to the feature vector to produce a vector of categorical logical values ​​corresponding to the categorical distribution of the B-scan.

[0148] IV.B. Exemplary Model Training

[0149] Fig. 9 An exemplary data flow diagram with data segmentation statistics according to one or more exemplary embodiments is depicted. The data flow diagram tracks a model (e.g., Fig. 8A and 8B The training data 900 may be Figure 7An example of a dataset 725 in is generated from 1,910 OCT volumes of 280 eyes of 140 participants with intermediate age-related macular degeneration (iAMD), each volume being generated by visiting each eye every six months with 1 volume for up to 3 years. Volumes graded as neovascular age-related macular degeneration were excluded. Of the remaining 1,884 volumes, 118 volumes from 40 eyes of 28 participants were graded as nGA positive. A 5-fold cross validation 902 was performed on the training data 900, where 5 models were trained in 5 different splits in a "cross validation" manner. In each split, the training data 900 was split by patient into a training set, a validation set, and a test set. Early stopping was used to monitor the F1 score on the validation set. Model performance evaluation was applied to the test set. Table 904 provides the segmentation statistics, number of volumes, and participants for the 5-fold cross validation 902. It should be understood that the number of eyes is twice the number of participants.

[0150] With the training data 900 covering a small number of participants, the performance of the deep learning model can be tested on the entire dataset in 5 test sets from 5 different segmentation folds. For each fold, a test set of optical coherence tomography volumes was obtained from approximately 20% of the participants stratified according to whether the patient had nGA. The OCT volumes of the remaining 80% of the participants were further divided into training sets (64%) and validation sets (16%), where the volume of one patient was only present in one of the datasets. Although the term "cross-validation" is used to describe the data split, the corresponding test sets were not used in the training and validation processes.

[0151] In some exemplary embodiments, at least some preprocessing may be performed on the B-scans for normalization. For example, the B-scans may be resized (e.g., to 512×496 pixels) before being rescaled to the intensity range [0,1]. Some data augmentation may be randomly applied, such as small angle rotations, horizontal flips, vertical flips, addition of Gaussian noise, and Gaussian blurring, to improve the invariance of the model to these transformations.

[0152] Reference again Figure 8B, the residual neural network (ResNet) backbone of the classifier 802 can be pre-trained on the ImageNet dataset. In one or more embodiments, during model training, the Adam optimizer can be used to minimize focal loss, while L2 weight decay regularization can be used to improve the ability of the model to generalize on the training data 900. In some cases, hyperparameter adjustment can be performed using the training data 900 and the validation set to find the optimal values ​​for the learning rate and weight decay. The model 800 trained with the optimal hyperparameters can be tested on the test set. Various indicators can be evaluated to indicate model performance. Such indicators include, for example, but are not limited to, the area under the curve (AUC), the area under the precision-recall curve (AUPRC), the recall rate, the precision, and the F1 score. In addition, a confusion matrix can be calculated.

[0153] IV.C. Example Graph Output Generation

[0154] Fig.10 1000 is an illustration of an output workflow for output generated from an OCT volume according to one or more exemplary embodiments. In workflow 1000, the output of the gradient weighted class activation map can be overlaid on an input OCT image to facilitate visualization of saliency as well as the original grayscale OCT image. Visually emphasized areas (e.g., by a particular color or highlight) can indicate the location of nGA lesions. The saliency map can be used to reason about the decisions of a model (e.g., model 800), to examine the generalizability of the model, and to examine and exploit the model's capabilities in the detection of neoplastic geographic atrophy lesions.

[0155] As shown in workflow 1000, a B-scan logical value is generated for a single OCTB-type scan of an OCT volume input into a model (e.g., model 800). These logical values ​​are used to classify the OCT volume as whether nGA is confirmed. The GradCAM output of the model for a single slice (e.g., slice 22) is shown. Before generating bounding boxes via connected component analysis, an adaptive threshold is applied to the corresponding GradCAM output of the gradient-weighted class activation map (in the viridis color map). The confidence score of the bounding box can be estimated based on the average significance and the corresponding B-scan logical value. A graph output can be generated by superimposing the B-scan with the GradCAM output, superimposing the B-scan with the bounding box and its associated confidence score, or both. Bounding boxes with confidence scores below a threshold (e.g., <0.6) can be removed from subsequent processing.

[0156] The GradCAM output and the map output help to intuitively locate nGA lesions. In one or more embodiments, each bounding box can be considered to potentially identify an nGA lesion. For example, bounding boxes with confidence scores exceeding a threshold can be considered to be the location of one or more nGA lesions. In one or more embodiments, the confidence score of each bounding box can be estimated based on a single classification logical value of the B-scan classifier as where S represents the sigmoid function, l represents the single B-scan classification logic value, n represents the number of B-scans in the volume, h represents the average significance of the detected regions, and ∑h represents the total average significance of all detected regions within the B-scan.

[0157] A higher confidence score may mean that there is a higher probability that the area detected in the bounding box covers a neonatal geographic atrophy lesion. Therefore, bounding boxes with confidence scores below a threshold (e.g., <0.6) can be removed by thresholding, so that B-scans with one or more remaining bounding boxes (after thresholding) can be identified as B-scans exhibiting neonatal geographic atrophy (nGA).

[0158] In some exemplary embodiments, the aforementioned confidence score threshold can be determined based on the presence of B-scans with newborn geographic atrophy in the validation set. For example, on the validation set, the number of classified newborn geographic atrophy B-scans and the recall rate of diagnostic newborn geographic atrophy B-scans relative to different thresholds can be plotted respectively. A lower threshold may cause the model to produce fewer false negatives (e.g., true newborn geographic atrophy B-scans are misclassified as non-newborn geographic atrophy) and more false positives (e.g., true non-newborn geographic atrophy presents B-scans misclassified as newborn geographic atrophy). In the case of deploying the detection controller 710 for patient screening, those B-scans classified as presenting newborn geographic atrophy can undergo further review and verification. Therefore, the model can be adjusted to improve the recall rate while maintaining the number of B-scans classified as newborn geographic atrophy at an acceptable number. In an example implementation, the threshold can be increased in steps of 0.02 starting from a small value. The threshold can be selected so that a further increase in the threshold causes the recall rate to decrease by more than 0.2, but only less than 1,000 additional B-scans are saved for further review.

[0159] In some exemplary embodiments, the detection controller 710 may generate an output of a B-scan exhibiting nascent geographic atrophy including one or more primary bounding boxes with an above-threshold confidence score. The confidence score may be considered a confidence score for the B-scan. A successful diagnosis of a nascent geographic atrophy B-scan with lesion localization is recorded if and only if the bounding box output overlaps with the ground truth and / or expert annotated bounding boxes.

[0160] In some exemplary embodiments, the detection controller 710 may be deployed for AI-assisted diagnosis of neonatal geographic atrophy (nGA). In an exemplary embodiment, the detection controller 710 may propose a neonatal geographic atrophy diagnosis and one or more lesion locations. For example, the detection controller 710 may identify within a set of optical coherence tomography volumes high-risk B-scans that the underlying deep learning model determines to exhibit neonatal geographic atrophy. These high-risk B-scans may be presented, for example, in a user interface 735 of a client device 730. The presence or absence of neonatal geographic atrophy and the suggested location of the neonatal geographic atrophy lesions may be confirmed based on one or more user inputs received at the client device 730.

[0161] Fig.11A FIG. 1 is a diagram of a confusion matrix 1100 according to one or more exemplary embodiments. The confusion matrix 1100 may be a matrix such as Fig. 8A and Figure 8B An example of a confusion matrix generated from a 5-fold cross validation of the performance of model 800. N represents negative normal volumes and P represents positive neoplastic atrophy volumes.

[0162] Fig. 11B is a statistical graph of 5-fold cross validation according to one or more exemplary embodiments. The area under the curve (AUC), area under the precision-recall curve (AUPRC), recall, precision, and F1 score of the model on the test set from the 5-fold cross validation are shown in FIG. Fig. 11B The average performance of the five folds is also given, and the error bars show the 95% confidence intervals (CI). The average precision and recall were 0.76 (95% CI 0.60-0.91) and 0.74 (95% CI 0.56-0.93), respectively.

[0163] Fig. 12A 1 is an illustration of an OCT image 1200 (e.g., a B-scan) in which an nGA lesion has been detected, according to one or more exemplary embodiments. The original OCT image is shown on the left, with boxes indicating where a human grader annotated the presence of nGA and where a system (e.g., neoplastic geographic atrophy detection system 700) identified the presence of nGA. A true positive may be a B-scan where the bounding box detected by the model overlaps with the bounding box annotated by the expert. A true negative may be a B-scan that contains neither the bounding box detected by the model nor the annotated bounding box.

[0164] Fig. 12B Graph 1202 is a precision-recall curve for 5-fold cross validation according to one or more exemplary embodiments.

[0165] Fig. 12C1 is a graphical representation of a confusion matrix 1204 according to one or more exemplary embodiments. In the confusion matrix 1204, N represents a negative B-scan without neonatal geographic atrophy (nGA) lesions and P represents a positive B-scan with neonatal geographic atrophy (nGA) lesions.

[0166] like FIG. 12A to FIG. 12C As shown, the detection controller 710 can achieve robust B-scan diagnosis and lesion localization performance without any B-scan level classification or bounding box annotation. Overall, the recall and precision of B-scan diagnosis with correctly located lesions by bounding boxes are 0.93 and 0.27, respectively, over the entire example dataset.

[0167] Using the type of new geographic atrophy detection system 700 described herein can achieve more accurate and efficient nGA detection while reducing the amount of time required to process B-scans. As an example, using the new geographic atrophy detection system 700 instead of 92,316 individual B-scans, a clinician may only need to review 1,550 B-scans (or some other number of B-scans) in which nGA was detected (e.g., about 2%). In addition, using the new geographic atrophy detection system 700, nGA can be detected in places where a human grader might not be able to detect nGA lesions.

[0168] In some exemplary embodiments, the detection controller 710 including the aforementioned deep learning model is capable of diagnosing newly emerged geographic atrophy on an optical coherence tomography volume. The detection controller 710 is capable of performing newly emerged geographic atrophy diagnosis in a population starting with intermediate age-related macular degeneration (iAMD) and without obvious geographic atrophy lesions. Newly emerged geographic atrophy is an important risk factor for developing geographic atrophy. The detection controller 710 is capable of providing a diagnosis on a single B-scan and locating the lesions present therein based on the optical coherence tomography volume-by-volume diagnostic labels.

[0169] While datasets such as dataset 725 or training data 900 may be highly imbalanced (e.g., a small fraction or 6.26% of cases have neo-geographic atrophy), using a B-scan classifier with a pre-trained artificial neural network (ANN) backbone (e.g., a 2D backbone pre-trained on Imagenet data) greatly improves model performance (F1 score increased from 0.25 to 0.74) for training datasets with a limited number of OCT volumes.

[0170] Fig.131 is an illustration of an OCT image 1300 that has been annotated with a bounding box according to one or more exemplary embodiments. The OCT image 1300 shows that the bounding box can be used to locate pathologies that present similarly to nGA (e.g., drusen or hyperreflective lesions that connect to the outer plexiform layer (OPL) with the retinal pigment epithelium (RPE); drusen, cysts, or hyperreflective lesions that produce a sunken-like structure). Such a bounding box can be further analyzed by a human grader. In some cases, a higher threshold confidence score can be used to exclude pathologies other than nGA.

[0171] However, when using neo-geographic atrophy as a biomarker for progression or early endpoints, weakly supervised methods for diagnosing neo-geographic atrophy and localizing neo-geographic atrophy lesions can help screen patients for neo-geographic atrophy enrichment or aid in the diagnosis of age-related macular degeneration stage. In clinical trials with neo-geographic atrophy as an inclusion / exclusion criterion or a clinical biomarker for progression or endpoint, grading neo-geographic atrophy on optical coherence tomography volumes on high-density B-scans is laborious and costly to perform, especially when screening large populations. The proposed AI-assisted diagnosis can greatly reduce the operational burden and improve the feasibility of such trials. Similar strategies can also be applied to other trials based on multiple anatomical biomarkers for clinical enrichment.

[0172] V. Exemplary Neural Networks

[0173] Fig.14 An exemplary neural network that can be used to implement a computer-based model according to various embodiments of the present disclosure is shown. For example, a neural network 1400 can be used to implement the model 132 of the health status identification system 101. As shown, the artificial neural network 1400 includes three layers, namely an input layer 1402, a hidden layer 1404, and an output layer 1407. Each layer in the layers 1402, 1404, and 1407 may include one or more nodes. For example, the input layer 1402 includes nodes 1408 to 1414, the hidden layer 1404 includes nodes 1417 and 1418, and the output layer 1407 includes a node 1422. In this example, each node in the layer is connected to each node in the adjacent layer. For example, the node 1408 in the input layer 1402 is connected to two nodes 1417 and 1418 in the hidden layer 1404. Similarly, the node 1417 in the hidden layer 1404 is connected to all nodes 1408 to 1414 in the input layer 1402 and the node 1422 in the output layer 1407. Although only one hidden layer is shown for artificial neural network 1400, it is contemplated that artificial neural network 1400 used to implement model 132 may include as many hidden layers as necessary or desired.

[0174] In this example, artificial neural network 1400 receives a set of input values ​​and produces output values. Each node in input layer 1402 may correspond to a different input value. For example, when artificial neural network 1400 is used to implement model 132, each node in input layer 1402 may correspond to a different attribute of an OCT volume image of the retina (e.g., from Figure 1 ).

[0175] In some embodiments, each node 1417 and 1418 in the hidden layer 1404 generates a representation that may include a mathematical calculation (or algorithm) to generate a value based on the input values ​​received from the nodes 1408 to 1414. The mathematical calculation may include assigning different weights to each of the data values ​​received from the nodes 1408-1414. The nodes 1417 and 1418 may include different algorithms and / or different weights assigned to the data variables from the nodes 1408 to 1414, so that each of the nodes 1417 and 1418 may generate different values ​​based on the same input values ​​received from the nodes 1408 to 1414. In some embodiments, the weights initially assigned to the features (or input values) of each of the nodes 1417 and 1418 may be randomly generated (e.g., using a computer random generator). The node 1422 in the output layer 1407 may use the values ​​generated by the nodes 1417 and 1418 to generate output values ​​for the artificial neural network 1400. When artificial neural network 1400 is used to implement model 132, output values ​​generated by artificial neural network 1400 may include a saliency map, such as but not limited to a heat map of an OCT volume image of the retina (e.g., saliency map 144), which identifies biomarkers therein.

[0176] The artificial neural network 1400 can be trained by using training data. For example, the training data herein can be an OCT volume image of the retina. The training data can be, for example Figure 1 1404. By providing the training data to the artificial neural network 1400, the nodes 1417 and 1418 in the hidden layer 1404 can be trained (adjusted) so that the optimal output is generated in the output layer 1407 based on the training data. By continuously providing different sets of training data and penalizing the artificial neural network 1400 when the output of the artificial neural network 1400 is incorrect (for example, when the biomarker in the OCT volume image is incorrectly identified), the artificial neural network 1400 (and specifically, the representation of the nodes in the hidden layer 1404) can be trained (adjusted) to improve its performance in data classification. Adjusting the artificial neural network 1400 may include adjusting the weights associated with each node in the hidden layer 1404.

[0177] Although the above discussion relates to an artificial neural network as an example of machine learning, it should be understood that other types of machine learning methods may also be applicable to various aspects of the present disclosure. For example, a support vector machine (SVM) can be used to implement machine learning. SVM is a group of related supervised learning methods for classification and regression. The SVM training algorithm (which may be a non-probabilistic binary linear classifier) ​​can build a model that predicts whether a new example belongs to one category or another category. As another example, a Bayesian network can be used to implement machine learning. A Bayesian network is a non-cyclic probabilistic graphical model that represents a set of random variables and their conditional independence with a directed acyclic graph (DAG). A Bayesian network can present a probabilistic relationship between one variable and another variable. Another example is a machine learning engine that uses a decision tree learning model to perform a machine learning process. In some cases, a decision tree learning model may include a classification tree model and a regression tree model. In some embodiments, the machine learning engine uses a gradient boosting machine (GBM) model (e.g., XGBoost) as a regression tree model. Other machine learning techniques can be used to implement a machine learning engine, such as via a random forest or a deep neural network. For reasons of simplicity, other types of machine learning algorithms are not discussed in detail herein, and it should be understood that the present disclosure is not limited to a particular type of machine learning.

[0178] VI. Exemplary Computing Systems

[0179] Fig.15 A block diagram illustrating an example of a computing system 1500 according to some exemplary embodiments is depicted. Referring to FIG. Figure 7 to Figure 1 7 and Fig.15 , the computing system 1500 may be used to implement Figure 7 The detection controller 710 in Figure 7 The client device 730, and / or any components therein.

[0180] like Fig.15 As shown, computing system 1500 may include processor 1510, memory 1520, storage device 1530, and input / output device 1540. Computing system 1500 may be Figure 1An example implementation of the health status identification system 101 in the embodiment of the present invention. The processor 1510, the memory 1520, the storage device 1530 and the input / output device 1540 can be interconnected via the system bus 1550. The processor 1510 is capable of processing instructions for execution within the computing system 1500. Such executed instructions can implement one or more components in, for example, the detection controller 710, the client device 730, etc. In some exemplary embodiments, the processor 1510 can be a single-threaded processor. Alternatively, the processor 1510 can be a multi-threaded processor. The processor 1510 is capable of processing instructions stored on the memory 1520 and / or the storage device 1530 to display a user interface for display via the input / output device 1540, for example Figure 1 The display system 106 or Figure 7 Graphic information of user interface 735 in.

[0181] Memory 1520 is a computer-readable medium, such as a volatile or non-volatile computer-readable medium, that stores information within computing system 1500. For example, memory 1520 may store a data structure representing a configuration object database. Storage device 1530 can provide persistent storage for computing system 1500. Storage device 1530 may be a floppy disk device, a hard disk device, an optical disk device, a tape device, or other suitable persistent storage device. Storage device 1530 may be Figure 1 1500. The input / output device 1540 provides input / output operations for the computing system 1500. In some exemplary embodiments, the input / output device 1540 includes a keyboard and / or a pointing device. In various specific implementations, the input / output device 1540 includes a display unit for displaying a graphical user interface.

[0182] According to some exemplary embodiments, the input / output device 1540 may provide input / output operations for network devices. For example, the input / output device 1540 may include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0183] In some exemplary embodiments, computing system 1500 can be used to execute various interactive computer software applications that can be used to organize, analyze and / or store data in various formats. Alternatively, computing system 1500 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents and / or any other objects, etc.), computing functions, communication functions, etc. The application can include various additional functions or can be an independent computing product and / or function. After activation within the application, the function can be used to generate a user interface provided via input / output device 1540. The user interface can be generated by computing system 1500 and presented to the user (e.g., on a computer screen monitor, etc.).

[0184] One or more aspects or features of the subject matter described herein can be implemented with digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include a specific implementation in one or more computer programs, which are executable and / or interpretable on a programmable system, which includes at least one programmable processor (which can be dedicated or general, coupled to receive data and instructions from it and send data and instructions to it), a storage system, at least one input device, and at least one output device. A programmable system or computing system can include a client and a server. Typically, the client and the server are remotely arranged from each other, and generally interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on each computer and a client-server relationship between each other.

[0185] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer product, device and / or equipment (such as, for example, disks, optical disks, memories and programmable logic devices (PLDs)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily (such as, for example, a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium). A machine-readable medium may store such machine instructions in a temporary manner (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores) alternatively or additionally.

[0186] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and a pointing device (such as, for example, a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive tracking pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0187] In the above description and claims, phrases such as "at least one" or "one or more" may appear, followed by a list of combinations of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, the phrase is intended to represent any element or feature listed alone, or any other described element or feature combined with any other described element or feature. For example, the phrases "at least one of A and B"; "one or more of A and B"; "A and / or B" are each intended to represent "single A, single B, or A and B together". A similar interpretation also applies to lists including three or more items. For example, the phrases "at least one of A and B"; "one or more of A, B, and C" and "A, B, and / or C" are each intended to represent "single A, single B, single C, A and B together, A and C together, B and C together, or A and B and C together". The use of the term "based on" above and in the claims is intended to represent "based at least in part", so that undescribed features or elements are also permissible.

[0188] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The embodiments described in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are only some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to those features and / or variations described herein, other features and / or variations may also be provided. For example, the above-mentioned specific implementations may be directed to various combinations and sub-combinations of the disclosed features and / or to combinations and sub-combinations of several further features disclosed above. In addition, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other specific implementations may be within the scope of the following claims.

[0189] VII. Exemplary Definitions and Context

[0190] The present disclosure is not limited to these exemplary embodiments and applications, nor is it limited to the operation of the exemplary embodiments and applications or the methods described herein.In addition, the drawings may show simplified or partial views, and the sizes of elements in the drawings may be exaggerated or not proportional.

[0191] Where reference is made to a list of elements (e.g., elements a, b, c), such reference is intended to include any one of the elements listed individually, any combination of less than all of the elements listed, and / or any combination of all of the elements listed. The section divisions in the specification are for ease of review only and do not limit any combination of the elements discussed.

[0192] The term "subject" may refer to a subject of a clinical trial, a human or animal being treated, a human or animal being treated for cancer, a human or animal being monitored for remission or recovery, a human or animal being monitored for preventive health (e.g., because of their medical history), or any other human or patient or animal of interest. In each case, "subject" and "patient" are used interchangeably herein.

[0193] The term "OCT image" may refer to an image of a tissue, organ, etc. (such as a retina) scanned or captured using optical coherence tomography (OCT) imaging technology. The term may refer to one or both of a 2D "slice" image and a 3D "volume" image. When not explicitly indicated, the term may be understood to include an OCT volume image.

[0194] Unless otherwise defined, scientific and technical terms used in conjunction with the present teachings described herein shall have the meanings commonly understood by those of ordinary skill in the art. In addition, unless the context otherwise requires, singular terms shall include the plural and plural terms shall include the singular. In general, nomenclature and techniques used in conjunction with chemistry, biochemistry, molecular biology, pharmacology, and toxicology are described herein, and these nomenclatures and techniques are those well known and commonly used in the art.

[0195] As used herein, "substantially" means sufficient to achieve the intended purpose. Thus, the term "substantially" allows for minor, inconsequential variations relative to an absolute or ideal state, dimension, measurement, result, etc., as would be expected by one of ordinary skill in the art, but without significantly affecting the overall performance. When used relative to a numerical value or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within ten percent.

[0196] As used herein, the term "about" used with respect to a numerical value or a parameter or feature that can be expressed as a numerical value means within ten percent of the numerical value. For example, "about 50" means a value within the range of 45 to 55, including the end values.

[0197] The term "ones" means more than one.

[0198] As used herein, the term "plurality" may be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.

[0199] As used herein, the term "a set" refers to one or more. For example, a set of items includes one or more items.

[0200] As used herein, the phrase "at least one of..." when used with a list of items means that different combinations of one or more of the listed items can be used, and only one item in the list may be required. An item can be a specific object, thing, step, operation, process, or category. In other words, "at least one of..." refers to any combination of items or number of items that can be used in the list, but not all items in the list are required. For example, but not limiting, "at least one of item A, item B, or item C" refers to item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and C. In some cases, "at least one of item A, item B, or item C" refers to, but is not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.

[0201] As used herein, a "model" may include one or more algorithms, one or more mathematical techniques, one or more machine learning (ML) algorithms, or a combination thereof.

[0202] As used herein, "machine learning" can include the practice of using algorithms to parse data, learn from it, and then make a determination or prediction about something in the world. Machine learning uses algorithms that can learn from data without relying on rule-based programming.

[0203] As used herein, "artificial neural network" or "neural network" may refer to a mathematical algorithm or computational model that simulates a set of interconnected artificial neurons that processes information based on a connectionist computational approach. A neural network (which may also be referred to as a neural network) may use one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input for the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. In various embodiments, references to "neural network" may be references to one or more neural networks.

[0204] Neural networks can process information in two ways, for example; when the neural network is being trained (e.g., using a training data set), the neural network is in training mode, and when the neural network puts what it has learned into practice (e.g., using a test data set), the neural network is in inference (or prediction) mode. Neural networks can learn through a feedback process (e.g., backpropagation) that allows the network to adjust the weight factors of various nodes in the intermediate hidden layers (modify their behavior) so that the output matches the output of the training data. In other words, a neural network can learn by being fed training data (learning examples) and ultimately learn how to obtain the correct output even if it is presented with a new range or set of inputs.

[0205] VIII. Description of Exemplary Embodiments

[0206] Embodiment 1: A system comprising: at least one data processor; and at least one memory storing instructions, which, when executed by the at least one data processor, cause operations including: applying a machine learning model, the machine learning model being trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on a significance map associated with the diagnosis of neonatal geographic atrophy; and verifying the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0207] Embodiment 2: The system of Embodiment 1, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with suprathreshold contributions to the diagnosis of neonatal geographic atrophy.

[0208] Embodiment 3: A system according to embodiment 1 or embodiment 2, wherein the saliency map is generated by applying Gradient Weighted Class Activation Mapping (GradCAM).

[0209] Embodiment 4: The system of any one of embodiments 1 to 3, wherein the saliency map comprises a heat map.

[0210] Embodiment 5: A system according to any one of embodiments 1 to 4, wherein the machine learning model comprises a classifier based on an artificial neural network (ANN).

[0211] Embodiment 6: A system according to any one of embodiments 1 to 5, wherein the machine learning model includes a classifier based on a residual neural network (RNN).

[0212] Embodiment 7: The system of any one of embodiments 1 to 6, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having multiple two-dimensional B-mode scans.

[0213] Embodiment 8: A system according to any one of embodiments 1 to 7, wherein the machine learning model is trained based on a dataset comprising multiple optical coherence tomography (OCT) volumes annotated with volume-by-volume labels.

[0214] Embodiment 9: The system according to any one of embodiments 1 to 8, wherein the location of the one or more newly formed geographic atrophic lesions is identified by one or more bounding boxes.

[0215] Embodiment 10: A system according to any one of embodiments 1 to 9, wherein the operation further comprises: generating a user interface that displays an indication of the location of the one or more newly formed geographic atrophy lesions on the optical coherence tomography volume of the patient; and verifying the diagnosis of newly formed geographic atrophy and / or the location of the one or more newly formed geographic atrophy lesions based on one or more user inputs received via the user interface.

[0216] Example 11: A computer-implemented method comprising: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on a significance map associated with the diagnosis of neonatal geographic atrophy; and verifying the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0217] Embodiment 12: The method of Embodiment 11, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a supra-threshold contribution to the diagnosis of neonatal geographic atrophy.

[0218] Embodiment 13: The method according to embodiment 11 or embodiment 12, wherein the saliency map is generated by applying gradient weighted class activation mapping (GradCAM).

[0219] Embodiment 14: A method according to any one of embodiments 11 to 13, wherein the saliency map comprises a heat map.

[0220] Embodiment 15: A method according to any one of embodiments 11 to 14, wherein the machine learning model comprises a classifier based on an artificial neural network (ANN).

[0221] Embodiment 16: A method according to any one of embodiments 11 to 15, wherein the machine learning model comprises a classifier based on a residual neural network (RNN).

[0222] Embodiment 17: The method of any one of embodiments 11 to 16, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having multiple two-dimensional B-mode scans.

[0223] Embodiment 18: A method according to any one of embodiments 11 to 17, wherein the machine learning model is trained based on a dataset comprising multiple optical coherence tomography (OCT) volumes annotated with volume-by-volume labels.

[0224] Embodiment 19: The method according to any one of embodiments 11 to 18, wherein the location of the one or more newly formed geographic atrophic lesions is identified by one or more bounding boxes.

[0225] Example 20: A method according to any one of Examples 11 to 19, wherein the operation further includes: generating a user interface that displays an indication of the location of the one or more newly formed geographic atrophy lesions on the optical coherence tomography volume of the patient; and verifying the diagnosis of newly formed geographic atrophy and / or the location of the one or more newly formed geographic atrophy lesions based on one or more user inputs received via the user interface.

[0226] Embodiment 21: A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause operations including: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on a significance map associated with the diagnosis of neonatal geographic atrophy; and verifying the diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions.

[0227] Embodiment 22. A method comprising: receiving an optical coherence tomography (OCT) volumetric image of a subject's retina; generating an output using the OCT volumetric image via a deep learning model, wherein the output indicates whether neoplastic geographic atrophy is detected; and generating a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volumetric image to the output generated by the deep learning model.

[0228] Example 23. A method according to Example 22, wherein the saliency mapping algorithm includes a gradient-weighted class activation mapping (GradCAM) algorithm, and wherein the graph output visually indicates the level of contribution of the set of regions in the OCT volume image to the output generated by the deep learning model.

[0229] Example 24. A method according to any one of Example 22 or Example 23, wherein the OCT volume image includes multiple two-dimensional OCT slice images, and the method further includes: generating an evaluation recommendation based on at least one of the output or the image output, wherein the evaluation recommendation identifies a subgroup of the multiple OCT slice images for further examination.

[0230] Embodiment 25. The method of Embodiment 24, wherein the subset includes less than 5% of the plurality of OCT slice images.

[0231] Example 26. A method according to any one of Examples 22 to 25, further comprising: displaying the image output, wherein the image output comprises a saliency map superimposed on a separate OCT slice image of the OCT volume image and a bounding box surrounding at least one area in the set of areas.

[0232] Example 27. A method according to Example 26, wherein the identification includes: identifying a potential biomarker region as associated with the neonatal geographic atrophy in association with a region in the set of regions; generating a scoring metric for the potential biomarker region; and when the scoring metric meets a selected threshold, identifying the biomarker region as including at least one biomarker for a selected diagnosis of neonatal geographic atrophy.

[0233] Embodiment 28. The method of embodiment 27, wherein the scoring metric comprises at least one of a size of the potential biomarker region or a confidence score for the potential biomarker region.

[0234] Example 29. A method according to any one of Examples 22 to 28, wherein generating the image output includes: using the saliency mapping algorithm to generate a saliency map for the OCT slice image of the OCT volume image, the saliency map indicating the importance of each pixel in the OCT slice image for the diagnosis of neoplastic geographic atrophy; filtering the saliency map to generate a modified saliency map; and superimposing the modified saliency map on the OCT slice image to generate the image output.

[0235] Example 30. A method according to any one of Examples 22 to 28, wherein generating an output via the deep learning model includes: generating an initial output for each of a plurality of OCT slice images forming the OCT volume image to form a plurality of initial outputs; and averaging the plurality of initial outputs to form a health indication output.

[0236] Embodiment 31. A system comprising:

[0237] Non-transitory memory; and

[0238] A hardware processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to: receive an optical coherence tomography (OCT) volumetric image of a subject's retina; generate an output using the OCT volumetric image via a deep learning model, wherein the output indicates whether newborn geographic atrophy is detected; generate a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volumetric image to an output generated by the deep learning model; and display the map output.

[0239] Embodiment 32. A method according to Embodiment 31, wherein the map output comprises a saliency map superimposed on a separate OCT slice image of the OCT volume image.

[0240] Embodiment 33. A system according to embodiment 31 or embodiment 32, wherein the saliency mapping algorithm comprises a gradient-weighted class activation mapping (GradCAM) algorithm.

[0241] Embodiment 34. A system according to any one of embodiments 31 to 33, wherein the deep learning model includes a residual neural network.

[0242] Embodiment 35. A system comprising:

[0243] Non-transitory memory; and

[0244] A hardware processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to: train a deep learning model using a training data set to form a trained deep learning model, the training data set comprising training OCT images that have been labeled as demonstrating neonatal geographic atrophy or not demonstrating neonatal geographic atrophy; receive an optical coherence tomography (OCT) volumetric image of a subject's retina; generate a classification score using the OCT volumetric image via the trained deep learning model, wherein the classification score indicates whether neonatal geographic atrophy is detected; generate a saliency volume map for the OCT volumetric image using a saliency mapping algorithm, wherein the saliency volume map indicates a level of contribution of a set of regions in the OCT volumetric image to a diagnosis of the geographic atrophy generated by the deep learning model; use the saliency volume map to detect a set of potential biomarker regions in the OCT volumetric image; and generate a report confirming detection of neonatal geographic atrophy when at least one of the set of potential biomarker regions meets a set of criteria and when the classification score meets a threshold.

[0245] Embodiment 36. A system according to embodiment 35, wherein the saliency mapping algorithm comprises a gradient-weighted class activation mapping (GradCAM) algorithm.

[0246] Example 37. A method according to Example 35 or Example 36, wherein the classification score is the probability that the OCT volume image demonstrates neoplastic geographic atrophy, and wherein the threshold is a value selected between 0.5 and 0.8.

[0247] Embodiment 38. A system according to any one of Embodiments 35 to 37, wherein the OCT volumetric image includes multiple two-dimensional OCT slice images, and wherein the hardware processor is further configured to read instructions from the non-volatile memory to cause the system to generate an evaluation recommendation based on at least one of the health indication output or the map output, wherein the evaluation recommendation identifies a subset of the multiple OCT slice images for further examination, the subset including less than 5% of the multiple OCT slice images.

[0248] IX. Other considerations

[0249] Although the present teaching is described in conjunction with various embodiments, the present teaching is not intended to be limited to such embodiments. On the contrary, the present teaching encompasses various alternatives, modifications and equivalents that will be understood by those skilled in the art.

[0250] In describing various embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, if the method or process does not rely on the specific order of steps described herein, the method or process should not be limited to the specific sequence of steps listed, and those skilled in the art can readily understand that these sequences can be different and still remain within the spirit and scope of the various embodiments.

Claims

1. A system comprising: at least one data processor; and at least one memory storing instructions that, when executed by the at least one data processor, result in operations comprising: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on the saliency map associated with the diagnosis of neonatal geographic atrophy; and The diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions are verified.

2. The system of claim 1, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with a suprathreshold contribution to the diagnosis of neoplastic geographic atrophy.

3. The system of claim 1, wherein the saliency map is generated by applying Gradient Weighted Class Activation Mapping (GradCAM). The system of claim 1 , wherein the saliency map comprises a heat map.

5. The system of claim 1, wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

6. The system of claim 1, wherein the machine learning model comprises a residual neural network (RNN) based classifier.

7. The system of claim 1, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-mode scans.

8. The system of claim 1, wherein the machine learning model is trained based on a dataset comprising a plurality of optical coherence tomography (OCT) volumes annotated with volume-wise labels.

9. The system of claim 1, wherein the locations of the one or more newly formed geographic atrophic lesions are identified by one or more bounding boxes.

10. The system of claim 1, wherein the operations further comprise: generating a user interface that displays an indication of the location of the one or more nascent geographic atrophic lesions on the optical coherence tomography volume of the patient; as well as The diagnosis of newborn geographic atrophy and / or the location of the one or more newborn geographic atrophy lesions are verified based on one or more user inputs received via the user interface.

11. A computer-implemented method comprising: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on the saliency map associated with the diagnosis of neonatal geographic atrophy; and The diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions are verified.

12. The method of claim 11, wherein the saliency map identifies one or more regions of the optical coherence tomography volume associated with suprathreshold contributions to the diagnosis of neonatal geographic atrophy.

13. The method of claim 11, wherein the saliency map is generated by applying Gradient Weighted Class Activation Mapping (GradCAM). The method of claim 11 , wherein the saliency map comprises a heat map.

15. The method of claim 11, wherein the machine learning model comprises an artificial neural network (ANN) based classifier.

16. The method of claim 11, wherein the machine learning model comprises a residual neural network (RNN) based classifier.

17. The method of claim 11, wherein the optical coherence tomography (OCT) volume comprises a three-dimensional volume having a plurality of two-dimensional B-mode scans.

18. The method of claim 11, wherein the machine learning model is trained based on a dataset comprising a plurality of optical coherence tomography (OCT) volumes annotated with volume-wise labels.

19. The method of claim 11, wherein the locations of the one or more newly formed geographic atrophic lesions are identified by one or more bounding boxes.

20. The method of claim 11, wherein the operations further comprise: generating a user interface that displays an indication of the location of the one or more nascent geographic atrophic lesions on the optical coherence tomography volume of the patient; as well as The diagnosis of newborn geographic atrophy and / or the location of the one or more newborn geographic atrophy lesions are verified based on one or more user inputs received via the user interface.

21. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, result in operations comprising: applying a machine learning model trained to determine a diagnosis of neonatal geographic atrophy (nGA) for a patient based at least on an optical coherence tomography (OCT) volume of the patient; determining the location of one or more neonatal geographic atrophy lesions based at least on the saliency map associated with the diagnosis of neonatal geographic atrophy; and The diagnosis of neonatal geographic atrophy and / or the location of the one or more neonatal geographic atrophy lesions are verified.

22. A method comprising: receiving an optical coherence tomography (OCT) volumetric image of the subject's retina; generating an output using the OCT volume image via a deep learning model, wherein the output indicates whether new geographic atrophy is detected; as well as using a saliency mapping algorithm to generate a graph output for the deep learning model, Wherein the map output indicates the level of contribution of a set of regions in the OCT volume image to the output generated by the deep learning model.

23. A method according to claim 22, wherein the saliency mapping algorithm comprises a gradient-weighted class activation mapping (GradCAM) algorithm, and wherein the map output visually indicates the level of contribution of the set of regions in the OCT volume image to the output generated by the deep learning model.

24. The method according to claim 22 or claim 23, wherein the OCT volume image comprises a plurality of two-dimensional OCT slice images, and the method further comprises: An evaluation recommendation is generated based on at least one of the output or the map output, wherein the evaluation recommendation identifies a subset of the plurality of OCT slice images for further examination.

25. The method of claim 24, wherein the subset includes less than 5% of the plurality of OCT slice images.

26. The method according to any one of claims 22 to 25, further comprising: The map output is displayed, wherein the map output includes a saliency map superimposed on an individual OCT slice image of the OCT volume image and a bounding box surrounding at least one region of the set of regions.

27. The method of claim 26, wherein the identification comprises: identifying, in association with a region in the set of regions, a potential biomarker region as being associated with the neoplastic geographic atrophy; generating a scoring metric for the potential biomarker region; as well as When the scoring metric satisfies a selected threshold, the biomarker region is identified as including at least one biomarker for a selected diagnosis of neonatal geographic atrophy.

28. The method of claim 27, wherein the scoring metric comprises at least one of a size of the potential biomarker region or a confidence score for the potential biomarker region.

29. The method of any one of claims 22 to 28, wherein generating the graph output comprises: generating a saliency map of the OCT slice image for the OCT volume image using the saliency mapping algorithm, the saliency map indicating the importance of each pixel in the OCT slice image for the diagnosis of neoplastic geographic atrophy; filtering the saliency map to generate a modified saliency map; as well as The modified saliency map is superimposed on the OCT slice image to generate the map output.

30. The method of any one of claims 22 to 28, wherein generating an output via the deep learning model comprises: generating an initial output for each of a plurality of OCT slice images forming the OCT volume image to form a plurality of initial outputs; as well as The plurality of initial outputs are averaged to form a health indication output.

31. A system comprising: non-transitory memory; and a hardware processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to: receiving an optical coherence tomography (OCT) volumetric image of the subject's retina; generating an output using the OCT volume image via a deep learning model, wherein the output indicates whether new geographic atrophy is detected; using a saliency mapping algorithm to generate a graph output for the deep learning model, wherein the map output indicates the level of contribution of a set of regions in the OCT volume image to an output generated by the deep learning model; and Displays the graph output.

32. The method of claim 31, wherein the map output comprises a saliency map superimposed on individual OCT slice images of the OCT volume image.

33. A system according to claim 31 or claim 32, wherein the saliency mapping algorithm comprises a Gradient Weighted Class Activation Mapping (GradCAM) algorithm.

34. A system according to any one of claims 31 to 33, wherein the deep learning model comprises a residual neural network.

35. A system comprising: non-transitory memory; and a hardware processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to: training a deep learning model using a training dataset to form a trained deep learning model, the training dataset comprising training OCT images that have been labeled as demonstrating neonatal geographic atrophy or not demonstrating neonatal geographic atrophy; receiving an optical coherence tomography (OCT) volumetric image of the subject's retina; generating a classification score using the OCT volume image via the trained deep learning model, wherein the classification score indicates whether neoplastic geographic atrophy is detected; generating a saliency volume map for the OCT volume image using a saliency mapping algorithm, wherein the saliency volume map indicates a level of contribution of a set of regions in the OCT volume image to the diagnosis of geographic atrophy generated by the deep learning model; Using the saliency volume map to detect a set of potential biomarker regions in the OCT volume image; as well as A report is generated confirming detection of neonatal geographic atrophy when at least one potential biomarker region in the set of potential biomarker regions satisfies a set of criteria and when the classification score satisfies a threshold.

36. The system of claim 35, wherein the saliency mapping algorithm comprises a Gradient Weighted Class Activation Mapping (GradCAM) algorithm.

37. The method of claim 35 or claim 36, wherein the classification score is the probability that the OCT volume image demonstrates neoplastic geographic atrophy, and wherein the threshold is a value selected between 0.5 and 0.

8.

38. A system according to any one of claims 35 to 37, wherein the OCT volumetric image includes a plurality of OCT slice images in two dimensions, and wherein the hardware processor is further configured to read instructions from the non-transitory memory to cause the system to generate an evaluation recommendation based on at least one of the health indication output or the map output, wherein the evaluation recommendation identifies a subset of the plurality of OCT slice images for further examination, the subset including less than 5% of the plurality of OCT slice images.

Images