Segmentation of Optical Coherence Tomography (OCT) Images

Abstract

A system and method for performing automatic retinal segmentation. Performing the automatic retinal segmentation includes receiving an image input relating to a subject's retina. Layer element data is generated using the image input and a first neural network. The layer element data identifies a set of retinal layer elements. Initial pathological element data is generated using the image input and a second neural network. The initial pathological element data identifies the set of retinal pathological elements. The initial pathological element data is refined with the layer element data to generate refined pathological element data. The refined pathological element data more accurately identifies the set of retinal pathological elements compared to the initial pathological element data.

Description

[Technical field]

[0001] Inventor: Andreas Mountz;Huanxiang Lu;Thomas Felix Albrecht;Fethala Benmansauer;Yvonna Yun Li This application relates to retinal segmentation used in the diagnosis and / or treatment of ophthalmic diseases (or conditions), and more particularly to automated retinal segmentation of optical coherence tomography (OCT) images using machine learning based algorithms for the diagnosis and / or treatment of ophthalmic diseases (e.g., age-related macular degeneration (AMD), diabetic macular edema (DME), etc.). [Background technology]

[0002] Ophthalmological diseases and conditions are varied and may include retinal diseases and conditions. Retinal diseases may affect one or more parts of the retina, the tissue at the back of the eye used to capture light and convert it into signals (e.g., electrical, chemical) that are sent to the brain. Retinal diseases may result in complications such as swelling of the macula (called macular edema). Many retinal diseases may affect vision and lead to vision loss or even blindness. Treatment may involve halting or slowing the disease to maintain, improve, or restore vision.

[0003] Age-related macular degeneration (AMD) is the leading cause of vision loss in subjects over the age of 50. AMD may initially manifest as dry AMD and progress to wet AMD. In the dry form, small deposits (drusen) form on the retina under the macula, which cause the retina to deteriorate over time. In the wet form, which may also be called neovascular AMD (nAMD), abnormal blood vessels originating from the choroid layer of the eye grow into the retina and leak fluid from the blood into the retina. Once the fluid enters the retina, it may immediately distort the subject's vision and over time it may damage the retina itself, for example by causing loss of retinal photoreceptors. The fluid may cause the macula to separate from its base, resulting in severe and rapid vision loss.

[0004] Diabetic macular edema (DME), a complication of diabetic retinopathy (DR), is a frequent cause of vision loss experienced by patients with diabetes. In DME, excess fluid accumulates in the extracellular spaces within the retina in the macular area (e.g., in the inner nuclear layer, outer plexiform layer, Henle fiber layer, and subretinal space).

[0005] Optical coherence tomography (OCT) can provide detailed scans of the macula to help detect macular degeneration, diabetic macular edema, and other eye problems much earlier than was previously possible.

[0006] To investigate the extent of retinal deterioration, for example with AMD or DME, OCT images of the retina (e.g., time-domain optical coherence tomography (TD-OCT) or spectral-domain optical coherence tomography (SD-OCT) images) may be acquired and used to identify features that may be associated with various degenerative levels of disease (e.g., AMD, DME). SD-OCT is an imaging technique in which light is directed at the retina at various optical frequencies and the reflected light is collected to capture two-dimensional or three-dimensional high-resolution cross-sectional images of the retina via a detected interference signal as a function of frequency. Different features captured in SD-OCT images can be used in identifying and determining the severity of retinal disease via retinal segmentation, which may help guide disease diagnosis and / or treatment. However, currently available techniques used to extract, understand, and / or interpret such features may be tedious and / or prone to error. Thus, the cumbersome nature of the retinal disease investigation process may be a limiting factor in disease diagnosis and / or treatment. Therefore, it may be desirable to have one or more methods and / or systems that recognize and take these issues into account. Summary of the Invention

[0007] In one or more embodiments, a method for performing retinal segmentation is provided. The method includes receiving an optical coherence tomography (OCT) image of the retina. A layer element image is generated using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indices. An initial pathological element image is generated using the OCT image and a second neural network, the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indices that assign a different group of pixels to each retinal pathological element of the set of retinal pathological elements. The initial pathological element image is refined with the layer element image to generate a refined pathological element image. The refined pathological element data visually identifying a set of retinal pathological elements using the set of pathological element indices, the set of pathological element indices assigning an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0008] In one or more embodiments, a method for performing retinal segmentation is provided. The method includes receiving an optical coherence tomography (OCT) image of the retina and generating, via a neural network, a multi-channel map using the OCT image. The multi-channel map includes a plurality of segmented images, each segmented image of the plurality of segmented images identifying a corresponding retinal layer of interest. A layer element image is generated using the multi-channel map, the layer element image identifying a set of retinal layer elements using a set of layer element indices. An initial pathological element image is refined using the layer element image to generate a refined pathological element image that visually identifies the set of retinal pathological elements using the set of pathological element indices, the refined pathological element image identifying at least one retinal pathological element in the set of retinal pathological elements more accurately than the initial pathological element image.

[0009] In one or more embodiments, a system for performing automated retinal segmentation is provided, the system comprising a non-transitory memory and a data processor coupled to the non-transitory memory. The data processor is configured to read instructions from the non-transitory memory to cause the system to perform steps including receiving an optical coherence tomography (OCT) image of the retina; generating a layer element image using the OCT image and a first neural network, where the layer element image identifies a set of retinal layer elements using a set of layer element indices; generating an initial pathological element image using the OCT image and a second neural network, where the initial pathological element image visually identifies the set of retinal pathological elements using a set of pathological element indices that assigns a different group of pixels to each retinal pathological element of the set of retinal pathological elements; and refining the initial pathological element image using the layer element image to generate a refined pathological element image, where the refined pathological element image visually identifies the set of retinal pathological elements using the set of pathological element indices and the set of pathological element indices assigns an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0010] In one or more embodiments, a method for performing automatic retinal segmentation is provided. The method includes receiving an image input relating to a subject's retina. Layer element data is generated using the image input and a first neural network. The layer element data identifies a set of retinal layer elements. Initial pathological element data is generated using the image input and a second neural network. The initial pathological element data identifies a set of retinal pathological elements. The initial pathological element data is refined with the layer element data to generate refined pathological element data. The refined pathological element data more accurately identifies the set of retinal pathological elements compared to the initial pathological element data. [Brief description of the drawings]

[0011] For a more complete understanding of the principles disclosed herein and their advantages, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0012] [Figure 1] 1 is a block diagram of a retinal segmentation system according to various embodiments.

[0013] [Diagram 2] 1 illustrates an example of a process flow for performing retinal segmentation of optical coherence tomography (OCT) images using machine learning based algorithms, according to various embodiments.

[0014] [Diagram 3] FIG. 1 is a block diagram illustrating a neural network with a multi-channel training method that may be used in a retina segmentation system according to various embodiments.

[0015] [Figure 4] 1 is a flowchart of a method for performing retinal segmentation, according to various embodiments.

[0016] [Diagram 5] 4 is a flowchart of a method for generating a layer element image according to various embodiments.

[0017] [Figure 6] 1 is a flowchart of a method for performing retinal segmentation, according to various embodiments.

[0018] [Figure 7] 11 is a flowchart of another method for performing automatic retinal segmentation, according to various embodiments.

[0019] [Figure 8A] 13A-13C are illustrations of retinal segmentation results according to various embodiments. [Figure 8B]13A-13C are illustrations of retinal segmentation results according to various embodiments.

[0020] [Figure 9] FIG. 1 is a schematic diagram of an example of a neural network that may be used to implement a computer-based model, according to various embodiments.

[0021] [Figure 10] FIG. 1 is a block diagram of a computer system according to various embodiments.

[0022] It should be understood that the figures are not necessarily drawn to scale, and that objects in the figures are not necessarily drawn to scale relative to each other. The figures are intended to provide clarity and understanding of various embodiments of the devices, systems, and methods disclosed herein. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like parts. Furthermore, it should be appreciated that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] I. Overview Various types of ophthalmic diseases (or conditions) may be detected, diagnosed, and / or treated using detailed scans of the retina in the macular region. As an example, neovascular age-related macular degeneration (nAMD) may be detected, diagnosed, and / or treated using detailed scans of the retina in the macular region. As another example, diabetic macular edema (DME) may be detected, diagnosed, and / or treated using detailed scans of the retina in the macular region. The embodiments described herein provide improved techniques for automatic retinal segmentation of retinal images (e.g., retinal scans) that are more accurate and reliable than existing methods for processing retinal images. More accurate and more reliable retinal segmentation may help ensure more accurate and complete diagnostic and / or treatment solutions for patients with ophthalmic diseases, such as, but not limited to, nAMD and DME.

[0024] Retinal segmentation involves the detection and identification of one or more retinal (e.g., retina-related) elements in a retinal image. A retinal element may consist of at least one of a retinal layer element or a retinal pathology element. The detection and identification of one or more retinal layer elements may be referred to as layer element (or retinal layer element) segmentation. The detection and identification of one or more retinal pathology elements may be referred to as pathology element (or retinal pathology element) segmentation.

[0025] The retinal layer element may be, for example, a retinal layer or a boundary associated with a retinal layer. Examples of retinal layers include, but are not limited to, an inner limiting membrane (ILM) layer, a retinal nerve fiber layer, a ganglion cell layer, an inner plexiform layer, an inner nuclear layer, an outer plexiform layer, an outer nuclear layer, an outer limiting membrane (ELM) layer, a photoreceptor layer, a retinal pigment epithelium (RPE) layer, a layer of RPE detachment, a Bruch's membrane (BM) layer, a choriocapillaris layer, a choroidal stromal layer, an ellipsoid zone (EZ), and other types of retinal layers. In some cases, a retinal layer may be composed of one or more layers. As an example, a retinal layer may be an outer plexiform layer-Henle fiber layer (OPL-HFL). A boundary associated with a retinal layer may be, for example, an inner boundary of a retinal layer, an outer boundary of a retinal layer, a boundary associated with a pathological feature of a retinal layer (e.g., an inner or outer boundary of a retinal layer detachment), or some other type of boundary. For example, the boundary may be an inner boundary of a peeling layer of the RPE (IB-RPE), an outer boundary of a peeling layer of the RPE (OB-RPE), or another type of boundary.

[0026] Retinal pathology elements may include, for example, fluid (e.g., fluid pockets), cells, solid material, or combinations thereof, indicative of retinal pathology (e.g., a disease or condition such as AMD or DME). For example, the presence of a particular retinal region may be indicative of nAMD or DME. Examples of retinal pathology elements include, but are not limited to, intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pockets, drusen, fibrotic development, and disruption. In some cases, the retinal pathology element may be a disruption (e.g., discontinuity, delamination, loss, etc.) of a retinal layer or zone. For example, the disruption may be a disruption of the ellipsoid region, ELM, RPE, or another layer or region. The disruption may represent damage or loss of cells (e.g., photoreceptors) in the area of ​​the disruption.

[0027] Further, the retinal pathology element may include a characteristic or subtype of one of the following: fluid (e.g., IRF, SRF, PED-associated fluid), material (e.g., HRM, SHRM, IHRM), lesion (e.g., HRF, SHRM lesion), or disruption. In particular, examples of retinal pathology elements may include the characteristics and / or subtypes of the different types of elements and disruptions described above that may be detected and identified via retinal segmentation. For example, whether the retinal fluid is clear or cloudy may be a detectable and identifiable characteristic of the retinal fluid. Thus, in some examples, the retinal pathology element may be a clear IRF, a cloudy IRF, a clear SRF, a cloudy SRF, some other type of clear retinal fluid, some other type of cloudy retinal fluid, or a combination thereof. In some cases, with respect to SHRM, shape characteristics (e.g., tall SHRM, dome-shaped SHRM at the fovea, flat SHRM near the fovea, irregular shapes, etc.), boundary characteristics (e.g., poorly defined SHRM, well-defined SHRM), reflectance (e.g., increased reflectance or other levels of reflectance), layering characteristics (e.g., hyperreflective bands in SHRM lesions), and lesion characteristics (e.g., height, width and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that may be detected and identified via retinal segmentation.

[0028] Existing methodologies and systems for performing retinal segmentation may be more time consuming than desired. For example, some currently available methodologies require manual annotation of images (e.g., a human grader who detects retinal elements and annotates the images to identify retinal elements). This type of process can be tedious, prone to human error, and can be a bottleneck that increases the overall time and effort required for overall image analysis in the detection, diagnosis, and / or treatment of ophthalmic diseases or conditions. Furthermore, in some cases, manual grading of pathologies via images may be too tedious or otherwise infeasible for a human grader (e.g., when there are hundreds or thousands of images to grade, when very small objects are scattered throughout the images, etc.). Furthermore, manual grading by a human grader may introduce undesirable bias or variability into the grading.

[0029] Some currently available methodologies use computer processing to perform segmentation of retinal layers and perform segmentation of retinal fluid, but these methodologies are not as accurate as desired. For example, some currently available methodologies use algorithms built into OCT imaging devices that are less reliable than desired. These algorithms may not be able to perform retinal segmentation accurately in cases of, for example, choroidal hyperpermeability atrophy. Furthermore, using data generated by retinal segmentation algorithms included within OCT imaging devices provided by different vendors may cause problems because different vendors have different definitions for central subfield thickness (CST). Thus, CST measurements generated using one OCT imaging device may not be comparable to CST measurements generated using another OCT imaging device.

[0030] Thus, the embodiments described herein provide methodologies and systems for performing automatic retinal segmentation of retinal elements in a manner that improves accuracy and reduces processing time. In particular, the methodologies and systems disclosed herein relate to automatic retinal segmentation of retinal scans based on algorithms that use machine learning. The embodiments described herein enable grading and retinal segmentation of large numbers of images and across images more accurately and efficiently than is possible with currently available methodologies and systems. Furthermore, the embodiments described herein enable a finer level of detail in retinal segmentation because retinal segmentation is performed at the pixel level. The embodiments described herein also provide greater predictability and reliability because overall bias and variability are reduced.

[0031] The disclosed methodology and system uses machine learning to automatically perform retinal segmentation of OCT images. The OCT images may take the form of, but are not limited to, time-domain optical coherence tomography (TD-OCT) images, spectral-domain optical coherence tomography (SD-OCT) images, two-dimensional OCT images, three-dimensional OCT images, OCT angiography (OCT-A) images, or combinations thereof. SD-OCT, also known as Fourier-domain OCT, may be referenced with respect to the embodiments described herein, although other types of OCT images are contemplated for use with the methodologies and systems described herein. Thus, the description of the embodiments with respect to images, image types, and techniques provides only non-limiting examples of such images, image types, and techniques.

[0032] In one or more embodiments, one or more OCT images are processed to automatically perform retinal segmentation to generate one or more segmented OCT images. The segmented OCT images use one or more graphical indices to identify one or more retinal elements on the segmented OCT images. For example, one or more color indices, shape indices, pattern indices, shading indices, lines, curves, markers, labels, tags, text features, other types of graphical indices, or combinations thereof, may be used to identify portions (e.g., in pixels) of the OCT images identified as retinal elements.

[0033] As one specific example, a group of pixels may be identified as capturing a particular retinal fluid (e.g., IRF or SRF). The segmented OCT image may identify this group of pixels using a color index. For example, each pixel of the group of pixels may be assigned a color that is unique to a particular retinal fluid and thus assigns each pixel to a particular retinal fluid. As another example, the segmented OCT image may identify the group of pixels by applying a patterned region or shape (continuous or discontinuous) across the group of pixels.

[0034] The segmented OCT image may be used to extract feature data for one or more retinal elements identified in the segmented OCT image. The feature data may include values ​​of any number or combination of features (e.g., quantitative features). Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a boundary associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, height of a lesion (e.g., a SHRM lesion), width of a lesion, area of ​​a lesion, calculated reflectance (e.g., reflectance category or score of a SHRM lesion), and number of hyperreflective foci.

[0035] The methodologies and systems described herein use retinal images, such as OCT images, to identify retinal elements and detect, diagnose, and / or treat ophthalmic diseases, such as AMD, diabetic retinopathy (DR), or DME. For example, an OCT image is first generated (or captured) using a retinal scanner or other type of OCT imaging device. The OCT image may be a TD-OCT image, a SD-OCT image, or some other type of OCT image. In other examples, the OCT image is received (or acquired) from a retinal scanner (or other OCT imaging device) or other source (e.g., data storage, computer, etc.). Once acquired, the OCT image is processed using an algorithm, including one or more artificial intelligence (AI)-based machine learning (ML) algorithms to perform retinal segmentation. For example, the algorithm may process the OCT image using a neural network to perform layer element segmentation and pathological element segmentation.

[0036] The methodologies and systems described herein use layer element segmentation to generate layer element data that is used to refine the pathological element data generated by the pathological element segmentation. For example, an OCT image may be processed through two paths, each of which may be implemented using one or more neural networks. The first path includes performing an automated layer element segmentation to generate layer element data, such as, for example, a layer element image. The layer element image is a segmented OCT image that identifies a set of retinal layer elements using one or more graphical indices (which may be referred to as layer element indices). The second path includes performing an automated pathological element segmentation to generate pathological element data, such as, for example, a pathological element image. The pathological element image is a segmented OCT image that identifies a set of retinal pathological elements using one or more graphical indices (which may be referred to as pathological element indices). In one or more embodiments, the second path includes using the layer element data generated through the first path to refine the pathological element data generated along the second path, such that the refined pathological element data more accurately identifies and locates the set of retinal pathological elements.

[0037] This type of refinement of the pathological element image based on the layer element image ensures a more accurate pathological element segmentation, and thus a more accurate detection, diagnosis, and / or treatment. For example, this type of refinement may automatically correct imaging artifacts and / or defects to improve accuracy and reduce or prevent false positive results that would otherwise occur, as with previous processing methods. In addition to improving the accuracy with which the computer system can perform retinal segmentation, the layer element image generated using neural network processing to refine the pathological element image generated using neural network processing may reduce the overall processing time for retinal segmentation, and therefore the overall time for detection, diagnosis, and / or treatment.

[0038] Recognizing and taking into account the importance and usefulness of methodologies and systems that can provide the aforementioned improvements, the present specification describes various embodiments for performing automatic retinal segmentation, which may include layer element segmentation and pathological element segmentation, using ML-based algorithms. The embodiments described herein may enable more accurate and more reliable retinal segmentation, thereby improving the accuracy and reliability of any detection, diagnosis, and / or treatment methodology that relies on the results of this retinal segmentation.

[0039] II. Machine Learning (ML)-Based Retinal Segmentation II.A. Example System for Automatic Retinal Segmentation FIG. 1 is a block diagram of an image processing system 100 according to various embodiments. The image processing system 100 is used to automatically perform retinal segmentation of a retinal image to aid in the evaluation, detection, diagnosis, and / or treatment of a patient having one or more ophthalmic diseases (or conditions), such as, but not limited to, nAMD, DME, and DR. The image processing system 100 may include a computing platform 102, a data storage 104, and a display system 106. The computing platform 102 may take various forms. In one or more embodiments, the computing platform 102 includes a single computer (or computer system) or multiple computers that communicate with each other. In other examples, the computing platform 102 takes the form of a cloud computing platform, a mobile computing platform (e.g., a smartphone, a tablet, etc.), or a combination thereof.

[0040] The data storage 104 and the display system 106 are each in communication with the computing platform 102. In some examples, the data storage 104, the display system 106, or both may be considered part of the computing platform 102 or may be otherwise integrated with the computing platform 102. Thus, in some examples, the computing platform 102, the data storage 104, and the display system 106 may be separate components that communicate with each other, while in other examples some combination of these components may be integrated together.

[0041] The image processing system 100 includes a retinal segmentation system 108, which may be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, the retinal segmentation system 108 is implemented on the computing platform 102. The retinal segmentation system 108 is used to perform automatic retinal segmentation of an input 110 received for processing. The input 110 may be received from another computing platform, retrieved from a database, uploaded from a cloud computing platform, received via an electronic message (e.g., email), received from a data storage device, retrieved from a data structure, or received in some other manner. In one or more embodiments, the input 110 is retrieved from the data storage 104.

[0042] The input 110 may include an image input, such as, for example, one or more retinal images. In one or more embodiments, the input 110 includes an OCT image 112. The OCT image 112 may be, for example, an SD-OCT image or a TD-OCT image of the retina of a subject experiencing and / or diagnosed with an ophthalmic disorder (e.g., AMD, DR, or DME).

[0043] In some embodiments, input 110 may further include one or more color fundus (CF) images, one or more fundus autofluorescence (FAF) images, one or more fluorescein angiography (FA) images, one or more other types of OCT images (e.g., OCT-A images), one or more other types of retinal images, or combinations thereof. In this manner, input 110 may include multi-modal image input. Use of multi-modal image input may increase the accuracy of retinal segmentation.

[0044] The retina segmentation system 108 includes a layer element segmentation module 114 and a pathological element segmentation module 116, each of which may be implemented using software, firmware, hardware, or a combination thereof. In one or more embodiments, the layer element segmentation module 114 and the pathological element segmentation module 116 are separate modules that work together to perform automatic retinal segmentation. In other embodiments, the layer element segmentation module 114 and the pathological element segmentation module 116 may be integrated together into a single module. The layer element segmentation module 114 and the pathological element segmentation module 116 are used in two different paths of processing.

[0045] The layer element segmentation module 114 is used to perform layer element segmentation to detect and identify retinal layer elements. As previously described in Section I, a retinal layer element may be, for example, a retinal layer or a boundary associated with a retinal layer. Examples of retinal layers include, but are not limited to, an inner limiting membrane (ILM) layer, an outer limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, a layer of an RPE detachment, a Bruch's membrane (BM) layer, an ellipsoid zone (EZ), and other types of retinal layers. A boundary associated with a retinal layer may be, for example, an inner boundary of a retinal layer, an outer boundary of a retinal layer, a boundary associated with a pathological feature of a retinal layer (e.g., an inner or outer boundary of a retinal layer detachment), or some other type of boundary. For example, the boundary may be an inner boundary of an RPE (IB-RPE) detachment layer, an outer boundary of an RPE (OB-RPE) detachment layer, or another type of boundary.

[0046] The pathological element segmentation module 116 is used to perform pathological element segmentation to detect and identify retinal pathological elements. As previously described in Section I, retinal pathological elements may include, for example, fluids, cells, solid materials, or combinations thereof that evidence retinal pathology associated with an ophthalmic disease or condition. For example, the presence of a particular retinal fluid may be a sign of leakage from retinal blood vessels, which may be a sign of nAMD. As another example, the presence of a particular retinal fluid, such as intraretinal fluid, may be a sign of DME. Examples of retinal pathological elements include, but are not limited to, intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pockets, and disruptions. In some cases, retinal pathological elements may be disruptions (e.g., discontinuities, delaminations, losses, etc.) of retinal layers or zones. For example, the disruption may be disruption of the ellipsoid region, the ELM, the RPE, or another layer or region. The disruption may represent damage or loss of cells (e.g., photoreceptors) in the area of ​​the disruption.

[0047] Further, the retinal pathology element may include a characteristic or subtype of one of the following: fluid (e.g., IRF, SRF, PED-associated fluid), material (e.g., HRM, SHRM, IHRM), lesion (e.g., HRF, SHRM lesion), or disruption. In particular, examples of retinal pathology elements may include the characteristics and / or subtypes of the different types of elements and disruptions described above that may be detected and identified via retinal segmentation. For example, whether the retinal fluid is clear or cloudy may be a detectable and identifiable characteristic of the retinal fluid. Thus, in some examples, the retinal pathology element may be a clear IRF, a cloudy IRF, a clear SRF, a cloudy SRF, some other type of clear retinal fluid, some other type of cloudy retinal fluid, or a combination thereof. In some cases, with respect to SHRM, shape characteristics (e.g., tall SHRM, dome-shaped SHRM at the fovea, flat SHRM near the fovea, irregular shapes, etc.), boundary characteristics (e.g., poorly defined SHRM, well-defined SHRM), reflectance (e.g., increased reflectance or other levels of reflectance), layering characteristics (e.g., hyperreflective bands in SHRM lesions), and lesion characteristics (e.g., height, width and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that may be detected and identified via retinal segmentation.

[0048] In some cases, a retinal layer element is associated with a retinal pathology element. For example, a retinal layer element, RPE detachment layer, is associated with a retinal pathology element, PED. Thus, the layer element segmentation module 114 and the pathology element segmentation module 116 may communicate with each other to perform retinal segmentation automatically and more accurately.

[0049] In one or more embodiments, the retina segmentation system 108 performs the automated segmentation using a machine learning system. The machine learning system may include a deep learning system, such as, for example, but not limited to, a neural network system 118. The neural network system 118 may include any number or combination of neural networks. In one or more embodiments, the neural network system 118 takes the form of a convolutional neural network (CNN) system that includes one or more convolutional neural networks. For example, a CNN may include multiple neural networks, each of which may itself be a convolutional neural network.

[0050] In one or more embodiments, a first portion of the neural network system 118 is implemented in the layer element segmentation module 114, while a second portion of the neural network system 118 is implemented in the pathological element segmentation module 116. For example, the layer element segmentation module 114 may include a first neural network 120 of the neural network system 118, and the pathological element segmentation module 116 may include a second neural network 122 of the neural network system 118.

[0051] Each of the first neural network 120 and the second neural network 122 may itself be composed of a set of neural networks. In one or more embodiments, the first neural network 120 and the second neural network 122 differ in at least one neural network. In other words, the second neural network 122 may include at least one neural network that is different from one or more neural networks in the first neural network 120. In other embodiments, the first neural network 120 and the second neural network 122 may include the same one or more types of neural networks. For example, the same one or more types of neural networks may be used to perform both layer element segmentation and pathological element segmentation. In some cases, the first neural network 120, the second neural network 122, or both may include one or more mathematical algorithms or functions in addition to a set of neural networks.

[0052] In one or more embodiments, the input 110 is processed along a first path using a layer element segmentation module 114, which performs automatic layer element segmentation using a first neural network 120. For example, the layer element segmentation module 114 may receive the input 110 (e.g., an OCT image 112) at the first neural network 120 for processing. In some embodiments, the layer element segmentation module 114 pre-processes the input 110 prior to inputting the input 110 to the first neural network 120 to enable focused attention on specific regions of interest. This pre-processing may include, for example, reducing noise and / or artifacts in the input 110 that may otherwise impair the ability to properly evaluate the specific regions of interest. In some cases, the first neural network 120 is trained to pre-process the input 110.

[0053] The layer element segmentation module 114 uses a first neural network 120 to process a received input (e.g., the input 110 or a preprocessed image input) to perform automatic layer element segmentation and generate layer element data 124 of a set of retinal layer elements detected in the input 110 or the preprocessed image input. The layer element data 124 may include, for example, but is not limited to, a layer element image (which may also be referred to as a layer element segmentation image), pixel data assigning each pixel or pixel section to a retinal layer element, image coordinates defining each retinal layer element, other information of the set of detected retinal layer elements, or combinations thereof.

[0054] The layer element image, which may be a layer element OCT image, includes a set of graphical indices, which may be referred to as a set of layer element indices. The set of layer element indices identifies a set of retinal layer elements. The layer element indices may take the form of, for example, but not limited to, a color indices, a shape indices, a pattern indices, a shading indices, a line, a curve, a marker, a label, a tag, text, another type of graphical indices, or a combination thereof. In some cases, two or more layer element indices may identify the same retinal layer element. For example, a particular color may be used to identify pixels representing a particular retinal layer element, while a label may be used to name or identify a particular retinal layer element associated with a particular color.

[0055] In some examples, the layer element indicia for identifying a retinal layer element that is a boundary associated with a retinal layer takes the form of a colored and / or patterned curve (continuous or discontinuous) on the layer element image, the curve representing the boundary. In other examples, the layer element indicia for identifying a retinal layer element that is a retinal layer may take the form of a colored and / or patterned area or shape (continuous or discontinuous) on the layer element image. The area or shape may represent, for example, the total thickness of the corresponding retinal layer.

[0056] In some embodiments, the first neural network 120 receives the input 110 (or a pre-processed image input) and generates a multi-channel map 125, which is then used to generate the layer element data 124. The multi-channel map 125 may be composed of a plurality of segmented images, each segmented image of the plurality of segmented images corresponding to a different retinal layer element or different retinal layer of interest. As an example, the plurality of segmented images may include a different segmented image for each retinal layer element of interest. As another example, the plurality of segmented images may include a different segmented image for each retinal layer of interest. In some cases, there may be two or more retinal layer elements of interest that correspond to the same retinal layer (e.g., inner and outer boundaries of the same retinal layer).

[0057] The first neural network 120 may output a multi-channel map 125, and the layer segmentation module 114 may further process the multi-channel map 125 using any number or combination of various mathematical techniques (e.g., curve fitting, logistic function(s), smoothing function(s), another type of function or algorithm, or a combination thereof) to generate the layer element data 124. In other embodiments, the multi-channel map 125 may be generated as an intermediate output by the first neural network 120, and the first neural network 120 then uses the multi-channel map 125 to generate the layer element data 124 as an output of the first neural network 120.

[0058] In yet other embodiments, the multi-channel map 125 may be processed to generate initial layer element data 126, which is then refined to form the layer element data 124 (which may then be referred to as refined layer element data). The initial layer element data 126 may include, but is not limited to, a layer element image (which may also be referred to as a layer element segmentation image), pixel data that assigns each pixel or pixel section to a retinal layer element, image coordinates that map each retinal layer element, other information for the set of detected retinal layer elements, or a combination thereof. However, in these examples, the initial layer element data 126 may be a first approximation.

[0059] As an example, the initial layer element data 126 may include an initial layer element image having at least one layer element index that identifies a boundary associated with a retinal layer of interest. This initial layer element image may be processed using any number or combination of various mathematical techniques (e.g., curve fitting, smoothing functions, another type of function or algorithm, or combinations thereof) to refine the initial layer element image to generate a refined layer element image that forms at least a portion of the layer element data 124. In one or more embodiments, this refinement may be smoothing of the identified boundary.

[0060] In other embodiments, the initial layer element data 126 may be generated as an intermediate output by the first neural network 120, which then uses the initial layer element data 126 to generate the layer element data 124 as an output of the first neural network 120. In this manner, the layer element data 124 may be generated in any number of different ways by the layer element segmentation module 114 within the first processing pass.

[0061] In one or more embodiments, the input 110 is also processed along a second path using a pathological element segmentation module 116, which performs pathological element segmentation using a second neural network 122 of the neural network system 118. For example, the pathological element segmentation module 116 may receive the input 110 (e.g., the OCT image 112) at the second neural network 122 for processing. In some embodiments, the pathological element segmentation module 116 pre-processes the input 110 before inputting the input 110 to the second neural network 122 to allow focused attention to specific regions of interest. This pre-processing may include, for example, reducing noise and / or artifacts in the input 110 that may otherwise impair the ability to properly evaluate the specific regions of interest. In some cases, the second neural network 122 is trained to pre-process the input 110.

[0062] The pathological element segmentation module 116 uses a second neural network 122 to process a received input (e.g., the input 110 or a preprocessed image input) to perform automatic pathological element segmentation and generate initial pathological element data 128 related to a set of pathological layer elements detected in the input 110 or the preprocessed image input. The initial pathological element data 128 may include, for example, but is not limited to, a pathological element image (which may also be referred to as a pathological element segmented image), pixel data assigning each pixel or pixel section to a retinal pathological element, image coordinates defining each retinal pathological element, other information about the set of detected retinal pathological elements, or a combination thereof.

[0063] The pathological element image, which may be a pathological element OCT image, includes a set of graphical indices, which may be referred to as a set of pathological element indices. The set of pathological element indices identifies a set of retinal pathological elements. The pathological element indices may take the form of, for example, but not limited to, color indices, shape indices, pattern indices, shading indices, lines, curves, markers, labels, tags, text, other types of graphical indices, or combinations thereof. In some cases, two or more pathological element indices may identify the same retinal pathological element. For example, a particular color may be used to identify pixels representing a particular retinal pathological element, while a label may be used to name or identify a particular retinal pathological element associated with a particular color.

[0064] In some instances, the pathological element indicia for identifying retinal pathological elements that are retinal fluid may take the form of colored and / or patterned regions or shapes (continuous or discontinuous) on the pathological element image, which may represent, for example, pockets formed by retinal fluid.

[0065] The initial pathological element data 128 output from the second neural network 122 may then be further processed and refined by the pathological element segmentation module 116. For example, the pathological element segmentation module 116 receives the layer element data 124 (or at least a portion of the layer element data 124) from the layer element segmentation module 114. The pathological element segmentation module 116 uses both the initial pathological element data 128 and the layer element data 124 to refine the initial pathological element 128 and generate pathological element data 132, which may be referred to as refined pathological element data.

[0066] Similar to the initial pathological element data 128, the pathological element data 132 may include, but is not limited to, for example, a pathological element image (which may also be referred to as a pathological element segmentation image), pixel data that assigns each pixel or pixel section to a retinal pathological element, image coordinates that define each retinal pathological element, other information about the set of detected retinal pathological elements, or combinations thereof. The pathological element data 132 more accurately identifies and locates the set of retinal pathological elements of interest compared to the initial pathological element data 128. For example, the pathological element data 132 may include a refined pathological element image with a set of pathological element indices that more accurately identifies at least one corresponding retinal pathological element compared to the initial pathological element data 128.

[0067] In one or more embodiments, the pathological element segmentation module 116 uses the layer element data 124 to constrain the allowable area for the set of retinal pathological elements identified in the pathological element data 132. For example, a portion of the layer element data 124 corresponding to two retinal layers may be used to constrain the allowable area for the retinal pathological elements such that the retinal pathological elements are not identified as extending beyond the allowable area for the retinal pathological elements. As one specific example, the layer element data 124 may be used to constrain the allowable area for intraretinal fluid in the pathological element image such that the intraretinal fluid is not identified by the corresponding pathological element indicator as crossing into the subretinal space.

[0068] Thus, the layer element data 124 can be used to refine anatomical characterizations of the set of retinal pathological elements identified in the pathological element data 132 using one or more pathological element indices corresponding to the retinal pathological elements. The anatomical characterizations of the retinal pathological elements can include, for example, but are not limited to, at least one of the location, size, shape, length, width, thickness, volume, or other characteristics of the retinal pathological elements.

[0069] In other embodiments, the initial pathological element data 128 may be an intermediate output of the second neural network 122, and the layer element data 124 may be input to the second neural network 122 to refine the initial pathological element data 128. In these examples, the second neural network 122 outputs pathological element data 132.

[0070] By refining the initial pathological element data 128 using the layer element data 124, the overall accuracy of the pathological element segmentation module 116 that generates the pathological element data 132 is improved. This improvement in accuracy may be achieved in any future analyses performed using the pathological element data 132.

[0071] For example, the feature extraction system 134 may be implemented on the computing platform 102. The feature extraction system 134 may be used to automatically extract feature data 136 from the pathological element data 132 and possibly the layer element data 124. The feature data 136 may include values ​​for any number or combination of features (e.g., quantitative features). Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a boundary associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of hyperreflective foci.

[0072] Refining the initial pathological element data 128 to form (refined) pathological element data 132 increases the accuracy of the extracted feature data 136. Furthermore, any detection, diagnosis, and / or treatment methodologies that rely on the pathological element data 132 and / or the feature data 136 extracted from the pathological element data 132 may be more accurate.

[0073] In one or more embodiments, the feature data 136 includes values ​​of features associated with an ETDRS (Early Treatment of Diabetic Retinopathy Score) grid. The ETDRS grid divides the retina into nine regions defined by two rings and a central region. The central region represents the foveal center. The two rings include an inner macular ring and an outer macular ring. The inner macular ring is divided into four regions: a superior medial region, a temporal medial region, an inferior medial region, and a nasal medial region. The outer macular ring is divided into four regions: a superior lateral region, a temporal lateral region, an inferior lateral region, and a nasal lateral region.

[0074] In one or more embodiments, feature values ​​(e.g., number of fluid pockets, volume of fluid, etc.) may be generated for the foveal center, the inner macular ring, or the outer macular ring. Feature values ​​may be generated for a particular region (e.g., quadrant) of the inner macular ring or the outer macular ring. In some cases, feature values ​​may be generated for two corresponding regions of two rings (e.g., the upper inner region of the inner macular ring and the upper outer region of the outer macular ring). Thus, feature values ​​may be generated for any single region of the ETDRS grid, a ring of the ETDRS grid, a multi-region area formed by multiple regions of the ETDRS grid, or the central region of the ETDRS grid. More accurate retinal segmentation, as provided by the embodiments described herein, allows for more accurate extraction of feature data for various regions and multi-region areas of the ETDRS grid.

[0075] In one or more embodiments, the neural network system 118 is trained using the training data 140. For example, the first neural network 120 may be trained using a first training data set of the training data 140, and the second neural network 122 may be trained using a second training data set of the training data 140. The first training data set may include, for example, but not limited to, a plurality of training OCT images and training layer element data (e.g., a plurality of training layer element images). The second training data set may include a plurality of training OCT images (which may be the same, partially the same, or different from the plurality of training OCT images in the first training data set) and training pathological element data (e.g., a plurality of training pathological element images).

[0076] 2 is a schematic diagram of an example workflow 200 for performing automatic retinal segmentation using OCT images, according to various embodiments. Workflow 200 is an example of an implementation for automatic retinal segmentation that may be performed using retinal segmentation system 108 of FIG 1. For example, workflow 200 may be implemented using layer component segmentation module 114 and pathological component segmentation module 116 of FIG 1.

[0077] The retina segmentation system 108 receives an input 201 for processing. In one or more embodiments, the input 201 includes an OCT image (e.g., OCT image 112 of FIG. 1). In some embodiments, the OCT image may be a pre-processed OCT image. The input 201 may be sent to a first processing path using a layer component segmentation module 114 and a second processing path using a pathological component segmentation module 116.

[0078] The layer element segmentation module 114 receives an input 201 and processes the input 201 via a neural network step 202. The neural network step 202 may be implemented using a first neural network 120. In one or more embodiments, the first neural network 120 includes a CNN, such as, for example, but not limited to, a U-Net for performing forward prediction.

[0079] The layer element segmentation module 114 may use at least a portion of the first neural network 120 to process the input 201 through a neural network process 202 to generate a multi-channel map 204. The multi-channel map 204 is an example of an implementation for the multi-channel map 125 of Figure 1. The multi-channel map 204 includes a number of segmented images 205.

[0080] In one or more embodiments, each segmented image of the plurality of segmented images 205 corresponds to a different retinal layer. For example, a different segmented image may be generated for each different retinal layer of interest. Additionally, each segmented image may identify the corresponding retinal layer of interest using at least one graphical indicator (e.g., a color indicator, a shape indicator, a pattern indicator, a shading indicator, a marker, a label, a tag, text, other types of graphical indicators, or a combination thereof). The one or more graphical indicators, which may be referred to as layer element indicators, visually identify the portion of the segmented image that represents the corresponding retinal layer of interest. For example, a group of pixels that represent the corresponding retinal layer of interest may be assigned to the corresponding retinal layer of interest and visually identified via a color indicator. This coloring of the pixel groups may visually identify the region (continuous or discontinuous) of the segmented image that represents the corresponding retinal layer of interest.

[0081] The layer element segmentation module 114 processes the multi-channel map 204 via a curve fitting process 206 to generate an initial layer element image 208. The initial layer element image 208 may be an example of an implementation of the initial layer element data 126 of FIG. 1. The curve fitting process 206 may include, for example, performing piecewise logistic curve fitting to approximate at least one boundary associated with each retinal layer of interest identified in the multi-channel map 204. In one or more embodiments, the boundary associated with the retinal layer may be an inner boundary (e.g., an anatomically innermost boundary) in the retinal layer.

[0082] If a region representing the corresponding retinal layer of interest in the segmented image of the multi-channel map 204 is discontinuous (e.g., formed by multiple regions separated by gaps), the curve fitting process 206 approximates a continuous or nearly continuous boundary (e.g., inner boundary, outer boundary, etc.) that extends across the discontinuous region. In this manner, the curve fitting process 206 may be used to identify a single continuous or nearly continuous boundary of the corresponding retinal layer of interest identified in the initial layer element image 208 using at least one graphical indicator (e.g., a colored and / or patterned line that highlights the boundary in the initial layer element image 208).

[0083] In one or more embodiments, one or more boundaries are identified for each retinal layer of interest identified in the multi-channel map 204 and identified on the initial layer element image 208 using any number of layer element indices. In this manner, the multi-channel map 204, comprised of multiple segmented images 205, may be processed to form a single initial layer element image 208. When the initial layer element image 208 identifies such boundaries (e.g., as opposed to the complete thickness of a retinal layer), the initial layer element image 208 may be referred to as a height map.

[0084] Additionally, the layer element segmentation module 114 may process the initial layer element image 208 via a smoothing step 210 to generate a refined layer element image 212. The refined layer element image 212 is an example of an implementation of the layer element data 124 of FIG. 1. The refined layer element image 212 includes a set of layer element indices that more accurately identify the location of boundaries in the refined layer element image 212 as compared to the initial layer element image 208. The smoothing step 210 may be performed, for example, using n-dimensional Gaussian smoothing. This smoothing serves to smooth the curves generated via the curve fitting step 206 in the initial layer element image 208, reduce noise, or both, to generate the refined layer element image 212.

[0085] In the second processing path, the pathological element segmentation module 116 receives the input 201 and processes the input 201 via a neural network step 214. The neural network step 214 may be implemented using a second neural network 122. In one or more embodiments, the second neural network 122 includes a CNN, such as, for example, but not limited to, a U-Net for performing forward prediction.

[0086] The pathological element segmentation module 116 may use at least a portion of the second neural network 122 to process the input 201 through a neural network step 214 to generate an initial pathological element image 216. The initial pathological element image 216 is an example of an implementation of the initial pathological element data 128 of FIG.

[0087] The initial pathological element image 216 identifies a set of retinal pathological elements using one or more graphical indices (e.g., color indices, shape indices, pattern indices, shading indices, markers, labels, tags, text, other types of graphical indices, or combinations thereof). The one or more graphical indices, which may be referred to as pathological element indices, visually identify one or more portions of the initial pathological element image 216 that have been identified as representing a set of retinal pathological elements of interest. For example, a group of pixels that represents a retinal pathological element of interest may be assigned to that retinal pathological element and visually identified via a color indices. The coloring of this group of pixels may visually identify an area (contiguous or discontinuous) of the initial pathological element image 216 that represents a retinal pathological element. This identification is approximate.

[0088] The pathological element segmentation module 116 proceeds to refine the initial pathological element image 216 using the refined layer element image 212. For example, the pathological element segmentation module 116 may receive the refined layer element image 212 from the layer element segmentation module 114. The pathological element segmentation module 116 uses the refined layer element image 212 to perform a refinement process 218 on the initial pathological element image 216, thereby generating a refined pathological element image 220. The refined pathological element image 220 may be an example of an implementation of the pathological element data 132 of FIG. 1. The refined pathological element image 220 identifies a set of retinal pathological elements using one or more pathological element indices more accurately than the initial pathological element image 216.

[0089] The refinement step 218 may refine the initial pathological element image 216, for example, by constraining an allowable area for the set of retinal pathological elements using the refinement layer element image 212 (or data extracted from the refinement layer element image 212). For example, one or more boundaries identified in the refinement layer element image 212 may be used to constrain an allowable area for a retinal pathological element such that a retinal pathological element is not identified as extending beyond the allowable area of ​​the retinal pathological element. As one specific example, one or more boundaries in the refinement layer element image 212 may be used to constrain an allowable area of ​​intraretinal fluid such that intraretinal fluid is not identified by a corresponding pathological element indicator as crossing into the subretinal space in the refined pathological element image 220. In this manner, the refinement step 218 uses the set of pathological element indicators to ensure that the anatomical characterization of the set of retinal pathological elements in the refined pathological element image 220 is accurate (e.g., anatomically feasible, clinically relevant, and / or otherwise appropriate).

[0090] 1 and 2 illustrates a system for automated and reliable identification of retinal layer elements and retinal pathology elements (e.g., nAMD-associated retinal elements, DME-associated retinal elements, etc.) in OCT images of the retina. Improved accuracy of retinal segmentation may enable more accurate and / or clinically relevant diagnostic and / or treatment solutions for patients suffering from, for example, nAMD, DR, DME, or other ophthalmic diseases or conditions.

[0091] FIG. 3 is a schematic diagram illustrating a neural network that can be used in the retina segmentation system 108 of FIG. 1 according to various embodiments. The neural network 300 is an example of an implementation of a neural network in the neural network system 118 of FIG. 1 that can be implemented in the retina segmentation system 108 of FIG. 1. In particular, the neural network 300 may be an example of an implementation of a neural network in the first neural network 120 of FIG. 1 or an example of an implementation of the second neural network 122 of FIG. 1. The neural network 300 may be used in performing automated layer element segmentation. For example, the neural network system 300 may be used to generate a multi-channel map, such as the multi-channel map 125 of FIG. 1 or the multi-channel map 204 of FIG. 2.

[0092] Neural network 300 may include an initial neural network 302, a background neural network 304, and a foreground neural network 306. In one or more embodiments, each of initial neural network 302, background neural network 304, and foreground neural network 306 may be implemented as a fully convolutional network (FCN) (e.g., a stacked FCN). In other embodiments, neural network 300 may include one or more other types or combinations of neural networks.

[0093] The initial neural network 302 receives an image input 308 for processing. The image input 308 may be an example of an implementation of the image input included in input 110 of FIG. 1 or input 201 of FIG. 2. The image input 308 takes the form of an OCT image (e.g., OCT image 112 of FIG. 1). In some cases, the OCT image may be an image received directly from a retinal scanner or other type of OCT imaging device, or may be a pre-processed OCT image. The initial neural network 302 processes the image input 308 to generate a background probability map 310 and a foreground probability map 312.

[0094] The background probability map 310 identifies (or segments) the background of the image input 308. In one or more embodiments, this background may be anything in the image input 308 that is not of interest. For example, the background may be any portion of the image input 308 that is not a retinal layer of interest. In one or more embodiments, the background probability map 310 includes a separate background probability image for each retinal layer of interest, such that the background probability image at the corresponding retinal layer of interest identifies the background of the image relative to the corresponding retinal layer of interest using at least one graphical indicator.

[0095] As an example, the background probability map 310 may include a first background probability image and a second background probability image. The first background probability image identifies background for a first retinal layer of interest by coloring (or shading, patterning, etc.) a group of pixels identified as representing background differently from the remaining pixels in the image. The second background probability image identifies background for a second retinal layer of interest by coloring (or shading, patterning, etc.) a group of pixels identified as representing background differently from the remaining pixels in the image.

[0096] The foreground probability map 312 identifies (or segments) the foreground of the image input 308. In one or more embodiments, this foreground may be anything within the image input 308 that is of interest. For example, the foreground may be any portion of the image input 308 that represents a retinal layer of interest. In one or more embodiments, the foreground probability map 312 includes a separate foreground probability image for each retinal layer of interest, such that the foreground probability image for the corresponding retinal layer of interest uses at least one graphical indicator to identify the corresponding retinal layer of interest.

[0097] As an example, the foreground probability map 312 may include a first foreground probability image and a second foreground probability image. The first foreground probability image identifies a first retinal layer of interest by coloring (or shading, patterning, etc.) a group of pixels identified as representing the first retinal layer of interest differently than the remaining pixels in the image. The second foreground probability image identifies a second retinal layer of interest by coloring (or shading, patterning, etc.) a group of pixels identified as representing the second retinal layer of interest differently than the remaining pixels in the image.

[0098] The background probability map 310 and the image input 308 are combined and sent as input to the background neural network 304 to generate a refined background map 314. The refined background map 314 more accurately identifies (segments) portions of the image input 308 that do not represent retinal layers of interest. For example, the refined background map 314 may include multiple refined background images, each of which identifies the background of the image for a respective retinal layer of interest more accurately than a corresponding background probability image in the background probability map 310.

[0099] The foreground probability map 312 and the image input 308 are combined and sent as an input to the foreground neural network 306 to generate a refined foreground map 316. The refined foreground map 316 more accurately identifies (segments) the portion of the image input 308 that represents one or more retinal layers of interest. For example, the refined foreground map 316 may include multiple refined foreground images, each of which identifies a corresponding retinal layer of interest more accurately than a corresponding foreground probability image in the foreground probability map 312.

[0100] The refined background map 314 and the refined foreground map 316 are then integrated to form a multi-channel map 318. This integration may be performed using a concatenation and / or stacking process of both maps. The multi-channel map 318 may be an example of an implementation of the multi-channel map 125 of FIG. 1 or the multi-channel map 204 of FIG. 2. In one or more embodiments, the multi-channel map 318 includes a separate segmented image for each retinal layer of interest. In other words, each segmented image clearly and precisely identifies the portion of that image that represents the corresponding retinal layer of interest.

[0101] II.B. Example of a Methodology for Automatic Retinal Segmentation 4 is a flowchart of a method 400 for performing retinal segmentation, according to various embodiments. In various embodiments, the method 400 can be implemented using the image processing system 100 of FIG. 1. For example, the method 400 may be implemented using the retinal segmentation system 108 described with respect to FIGS. 1 and 2. In one or more embodiments, portions of the method 400 may be implemented using the neural network 300 of FIG. 3.

[0102] As shown in FIG. 4, the method 400 includes receiving an optical coherence tomography (OCT) image of a subject's retina at step 402. The subject may be suffering from an ophthalmic disease or condition. For example, the subject may be diagnosed as experiencing and / or having AMD (e.g., nAMD), DR, DME, or another ophthalmic disease or condition. In various embodiments, the OCT image may be the OCT image 112 of the input 110 as described with respect to FIG. 1. The OCT image may be, for example, an SD-OCT image or a TD-OCT image.

[0103] The method 400 further includes, in step 404, generating a layer element image using the OCT image and the first neural network, where the layer element image identifies a set of retinal layer elements using a set of layer element indices. The retinal layer element may be, for example, a retinal layer or a boundary associated with a retinal layer. The retinal layer may be, for example, but not limited to, an inner limiting membrane (ILM) layer, an outer limiting membrane (ELM) layer, an ellipsoid zone (EZ), an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, a layer of an RPE detachment, a Bruch's membrane (BM) layer, or another type of retinal layer. The boundary associated with a retinal layer may be, for example, an inner boundary of a retinal layer, an outer boundary of a retinal layer, a boundary associated with a pathological feature of a retinal layer (e.g., an inner or outer boundary of a retinal layer detachment), or some other type of boundary. For example, the boundary may be an inner boundary of an RPE (IB-RPE) detachment layer, an outer boundary of an RPE (OB-RPE) detachment layer, or another type of boundary.

[0104] The set of layer element indices used in the layer element image may be a set of graphical indices, such as, but not limited to, a color indices, a shape indices, a pattern indices, a shading indices, lines, curves, markers, labels, tags, text, or other types of graphical indices.

[0105] In one or more embodiments, the layer element image visually identifies one or more portions of the layer element image identified as representing a retinal layer element of interest. The retinal layer element of interest may be, for example, a boundary associated with a retinal layer. In one or more embodiments, the layer element image may visually identify this retinal layer of interest by assigning a group of pixels representing a boundary to a color assigned to that retinal boundary. When each retinal layer element of interest in the layer element image is a boundary, the layer element image may be referred to as a height map.

[0106] Step 404 may be performed using a first neural network, such as first neural network 120 of FIG. 1. The first neural network may include, for example, but not limited to, at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network. In one or more embodiments, the first neural network is used to perform all of the steps involved in step 404. In other embodiments, the first neural network is used to perform a portion of the steps included in step 404.

[0107] Step 404 may be performed in a variety of ways. Method 500 of FIG. 5 below is one example of a method that may be used to implement step 404.

[0108] The method 400 further includes, at step 406, generating an initial pathological element image using the OCT image and a second neural network, the initial pathological element image visually identifying the set of retinal pathological elements using the set of pathological element indices that assign a different pixel group to each retinal pathological element of the set of retinal pathological elements, which identification may be approximate.

[0109] Retinal pathology elements may include, for example, fluids, cells, solid material, or combinations thereof that evidence retinal pathology associated with an ophthalmic disease or condition. For example, the presence of certain retinal fluids may be a sign of leakage from retinal blood vessels, which may be a sign of nAMD. As another example, the presence of certain retinal fluids, such as intraretinal fluid, may be a sign of DME. Examples of retinal pathology elements include, but are not limited to, intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pockets, and disruptions. In some cases, retinal pathology elements may be disruptions (e.g., discontinuities, delaminations, losses, etc.) of retinal layers or zones. For example, disruptions may be disruptions of the ellipsoid region, ELM, RPE, or another layer or region. The disruption may represent damage or loss of cells (eg, photoreceptors) in the area of ​​the disruption.

[0110] Further, the retinal pathology element may include a characteristic or subtype of one of the following: fluid (e.g., IRF, SRF, PED-associated fluid), material (e.g., HRM, SHRM, IHRM), lesion (e.g., HRF, SHRM lesion), or disruption. In particular, examples of retinal pathology elements may include the characteristics and / or subtypes of the different types of elements and disruptions described above that may be detected and identified via retinal segmentation. For example, whether the retinal fluid is clear or cloudy may be a detectable and identifiable characteristic of the retinal fluid. Thus, in some examples, the retinal pathology element may be a clear IRF, a cloudy IRF, a clear SRF, a cloudy SRF, some other type of clear retinal fluid, some other type of cloudy retinal fluid, or a combination thereof. In some cases, with respect to SHRM, shape characteristics (e.g., tall SHRM, dome-shaped SHRM at the fovea, flat SHRM near the fovea, irregular shapes, etc.), boundary characteristics (e.g., poorly defined SHRM, well-defined SHRM), reflectance (e.g., increased reflectance or other levels of reflectance), layering characteristics (e.g., hyperreflective bands in SHRM lesions), and lesion characteristics (e.g., height, width and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that may be detected and identified via retinal segmentation.

[0111] In one or more embodiments, the initial pathological element image visually identifies one or more portions of the initial pathological element image that have been identified as representing a retinal pathological element of interest. The retinal pathological element of interest may be, for example, subretinal fluid. In one or more embodiments, the initial pathological element image visually identifies the retinal subbody by assigning a group of pixels representing subretinal fluid to a color assigned to the retinal subbody.

[0112] Step 406 may be performed using a second neural network, such as second neural network 122 of FIG. 1. The second neural network may include, for example, but not limited to, at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network. In one or more embodiments, the second neural network is used to perform all of the steps involved in step 406. In other embodiments, the second neural network is used to perform a portion of the steps included in step 406.

[0113] The method 400 further includes, at step 408, refining the initial pathological element image using the layer element image to generate a refined pathological element image, the refined pathological element image visually identifying the set of retinal pathological elements using the set of pathological element indicators, the set of pathological element indicators assigning the updated groups of pixels to at least one retinal pathological element of the set of retinal pathological elements, the refined pathological element image more accurately representing the at least one retinal pathological element of the set of retinal pathological elements compared to the initial pathological element image.

[0114] The refinement in step 408 may be performed in different ways. For example, the refinement in step 408 may include updating a group of pixels in the initial pathological element image that are assigned to a particular retinal pathological element to form an updated group of pixels in the retinal pathological element in the refined pathological element image by constraining an acceptable area for the retinal pathological element based on the layer element image. The acceptable area may be constrained based on what is anatomically feasible, clinically relevant, and / or other suitable. The updated group of pixels includes fewer pixels than the group of pixels.

[0115] In one or more embodiments, a group of pixels in the initial pathological element image may be assigned to a first retinal pathological element of the set of retinal pathological elements using a pathological element index of the set of pathological element indexes. Refining the initial pathological element image may include reassigning a portion of the group of pixels in the initial pathological element image based on whether an anatomical characterization of the first retinal pathological element identified by the pathological element index is anatomically feasible. The anatomical characterization of the first retinal pathological element of the set of retinal pathological elements may include at least one of a position, a size, a shape, a length, a width, a thickness, a volume, or another characteristic of the retinal pathological element.

[0116] In one or more embodiments, the reassignment of the portion of the group of pixels may include, for example, reassigning a first pixel of the group of pixels from a first retinal pathological element of the set of retinal pathological elements to a second retinal pathological element based on the layer element image. Reassigning a pixel to a different retinal pathological element may include, for example, but not limited to, changing the application of a pathological element index associated with the pixel. For example, a pixel may be changed from a first color in the initial pathological element image to a second color in the refined pathological element image.

[0117] In one or more embodiments, the reassignment of the portion of the group of pixels may include, for example, reassigning a second pixel of the group of pixels from a first retinal pathological element to a background based on the layer element image. Reassigning a pixel to the background may include, for example, but not limited to, removing application of a pathological element indicator associated with that pixel. For example, a color previously applied to that pixel in the initial pathological element image may be removed in the refined pathological element image.

[0118] The above examples of reassigning pixels are merely illustrative and are not intended to impose any limitations on the manner in which pixels may be reassigned. The reassignment of pixels in step 408 may be performed based on whether the anatomical characterization of the set of retinal pathological elements presented by the set of pathological element indices in the initial pathological element image is acceptable (e.g., anatomically feasible, clinically appropriate, and / or otherwise appropriate). For example, a pixel annotated with a particular pathological element indices that assigns the pixel to a particular retinal pathological element may be reassigned if the location of the pixel makes it anatomically infeasible to associate with the particular retinal pathological element. Such a determination is made using the layer element image and / or data extracted from the layer element image.

[0119] Method 400 may optionally include, in step 410, using the refined pathological element image to perform an analysis for use in detecting, diagnosing, and / or treating an ophthalmic disease or condition. The ophthalmic disease or condition may be, for example, nAMD, DME, or DR. The analysis in step 410 may include, for example, extracting feature data from the refined pathological element image, and possibly from the layer element image. The feature data may include values ​​for any number or combination of features (e.g., quantitative features). Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a boundary associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of hyperreflective foci.

[0120] In various embodiments, the first neural network described in step 404, the second neural network described in step 406, or both may be trained using training data, such as training data 140 of Figure 1. The first neural network may be trained using a first training data set including, for example, a first plurality of training OCT images and a plurality of training layer element images. The plurality of training layer element images may include training multi-channel maps, training initial layer element images, training refinement layer element images, or a combination thereof.

[0121] 5 is a flow chart of a method 500 for generating a layer element image, according to various embodiments. In various embodiments, the method 500 can be implemented using the image processing system 100 of FIG. 1. For example, the method 500 can be implemented using the retina segmentation system 108 described with respect to FIGS. 1 and 2. In one or more embodiments, the method 500 can be implemented using the neural network 300 of FIG. 3. The method 500 can be an example of a method that can be used to implement step 404 of FIG. 4. The method 500 can include one or more steps or processes of the workflow 200 of FIG. 2.

[0122] As shown in FIG. 5, the method 500 includes generating a multi-channel map using the OCT image via a neural network at step 502. The multi-channel map includes a plurality of segmented images, each segmented image of the plurality of segmented images identifying a corresponding retinal layer of interest. The multi-channel map may be, for example, the multi-channel map 125 of FIG. 1, the multi-channel map 204 of FIG. 2, or the multi-channel map 318 of FIG. 3. The OCT image, which may be the OCT image received at step 402 of the process 400 of FIG. 4, may be, for example, the OCT image 112 of FIG. 1. The neural network may be, for example, the first neural network 120 of FIG. 1 or the neural network 300 of FIG. 3. In one or more embodiments, the neural network may include at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network.

[0123] The method 500 further includes converting the multi-channel map to an initial layer element image that identifies a set of retinal layer elements using the set of layer element indices at step 504. The set of layer element indices assigns different pixel groups in the initial layer element image to each retinal layer element of the set of retinal layer elements. The conversion at step 504 may be performed, for example, by applying a piecewise logistic curve fit to the multi-channel map to generate the initial layer heights. The set of retinal layer elements may relate to the various retinal layers identified in the multi-channel map 204. In some cases, two or more retinal layer elements may correspond to the same retinal layer of interest.

[0124] The method 500 further includes applying smoothing to the initial layer element image to generate the layer element image, at step 506. The smoothing may be performed, for example, using Gaussian smoothing (e.g., n-dimensional (nD) Gaussian smoothing). In one or more embodiments, the initial layer element image and the layer element image both take the form of height maps.

[0125] 6 is a flowchart of another method 600 for performing retinal segmentation, according to various embodiments. In various embodiments, the method 600 can be implemented using the image processing system 100 of FIG. 1. For example, the method 600 may be implemented using the retinal segmentation system 108 described with respect to FIGS. 1 and 2. In one or more embodiments, portions of the method 600 may be implemented using the neural network 300 of FIG. 3.

[0126] The method 600 includes receiving an optical coherence tomography (OCT) image of a retina at step 602. The OCT image may be, for example, the OCT image 112 of FIG.

[0127] The method 600 further includes generating a multi-channel map using the OCT image via a neural network at step 604, the multi-channel map including a plurality of segmented images, each segmented image of the plurality of segmented images identifying a corresponding retinal layer of interest. In one or more embodiments, the neural network includes one or more fully convolutional networks (FCN). In one or more embodiments, the neural network includes a U-Net. The neural network may be, for example, the neural network 300 of FIG. 3.

[0128] Method 600 further includes, at step 606, generating a layer element image using the multi-channel map that identifies a set of retinal layer elements using the set of layer element indices. In one or more embodiments, step 606 includes converting the multi-channel map to an initial layer element image that identifies a boundary associated with a retinal layer of interest identified by the multi-channel map. In some cases, the boundary associated with the retinal layer of interest estimates an inner boundary of the retinal layer. The conversion at step 606 may be performed by applying a piecewise logistic curve fit to the multi-channel map to generate the initial layer element image. In some embodiments, step 606 includes applying smoothing to the initial layer height map to generate the layer element image. In other embodiments, the initial layer element image is used as the layer element image.

[0129] The method 600 may further include, at step 608, refining the initial pathological element image using the layer element image to generate a refined pathological element image that visually identifies the set of retinal pathological elements using the set of pathological element indices. The refined pathological element image identifies at least one retinal pathological element in the set of retinal pathological elements more accurately than the initial pathological element image. The initial pathological element image may have been generated using a different neural network.

[0130] 7 is a flowchart of another method 700 for performing automatic retinal segmentation, according to various embodiments. In various embodiments, the method 700 can be implemented using the image processing system 100 described in FIG 1. For example, the method 700 may be implemented using the retinal segmentation system 108 described with respect to FIGS. 1 and 2.

[0131] Step 702 includes receiving an image input relating to a subject's retina. The image input may be, for example, input 201 of FIG. 2 (or input 110 of FIG. 1). The image input may include an OCT image (e.g., an SD-OCT image).

[0132] Step 704 includes generating layer element data using the image input and the first neural network, the layer element data identifying a set of retinal layer elements. The layer element data may be, for example, layer element data 124 of FIGURE 1. In one or more embodiments, the layer element data includes a layer element image that identifies the set of retinal layer elements using a set of layer element indices.

[0133] A retinal layer element of a set of retinal layer elements is either a retinal layer or a boundary associated with a retinal layer, which may be, for example, but not limited to, an inner limiting membrane (ILM) layer, an outer limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, a layer of an RPE detachment, a Bruch's membrane (BM) layer, an ellipsoid zone (EZ), or other types of retinal layers.

[0134] Step 706 includes generating initial pathological element data using the image input and a second neural network, the initial pathological element data identifying a set of retinal pathological elements. The initial pathological element data may be, for example, initial pathological element data 128 of FIG. 1. In one or more embodiments, the initial pathological element data includes an initial pathological element image that visually identifies the set of retinal pathological elements using a set of pathological element indices that assign different groups of pixels in the initial pathological element image to each retinal pathological element of the set of retinal pathological elements.

[0135] The set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective focus (HRF), retinal fluid pocket, or disruption. In some cases, the retinal pathological element may be a disruption (e.g., discontinuity, delamination, loss, etc.) of a retinal layer or retinal zone. For example, the disruption may be a disruption of the ellipsoid region, ELM, RPE, or another layer or region. The disruption may represent damage or loss of cells (e.g., photoreceptors) in the area of ​​the disruption.

[0136] Step 708 includes refining the initial pathological element data using the layer element data to generate refined pathological element data, where the refined pathological element data more accurately identifies the set of retinal pathological elements compared to the initial pathological element data. The refined pathological element data may be, for example, refined pathological element data 132 of FIG. 1. In one or more embodiments, the refined pathological element data includes a refined pathological element image that visually identifies the set of retinal pathological elements using a set of pathological element indicators, where the set of pathological element indicators assign an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0137] Step 710 may optionally include performing an analysis using the refined pathological element data for use in detecting, diagnosing, and / or treating an ophthalmic disease or condition (e.g., nAMD, DR, or DME). The analysis of step 710 may include, for example, extracting feature data from the refined pathological element data, and possibly from the layer element data. The feature data may include values ​​for any number or combination of features (e.g., quantitative features). Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a boundary associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of hyperreflective foci.

[0138] In one or more embodiments, the retinal pathology may be a biomarker for one or more ophthalmic diseases or conditions. For example, detection of the retinal pathology may indicate the presence of one or more ophthalmic diseases or conditions. The refinement in step 708 improves the accuracy of any disease detection and / or diagnosis made based on the identification of the retinal pathology via the refined pathology data. In some embodiments, performing the refinement in step 708 helps to improve the accuracy of the analysis made in step 710, thereby improving the accuracy of any detection, diagnosis, and / or treatment method or solution based on this analysis.

[0139] III. Exemplary Results of the Systems and Methods Disclosed Herein 8A and 8B are examples of retinal segmentation results according to various embodiments. Fig. 8A is an example of a manual retinal segmentation result 800A according to various embodiments. Fig. 8B is an example of an automated (e.g., automated ML-based) retinal segmentation result 800B according to various embodiments.

[0140] The manual retinal segmentation result 800A is based on annotations performed by experts, such as the Liverpool Reading Center, according to their standard operating procedures. In contrast, the automatic retinal segmentation result 800B is generated via an automatic retinal segmentation system, such as the retinal segmentation system 108 described with respect to FIG. 1 and FIG. 2. Comparing the manual retinal segmentation result 800A with the automatic retinal segmentation result 800B verifies that the embodiments disclosed herein can provide accurate and reliable results using ML-based algorithms. Furthermore, the embodiments disclosed herein can be used to automatically correct image artifacts and / or defects. In some cases, the embodiments described herein provide a fully automatic diagnosis solution for nAMD based on the automatic detection of retinal pathological elements known to be associated with nAMD. In other cases, the embodiments described herein can provide a fully automatic diagnosis solution for other ophthalmic diseases or conditions (e.g., DR, DME) based on the automatic detection of retinal pathological elements known to be associated with such ophthalmic diseases or conditions.

[0141] IV. Artificial Neural Networks 9 is a schematic diagram of an example of a neural network that can be used to implement a computer-based model, according to various embodiments. For example, neural network 900 may be an example of an implementation of a neural network that may be included in first neural network 120, second neural network 122, or both of FIG. 1.

[0142] As shown, neural network 900 includes three layers: an input layer 902, a hidden layer 904, and an output layer 906. Each of the input layer 902, hidden layer 904, and output layer 906 may include one or more nodes. In this example, the input layer 902 includes node 908, node 910, node 912, and node 914, the hidden layer 904 includes node 916 and node 918, and the output layer 906 includes node 920.

[0143] In this example, each node in a layer is connected to all nodes in the adjacent layer. For example, node 908 in the input layer 902 is connected to both node 916 and node 918 in the hidden layer 904. Similarly, node 916 in the hidden layer 904 is connected to all of nodes 908, 910, 912, and 914 in the input layer 902, as well as node 920 in the output layer 906. Although only one hidden layer is shown for neural network 900, neural network 900 may include any number of hidden layers between the input layer 902 and the output layer 906.

[0144] In this example, neural network 900 receives a set of input values ​​(e.g., inputs 1-4) and generates an output value (e.g., output 5). Each node in input layer 902 may correspond to a distinct input value. As an example, the set of input values ​​may include a set of attributes of an image, such as OCT image 112 of FIG. 1. In this example, each node in input layer 902 may correspond to and receive a distinct attribute of the image.

[0145] In some embodiments, each of the nodes 916 and 918 in the hidden layer 904 generates a representation, which may include a mathematical calculation (or algorithm) that generates a value based on input values ​​received from nodes 908, 910, 912, and 914. The mathematical calculation may include assigning different weights to each of the data values ​​received from nodes 908, 910, 912, and 914. The nodes 916 and 918 may include different algorithms and / or different weights assigned to data variables from nodes 908, 910, 912, and 914 such that each of the nodes 916 and 918 may generate different values ​​based on the same input values ​​received from nodes 908, 910, 912, and 914. In some embodiments, the weights initially assigned to the features (or input values) for each of the nodes 916 and 918 may be randomly generated (e.g., using a computer randomizer). The values ​​produced by nodes 916 and 918 may be used by node 920 in the output layer 906 to generate an output value in the neural network 900.

[0146] The neural network 900 may be trained using training data. For example, the training data may include various OCT images. By providing the training data to the neural network 900, the nodes 916 and 918 in the hidden layer 904 may be trained (tuned) to generate optimal outputs in the output layer 906 based on the training data. By successively providing different training data sets and penalizing the neural network 900 when its output is incorrect, the neural network 900 (specifically, the representations of the nodes in the hidden layer 904) may be trained (tuned) to improve its performance in data classification. Tuning the neural network 900 may include adjusting weights associated with each node in the hidden layer 904.

[0147] Although the above description concerns artificial neural networks as an example of machine learning, it is understood that other types of machine learning methods may also be suitable for implementing various aspects of the present disclosure. For example, machine learning may be implemented using a support vector machine (SVM). SVM is a set of related supervised learning methods used for classification and regression. The SVM training algorithm, which may be a non-probabilistic binary linear classifier, may build a model that predicts whether a new example falls into one category or another. As another example, machine learning may be implemented using a Bayesian network. A Bayesian network is an acyclic probabilistic graphical model that represents a set of random variables and their conditional independence in a directed acyclic graph (DAG). A Bayesian network may present a probabilistic relationship between one variable and another variable. Another example is a machine learning engine that uses a decision tree learning model to perform a machine learning process. In some cases, the decision tree learning model may include a classification tree model and a regression tree model.

[0148] In some embodiments, the machine learning engine uses a gradient boosting machine (GBM) model (e.g., XGBoost) as the regression tree model. Other machine learning techniques may be used to implement the machine learning engine, for example, via random forests or deep neural networks. Other types of machine learning algorithms are not described in detail herein for simplicity, and it is understood that the present disclosure is not limited to any particular type of machine learning.

[0149] V. Computer Implemented Systems FIG. 10 is a block diagram of a computer system according to various embodiments. The computer system 1000 may be an example of one implementation for the computing platform 102 previously described in FIG. 1. In one or more examples, the computer system 1000 may include a bus 1002 or other communication mechanism for communicating information and a processor 1004 coupled with the bus 1002 for processing information. In various embodiments, the computer system 1000 may also include memory, which may be a random access memory (RAM) 1006 or other dynamic storage device, coupled to the bus 1002 for determining instructions to be executed by the processor 1004. The memory may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 1004. In various embodiments, the computer system 1000 may further include a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010, such as a magnetic disk or optical disk, may be provided and coupled to bus 1002 for storing information and instructions.

[0150] In various embodiments, the computer system 1000 may be coupled via the bus 1002 to a display 1012, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. An input device 1014, including alphanumeric and other keys, may be coupled to the bus 1002 for communicating information and command selections to the processor 1004. Another type of user input device is a cursor control device 1016, such as a mouse, joystick, trackball, gesture input device, eye gaze-based input device, or cursor direction keys, for communicating directional information and command selections to the processor 1004 and for controlling cursor movement on the display 1012. This input device 1014 typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify a position in a plane. However, it should be understood that input devices 1014 that allow three-dimensional (e.g., x, y, and z) cursor movement are also contemplated herein.

[0151] Consistent with certain implementations of the present teachings, results may be provided by computer system 1000 in response to processor 1004 executing one or more sequences of one or more instructions contained in RAM 1006. Such instructions may be read into RAM 1006 from another computer-readable medium or computer-readable storage medium, such as storage device 1010. Execution of the sequences of instructions contained in RAM 1006 may cause processor 1004 to perform the processes described herein. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present teachings. Thus, implementations of the present teachings are not limited to any specific combination of hardware circuitry and software.

[0152] As used herein, the term "computer-readable medium" (e.g., data store, data storage, storage device, data storage device, etc.) or "computer-readable storage medium" refers to any medium that participates in providing instructions to the processor 1004 for execution. Such media may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Examples of non-volatile media may include, but are not limited to, optical, solid-state, and magnetic disks, such as the storage device 1010. Examples of volatile media may include, but are not limited to, dynamic memory, such as the RAM 1006. Examples of transmission media may include, but are not limited to, coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 1002.

[0153] Common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROMs, and EPROMs, flash EPROMs, any other memory chip or cartridge, or any other tangible medium from which a computer can read.

[0154] In addition to computer-readable media, instructions or data may be provided as signals on a transmission medium included in a communication device or system to provide a sequence of one or more instructions to the processor 1004 of the computer system 1000 for execution. For example, a communication device may include a transceiver having signals indicative of instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in the disclosure herein. Representative examples of data communication transmission connections may include, but are not limited to, a telephone modem connection, a wide area network (WAN), a local area network (LAN), an infrared data connection, an NFC connection, an optical communication connection, and the like.

[0155] It should be understood that the methodologies, flowcharts, diagrams, and accompanying disclosure described herein can be implemented using computer system 1000 as a standalone device or on a distributed network of shared computer processing resources, such as a cloud computing network.

[0156] The methodologies described herein may be implemented by various means depending on the application. For example, the methodologies may be implemented in hardware, firmware, software, or any combination thereof. In the case of a hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or combinations thereof.

[0157] In various embodiments, the methods of the present teachings may be implemented as firmware and / or software programs and applications written in conventional programming languages ​​such as C, C++, Python, etc. When implemented as firmware and / or software, the embodiments described herein may be implemented on a non-transitory computer-readable medium having a program stored thereon for causing a computer to perform the aforementioned methods. It should be understood that the various engines described herein may be provided on a computer system such as computer system 1000, whereby processor 1004 performs the analysis and decisions provided by these engines according to instructions provided by any one or combination of memory components RAM 1006, ROM 1008, or storage device 1010, and user input provided via input device 1014.

[0158] VI. Definitions and Context Examples The present disclosure is not limited to these exemplary embodiments and applications or to the manner in which the exemplary embodiments and applications function or are described herein. Additionally, the figures may depict simplified or partial views, and dimensions of elements in the figures may be exaggerated or otherwise out of proportion.

[0159] Unless otherwise defined, scientific and technical terms used in connection with the present teachings described herein shall have the meanings commonly understood by those of ordinary skill in the art. Further, unless otherwise required by context, singular terms shall include the plural and plural terms shall include the singular. In general, the nomenclature utilized in connection with, and techniques of, chemistry, biochemistry, molecular biology, pharmacology and toxicology described herein are those well known and commonly used in the art.

[0160] Additionally, when the terms "on," "attached to," "connected to," "coupled to," or similar terms are used herein, an element (e.g., a component, material, layer, substrate, etc.) may be "on," "attached to," "connected to," or "coupled to" another element, regardless of whether the element is directly on, directly attached to, connected to, or coupled to another element, or whether there are one or more intervening elements between the one element and the other element. Additionally, when a reference is made to a list of elements (e.g., elements a, b, c), such reference is intended to include any one of the listed elements by itself, any combination of less than all of the listed elements, and / or all combinations of the listed elements. The division of sections herein is for ease of viewing only and does not limit any combination of the elements described.

[0161] The term "subject" may refer to a subject of a clinical trial, a person receiving treatment, a person undergoing anti-cancer drug therapy, a person being monitored for remission or recovery, a person undergoing preventive health analysis (e.g., by medical history), or any other person or patient of interest. In various instances, "subject" and "patient" may be used interchangeably herein.

[0162] As used herein, "substantially" means sufficient to function for the intended purpose. Thus, the term "substantially" allows for small, insignificant variations from an absolute or perfect state, dimension, measurement, or result, such as would be expected by one of ordinary skill in the art, but which have no visible effect on overall performance. When used in reference to a numerical value, or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within 10 percent.

[0163] As used herein, the term "about" when used with respect to a numerical value or a parameter or characteristic that can be expressed as a numerical value means within 10% of the numerical value. For example, "about 50" means a value in the range of 45 to 55.

[0164] The term "ones" means two or more.

[0165] As used herein, the term "plurality" can be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.

[0166] As used herein, the term "set of" means one or more. For example, a set of items includes one or more items.

[0167] As used herein, the phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used, or only one of the items in the list may be used. The items may be specific objects, things, steps, operations, processes, or categories. In other words, "at least one of" means that any combination or number of items from the list may be used, but not all of the items in the list may be used. For example, without limitation, "at least one of item A, item B, or item C" means item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and item C. In some cases, "at least one of item A, item B, or item C" may mean, but is not limited to, two item A, one item B, ten item C, four item B, seven item C, or other suitable combinations.

[0168] As used herein, a "model" may include one or more algorithms, one or more mathematical techniques, one or more machine learning algorithms, or a combination thereof.

[0169] As used herein, "machine learning" may include the practice of using algorithms to analyze data, learn from it, and then make decisions or predictions about something in the world. Machine learning may use algorithms that can learn from data without relying on rule-based programming. Deep learning may be a form of machine learning.

[0170] As used herein, "artificial neural network" or "neural network" (NN) may refer to a mathematical algorithm or computational model that mimics an interconnected group of artificial neurons that process information based on a connectionistic approach to computation. A neural network, also called a neural net, may use one or more layers of nonlinear units to predict an output for a received input. Some neural networks may include one or more hidden layers in addition to an output layer. The output of each hidden layer may be used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current value of each parameter set. In various embodiments, a reference to a "neural network" may be a reference to one or more neural networks.

[0171] Neural networks may process information in two ways: when the neural network is being trained, it is in training mode, and when the neural network puts into practice what it has learned, it is in inference (or prediction) mode. Neural networks may learn through a feedback process (e.g., backpropagation) that allows the network to adjust the weight coefficients of individual nodes in the intermediate hidden layers (modify the behavior of individual nodes) so that the output matches the output of the training data. In other words, the neural network may learn by being fed training data (training examples), and eventually it learns how to arrive at the correct output, even when presented with a new range or set of inputs. The neural network may include, for example, but is not limited to, at least one of a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a convolutional neural network (CNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural ODE), a U-Net, a fully convolutional network (FCN), a stacked FCN, a stacked FCN with multi-channel learning, a squeeze-and-excite embedded neural network, a MobileNet, or another type of neural network.

[0172] As used herein, "deep learning" may refer to the use of multi-layer artificial neural networks to automatically learn representations from input data such as images, videos, text, etc., without human-provided knowledge, to make highly accurate predictions in tasks such as object detection / identification, speech recognition, and language translation.

[0173] VII. Further Considerations While the present teachings will be described in conjunction with various embodiments, it is not intended that the present teachings be limited to such embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those skilled in the art.

[0174] For example, the flowcharts and block diagrams described above illustrate the architecture, functionality, and / or operation of possible implementations of various method and system embodiments. Each block in the flowcharts or block diagrams may represent a module, segment, function, portion of an operation or step, or a combination thereof. In some alternative implementations of an embodiment, one or more functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be executed substantially simultaneously. In other cases, the blocks may be executed in the reverse order. Furthermore, in some cases, one or more blocks may be added to replace or supplement one or more other blocks in the flowcharts or block diagrams.

[0175] Thus, in describing various embodiments, the specification may present a method and / or process as a particular sequence of steps. However, to the extent that the method or process does not depend on the particular order of steps described herein, the method or process should not be limited to the particular order of steps described, and as one of ordinary skill in the art will readily recognize, the order may be varied and still remain within the spirit and scope of the various embodiments.

[0176] VIII. Enumeration of Embodiments Embodiment 1. A method for performing retinal segmentation, comprising: receiving an optical coherence tomography (OCT) image of the retina; generating a layer element image using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indices; generating an initial pathological element image using the OCT image and a second neural network, the initial pathological element image visually identifying the set of retinal pathological elements using a set of pathological element indices that assign a different group of pixels to each retinal pathological element of the set of retinal pathological elements; and refining the initial pathological element image using the layer element image to generate a refined pathological element image, the refined pathological element image visually identifying the set of retinal pathological elements using the set of pathological element indices and the set of pathological element indices assigning an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0177] Embodiment 2. The method of embodiment 1, wherein one pathological element index of the set of pathological element indexes is used to assign a group of pixels in the initial pathological element image to a first retinal pathological element of the set of retinal pathological elements, and the refining includes reallocating a portion of the group of pixels in the initial pathological element image based on whether an anatomical characterization of the first retinal pathological element identified by the pathological element index is anatomically feasible, and the anatomical characterization of the first retinal pathological element includes at least one of a position, a size, a shape, a length, a width, a thickness, or a volume of the retinal pathological element.

[0178] Embodiment 3. The method of embodiment 2, wherein the reassigning includes at least one of reassigning a first pixel of the group of pixels from a first retinal pathological element to a second retinal pathological element of the set of retinal pathological elements based on the layer element image, or reassigning a second pixel of the group of pixels from the first retinal pathological element to background based on the layer element image.

[0179] Embodiment 4. The method of any one of embodiments 1 to 2, wherein the refining includes updating a group of pixels in the initial pathological element image that is assigned to one retinal pathological element of the set of retinal pathological elements to form an updated group of pixels in the retinal pathological element in the refined pathological element image by constraining an allowable area for the retinal pathological element based on the layer element image, wherein the updated group of pixels includes fewer pixels than the group of pixels.

[0180] Embodiment 5. The method of any one of embodiments 1 to 4, wherein generating the layer element image includes generating a multi-channel map using the OCT image via a first neural network, the multi-channel map including a plurality of segmented images, each segmented image of the plurality of segmented images identifying a corresponding retinal layer of interest.

[0181] Embodiment 6. The method of embodiment 5, wherein generating a layer element image further includes converting the multi-channel map into an initial layer element image that identifies a set of retinal layer elements using a set of layer element indices, the set of layer element indices assigning different groups of pixels in the initial layer element image to each retinal layer element of the set of retinal layer elements.

[0182] Embodiment 7. The method of embodiment 6, wherein the converting includes applying piecewise logistic curve approximation to the multi-channel map to generate the initial layer element image.

[0183] Embodiment 8. The method of embodiment 6 or embodiment 7, wherein generating the layer element image further comprises applying smoothing to the initial layer element image to generate the layer element image.

[0184] Embodiment 9. The method of embodiment 8, wherein applying smoothing to the initial layer element image includes applying Gaussian smoothing to the initial layer element image to generate the layer element image.

[0185] Embodiment 10. The method of any one of embodiments 1 to 9, wherein one retinal layer element of the set of retinal layer elements is either a retinal layer or a boundary associated with a retinal layer.

[0186] Embodiment 11. The method of embodiment 10, wherein the retinal layer is selected from the group consisting of the inner limiting membrane (ILM) layer, the outer limiting membrane (ELM) layer, the outer plexiform layer-Henle fiber layer (OPL-HFL), the retinal pigment epithelium (RPE) layer, the layer of the RPE detachment, the Bruch's membrane (BM) layer, and the ellipsoid zone (EZ).

[0187] Embodiment 12. The method of any one of embodiments 10 to 11, wherein the set of retinal pathology elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pockets, or disruptions, or characteristics or subtypes of fluid, material, or disruptions.

[0188] Embodiment 13. The method of any one of embodiments 10 to 12, wherein each of the set of layer element indicators and the set of pathological element indicators includes at least one of a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a marker, a label, a tag, or text.

[0189] Embodiment 14. The method of any one of embodiments 10 to 13, wherein the first neural network comprises a first U-Net and the second neural network comprises a second U-Net.

[0190] Embodiment 15. The method of any one of embodiments 10 to 14, wherein the first neural network is trained using a first training data set including a first plurality of training OCT images and a plurality of training layer element images, and the second neural network is trained using a second training data set including a second plurality of training OCT images and a plurality of training pathological element images.

[0191] Embodiment 16. The method of embodiment 15, wherein at least a portion of the first plurality of training OCT images is included in the second plurality of training OCT images.

[0192] Embodiment 17. A method for performing retinal segmentation comprising: receiving an optical coherence tomography (OCT) image of the retina; generating a multi-channel map using the OCT image via a neural network, the multi-channel map including a plurality of segmented images, each segmented image of the plurality of segmented images identifying a corresponding retinal layer of interest; generating a layer element image using the multi-channel map that identifies a set of retinal layer elements using a set of layer element indicators; and refining an initial pathological element image using the layer element image to generate a refined pathological element image that visually identifies the set of retinal pathological elements using the set of pathological element indicators, the refined pathological element image identifying at least one retinal pathological element in the set of retinal pathological elements more accurately than the initial pathological element image.

[0193] Embodiment 18. The method of embodiment 17, wherein generating the layer element image includes converting the multi-channel map into the initial layer element image using piecewise logistic curve approximation.

[0194] Embodiment 19. The method of embodiment 18, wherein generating a layer element image further includes applying smoothing to the initial layer element image to generate a layer element image that is subsequently used to refine the initial pathological element image.

[0195] Embodiment 20. The method of any one of embodiments 17 to 19, wherein refining the initial pathological element image includes at least one of reassigning a first portion of pixels in the initial pathological element image from one retinal pathological element to a different retinal pathological element in the refined pathological element image based on the layer element image, or reassigning a second portion of pixels in the initial pathological element image to background in the refined pathological element image based on the layer element image.

[0196] Embodiment 21. A system for performing automatic retinal segmentation, comprising: a non-transient memory; a first neural network coupled to the non-transient memory, receiving an optical coherence tomography (OCT) image of the retina; generating a layer element image using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indices; and generating an initial pathological element image using the OCT image and a second neural network, the initial pathological element image identifying the set of retinal pathological elements and assigning a different group of pixels to each retinal pathological element of the set of retinal pathological elements. and a data processor configured to read from a non-transitory memory instructions to cause the system to perform steps including: generating an initial pathological element image that visually identifies a set of retinal pathological elements using a set of pathological element indicators that assign an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements; and refining the initial pathological element image using the layer element image to generate a refined pathological element image, wherein the refined pathological element image visually identifies a set of retinal pathological elements using the pathological element indicators, and the set of pathological element indicators assigns an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0197] Embodiment 22. A method for performing automatic retinal segmentation, comprising: receiving an image input relating to a subject's retina; generating layer element data using the image input and a first neural network, the layer element data identifying a set of retinal layer elements; generating initial pathological element data using the image input and a second neural network, the initial pathological element data identifying the set of retinal pathological elements; and refining the initial pathological element data using the layer element data to generate refined pathological element data, the refined pathological element data more accurately identifying the set of retinal pathological elements compared to the initial pathological element data.

[0198] Embodiment 23. The method of embodiment 22, wherein the initial pathological element data includes an initial pathological element image that visually identifies a set of retinal pathological elements using a set of pathological element indices that assign different groups of pixels in the initial pathological element image to each retinal pathological element of the set of retinal pathological elements.

[0199] Embodiment 24. The method of embodiment 23, wherein the refined pathological element data includes a refined pathological element image that visually identifies a set of retinal pathological elements using a set of pathological element indicators, the set of pathological element indicators assigning the updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements.

[0200] Embodiment 25. The method of any one of embodiments 22 to 24, wherein the layer element data includes a layer element image that identifies a set of retinal layer elements using a set of layer element indices.

[0201] Embodiment 26. The method of any one of embodiments 22 to 25, wherein the image input includes an SD-OCT image.

[0202] Embodiment 27. The method of any one of embodiments 22 to 26, wherein one retinal layer element of the set of retinal layer elements is either a retinal layer or a boundary associated with a retinal layer, and the retinal layer is selected from the group consisting of an inner limiting membrane (ILM) layer, an outer limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, a layer of an RPE detachment, a Bruch's membrane (BM) layer, and an ellipsoid zone (EZ).

[0203] Embodiment 28. The method of any one of embodiments 22 to 27, wherein the set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pockets, or disruptions, or characteristics or subtypes of fluid, material, or disruptions.

[0204] Embodiment 29. A system comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more of the methods described in embodiments 1 to 20 and 22 to 28.

[0205] Embodiment 30. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to execute part or all of one or more of the methods recited in embodiments 1 to 20 and 22 to 28.

Claims

1. A method for performing retinal segmentation, Receiving optical coherence tomography (OCT) images of the retina, The process involves generating a layer element image using the OCT image and a first neural network, wherein the layer element image identifies a set of retinal layer elements using a set of layer element indices. The method of generating an initial pathological element image using the OCT image and a second neural network, wherein the initial pathological element image visually identifies a set of retinal pathological elements using a set of pathological element indices that assign different groups of pixels to each retinal pathological element in the set of retinal pathological elements, Elaborating the initial pathological element image using the layered element image to generate an elaborated pathological element image, wherein the elaborated pathological element image visually identifies the set of retinal pathological elements using the set of pathological element indicators, and the set of pathological element indicators assigns an updated group of pixels to at least one retinal pathological element from the set of retinal pathological elements; Methods that include...

2. One pathological element index from the set of pathological element indexes is used to assign a group of pixels in the initial pathological element image to a first retinal pathological element of the set of retinal pathological elements, and the refinement is as follows: This includes reallocating a portion of the group of pixels in the initial pathological element image based on whether the anatomical characterization of the first retinal pathological element identified by the pathological element index is anatomically feasible, The anatomical characterization of the first retinal pathological element includes at least one of the location, size, shape, length, width, thickness, or volume of the retinal pathological element. The method according to claim 1.

3. The aforementioned reallocation is, Based on the layer element image, reassigning a first pixel of the group of pixels from the first retinal pathological element to a second retinal pathological element of the set of retinal pathological elements, or Based on the layer element image, the second pixel of the group of pixels is reassigned from the first retinal pathological element to the background. The method according to claim 2, comprising at least one of the above.

4. The aforementioned refinement is, Forming the updated group of pixels in the retinal pathological element in the refined pathological element image by updating a group of pixels in the initial pathological element image assigned to one retinal pathological element of the set of retinal pathological elements, thereby constraining the permissible area for the retinal pathological element based on the layered element image, wherein the updated group of pixels contains fewer pixels than the group of pixels. The method according to claim 1, including the method described in claim 1.

5. The generation of the aforementioned layer element image is The method for generating a multichannel map using the OCT image via the first neural network is to generate a multichannel map comprising a plurality of segmented images, wherein each of the plurality of segmented images identifies a corresponding retinal layer of interest. The method according to claim 1, including the method described in claim 1.

6. The generation of the aforementioned layer element image is Transforming the multichannel map into an initial layer element image that identifies the set of retinal layer elements using the set of layer element indices, wherein the set of layer element indices assigns different groups of pixels in the initial layer element image to each retinal layer element of the set of retinal layer elements. The method according to claim 5, further comprising:

7. The aforementioned conversion is performed by Applying piecewise logistic curve approximation to the multi-channel map to generate the initial layer element image, The method according to claim 6, including the method described in claim 6.

8. The generation of the aforementioned layer element image is Applying smoothing to the initial layer element image to generate the layer element image, The method according to claim 6, further comprising:

9. The method according to claim 8, wherein applying smoothing to the initial layer element image includes applying Gaussian smoothing to the initial layer element image to generate the layer element image.

10. The method according to claim 1, wherein one of the set of retinal layer elements is either a retinal layer or a boundary associated with the retinal layer.

11. The method according to claim 10, wherein the retinal layer is selected from the group consisting of the internal limiting membrane (ILM) layer, the external limiting membrane (ELM) layer, the outer plexiform layer-Henle fiber layer (OPL-HFL), the retinal pigment epithelium (RPE) layer, the RPE detachment layer, the Bruch's membrane (BM) layer, and the ellipsoidal region (EZ).

12. The method according to claim 1, wherein the set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pocket, or rupture.

13. The method according to claim 1, wherein each of the set of layer element indicators and the set of pathological element indicators includes at least one of a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a marker, a label, a tag, or text.

14. The method according to claim 1, wherein the first neural network comprises a first U-Net, and the second neural network comprises a second U-Net.

15. The first neural network is trained using a first training dataset which includes a first set of training OCT images and a set of training layer element images. The second neural network is trained using a second training dataset which includes a second set of training OCT images and a second set of training pathological element images. The method according to claim 1.

16. The method according to claim 15, wherein at least a portion of the first plurality of training OCT images is included in the second plurality of training OCT images.

17. A method for performing retinal segmentation, Receiving optical coherence tomography (OCT) images of the retina, The method of generating a multichannel map using the OCT image via a neural network, wherein the multichannel map comprises a plurality of segmented images, and each of the plurality of segmented images identifies a corresponding retinal layer of interest. Using a set of layer element indices, a layer element image is generated using the multi-channel map to identify a set of retinal layer elements, The method of refining an initial pathological element image using a layered element image to generate an elaborated pathological element image that visually identifies a set of retinal pathological elements using a set of pathological element indicators, wherein the elaborated pathological element image more accurately identifies at least one retinal pathological element in the set of retinal pathological elements than the initial pathological element image. Methods that include...

18. The generation of the aforementioned layer element image is Converting the multichannel map to an initial layer element image using piecewise logistic curve approximation, The method according to claim 17, including the method described in claim 17.

19. The generation of the aforementioned layer element image is Applying smoothing to the initial layer element image to generate the layer element image to be used thereafter to refine the initial pathological element image, The method according to claim 18, further comprising:

20. Refining the aforementioned initial pathological element images is Based on the layered element image, a first portion of pixels in the initial pathological element image is reassigned from one retinal pathological element to one different retinal pathological element in the refined pathological element image, or Based on the layered element image, a second portion of pixels in the initial pathological element image is reassigned to the background in the refined pathological element image. The method according to claim 17, comprising at least one of the above.

21. A system for performing automated retinal segmentation, Non-temporary memory and The aforementioned non-temporary memory is created, Receiving optical coherence tomography (OCT) images of the retina, The process involves generating a layer element image using the OCT image and a first neural network, wherein the layer element image identifies a set of retinal layer elements using a set of layer element indices. The method of generating an initial pathological element image using the OCT image and a second neural network, wherein the initial pathological element image visually identifies a set of retinal pathological elements using a set of pathological element indices that assign different groups of pixels to each retinal pathological element in the set of retinal pathological elements, Elaborating the initial pathological element image using the layered element image to generate an elaborated pathological element image, wherein the elaborated pathological element image visually identifies the set of retinal pathological elements using the set of pathological element indicators, and the set of pathological element indicators assigns an updated group of pixels to at least one retinal pathological element from the set of retinal pathological elements; A data processor configured to read instructions from the non-temporary memory for the system to execute a process including the following: A system that includes these features.

22. A method for performing automated retinal segmentation, Receiving image input related to the target retina, The process involves generating layer element data using the aforementioned image input and a first neural network, wherein the layer element data identifies a set of retinal layer elements. The process involves generating initial pathological element data using the aforementioned image input and a second neural network, wherein the initial pathological element data identifies a set of retinal pathological elements. The process involves refining the initial pathological element data using the layer element data to generate refined pathological element data, wherein the refined pathological element data more accurately identifies the set of retinal pathological elements compared to the initial pathological element data. Methods that include...

23. The method according to claim 22, wherein the initial pathological element data includes the initial pathological element image, which visually identifies the set of retinal pathological elements using a set of pathological element indices that assign different groups of pixels in the initial pathological element image to each retinal pathological element in the set of retinal pathological elements.

24. The method according to claim 23, wherein the refined pathological element data includes refined pathological element images that visually identify the set of retinal pathological elements using the set of pathological element indices, the set of pathological element indices assigns updated groups of pixels to at least one retinal pathological element of the set of retinal pathological elements.

25. The method according to claim 22, wherein the layer element data includes a layer element image that identifies a set of retinal layer elements using a set of layer element indices.

26. The method according to claim 22, wherein the image input includes an SD-OCT image.

27. The method according to claim 22, wherein one of the set of retinal layer elements is either a retinal layer or a boundary associated with the retinal layer, and the retinal layer is selected from the group consisting of an internal limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, a layer of RPE detachment, a Bruch's membrane (BM) layer, and an ellipsoidal region (EZ).

28. The method according to claim 22, wherein the set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pocket, or rupture.

29. One or more data processors, A non-temporary computer-readable storage medium containing instructions, wherein when the instructions are executed by the one or more data processors, the one or more data processors cause the one or more data processors to execute part or all of the method described in any one of claims 1 to 20 and 22 to 28; A system that includes these features.

30. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, comprising instructions configured to cause one or more data processors to execute some or all of the methods described in any one of claims 1 to 20 and 22 to 28.

Images