Segmentation of optical coherence tomography (OCT) images

Abstract

The invention discloses a system and method for automatic retinal segmentation. Performing the automatic retina segmentation includes receiving an image input for a retina of a subject. Layer element data is generated using the image input and a first neural network. The layer element data identifies a set of retinal layer elements. Initial pathological element data is generated using the image input and a second neural network. The initial pathological element data identifies a set of retinal pathological elements. The initial pathological element data is refined using the layer element data to generate refined pathological element data. The refined pathological element data identifies the set of retinal pathological elements more accurately than the initial pathological element data.

Description

[0001] Inventors:

[0002] A. Maunz; H. Lu; T. F. Albrecht; F. Benmansor; Y. Y. Li Technical Field

[0003] The present application relates to retinal segmentation for the diagnosis and / or treatment of ophthalmic diseases (or conditions), and more specifically to automatic retinal segmentation of optical coherence tomography (OCT) images using a machine learning-based algorithm for the diagnosis and / or treatment of ophthalmic diseases (e.g., age-related macular degeneration (AMD), diabetic macular edema (DME), etc.). Background Art

[0004] Ophthalmic diseases and conditions are varied and may include retinal diseases and conditions. Retinal diseases may affect one or more parts of the retina, which is the tissue at the back of the eye that captures light and converts it into signals (e.g., electrical signals, chemical signals) that are sent to the brain. Retinal diseases may cause complications, such as, for example, swelling of the macula (called macular edema). Many retinal diseases can affect vision and may cause loss of vision or, in some cases, blindness. Treatment may involve stopping or slowing the disease to maintain, improve, or restore vision.

[0005] Age-related macular degeneration (AMD) is the main cause of 50 years old and above patient's vision loss.AMD initially manifests as dry AMD, and may progress to wet AMD.For dry, small deposits (drusen) can be formed under the macula on the retina, causing the final degeneration of the retina.For wet, it may also be referred to as neovascular AMD (nAMD), and the abnormal blood vessels originating from the choroidal layer of eyes grow into the retina, and effusion is leaked into the retina from the blood.Once entering the retina, fluid may distort the vision of the experimenter immediately, and over time, may damage the retina itself, such as causing the loss of the photoreceptors in the retina.Fluid may cause the macula to separate from its base, thereby causing serious and acute vision loss.

[0006] Diabetic macular edema (DME) is a complication of diabetic retinopathy (DR) and is often the cause of vision loss in diabetic patients. In DME, excess fluid accumulates in the extracellular spaces within the retina in the macula (e.g., inner nuclear layer, outer plexiform layer, Henle fiber layer, and subretinal space).

[0007] Optical coherence tomography (OCT) can provide detailed scans of the macula to help detect macular degeneration, diabetic macular edema, and other eye problems earlier than was possible in the past.

[0008] In order to investigate the degree of deterioration of the retina when suffering from, for example, AMD or DME, OCT images of the retina (e.g., time domain optical coherence tomography (TD-OCT) or frequency domain optical coherence tomography (SD-OCT) images) can be acquired and used to identify features that may be associated with different levels of degenerative diseases (e.g., AMD, DME). SD-OCT is an imaging technique in which light is directed to the retina at various optical frequencies, and in which reflected light is collected to capture a two-dimensional or three-dimensional high-resolution cross-sectional image of the retina via an interferometric measurement signal detected as a function of frequency. The different features captured in the SD-OCT image can be identified via retinal segmentation and used to determine the severity of retinal disease, which may help guide the diagnosis and / or treatment of the disease. However, currently available techniques for extracting, understanding and / or interpreting such features may be tedious and / or error-prone. Therefore, the cumbersome nature of the retinal disease investigation process may be a limiting factor in the diagnosis and / or treatment of the disease. Therefore, it is desirable to have one or more methods and / or systems that recognize and take into account these problems. Summary of the invention

[0009] In one or more embodiments, a method for performing retinal segmentation is provided. The method includes receiving an optical coherence tomography (OCT) image of the retina. A layer element image is generated using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indicators. An initial pathological element image is generated using the OCT image and a second neural network, the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indicators, the set of pathological element indicators assigning a different pixel group to each retinal pathological element in the set of retinal pathological elements. The initial pathological element image is refined using the layer element image to generate a refined pathological element image. The refined pathological element image visually identifies the set of retinal pathological elements using the set of pathological element indicators, the set of pathological element indicators assigning an updated pixel group to at least one retinal pathological element in the set of retinal pathological elements.

[0010] In one or more embodiments, a method for performing retinal segmentation is provided. The method includes receiving an optical coherence tomography (OCT) image of the retina, and using the OCT image to generate a multi-channel map via a neural network. The multi-channel map includes a plurality of segmented images, wherein each segmented image in the plurality of segmented images identifies a corresponding target retinal layer. The multi-channel map is used to generate a layer element image, which uses a set of layer element indicators to identify a set of retinal layer elements. The layer element image is used to refine an initial pathological element image to generate a refined pathological element image, which uses a set of pathological element indicators to visually identify a set of retinal pathological elements, wherein the refined pathological element image more accurately identifies at least one retinal pathological element in the set of retinal pathological elements than the initial pathological element image.

[0011] In one or more embodiments, a system for performing automatic retinal segmentation is provided. The system includes a non-transitory memory and a hardware processor coupled to the non-transitory memory. The data processor is configured to read instructions from the non-transitory memory to cause the system to perform operations, including: receiving an optical coherence tomography (OCT) image of the retina; using the OCT image and a first neural network to generate a layer element image, the layer element image using a set of layer element indicators to identify a set of retinal layer elements; using the OCT image and a second neural network to generate an initial pathological element image, the initial pathological element image using a set of pathological element indicators to visually identify a set of retinal pathological elements, the set of pathological element indicators assigning different pixel groups to each retinal pathological element in the set of retinal pathological elements; and using the layer element image to refine the initial pathological element image to generate a refined pathological element image, the refined pathological element image using the set of pathological element indicators to visually identify the set of retinal pathological elements, the set of pathological element indicators assigning updated pixel groups to at least one retinal pathological element in the set of retinal pathological elements.

[0012] In one or more embodiments, a method for performing automatic retinal segmentation is provided. The method includes receiving an image input of a retina of a subject. Using the image input and a first neural network to generate layer element data. The layer element data identifies a set of retinal layer elements. Using the image input and a second neural network to generate initial pathology element data. The initial pathology element data identifies a set of retinal pathology elements. Using the layer element data to refine the initial pathology element data to generate refined pathology element data. The refined pathology element data more accurately identifies the set of retinal pathology elements than the initial pathology element data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more fully understand the principles and advantages disclosed herein, reference is now made to the following description in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a block diagram of a retinal segmentation system in accordance with various embodiments.

[0015] Figure 2 An example process flow for retinal segmentation of optical coherence tomography (OCT) images using a machine learning based algorithm is shown in accordance with various embodiments.

[0016] Figure 3 is a block diagram illustrating a neural network with a multi-channel learning method that may be used in a retinal segmentation system according to various embodiments.

[0017] Figure 4 is a flow chart of a method of performing retinal segmentation according to various embodiments.

[0018] Figure 5 is a flow chart of a method for generating a layer element image according to various embodiments.

[0019] Figure 6 is a flow chart of a method for performing retinal segmentation according to various embodiments.

[0020] Figure 7 is a flow chart of another method for performing automatic retinal segmentation in accordance with various embodiments.

[0021] Fig. 8A and Figure 8B is a diagram of retinal segmentation results according to various embodiments.

[0022] Fig. 9 is a schematic diagram of an example neural network that may be used to implement a computer-based model in accordance with various embodiments.

[0023] Fig.10 is a block diagram of a computer system according to various embodiments.

[0024] It should be understood that the drawings are not necessarily drawn to scale, nor are the objects in the drawings necessarily drawn to scale relative to each other. The drawings are depictions of various embodiments intended for clarity and understanding of the devices, systems, and methods disclosed herein. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. In addition, it should be understood that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION

[0025] I. Overview

[0026] Detailed scans of the retina can be used to detect, diagnose and / or treat various types of ophthalmic diseases (or illnesses). As an example, detailed scans of the retina in the macula can be used to detect, diagnose and / or treat neovascular age-related macular degeneration (nAMD). As another example, detailed scans of the retina in the macula can be used to detect, diagnose and / or treat diabetic macular edema (DME). Embodiments described herein provide improved techniques for automatic retinal segmentation of retinal images (e.g., retinal scans), which are more accurate and more reliable than existing methods for processing retinal images. More accurate and more reliable retinal segmentation can help ensure that more accurate and thorough diagnosis and / or treatment solutions are provided for patients suffering from ophthalmic diseases (such as, for example, but not limited to, nAMD and DME).

[0027] Retinal segmentation includes detecting and identifying one or more retinal (e.g., retina-related) elements in a retinal image. The retinal elements may be composed of at least one of retinal layer elements or retinal pathological elements. Detection and identification of one or more retinal layer elements may be referred to as layer element (or retinal layer element) segmentation. Detection and identification of one or more retinal pathological elements may be referred to as pathological element (or retinal pathological element) segmentation.

[0028] Retinal layer elements can be, for example, retinal layers or borders associated with retinal layers. The example of retinal layers includes, but is not limited to, inner limiting membrane (ILM) layer, retinal nerve fiber layer, ganglion cell layer, inner plexiform layer, inner nuclear layer, outer plexiform layer, outer nuclear layer, external limiting membrane (ELM) layer, photoreceptor layer, retinal pigment epithelium (RPE) layer, RPE detachment layer, Bruch's membrane (BM) layer, choroid capillaris layer, choroidal stroma layer, ellipsoidal band (EZ) and other types of retinal layers. In some cases, retinal layers can be made up of one or more layers. As an example, retinal layers can be outer outer plexiform layer-Henle fiber layer (OPL-HFL). The border associated with retinal layers can be, for example, the inner border of retinal layers, the outer border of retinal layers, the border associated with the pathological features of retinal layers (for example, the inner border or outer border of retinal layer detachment) or a certain other type of border. For example, the border can be the inner border of RPE (IB-RPE) detachment layer, the outer border of RPE (OB-RPE) detachment layer or the border of another type.

[0029] Retinal pathology elements can include, for example, effusion (for example, effusion bag), cell, solid material or their combination that prove retinal pathology (for example, disease or illness, such as AMD or DME).For example, the existence of some retinal effusion may be a sign of nAMD or DME.The example of retinal pathology elements includes but is not limited to intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelium detachment (PED), high reflective material (HRM), subretinal high reflective material (SHRM), intraretinal high reflective material (IHRM), high reflective focus (HRF), retinal fluid bag, drusen, fibrosis development and destruction.In some cases, retinal pathology elements can be the destruction (for example, discontinuity, stratification, loss etc.) of retinal layer or retinal area.For example, destruction may be the destruction of ellipsoid band, ELM, RPE or another layer or district.Destruction may represent the damage or loss of the cell (for example, photoreceptor) in the destruction area.

[0030] Additionally, retinal pathology elements may include features or subtypes of one of an effusion (e.g., IRF, SRF, effusion associated with PED), a material (e.g., HRM, SHRM, IHRM), a lesion (e.g., HRF, SHRM lesion), or a disruption. Specifically, examples of retinal pathology elements may include features and / or subtypes of the above-described different types of elements and disruptions that may be detected and identified via retinal segmentation. For example, whether a retinal effusion is clear or turbid may be a detectable and identifiable feature of a retinal effusion. Thus, in some examples, a retinal pathology element may be a clear IRF, a turbid IRF, a clear SRF, a turbid SRF, some other type of clear retinal fluid, some other type of turbid retinal fluid, or a combination thereof. In some cases, for SHRM, shape features (e.g., high SHRM, dome-shaped SHRM centered at the fovea, flat SHRM near the fovea center, deformation, etc.), boundary features (e.g., poorly defined SHRM, well-defined SHRM), reflectivity (e.g., increased reflectivity or other levels of reflectivity), layering features (e.g., high reflective bands in SHRM lesions), and lesion features (e.g., height, width, and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that can be detected and identified via retinal segmentation.

[0031] Existing methods and systems for performing retinal segmentation may be more time consuming than desired. For example, some currently available methods require manual annotation of images (e.g., a human grader detects retinal elements and annotates the images to identify the retinal elements). This type of process can be lengthy, can be prone to human error, and can be a bottleneck that increases the overall time and effort required for overall image analysis in the detection, diagnosis, and / or treatment of ophthalmic diseases or conditions. Furthermore, in some cases, manual grading of pathologies via images may be too cumbersome or infeasible for human graders (e.g., when there are hundreds or thousands of images to grade, when there are very small objects scattered throughout the images, etc.). Furthermore, manual grading by human graders may introduce undesirable bias or variability to the grading.

[0032] Some currently available methods use computer processing to perform segmentation of retinal layers and to perform segmentation of retinal fluid, but these methods are not as accurate as desired. For example, some currently available methods use algorithms built into OCT imaging devices, which are less reliable than expected. For example, in cases of atrophy with choroidal hypertransmission, these algorithms may not accurately perform retinal segmentation. In addition, data generated using retinal segmentation algorithms included in OCT imaging devices provided by different vendors may cause problems because different vendors have different definitions of central subfield thickness (CST). Therefore, CST measurements generated using one OCT imaging device may not be comparable to CST measurements generated using another OCT imaging device.

[0033] Thus, embodiments described herein provide methods and systems for automatic retinal segmentation of retinal elements in a manner that improves accuracy and reduces processing time. Specifically, the methods and systems disclosed herein relate to automatic retinal segmentation of retinal scans based on algorithms that use machine learning. The embodiments described herein enable more accurate and efficient grading and retinal segmentation of a larger number of images, as well as entire images, than currently available methods and systems. In addition, the embodiments described herein enable retinal segmentation to achieve a finer level of detail because retinal segmentation is performed at the pixel level. The embodiments described herein also provide greater predictability and reliability because overall bias and variability are reduced.

[0034] The disclosed methods and systems use machine learning to automatically perform retinal segmentation of OCT images. The OCT images may take the form of, but are not limited to, time-domain optical coherence tomography (TD-OCT) images, frequency-domain optical coherence tomography (SD-OCT) images, two-dimensional OCT images, three-dimensional OCT images, OCT angiography (OCT-A) images, or combinations thereof. Although SD-OCT (also known as Fourier domain OCT) may be mentioned with respect to the embodiments described herein, other types of OCT images are also contemplated for use with the methods and systems described herein. Therefore, the description of the embodiments with respect to images, image types, and techniques merely provides non-limiting examples of such images, image types, and techniques.

[0035] In one or more embodiments, one or more OCT images are processed to automatically perform retinal segmentation and generate one or more segmented OCT images. The segmented OCT images use one or more graphical indicators to identify one or more retinal elements on the segmented OCT images. For example, one or more color indicators, shape indicators, pattern indicators, shaded indicators, lines, curves, logos, marks, labels, text features, other types of graphical indicators, or combinations thereof can be used to identify one or more portions of the OCT image that have been identified as retinal elements (e.g., based on pixels).

[0036] As a specific example, a group of pixels may be identified as capturing a particular retinal fluid (e.g., an IRF or SRF). The segmented OCT image may use a color indicator to identify this group of pixels. For example, a color unique to a particular retinal fluid may be assigned to each pixel in the group of pixels, thereby assigning each pixel to a particular retinal fluid. As another example, the segmented OCT image may identify the group of pixels by applying a patterned area or shape (continuous or discontinuous) to the group of pixels.

[0037] The segmented OCT image can be used to extract feature data of the one or more retinal elements identified in the segmented OCT image. The feature data may include values ​​of any number of features (e.g., quantitative features) or combinations thereof. Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a border associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, height of a lesion (e.g., a SHRM lesion), width of a lesion, area of ​​a lesion, calculated reflectivity (reflectivity class or score for a SHRM lesion), and number of highly reflective foci.

[0038] The methods and systems described herein use retinal images (such as OCT images) to identify retinal elements to detect, diagnose and / or treat ophthalmic diseases, such as AMD, diabetic retinopathy (DR) or DME. For example, first, an OCT image is generated (or captured) using a retinal scanner or another type of OCT imaging device. The OCT image can be a TD-OCT image, an SD-OCT image, or some other type of OCT image. In other examples, the OCT image is received (or acquired) from a retinal scanner (or other OCT imaging device) or from another source (e.g., a data storage device, a computer, etc.). After the OCT image is obtained, it is processed using an algorithm including one or more machine learning (ML) algorithms based on artificial intelligence (AI) to perform retinal segmentation. For example, the algorithm can use a neural network to process the OCT image and perform layer element segmentation and pathological element segmentation.

[0039] The methods and systems described herein use layer element segmentation to generate layer element data that is used to refine the pathological element data generated by pathological element segmentation. For example, an OCT image can be processed via two paths, each of which can be implemented using one or more neural networks. The first path includes performing automatic layer element segmentation to generate layer element data, such as, for example, a layer element image. The layer element image is a segmented OCT image that uses one or more graphical indicators (which can be referred to as layer element indicators) to identify a set of retinal layer elements. The second path includes performing automatic pathological element segmentation to generate pathological element data, such as, for example, a pathological element image. The pathological element image is a segmented OCT image that uses one or more graphical indicators (which can be referred to as pathological element indicators) to identify a set of retinal pathological elements. In one or more embodiments, the second path includes using the layer element data generated via the first path to refine the pathological element data generated along the second path, so that the refined pathological element data more accurately identifies and locates the set of retinal pathological elements.

[0040] This type of refinement of the pathological element image based on the layer element image ensures more accurate pathological element segmentation, thereby ensuring more accurate detection, diagnosis and / or treatment. For example, this type of refinement can enable automatic correction for imaging artifacts and / or defects to improve accuracy and reduce or prevent false positive results that would otherwise occur as with previous processing methods. In addition to improving the accuracy with which a computer system is able to perform retinal segmentation, using the layer element image generated using the neural network processing to refine the pathological element image generated using the neural network processing can reduce the overall processing time for retinal segmentation, thereby reducing the overall time for detection, diagnosis and / or treatment.

[0041] Recognizing and considering the importance and usefulness of methods and systems that can provide the above improvements, the present specification describes various embodiments of using ML-based algorithms to perform automatic retinal segmentation, which can include layer element segmentation and pathological element segmentation. The embodiments described herein can achieve more accurate and reliable retinal segmentation, which can improve the accuracy and reliability of any detection, diagnosis and / or treatment methods that rely on the results of the retinal segmentation.

[0042] II. Retinal Segmentation Based on Machine Learning (ML)

[0043] II.A. Example System for Automatic Retinal Segmentation

[0044] Figure 1 1 is a block diagram of an image processing system 100 according to various embodiments. The image processing system 100 is used to automatically perform retinal segmentation of retinal images to help evaluate, detect, diagnose and / or treat patients suffering from one or more ophthalmic diseases (or conditions) such as, for example, but not limited to, nAMD, DME and DR. The image processing system 100 may include a computing platform 102, a data storage device 104 and a display system 106. The computing platform 102 can take various forms. In one or more embodiments, the computing platform 102 includes a single computer (or computer system) or multiple computers that communicate with each other. In other instances, the computing platform 102 takes the form of a cloud computing platform, a mobile computing platform (e.g., a smart phone, a tablet computer, etc.), or a combination thereof.

[0045] The data storage 104 and the display system 106 are each in communication with the computing platform 102. In some instances, the data storage 104, the display system 106, or both may be considered part of or otherwise integral to the computing platform 102. Thus, in some instances, the computing platform 102, the data storage 104, and the display system 106 may be separate components that communicate with each other, but in other instances, some combination of these components may be integrated together.

[0046] The image processing system 100 includes a retinal segmentation system 108, which can be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, the retinal segmentation system 108 is implemented in the computing platform 102. The retinal segmentation system 108 is used to perform automatic retinal segmentation of the input 110 received for processing. The input 110 can be received from another computing platform, retrieved from a database, uploaded from a cloud computing platform, received via an electronic message (e.g., email), received from a data storage device, retrieved from a data structure, or received in some other manner. In one or more embodiments, the input 110 is retrieved from the data storage 104.

[0047] The input 110 may include an image input, such as, for example, one or more retinal images. In one or more embodiments, the input 110 includes an OCT image 112. The OCT image 112 may be, for example, an SD-OCT image or a TD-OCT image of the retina of a subject who is experiencing and / or has been diagnosed with an ophthalmic disease (e.g., AMD, DR, or DME).

[0048] In some embodiments, the input 110 may also include one or more color fundus (CF) images, one or more fundus autofluorescence (FAF) images, one or more fluorescein angiography (FA) images, one or more other types of OCT images (e.g., OCT-A images), one or more other types of retinal images, or a combination thereof. In this way, the input 110 may include a multimodal image input. Using a multimodal image input may improve the accuracy of retinal segmentation.

[0049] The retinal segmentation system 108 includes a layer element segmentation module 114 and a pathological element segmentation module 116, each of which can be implemented using software, firmware, hardware, or a combination thereof. In one or more embodiments, the layer element segmentation module 114 and the pathological element segmentation module 116 are separate modules that work together to perform automatic retinal segmentation. In other embodiments, the layer element segmentation module 114 and the pathological element segmentation module 116 can be integrated together in a single module. The layer element segmentation module 114 and the pathological element segmentation module 116 are used in two different processing paths.

[0050] Layer element segmentation module 114 is used to perform layer element segmentation to detect and identify retinal layer elements. As previously described in Section I, retinal layer elements can be, for example, retinal layers or boundaries associated with retinal layers. Examples of retinal layers include, but are not limited to, inner limiting membrane (ILM) layer, external limiting membrane (ELM) layer, outer plexiform layer-Henle fiber layer (OPL-HFL), retinal pigment epithelium (RPE) layer, RPE detachment layer, Bruch's membrane (BM) layer, ellipsoid zone (EZ) and other types of retinal layers. The boundary associated with the retinal layer can be, for example, the inner boundary of the retinal layer, the outer boundary of the retinal layer, the boundary associated with the pathological features of the retinal layer (for example, the inner boundary or outer boundary of the retinal layer detachment) or a boundary of some other type. For example, the boundary can be the inner boundary of the RPE (IB-RPE) detachment layer, the outer boundary of the RPE (OB-RPE) detachment layer or the boundary of another type.

[0051] Pathological element segmentation module 116 is used to perform pathological element segmentation to detect and identify retinal pathological elements. As previously described in Section I, retinal pathological elements may include, for example, effusion, cells, solid materials or combinations thereof that prove retinal pathology associated with ophthalmic diseases or conditions. For example, the presence of some retinal effusions may be a sign of retinal vascular leakage, which may be a sign of nAMD. As another example, the presence of some retinal effusions (such as intraretinal effusions) may be a sign of DME. Examples of retinal pathological elements include, but are not limited to, intraretinal effusions (IRF), subretinal effusions (SRF), effusions associated with pigment epithelial detachment (PED), high reflective materials (HRM), subretinal high reflective materials (SHRM), intraretinal high reflective materials (IHRM), high reflective foci (HRF), retinal effusion bags and destruction. In some cases, retinal pathological elements may be the destruction (e.g., discontinuity, stratification, loss, etc.) of retinal layers or retinal zones. For example, destruction may be the destruction of ellipsoid bands, ELM, RPE or another layer or zone. Destruction may represent the damage or loss of cells (e.g., photoreceptors) in the destruction zone.

[0052] Additionally, retinal pathology elements may include features or subtypes of one of an effusion (e.g., IRF, SRF, effusion associated with PED), a material (e.g., HRM, SHRM, IHRM), a lesion (e.g., HRF, SHRM lesion), or a disruption. Specifically, examples of retinal pathology elements may include features and / or subtypes of the above-described different types of elements and disruptions that may be detected and identified via retinal segmentation. For example, whether a retinal effusion is clear or turbid may be a detectable and identifiable feature of a retinal effusion. Thus, in some examples, a retinal pathology element may be a clear IRF, a turbid IRF, a clear SRF, a turbid SRF, some other type of clear retinal fluid, some other type of turbid retinal fluid, or a combination thereof. In some cases, for SHRM, shape features (e.g., high SHRM, dome-shaped SHRM centered at the fovea, flat SHRM near the fovea center, deformation, etc.), boundary features (e.g., poorly defined SHRM, well-defined SHRM), reflectivity (e.g., increased reflectivity or other levels of reflectivity), layering features (e.g., high reflective bands in SHRM lesions), and lesion features (e.g., height, width, and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that can be detected and identified via retinal segmentation.

[0053] In some cases, retinal layer elements are associated with retinal pathological elements. For example, RPE detachment layer (which is a retinal layer element) is associated with PED (which is a retinal pathological element). Therefore, layer element segmentation module 114 and pathological element segmentation module 116 can communicate with each other to automatically and more accurately perform retinal segmentation.

[0054] In one or more embodiments, the retinal segmentation system 108 uses a machine learning system to perform automatic segmentation. The machine learning system may include, for example, a deep learning system, such as, but not limited to, a neural network system 118. The neural network system 118 may include any number of neural networks or any combination of neural networks. In one or more embodiments, the neural network system 118 takes the form of a convolutional neural network (CNN) system that includes one or more convolutional neural networks. For example, a CNN may include multiple neural networks, each of which may itself be a convolutional neural network.

[0055] In one or more embodiments, a first portion of the neural network system 118 is implemented in the layer element segmentation module 114, and a second portion of the neural network system 118 is implemented in the pathology element segmentation module 116. For example, the layer element segmentation module 114 may include a first neural network 120 of the neural network system 118; and the pathology element segmentation module 116 may include a second neural network 122 of the neural network system 118.

[0056] Each of the first neural network 120 and the second neural network 122 may itself be composed of a set of neural networks. In one or more embodiments, the first neural network 120 and the second neural network 122 differ in at least one neural network. In other words, the second neural network 122 may include at least one neural network that is different from the one or more neural networks in the first neural network 120. In other embodiments, the first neural network 120 and the second neural network 122 may include the same one or more types of neural networks. For example, the same one or more types of neural networks may be used for both layer element segmentation and pathological element segmentation. In some cases, in addition to a set of neural networks, the first neural network 120, the second neural network 122, or both may include one or more mathematical algorithms or functions.

[0057] In one or more embodiments, the input 110 is processed along a first path using a layer element segmentation module 114 that uses a first neural network 120 to perform automatic layer element segmentation. For example, the layer element segmentation module 114 can receive the input 110 (e.g., an OCT image 112) for processing at the first neural network 120. In some embodiments, the layer element segmentation module 114 pre-processes the input 110 to enable focusing on a particular target region before inputting the input 110 into the first neural network 120. The pre-processing can include, for example, reducing noise and / or artifacts in the input 110 that might otherwise impair the ability to correctly assess a particular target region. In some cases, the first neural network 120 is trained to pre-process the input 110.

[0058] The layer element segmentation module 114 uses the first neural network 120 to process the received input (e.g., the input 110 or the pre-processed image input) to perform automatic layer element segmentation and generate layer element data 124 of a set of retinal layer elements detected within the input 110 or the pre-processed image input. The layer element data 124 may include, for example but not limited to: a layer element image (which may also be referred to as a layer element segmentation image), pixel data that assigns each pixel or pixel portion to a retinal layer element, image coordinates that map out each retinal layer element, other information of the set of retinal layer elements that have been detected, or a combination thereof.

[0059] A layer element image, which may be a layer element OCT image, includes a set of graphical indicators, which may be referred to as a set of layer element indicators. The set of layer element indicators identifies a set of retinal layer elements. The layer element indicators may take the form of, for example, but not limited to, a color indicator, a shape indicator, a pattern indicator, a shade indicator, a line, a curve, a logo, a mark, a label, text, another type of graphical indicator, or a combination thereof. In some cases, two or more layer element indicators may identify the same retinal layer element. For example, a particular color may be used to identify pixels representing a particular retinal layer element, while a label may be used to name or identify a particular retinal layer element associated with a particular color.

[0060] In some examples, a layer element indicator for identifying a retinal layer element as a boundary associated with a retinal layer takes the form of a colored and / or patterned curve (continuous or discontinuous) on a layer element image. The curve represents the boundary. In other examples, a layer element indicator for identifying a retinal layer element as a retinal layer may take the form of a colored and / or patterned area or shape (continuous or discontinuous) on a layer element image. The area or shape may represent, for example, the entire thickness of the corresponding retinal layer.

[0061] In some embodiments, the first neural network 120 receives the input 110 (or preprocessed image input) and generates a multi-channel map 125, which is then used to generate layer element data 124. The multi-channel map 125 can be composed of a plurality of segmented images, each of which corresponds to a different retinal layer element or a different target retinal layer. As an example, the plurality of segmented images may include a different segmented image for each target retinal layer element. As another example, the plurality of segmented images may include a different segmented image for each target retinal layer. In some cases, there may be two or more target retinal layer elements corresponding to the same retinal layer (e.g., the inner boundary and the outer boundary of the same retinal layer).

[0062] The first neural network 120 may output a multi-channel map 125, and the layer segmentation module 114 may further process the multi-channel map 125 using any number of various mathematical techniques (e.g., curve approximation, a logistic function, a smoothing function, another type of function or algorithm) to generate layer element data 124. In other embodiments, the multi-channel map 125 may be produced by the first neural network 120 as an intermediate output, and then the first neural network uses the multi-channel map 125 to generate the layer element data 124 as an output of the first neural network 120.

[0063] In still other embodiments, the multi-channel map 125 may be processed to generate initial layer element data 126, which is then refined to form the layer element data 124 (which may then be referred to as refined layer element data). The initial layer element data 126 may include, but is not limited to, a layer element image (which may also be referred to as a layer element segmentation image), pixel data assigning each pixel or pixel portion to a retinal layer element, image coordinates mapping each retinal layer element, other information of the set of retinal layer elements that have been detected, or a combination thereof. However, in these examples, the initial layer element data 126 may be a first approximation.

[0064] As one example, the initial layer element data 126 may include an initial layer element image having at least one layer element indicator that identifies a boundary associated with a target retinal layer. The initial layer element image may be processed using any number of various mathematical techniques (e.g., curve approximation, a smoothing function, another type of function or algorithm, or a combination thereof) to refine the initial layer element image and generate a refined layer element image that forms at least a portion of the layer element data 124. In one or more embodiments, this refinement may be a smoothing of the identified boundary.

[0065] In other embodiments, initial layer element data 126 may be produced by first neural network 120 as an intermediate output, which is then used by first neural network 120 to generate layer element data 124 as an output of first neural network 120. In this manner, layer element data 124 may be generated by layer element segmentation module 114 within the first processing path in any number of different ways.

[0066] In one or more embodiments, the input 110 is also processed along the second path using a pathology element segmentation module 116 that uses a second neural network 122 of the neural network system 118 to perform pathology element segmentation. For example, the pathology element segmentation module 116 can receive the input 110 (e.g., OCT image 112) for processing at the second neural network 122. In some embodiments, the pathology element segmentation module 116 pre-processes the input 110 to enable focusing on a specific target area before inputting the input 110 into the second neural network 122. The pre-processing can include, for example, reducing noise and / or artifacts in the input 110 that otherwise might impair the ability to correctly assess a specific target area. In some cases, the second neural network 122 is trained to pre-process the input 110.

[0067] The pathological element segmentation module 116 uses the second neural network 122 to process the received input (e.g., the input 110 or the pre-processed image input) to perform automatic pathological element segmentation and generate initial pathological element data 128 of a set of pathological layer elements detected within the input 110 or the pre-processed image input. The initial pathological element data 128 may include, for example but not limited to: a pathological element image (which may also be referred to as a pathological element segmentation image), pixel data assigning each pixel or pixel portion to a retinal pathological element, image coordinates mapping each retinal pathological element, other information of the set of retinal pathological elements that have been detected, or a combination thereof.

[0068] A pathological element image, which may be a pathological element OCT image, includes a set of graphical indicators, which may be referred to as a set of pathological element indicators. The set of pathological element indicators identifies a set of retinal pathological elements. The pathological element indicators may take the form of, for example, but not limited to, a color indicator, a shape indicator, a pattern indicator, a shadow indicator, a line, a curve, a logo, a mark, a label, text, another type of graphical indicator, or a combination thereof. In some cases, two or more pathological element indicators may identify the same retinal pathological element. For example, a specific color may be used to identify pixels representing a specific retinal pathological element, and a label may be used to name or identify a specific retinal pathological element associated with a specific color.

[0069] In some examples, the pathological element indicator for identifying a retinal pathological element as retinal fluid can take the form of a colored and / or patterned area or shape (continuous or discontinuous) on the pathological element image. The area or shape can represent, for example, a bag formed by retinal fluid.

[0070] The initial pathology element data 128 output from the second neural network 122 may then be further processed and refined by the pathology element segmentation module 116. For example, the pathology element segmentation module 116 receives the layer element data 124 (or at least a portion of the layer element data 124) from the layer element segmentation module 114. The pathology element segmentation module 116 uses both the initial pathology element data 128 and the layer element data 124 to refine the initial pathology element data 128 and generate pathology element data 132, which may be referred to as refined pathology element data.

[0071] Similar to the initial pathological element data 128, the pathological element data 132 may include, for example but not limited to: a pathological element image (which may also be referred to as a pathological element segmentation image), pixel data assigning each pixel or pixel portion to a retinal pathological element, image coordinates mapping each retinal pathological element, other information of the group of retinal pathological elements that have been detected, or a combination thereof. The pathological element data 132 more accurately identifies and locates the group of target retinal pathological elements than the initial pathological element data 128. For example, the pathological element data 132 includes a refined pathological element image with a group of pathological element indicators that may more accurately identify at least one corresponding retinal pathological element than the initial pathological element data 128.

[0072] In one or more embodiments, the pathology element segmentation module 116 uses the layer element data 124 to constrain the allowed regions of the set of retinal pathology elements identified in the pathology element data 132. For example, a portion of the layer element data 124 corresponding to two retinal layers may be used to constrain the allowed regions of the retinal pathology elements such that the retinal pathology elements are not identified as extending beyond the allowed regions of the retinal pathology elements. As a specific example, the layer element data 124 may be used to constrain the allowed regions of intraretinal fluid in the pathology element image such that the intraretinal fluid is not identified by the corresponding pathology element indicator as spanning into the subretinal space.

[0073] Thus, the layer element data 124 may be used to refine the anatomical characterization of the retinal pathology elements in the set of retinal pathology elements identified in the pathology element data 132 using the one or more pathology element indicators corresponding to the retinal pathology elements. The anatomical characterization of the retinal pathology elements may include, for example, but not limited to, at least one of the location, size, shape, length, width, thickness, volume, or other characteristics of the retinal pathology elements.

[0074] In other embodiments, initial pathology element data 128 may be an intermediate output of second neural network 122, and layer element data 124 may be input into second neural network 122 to refine initial pathology element data 128. In these examples, second neural network 122 outputs pathology element data 132.

[0075] Using the layer element data 124 to refine the initial pathology element data 128 improves the overall accuracy of the pathology element segmentation module 116 in generating the pathology element data 132. This improvement in accuracy may be performed in any future analysis performed using the pathology element data 132.

[0076] For example, the feature extraction system 134 can be implemented in the computing platform 102. The feature extraction system 134 can be used to automatically extract feature data 136 from the pathology element data 132 and, in some cases, the layer element data 124. The feature data 136 may include values ​​for any number of features (e.g., quantitative features) or combinations thereof. Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a border associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of hyperreflective foci.

[0077] Refining the initial pathology element data 128 to form (refined) pathology element data 132 improves the accuracy of the extracted feature data 136. Furthermore, any detection, diagnosis, and / or treatment methods that rely on the pathology element data 132 and / or feature data 136 extracted from the pathology element data 132 may be more accurate.

[0078] In one or more embodiments, the feature data 136 includes feature values ​​associated with an ETDRS (Early Treatment Diabetic Retinopathy Score) grid. The ETDRS grid divides the retina into nine regions defined by two rings and a central region. The central region represents the center of the fovea. The two rings include an inner macular ring and an outer macular ring. The inner macular ring is divided into four regions: an upper inner region, an inner temporal region, a lower inner region, and an inner nasal region. The outer macular ring is divided into four regions: an upper outer region, an outer temporal region, a lower outer region, and an outer nasal region.

[0079] In one or more embodiments, the value of a feature (e.g., the number of fluid pockets, the volume of fluid, etc.) may be generated relative to the center of the fovea, the inner macular ring, or the outer macular ring. The value of a feature may be generated for a specific region (e.g., a quadrant) of the inner macular ring or the outer macular ring. In some cases, the value of a feature may be generated relative to two corresponding regions of the two rings (e.g., the upper inner region of the inner macular ring and the upper outer region of the outer macular ring). Thus, the value of a feature may be generated for any single region of the ETDRS grid, for a ring of the ETDRS grid, for a multi-region area formed by multiple regions of the ETDRS grid, or for the central region of the ETDRS grid. More accurate retinal segmentation as provided by the embodiments described herein allows for more accurate extraction of feature data about individual regions and multi-region areas of the ETDRS grid.

[0080] In one or more embodiments, neural network system 118 is trained using training data 140. For example, first neural network 120 may be trained using a first training data set of training data 140, while second neural network 122 may be trained using a second training data set of training data 140. The first training data set may include, for example, but not limited to, a plurality of training OCT images and training layer element data (e.g., a plurality of training layer element images). The second training data set may include a plurality of training OCT images (which may be the same, partially the same, or different from the plurality of training OCT images in the first training data set) and training pathology element data (e.g., a plurality of training pathology element images).

[0081] Figure 2 is a schematic diagram of an example workflow 200 for automatic retinal segmentation using OCT images according to various embodiments. The workflow 200 may be used Figure 1 An example of an implementation of automatic retinal segmentation using the retinal segmentation system 108 in FIG. 200 may be performed using, for example, Figure 1 It is implemented by the layer element segmentation module 114 and the pathological element segmentation module 116 in the embodiment.

[0082] The retinal segmentation system 108 receives an input 201 for processing. In one or more embodiments, the input 201 includes an OCT image (e.g., Figure 1 In some embodiments, the OCT image 112 may be a pre-processed OCT image. The input 201 may be sent to a first processing path using the layer element segmentation module 114 and a second processing path using the pathology element segmentation module 116.

[0083] The layer element segmentation module 114 receives input 201 and processes the input 201 via a neural network operation 202. The neural network operation 202 may be implemented using a first neural network 120. In one or more embodiments, the first neural network 120 includes a CNN, such as, for example but not limited to, a U-Net for forward prediction.

[0084] Layer element segmentation module 114 may use at least a portion of first neural network 120 to process input 201 via neural network operation 202 and generate multi-channel map 204. Multi-channel map 204 is Figure 1 An example of an implementation of the multi-channel map 125 in FIG. The multi-channel map 204 includes multiple segmented images 205.

[0085] In one or more embodiments, each segmented image in the plurality of segmented images 205 corresponds to a different retinal layer. For example, a different segmented image may be generated for each different target retinal layer. In addition, each segmented image may use at least one graphical indicator (e.g., a color indicator, a shape indicator, a pattern indicator, a shading indicator, a logo, a mark, a label, a text, another type of graphical indicator, or a combination thereof) to identify the corresponding target retinal layer. The one or more graphical indicators, which may be referred to as layer element indicators, visually identify the portion of the segmented image that represents the corresponding target retinal layer. For example, a pixel group representing the corresponding target retinal layer may be assigned to the corresponding target retinal layer and visually identified via a color indicator. The coloring of the pixel group may visually identify the region (continuous or discontinuous) of the segmented image that represents the corresponding target retinal layer.

[0086] The layer element segmentation module 114 processes the multi-channel map 204 via a curve approximation operation 206 to generate an initial layer element image 208. The initial layer element image 208 may be Figure 1 The curve approximation operation 206 may include performing, for example, a piecewise logistic curve approximation to approximate at least one boundary associated with each target retinal layer identified in the multi-channel map 204. In one or more embodiments, the boundary associated with the retinal layer may be an inner boundary of the retinal layer (e.g., an anatomically innermost boundary).

[0087] When the region representing the corresponding target retinal layer in the segmented image of the multi-channel map 204 is discontinuous (e.g., formed by multiple regions separated by gaps), the curve approximation operation 206 approximates a continuous or nearly continuous boundary (e.g., an inner boundary, an outer boundary, etc. extending through the discontinuous region). In this manner, the curve approximation operation 206 can be used to identify a single continuous or nearly continuous boundary of the corresponding target retinal layer, which is identified in the initial layer element image 208 using at least one graphical indicator (e.g., a colored and / or patterned line that highlights the boundary in the initial layer element image 208).

[0088] In one or more embodiments, any number of layer element indicators are used to identify one or more boundaries for each target retinal layer identified in the multi-channel map 204 and identified on the initial layer element image 208. In this manner, the multi-channel map 204 composed of multiple segmented images 205 can be processed to form a single initial layer element image 208. When the initial layer element image 208 identifies such boundaries (e.g., as opposed to the entire thickness of the retinal layer), the initial layer element image 208 can be referred to as a height map.

[0089] In addition, the layer element segmentation module 114 can process the initial layer element image 208 via a smoothing operation 210 to generate a refined layer element image 212. The refined layer element image 212 is Figure 1 2. The refined layer element image 212 includes a set of layer element indicators that more accurately identify the locations of boundaries within the refined layer element image 212 than the initial layer element image 208. The smoothing operation 210 may be performed using, for example, n-dimensional Gaussian smoothing. The smoothing helps smooth the curves generated in the initial layer element image 208 via the curve approximation operation 206, reduce noise, or both, to generate the refined layer element image 212.

[0090] In the second processing path, the pathological element segmentation module 116 receives the input 201 and processes the input 201 via the neural network operation 214. The neural network operation 214 can be implemented using the second neural network 122. In one or more embodiments, the second neural network 122 includes a CNN, such as, for example but not limited to, a U-Net for forward prediction.

[0091] The pathological element segmentation module 116 may use at least a portion of the second neural network 122 to process the input 201 via the neural network operation 214 and generate an initial pathological element image 216. The initial pathological element image 216 is Figure 1 An example of an implementation of the initial pathology element data 128 in FIG.

[0092] The initial pathological element image 216 identifies a set of retinal pathological elements using one or more graphical indicators (e.g., color indicators, shape indicators, pattern indicators, shading indicators, logos, marks, labels, text, another type of graphical indicators, or a combination thereof). The one or more graphical indicators (which may be referred to as pathological element indicators) visually identify the one or more portions of the initial pathological element image 216 that have been identified as representing the set of target retinal pathological elements. For example, a pixel group representing a target retinal pathological element may be assigned to the retinal pathological element and visually identified via a color indicator. The coloring of the pixel group may visually identify an area (continuous or discontinuous) of the initial pathological element image 216 representing the retinal pathological element. The identification is an approximation.

[0093] The pathological element segmentation module 116 continues to refine the initial pathological element image 216 using the refined layer element image 212. For example, the pathological element segmentation module 116 may receive the refined layer element image 212 from the layer element segmentation module 114. The pathological element segmentation module 116 performs a refinement operation 218 on the initial pathological element image 216 using the refined layer element image 212, and thereby generates a refined pathological element image 220. The refined pathological element image 220 may be Figure 1 The refined pathology element image 220 more accurately identifies the set of retinal pathology elements than the initial pathology element image 216 using one or more pathology element indicators.

[0094] The refinement operation 218 may refine the initial pathological element image 216 by, for example, constraining the allowed regions of the set of retinal pathological elements using the refinement layer element image 212 (or data extracted from the refinement layer element image 212). For example, one or more boundaries identified in the refinement layer element image 212 may be used to constrain the allowed regions of the retinal pathological elements such that the retinal pathological elements are not identified as extending beyond the allowed regions of the retinal pathological elements. As a specific example, one or more boundaries in the refinement layer element image 212 may be used to constrain the allowed regions of intraretinal fluid such that the intraretinal fluid is not identified by the corresponding pathological element indicators as crossing into the subretinal space in the refined pathological element image 220. In this manner, the refinement operation 218 ensures that the anatomical characterization of the set of retinal pathological elements in the refined pathological element image 220 using the set of pathological element indicators is accurate (e.g., anatomically feasible, clinically relevant, and / or otherwise appropriate).

[0095] therefore, Figure 1 and Figure 2The retinal segmentation system 108 described in the accompanying drawings shows a system for automatically and reliably identifying retinal layer elements and retinal pathology elements (e.g., nAMD-related retinal elements, DME-related retinal elements, etc.) in an OCT image of the retina. Improving the accuracy of retinal segmentation can provide more accurate and / or clinically relevant diagnostic and / or treatment solutions for patients suffering from, for example, nAMD, DR, DME, or other ophthalmic diseases or conditions.

[0096] Figure 3 is a diagram showing that various embodiments may be used Figure 1 Schematic diagram of a neural network used in the retinal segmentation system 108 in FIG. The neural network 300 is Figure 1 An example of an implementation of a neural network in the neural network system 118 in Figure 1 Specifically, the neural network 300 may be implemented in the retinal segmentation system 108 in FIG. Figure 1 An example of an implementation of a neural network in the first neural network 120 in is or Figure 1 The neural network system 300 may be used to perform automatic layer element segmentation. For example, the neural network system 300 may be used to generate a multi-channel map, such as Figure 1 Multi-channel graph 125 or Figure 2 Multi-channel graph 204 in.

[0097] Neural network 300 may include initial neural network 302, background neural network 304, and foreground neural network 306. In one or more embodiments, each of initial neural network 302, background neural network 304, and foreground neural network 306 may be implemented as a fully convolutional network (FCN) (e.g., a stacked FCN). In other embodiments, neural network 300 may include one or more other types of neural networks or combinations of neural networks.

[0098] The initial neural network 302 receives an image input 308 for processing. The image input 308 may be Figure 1 Input 110 or Figure 2 The image input 308 may be an OCT image (e.g., Figure 1 In some cases, the OCT image may be an image received directly from a retinal scanner or other type of OCT imaging device or may be a pre-processed OCT image. The initial neural network 302 processes the image input 308 to generate a background probability map 310 and a foreground probability map 312.

[0099] Background probability map 310 identifies (or segments out) the background of image input 308. In one or more embodiments, the background may be anything in image input 308 that is not a target. For example, the background may be any portion of image input 308 that is not a target retinal layer. In one or more embodiments, background probability map 310 includes a separate background probability image for each target retinal layer, such that the background probability image for a corresponding target retinal layer identifies the background of the image relative to the corresponding target retinal layer using at least one graphical indicator.

[0100] As an example, the background probability map 310 may include a first background probability image and a second background probability image. The first background probability image identifies the background relative to the first target retinal layer by coloring (or shading, patterning, etc.) the pixel group identified as representing the background differently from the rest of the pixels in the image. The second background probability image identifies the background relative to the second target retinal layer by coloring (or shading, patterning, etc.) the pixel group identified as representing the background differently from the rest of the pixels in the image.

[0101] The foreground probability map 312 identifies (or segments out) the foreground of the image input 308. In one or more embodiments, the foreground may be any content of the target in the image input 308. For example, the foreground may be any portion of the image input 308 representing a target retinal layer. In one or more embodiments, the foreground probability map 312 includes a separate foreground probability image for each target retinal layer, such that the foreground probability image for a corresponding target retinal layer identifies the corresponding target retinal layer using at least one graphical indicator.

[0102] As an example, foreground probability map 312 may include a first foreground probability image and a second foreground probability image. The first foreground probability image identifies a first target retinal layer by coloring (or shading, patterning, etc.) a pixel group identified as representing a target first retinal layer differently from the rest of the pixels in the image. The second foreground probability image identifies a second target retinal layer by coloring (or shading, patterning, etc.) a pixel group identified as representing a target second retinal layer differently from the rest of the pixels in the image.

[0103] The background probability map 310 and the image input 308 are combined and sent as input to the background neural network 304 to generate a refined background map 314. The refined background map 314 more accurately identifies (segments out) portions of the image input 308 that do not represent the target retinal layer. For example, the refined background map 314 may include a plurality of refined background images, each of which more accurately identifies the background of the image relative to the corresponding target retinal layer than the corresponding background probability image in the background probability map 310.

[0104] The foreground probability map 312 and the image input 308 are combined and sent as input to the foreground neural network 306 to generate a refined foreground map 316. The refined foreground map 316 more accurately identifies (segments out) the portion of the image input 308 that represents one or more target retinal layers. For example, the refined foreground map 316 may include a plurality of refined foreground images, each of which more accurately identifies a corresponding target retinal layer than the corresponding foreground probability image in the foreground probability map 312.

[0105] Then, the refined background image 314 and the refined foreground image 316 are integrated to form a multi-channel image 318. The integration may be performed using a cascade and / or stacking operation of the two images. The multi-channel image 318 may be Figure 1 Multi-channel graph 125 or Figure 2 An example of an implementation of the multi-channel map 204 in FIG. In one or more embodiments, the multi-channel map 318 includes a separate segmented image for each target retinal layer. In other words, each segmented image clearly and accurately identifies the portion of the image that represents the corresponding target retinal layer.

[0106] II.B. Example Method for Automated Retinal Segmentation

[0107] Figure 4 is a flow chart of a method 400 for performing retinal segmentation according to various embodiments. In various embodiments, the method 400 may use Figure 1 For example, the method 400 may be implemented using the image processing system 100 described in Figure 1 and Figure 2 In one or more embodiments, a portion of the method 400 may be implemented using the retinal segmentation system 108 described herein. Figure 3 The neural network 300 in FIG.

[0108] like Figure 4 As shown, method 400 includes receiving an optical coherence tomography (OCT) image of a subject's retina at step 402. The subject may suffer from an ophthalmic disease or condition. For example, the subject may be experiencing and / or diagnosed with AMD (e.g., nAMD), DR, DME, or another ophthalmic disease or condition. In various embodiments, the OCT image may be, for example, Figure 1 An OCT image 112 is depicted as input 110. The OCT image may be, for example, a SD-OCT image or a TD-OCT image.

[0109] Method 400 further includes generating a layer element image using the OCT image and the first neural network at step 404, the layer element image using a set of layer element indicators to identify a set of retinal layer elements. Retinal layer elements can be, for example, retinal layers or boundaries associated with retinal layers. Retinal layers can be, for example, but not limited to, inner limiting membrane (ILM) layer, external limiting membrane (ELM) layer, ellipsoid zone (EZ), outer plexiform layer-Henle fiber layer (OPL-HFL), retinal pigment epithelium (RPE) layer, RPE detachment layer, Bruch's membrane (BM) layer, or another type of retinal layer. The boundary associated with the retinal layer can be, for example, the inner boundary of the retinal layer, the outer boundary of the retinal layer, the boundary associated with the pathological features of the retinal layer (e.g., the inner boundary or outer boundary of the retinal layer detachment) or a boundary of some other type. For example, the boundary can be the inner boundary of the RPE (IB-RPE) detachment layer, the outer boundary of the RPE (OB-RPE) detachment layer, or another type of boundary.

[0110] The set of layer element indicators used in the layer element image can be a set of graphic indicators. For example, the layer element indicators can be, for example but not limited to, color indicators, shape indicators, pattern indicators, shadow indicators, lines, curves, logos, marks, labels, text or another type of graphic indicator.

[0111] In one or more embodiments, the layer element image visually identifies one or more portions of the layer element image that have been identified as representing a target retinal layer element. The target retinal layer element may be, for example, a boundary associated with a retinal layer. In one or more embodiments, the layer element image may visually identify the target retinal layer by assigning a group of pixels representing a boundary to a color that has been assigned to the retinal boundary. When each target retinal layer element in the layer element image is a boundary, the layer element image may be referred to as a height map.

[0112] Step 404 may use a first neural network (such as Figure 1 The first neural network may include, for example, but not limited to, at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network. In one or more embodiments, the first neural network is used to perform all operations involved in step 404. In other embodiments, the first neural network is used to perform part of the operations involved in step 404.

[0113] Step 404 can be performed in a variety of ways. Figure 5 Method 500 in FIG. 4 is an example of a method that can be used to implement step 404 .

[0114] The method 400 further includes generating an initial pathological element image using the OCT image and the second neural network, at step 406, the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indicators that assign a different set of pixels to each retinal pathological element in the set of retinal pathological elements. The identification may be an approximation.

[0115] Retinal pathology elements can include, for example, effusion, cells, solid materials or their combination that prove the retinal pathology associated with ophthalmic diseases or illness.For example, the existence of some retinal effusions may be a sign of retinal vascular leakage, which may be a sign of nAMD.As another example, the existence of some retinal effusions (such as intraretinal effusions) may be a sign of DME.The example of retinal pathology elements includes but is not limited to intraretinal effusions (IRF), subretinal effusions (SRF), effusions associated with pigment epithelial detachment (PED), high reflective materials (HRM), subretinal high reflective materials (SHRM), intraretinal high reflective materials (IHRM), high reflective foci (HRF), retinal effusion bags and destruction.In some cases, retinal pathology elements can be the destruction (for example, discontinuity, stratification, loss, etc.) of retinal layers or retinal areas.For example, destruction may be the destruction of ellipsoid bands, ELM, RPE or another layer or district.Destruction may represent the damage or loss of cells (for example, photoreceptors) in the destruction area.

[0116] Additionally, retinal pathology elements may include features or subtypes of one of an effusion (e.g., IRF, SRF, effusion associated with PED), a material (e.g., HRM, SHRM, IHRM), a lesion (e.g., HRF, SHRM lesion), or a disruption. Specifically, examples of retinal pathology elements may include features and / or subtypes of the above-described different types of elements and disruptions that may be detected and identified via retinal segmentation. For example, whether a retinal effusion is clear or turbid may be a detectable and identifiable feature of a retinal effusion. Thus, in some examples, a retinal pathology element may be a clear IRF, a turbid IRF, a clear SRF, a turbid SRF, some other type of clear retinal fluid, some other type of turbid retinal fluid, or a combination thereof. In some cases, for SHRM, shape features (e.g., high SHRM, dome-shaped SHRM centered at the fovea, flat SHRM near the fovea center, deformation, etc.), boundary features (e.g., poorly defined SHRM, well-defined SHRM), reflectivity (e.g., increased reflectivity or other levels of reflectivity), layering features (e.g., high reflective bands in SHRM lesions), and lesion features (e.g., height, width, and / or area of ​​SHRM lesions) may be examples of retinal pathological elements that can be detected and identified via retinal segmentation.

[0117] In one or more embodiments, the initial pathology element image visually identifies one or more portions of the initial pathology element image that have been identified as representing a target retinal pathology element. The target retinal pathology element may be, for example, subretinal fluid. In one or more embodiments, the initial pathology element image visually identifies subretinal fluid by assigning a group of pixels representing subretinal fluid to a color that has been assigned to subretinal fluid.

[0118] Step 406 may use a second neural network (such as Figure 1 The second neural network may include, for example, but not limited to, at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network. In one or more embodiments, the second neural network is used to perform all operations involved in step 406. In other embodiments, the second neural network is used to perform part of the operations involved in step 406.

[0119] The method 400 further includes refining the initial pathology element image using the layer element image to generate a refined pathology element image, the refined pathology element image visually identifying the set of retinal pathology elements using the set of pathology element indicators, the set of pathology element indicators assigning an updated set of pixels to at least one retinal pathology element in the set of retinal pathology elements, at step 408. The refined pathology element image more accurately represents at least one retinal pathology element in the set of retinal pathology elements than the initial pathology element image.

[0120] The refinement in step 408 can be performed in different ways. For example, the refinement in step 408 includes updating the set of pixels in the initial pathology element image assigned to a particular retinal pathology element by constraining the allowed region for the retinal pathology element based on the layer element image to form an updated set of pixels for the retinal pathology element in the refined pathology element image. The allowed region can be restricted based on what is anatomically feasible, clinically relevant, and / or otherwise appropriate. The updated set of pixels includes fewer pixels than the set of pixels.

[0121] In one or more embodiments, a pathological element indicator in the set of pathological element indicators may be used to assign a pixel group in the initial pathological element image to a first retinal pathological element in the set of retinal pathological elements. Refining the initial pathological element image may include reallocating a portion of the pixel group in the initial pathological element image based on whether the anatomical characterization of the first retinal pathological element identified by the pathological element indicator is anatomically feasible. The anatomical characterization of the first retinal pathological element in the set of retinal pathological elements may include at least one of a location, size, shape, length, width, thickness, volume, or another feature of the retinal pathological element.

[0122] In one or more embodiments, reallocating a portion of a pixel group may include, for example, reallocating a first pixel in the pixel group from a first retinal pathology element to a second retinal pathology element in the group of retinal pathology elements based on the layer element image. Reallocating a pixel to a different retinal pathology element may include, for example, but not limited to, changing the application of a pathology element indicator associated with the pixel. For example, a pixel may be changed from a first color in an initial pathology element image to a second color in a refined pathology element image.

[0123] In one or more embodiments, reallocating a portion of the pixel group may include, for example, reallocating a second pixel in the pixel group from the first retinal pathology element to the background based on the layer element image. Reallocating the pixel to the background may include, for example, but not limited to, removing the application of a pathology element indicator associated with the pixel. For example, a color previously applied to the pixel in the initial pathology element image may be removed in the refined pathology element image.

[0124] The above examples of reallocating pixels are merely illustrative and are not meant to impose any limitations on the manner in which pixels may be reallocated. The reallocation of pixels in step 408 may be based on whether the anatomical characterization of the set of retinal pathology elements presented by the set of pathology element indicators in the initial pathology element image is permissible (e.g., anatomically feasible, clinically relevant, and / or otherwise appropriate). For example, if the location of a pixel annotated with a specific pathology element indicator that assigns the pixel to a specific retinal pathology element makes it anatomically impossible to be associated with the specific retinal pathology element, the pixel may be reallocated. Such determinations are made using layer element images and / or data extracted from layer element images.

[0125] Method 400 may optionally include, at step 410, using the refined pathological element image to perform analysis for detection, diagnosis, and / or treatment of an ophthalmic disease or condition. The ophthalmic disease or condition may be, for example, nAMD, DME, or DR. The analysis in step 410 may include, for example, extracting feature data from the refined pathological element image and in some cases from the layer element image. The feature data may include values ​​of any number of features (e.g., quantitative features) or combinations thereof. Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a border associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of highly reflective foci.

[0126] In various embodiments, the first neural network described in step 404, the second neural network described in step 406, or both may use a neural network such as Figure 1 The first neural network may be trained using training data of training data 140 in . The first neural network may be trained using, for example, a first training data set including a first plurality of training OCT images and a plurality of training layer element images. The plurality of training layer element images may include training multi-channel images, training initial layer element images, training refined layer element images, or a combination thereof.

[0127] Figure 5 is a flow chart of a method 500 for generating a layer element image according to various embodiments. In various embodiments, the method 500 may use Figure 1 For example, the method 500 may be implemented using the image processing system 100 described in Figure 1 and Figure 2 In one or more embodiments, the method 500 may be implemented using the retinal segmentation system 108 described herein. Figure 3 The method 500 may be used to implement the neural network 300 in FIG. Figure 4 An example of a method of step 404 in FIG. 500 may include Figure 2 One or more steps or operations of workflow 200 in.

[0128] like Figure 5 As shown, method 500 includes, at step 502, using the OCT image to generate a multi-channel map via a neural network. The multi-channel map includes a plurality of segmented images, wherein each segmented image in the plurality of segmented images identifies a corresponding target retinal layer. The multi-channel map may be, for example, Figure 1 Multi-channel graph 125, Figure 2 Multi-channel graph 204 or Figure 3 The multi-channel image 318 in FIG. Figure 4The OCT image received in step 402 of method 400 may be, for example Figure 1 OCT image 112 in FIG. The neural network may be, for example, Figure 1 The first neural network 120 or Figure 3 In one or more embodiments, the neural network may include at least one of a CNN, an FCN, a stacked FCN, a stacked FCN with multi-channel learning, a U-Net, or another type of neural network.

[0129] The method 500 further includes, at step 504, converting the multi-channel map into an initial layer element image that uses a set of layer element indicators to identify a set of retinal layer elements. The set of layer element indicators assigns a different set of pixels in the initial layer element image to each retinal layer element of the set of retinal layer elements. The conversion in step 504 may be performed, for example, by applying a piecewise logistic curve approximation to the multi-channel map to generate initial layer heights. The set of retinal layer elements may be associated with respective retinal layers identified in the multi-channel map 204. In some cases, two or more retinal layer elements may correspond to the same target retinal layer.

[0130] The method 500 further includes applying smoothing to the initial layer element image to generate a layer element image at step 506. The smoothing may be performed using, for example, Gaussian smoothing (e.g., n-dimensional (nD) Gaussian smoothing). In one or more embodiments, the initial layer element image and the layer element image are both in the form of a height map.

[0131] Figure 6 is a flow chart of another method 600 for performing retinal segmentation according to various embodiments. In various embodiments, the method 600 may use Figure 1 For example, the method 600 may be implemented using the image processing system 100 described in Figure 1 and Figure 2 In one or more embodiments, a portion of the method 600 may be implemented using the retinal segmentation system 108 described herein. Figure 3 The neural network 300 in FIG.

[0132] Method 600 includes receiving an optical coherence tomography (OCT) image of the retina at step 602. The OCT image may be, for example, Figure 1 OCT image 112 in.

[0133] The method 600 further includes, at step 604, using the OCT image via a neural network to generate a multi-channel map, the multi-channel map comprising a plurality of segmented images, wherein each segmented image in the plurality of segmented images identifies a corresponding target retinal layer. In one or more embodiments, the neural network comprises one or more fully convolutional networks (FCNs). In one or more embodiments, the neural network comprises a U-Net. The neural network may be, for example, Figure 3 Neural Networks in 300.

[0134] The method 600 further includes, at step 606, using the multi-channel map to generate a layer element image, the layer element image identifying a set of retinal layer elements using a set of layer element indicators. In one or more embodiments, step 606 includes converting the multi-channel map into an initial layer element image, the initial layer element image identifying a boundary associated with a target retinal layer identified by the multi-channel map. In some cases, the boundary associated with the target retinal layer estimates an inner boundary of the retinal layer. The conversion in step 606 can be performed by applying a piecewise logistic curve approximation to the multi-channel map to generate the initial layer element image. In some embodiments, step 606 includes applying smoothing to the initial layer height map to generate the layer element image. In other embodiments, the initial layer element image is used as the layer element image.

[0135] The method 600 may further include refining the initial pathology element image using the layer element image to generate a refined pathology element image that visually identifies a set of retinal pathology elements using a set of pathology element indicators at step 608. The refined pathology element image more accurately identifies at least one retinal pathology element in the set of retinal pathology elements than the initial pathology element image. The initial pathology element image may have been generated using a different neural network.

[0136] Figure 7 is a flow chart of another method 700 for performing automatic retinal segmentation according to various embodiments. In various embodiments, the method 700 may use Figure 1 For example, the method 700 may be implemented using the image processing system 100 described in Figure 1 and Figure 2 The retinal segmentation system 108 described above is implemented.

[0137] Step 702 includes receiving an image input of the subject's retina. The image input may be, for example, Figure 2 Input 201 (or Figure 1 The image input may include an OCT image (eg, an SD-OCT image).

[0138] Step 704 includes using the image input and the first neural network to generate layer element data, the layer element data identifying a set of retinal layer elements. The layer element data may be, for example, Figure 1 In one or more embodiments, the layer element data includes a layer element image that identifies a set of retinal layer elements using a set of layer element indicators.

[0139] A retinal layer element in the set of retinal layer elements is a retinal layer or a boundary associated with a retinal layer. A retinal layer can be, for example, but not limited to, an inner limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, an RPE detachment layer, a Bruch's membrane (BM) layer, an ellipsoid zone (EZ), or another type of retinal layer.

[0140] Step 706 includes using the image input and the second neural network to generate initial pathology element data, the initial pathology element data identifying a set of retinal pathology elements. The initial pathology element data may be, for example, Figure 1 In one or more embodiments, the initial pathology element data 128 includes an initial pathology element image that visually identifies the set of retinal pathology elements using a set of pathology element indicators that assign different groups of pixels in the initial pathology element image to each retinal pathology element in the set of retinal pathology elements.

[0141] The set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), highly reflective material (HRM), subretinal highly reflective material (SHRM), intraretinal highly reflective material (IHRM), highly reflective foci (HRF), retinal fluid pockets, or disruption. In some cases, the retinal pathological element may be a disruption (e.g., discontinuity, delamination, loss, etc.) of a retinal layer or retinal zone. For example, the disruption may be a disruption of the ellipsoid band, ELM, RPE, or another layer or zone. The disruption may represent damage or loss of cells (e.g., photoreceptors) in the region of the disruption.

[0142] Step 708 includes refining the initial pathology element data using the layer element data to generate refined pathology element data that more accurately identifies the set of retinal pathology elements than the initial pathology element data. The refined pathology element data may be, for example, Figure 1In one or more embodiments, the refined pathology element data includes a refined pathology element image that visually identifies the set of retinal pathology elements using the set of pathology element indicators that assign an updated set of pixels to at least one retinal pathology element in the set of retinal pathology elements.

[0143] Step 710 may optionally include using the refined pathological element data to perform analysis for detection, diagnosis and / or treatment of an ophthalmic disease or condition (e.g., nAMD, DR or DME). The analysis in step 710 may include, for example, extracting feature data from the refined pathological element data and in some cases from the layer element data. The feature data may include values ​​for any number of features (e.g., quantitative features) or combinations thereof. Examples of such features may include, but are not limited to, maximum retinal layer thickness, minimum retinal layer thickness, average retinal layer thickness, maximum height of a border associated with a retinal layer, volume of a retinal fluid pocket, length of a fluid pocket, width of a fluid pocket, number of retinal fluid pockets, and number of highly reflective foci.

[0144] In one or more embodiments, the retinal pathology element may be a biomarker for one or more ophthalmic diseases or conditions. For example, detection of a retinal pathology element may indicate the presence of one or more ophthalmic diseases or conditions. The refinement in step 708 improves the accuracy of any disease detection and / or diagnosis based on the identification of the retinal pathology element via the refined pathology element data. In some embodiments, the refinement in step 708 helps improve the accuracy of the analysis performed in step 710, and thereby improves the accuracy of any detection, diagnosis, and / or treatment method or solution based on the analysis.

[0145] III. Example Results of the Systems and Methods Disclosed herein

[0146] Fig. 8A and Figure 8B is a diagram of retinal segmentation results according to various embodiments. Fig. 8A is a diagram of a manual retina segmentation result 800A according to various embodiments. Figure 8B is an illustration of an automatic (eg, ML-based automatic) retina segmentation result 800B according to various embodiments.

[0147] The manual retinal segmentation result 800A is based on annotations made by experts (such as Liverpool Central Library) according to their standard operating procedures. In contrast, the automatic retinal segmentation result 800B is obtained by an automatic retinal segmentation system (such as Figure 1 and Figure 2The retinal segmentation system 108 described herein is generated by the manual retinal segmentation result 800A and the automatic retinal segmentation result 800B. Comparing the manual retinal segmentation result 800A with the automatic retinal segmentation result 800B confirms that the embodiments disclosed herein are able to use ML-based algorithms to provide accurate and reliable results. In addition, the embodiments disclosed herein can be used to automatically correct image artifacts and / or defects. In some cases, the embodiments described herein provide a complete automated diagnostic solution for nAMD based on the automatic detection of retinal pathological elements known to be associated with nAMD. In other cases, the embodiments described herein can provide a complete automated diagnostic solution for other ophthalmic diseases or conditions (e.g., DR, DME) based on the automatic detection of retinal pathological elements known to be associated with such ophthalmic diseases or conditions.

[0148] IV. Artificial Neural Networks

[0149] Fig. 9 is a schematic diagram of an example neural network that can be used to implement a computer-based model according to various embodiments. For example, neural network 900 can be included in Figure 1 An example of an implementation of the first neural network 120, the second neural network 122, or a neural network in both.

[0150] As shown, the neural network 900 includes three layers, namely, an input layer 902, a hidden layer 904, and an output layer 906. Each of the input layer 902, the hidden layer 904, and the output layer 906 may include one or more nodes. In this example, the input layer 902 includes nodes 908, 910, 912, and 914; the hidden layer 904 includes nodes 916 and 918; and the output layer 906 includes nodes 920.

[0151] In this example, each node in a layer is connected to each node in an adjacent layer. For example, node 908 in input layer 902 is connected to both node 916 and node 918 in hidden layer 904. Similarly, node 916 in hidden layer 904 is connected to all of node 908, node 910, node 912, and node 914 in input layer 902, as well as node 920 in output layer 906. Although only one hidden layer is shown for neural network 900, neural network 900 may include any number of hidden layers between input layer 902 and output layer 906.

[0152] In this example, neural network 900 receives a set of input values ​​(e.g., inputs 1 to 4) and produces an output value (e.g., output 5). Each node in input layer 902 may correspond to a different input value. As an example, the set of input values ​​may include, for example, Figure 1A set of attributes of an image of the OCT image 112 in FIG. 1 . In this example, each node in the input layer 902 may correspond to and receive a different attribute of the image.

[0153] In some embodiments, each of the nodes 916 and 918 in the hidden layer 904 generates a representation that may include a mathematical calculation (or algorithm) that generates a value based on input values ​​received from the nodes 908, 910, 912, and 914. The mathematical calculation may include assigning different weights to each of the data values ​​received from the nodes 908, 910, 912, and 914. The nodes 916 and 918 may include different algorithms and / or different weights assigned to the data variables from the nodes 908, 910, 912, and 914, so that each of the nodes 916 and 918 may generate different values ​​based on the same input values ​​received from the nodes 908, 910, 912, and 914. In some embodiments, the weights initially assigned to the features (or input values) of each of the nodes 916 and 918 may be randomly generated (e.g., using a computer random generator). Node 920 in output layer 906 may use the values ​​generated by nodes 916 and 918 to produce an output value for neural network 900 .

[0154] The neural network 900 may be trained using training data. For example, the training data may include various OCT images. By providing the training data to the neural network 900, the nodes 916 and 918 in the hidden layer 904 may be trained (adjusted) so that the optimal output is generated in the output layer 906 based on the training data. By continuously providing different sets of training data and penalizing the neural network 900 when the output of the neural network 900 is incorrect, the neural network 900 (and specifically, the representation of the nodes in the hidden layer 904) may be trained (adjusted) to improve its performance in data classification. Adjusting the neural network 900 may include adjusting the weights associated with each node in the hidden layer 904.

[0155] Although the above discussion relates to an artificial neural network as an example of machine learning, it should be understood that other types of machine learning methods may also be applicable to various aspects of implementing the present disclosure. For example, a support vector machine (SVM) may be used to implement machine learning. SVM is a group of related supervised learning methods for classification and regression. An SVM training algorithm (which may be a non-probabilistic binary linear classifier) ​​may construct a model that predicts whether a new example belongs to one category or another category. As another example, a Bayesian network may be used to implement machine learning. A Bayesian network is a non-cyclic probabilistic graphical model that represents a set of random variables and their conditional independence with a directed acyclic graph (DAG). A Bayesian network may present a probabilistic relationship between a variable and another variable. Another example is a machine learning engine that uses a decision tree learning model to perform a machine learning process. In some cases, a decision tree learning model may include a classification tree model and a regression tree model.

[0156] In some embodiments, the machine learning engine uses a gradient boosting machine (GBM) model (e.g., XGBoost) as a regression tree model. Other machine learning techniques can be used to implement the machine learning engine, such as via a random forest or a deep neural network. For simplicity, other types of machine learning algorithms are not discussed in detail herein, and it should be understood that the present disclosure is not limited to a specific type of machine learning.

[0157] V. Computer-Implemented Systems

[0158] Fig.10 is a block diagram of a computer system according to various embodiments. Figure 1 1004 . In one or more examples, the computer system 1000 may include a bus 1002 or other communication mechanism for communicating information, and a processor 1004 coupled to the bus 1002 for processing information. In various embodiments, the computer system 1000 may also include a memory (which may be a random access memory (RAM) 1006 or other dynamic storage device) coupled to the bus 1002 for determining instructions to be executed by the processor 1004. The memory may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1004. In various embodiments, the computer system 1000 may further include a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010 (such as a magnetic disk or optical disk) may be provided and coupled to the bus 1002 for storing information and instructions.

[0159] In various embodiments, the computer system 1000 may be coupled via the bus 1002 to a display 1012, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), for displaying information to a computer user. An input device 1014, including alphanumeric and other keys, may be coupled to the bus 1002 for communicating information and command selections to the processor 1004. Another type of user input device is a cursor control 1016, such as a mouse, joystick, trackball, gesture input device, gaze-based input device, or cursor direction keys, for communicating direction information and command selections to the processor 1004 and for controlling cursor movement on the display 1012. The input device 1014 typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane. However, it should be understood that an input device 1014 that allows three-dimensional (e.g., x, y, and z) cursor movement is also contemplated herein.

[0160] Consistent with certain implementations of the present teachings, results may be provided by the computer system 1000 in response to the processor 1004 executing one or more sequences of one or more instructions contained in the RAM 1006. Such instructions may be read into the RAM 1006 from another computer-readable medium or a computer-readable storage medium, such as the storage device 1010. Execution of the sequences of instructions contained in the RAM 1006 may cause the processor 1004 to perform the processes described herein. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present teachings. Thus, implementations of the present teachings are not limited to any specific combination of hardware circuitry and software.

[0161] As used herein, the term "computer-readable medium" (e.g., data storage, data memory, storage device, data storage device, etc.) or "computer-readable storage medium" refers to any medium that participates in providing instructions to processor 1004 for execution. Such media can take a variety of forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Examples of non-volatile media can include, but are not limited to, optical disks, solid-state drives, magnetic disks (such as storage device 1010). Examples of volatile media can include, but are not limited to, dynamic memory, such as RAM 1006. Examples of transmission media can include, but are not limited to, coaxial cables, copper wire, and optical fiber, including the wires that make up bus 1002.

[0162] Common forms of computer readable media include, for example, floppy disks, foldable disks, hard disks, magnetic tapes, or any other magnetic medium; CD-ROMs, any other optical medium; punch cards, paper tapes, any other physical medium with a pattern of holes; RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge; or any other tangible medium from which a computer can read.

[0163] In addition to computer-readable media, instructions or data may also be provided as signals on a transmission medium included in a communication device or system to provide one or more instruction sequences to the processor 1004 of the computer system 1000 for execution. For example, a communication device may include a transceiver with signals indicating instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in this disclosure. Representative examples of data communication transmission connections may include, but are not limited to, telephone modem connections, wide area networks (WANs), local area networks (LANs), infrared data connections, NFC connections, optical communication connections, etc.

[0164] It should be appreciated that the methods, flow charts, diagrams, and accompanying disclosure described herein may be implemented using computer system 1000 as a stand-alone device or on a distributed network that shares computer processing resources, such as a cloud computing network.

[0165] Depending on the application, the methods described herein can be implemented in various ways. For example, these methods can be implemented in hardware, firmware, software, or any combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or combinations thereof.

[0166] In various embodiments, the methods of the present teachings may be implemented as firmware and / or software programs and applications written in conventional programming languages ​​such as C, C++, Python, etc. If implemented as firmware and / or software, the embodiments described herein may be implemented on a non-transitory computer readable medium in which a program is stored to cause a computer to perform the above methods. It should be understood that the various engines described herein may be provided on a computer system, such as computer system 1000, where the processor 1004 will perform the analysis and determination provided by these engines based on instructions provided by any of the memory components RAM 1006, ROM 1008, or storage device 1010, or a combination thereof, and user input provided via input device 1014.

[0167] VI. Example Definition and Context

[0168] The present disclosure is not limited to these example embodiments and applications, nor to the manner of operation of the example embodiments and applications or the manner described herein.In addition, the drawings may show simplified or partial views, and the sizes of elements in the drawings may be exaggerated or not in proportion.

[0169] Unless otherwise defined, scientific and technical terms used in conjunction with the present teachings described herein shall have the meanings commonly understood by those of ordinary skill in the art. In addition, unless the context otherwise requires, singular terms shall include the plural and plural terms shall include the singular. In general, nomenclature and techniques used in conjunction with chemistry, biochemistry, molecular biology, pharmacology, and toxicology are described herein, and these nomenclatures and techniques are those well known and commonly used in the art.

[0170] In addition, when the terms "on," "attached to," "connected to," "coupled to," or the like are used herein, an element (e.g., a component, material, layer, substrate, etc.) may be "on," "attached to," "connected to," or "coupled to" another element, regardless of whether one element is directly on, directly attached to, directly connected to, or directly coupled to another element, or whether there are one or more intervening elements between one element and another element. In addition, where a list of elements is referred to (e.g., elements a, b, c), such reference is intended to include any one of the elements listed individually, any combination of less than all of the listed elements, and / or combinations of all of the listed elements. The section divisions in the specification are for ease of review only and do not limit any combination of the elements discussed.

[0171] The term "subject" may refer to a subject of a clinical trial, a person undergoing treatment, a person undergoing anti-cancer treatment, a person undergoing remission or recovery monitoring, a person undergoing preventive health analysis (e.g., due to their medical history), or a person or patient of any other purpose. In various cases, "subject" and "patient" are used interchangeably herein.

[0172] As used herein, "substantially" means sufficient to achieve the intended purpose. Thus, the term "substantially" allows for minor, inconsequential variations relative to an absolute or ideal state, dimension, measurement, result, etc., as would be expected by one of ordinary skill in the art, but without significantly affecting the overall performance. When used relative to a numerical value or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within ten percent.

[0173] As used herein, the term "about" used with respect to a numerical value or a parameter or feature that can be expressed as a numerical value means within ten percent of the numerical value. For example, "about 50" means a value within the range of 45 to 55, including the end values.

[0174] The term "ones" means more than one.

[0175] As used herein, the term "plurality" may be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.

[0176] As used herein, the term "a set" refers to one or more. For example, a set of items includes one or more items.

[0177] As used herein, the phrase "at least one of..." when used with a list of items means that different combinations of one or more of the listed items may be used, and it is possible that only one item in the list may be used. An item may be a specific object, thing, step, operation, process, or category. In other words, "at least one of..." refers to any combination of items or number of items that can be used in the list, but not all of the items in the list may be used. For example, but not limiting, "at least one of item A, item B, or item C" refers to item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and C. In some cases, "at least one of item A, item B, or item C" refers to, but is not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.

[0178] As used herein, a "model" may include one or more algorithms, one or more mathematical techniques, one or more machine learning algorithms, or a combination thereof.

[0179] As used herein, "machine learning" can include the practice of using algorithms to parse data, learn from it, and then make a determination or prediction about something in the world. Machine learning can use algorithms that can learn from data without relying on rule-based programming. Deep learning may be a form of machine learning.

[0180] As used herein, an "artificial neural network" or "neural network" (NN) may refer to a mathematical algorithm or computational model that simulates a set of interconnected artificial neurons that process information based on a connectionist computational approach. A neural network (which may also be referred to as a neural network) may use one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks may also include one or more hidden layers. The output of each hidden layer may be used as an input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. In various embodiments, references to a "neural network" may be references to one or more neural networks.

[0181] A neural network can process information in two ways; when the neural network is being trained, the neural network is in training mode, and when the neural network puts what it has learned into practice, the neural network is in inference (or prediction) mode. A neural network can learn through a feedback process (e.g., backpropagation) that allows the network to adjust the weight factors of various nodes in the intermediate hidden layers (modify their behavior) so that the output matches the output of the training data. In other words, a neural network can learn by being fed training data (learning examples) and eventually learn how to obtain the correct output, even if it is presented with a new range or set of inputs. A neural network may include, for example, but not limited to, at least one of a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a convolutional neural network (CNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural-ODE), a U-Net, a fully convolutional network (FCN), a stacked FCN, a stacked FCN using multi-channel learning, a squeeze and excite embedding neural network, a MobileNet, or another type of neural network.

[0182] As used herein, “deep learning” may refer to the use of multi-layer artificial neural networks to automatically learn representations from input data (such as images, videos, text, etc.) without the need for human-provided knowledge to provide highly accurate predictions in tasks such as object detection / recognition, speech recognition, language translation, etc.

[0183] VII. Other considerations

[0184] Although the present teaching is described in conjunction with various embodiments, the present teaching is not intended to be limited to such embodiments. On the contrary, the present teaching encompasses various alternatives, modifications and equivalents that will be understood by those skilled in the art.

[0185] For example, the above-mentioned flow chart and block diagram illustrate the architecture, function and / or operation of the possible implementation of various methods and system embodiments.Each frame in the flow chart or block diagram can represent a part of a module, segment, function, operation or step or a combination thereof.In some alternative implementations of the embodiment, one or more functions marked in the frame may not occur in the order marked in the figure.For example, in some cases, two frames of continuous display can be performed substantially simultaneously.In other cases, frame can be performed in reverse order.In addition, in some cases, one or more frames can be added to replace or supplement one or more other frames in the flow chart or block diagram.

[0186] Thus, in describing various embodiments, the specification may have presented the method and / or process as a particular sequence of steps. However, if the method or process does not rely on the particular order of steps described herein, the method or process should not be limited to the particular sequence of steps listed, and those skilled in the art can readily understand that these sequences can be different and still remain within the spirit and scope of the various embodiments.

[0187] VIII. Presentation of Examples

[0188] Embodiment 1. A method for performing retinal segmentation, the method comprising: receiving an optical coherence tomography (OCT) image of a retina; using the OCT image and a first neural network to generate a layer element image, the layer element image using a set of layer element indicators to identify a set of retinal layer elements; using the OCT image and a second neural network to generate an initial pathological element image, the initial pathological element image using a set of pathological element indicators to visually identify a set of retinal pathological elements, the set of pathological element indicators assigning a different pixel group to each retinal pathological element in the set of retinal pathological elements; and using the layer element image to refine the initial pathological element image to generate a refined pathological element image, the refined pathological element image using the set of pathological element indicators to visually identify the set of retinal pathological elements, the set of pathological element indicators assigning an updated pixel group to at least one retinal pathological element in the set of retinal pathological elements.

[0189] Embodiment 2. A method according to embodiment 1, wherein a pathological element indicator in the set of pathological element indicators is used to assign a pixel group in the initial pathological element image to a first retinal pathological element in the set of retinal pathological elements, and wherein the refinement includes: reallocating a portion of the pixel group in the initial pathological element image based on whether the anatomical characterization of the first retinal pathological element identified by the pathological element indicator is anatomically feasible, wherein the anatomical feature of the first retinal pathological element includes at least one of the position, size, shape, length, width, thickness or volume of the retinal pathological element.

[0190] Embodiment 3. A method according to Embodiment 2, wherein the reallocation comprises at least one of: reallocating a first pixel in the pixel group from the first retinal pathology element to a second retinal pathology element in the group of retinal pathology elements based on the layer element image; or reallocating a second pixel in the pixel group from the first retinal pathology element to a background based on the layer element image.

[0191] Embodiment 4. A method according to any one of embodiments 1 to 2, wherein the refinement comprises: updating a pixel group in the initial pathology element image assigned to the retinal pathology element by constraining an allowed area of ​​the retinal pathology element in the set of retinal pathology elements based on the layer element image to form the updated pixel group for the retinal pathology element in the refined pathology element image, wherein the updated pixel group includes fewer pixels than the pixel group.

[0192] Embodiment 5. A method according to any one of embodiments 1 to 4, wherein generating the layer element image comprises: using the OCT image via the first neural network to generate a multi-channel image, wherein the multi-channel image comprises a plurality of segmented images, wherein each segmented image in the plurality of segmented images identifies a corresponding target retinal layer.

[0193] Embodiment 6. A method according to embodiment 5, wherein generating the layer element image further comprises: converting the multi-channel image into an initial layer element image, wherein the initial layer element image uses the set of layer element indicators to identify the set of retinal layer elements, wherein the set of layer element indicators assigns different pixel groups in the initial layer element image to each retinal layer element in the set of retinal layer elements.

[0194] Embodiment 7. The method of embodiment 6, wherein the converting comprises applying a piecewise logistic curve approximation to the multi-channel map to generate the initial layer element image.

[0195] Embodiment 8. The method according to embodiment 6 or embodiment 7, wherein generating the layer element image further comprises: applying smoothing to the initial layer element image to generate the layer element image.

[0196] Embodiment 9. The method according to Embodiment 8, wherein applying smoothing to the initial layer element image comprises applying Gaussian smoothing to the initial layer element image to generate the layer element image.

[0197] Embodiment 10. The method according to any one of embodiments 1 to 9, wherein the retinal layer element in the set of retinal layer elements is a retinal layer or a boundary associated with the retinal layer.

[0198] Example 11. A method according to Example 10, wherein the retinal layer is selected from the group consisting of: an inner limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, an RPE detachment layer, a Bruch's membrane (BM) layer and an ellipsoid zone (EZ).

[0199] Example 12. A method according to any one of Examples 10 to 11, wherein the set of retinal pathology elements includes intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), highly reflective material (HRM), subretinal highly reflective material (SHRM), intraretinal highly reflective material (IHRM), highly reflective foci (HRF), retinal fluid pockets, destruction, or characteristics or subtypes of fluid, material or destruction.

[0200] Embodiment 13. A method according to any one of Embodiments 10 to 12, wherein each of the set of layer element indicators and the set of pathology element indicators comprises at least one of a color indicator, a shape indicator, a pattern indicator, a shadow indicator, a line, a curve, a logo, a mark, a label or text.

[0201] Embodiment 14. A method according to any one of embodiments 10 to 13, wherein the first neural network includes a first U-Net and the second neural network includes a second U-Net.

[0202] Embodiment 15. A method according to any one of embodiments 10 to 14, wherein: a first training data set is used to train the first neural network, the first training data set includes a first plurality of training OCT images and a plurality of training layer element images; and a second training data set is used to train the second neural network, the second training data set includes a second plurality of training OCT images and a plurality of training pathological element images.

[0203] Embodiment 16. The method according to Embodiment 15, wherein at least a portion of the first plurality of training OCT images is included in the second plurality of training OCT images.

[0204] Embodiment 17. A method for performing retinal segmentation, the method comprising: receiving an optical coherence tomography (OCT) image of the retina; using the OCT image to generate a multi-channel image via a neural network, the multi-channel image comprising a plurality of segmented images, wherein each segmented image in the plurality of segmented images identifies a corresponding target retinal layer; using the multi-channel image to generate a layer element image, the layer element image using a set of layer element indicators to identify a set of retinal layer elements; and using the layer element image to refine an initial pathological element image to generate a refined pathological element image, the refined pathological element image using a set of pathological element indicators to visually identify a set of retinal pathological elements, wherein the refined pathological element image more accurately identifies at least one retinal pathological element in the set of retinal pathological elements than the initial pathological element image.

[0205] Embodiment 18. The method according to embodiment 17, wherein generating the layer element image comprises: converting the multi-channel map into an initial layer element image using piecewise logistic curve approximation.

[0206] Embodiment 19. The method according to embodiment 18, wherein generating the layer element image further comprises: applying smoothing to the initial layer element image to generate the layer element image, and the layer element image is subsequently used to refine the initial pathological element image.

[0207] Embodiment 20. A method according to any one of Embodiments 17 to 19, wherein refining the initial pathology element image comprises at least one of the following: reallocating a first portion of pixels in the initial pathology element image from one retinal pathology element to a different retinal pathology element in the refined pathology element image based on the layer element image; or reallocating a second portion of pixels in the initial pathology element image to a background in the refined pathology element image based on the layer element image.

[0208] Embodiment 21. A system for performing automatic retinal segmentation, the system comprising: a non-transitory memory; and a data processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations, the operations comprising: receiving an optical coherence tomography (OCT) image of a retina; using the OCT image and a first neural network to generate a layer element image, the layer element image using a set of layer element indicators to identify a set of retinal layer elements; using the OCT image and a second neural network to generate an initial pathological element image, the initial pathological element image using a set of pathological element indicators to visually identify a set of retinal pathological elements, the set of pathological element indicators assigning a different pixel group to each retinal pathological element in the set of retinal pathological elements; and using the layer element image to refine the initial pathological element image to generate a refined pathological element image, the refined pathological element image using the set of pathological element indicators to visually identify the set of retinal pathological elements, the set of pathological element indicators assigning an updated pixel group to at least one retinal pathological element in the set of retinal pathological elements.

[0209] Embodiment 22. A method for performing automatic retinal segmentation, the method comprising: receiving an image input of a subject's retina; using the image input and a first neural network to generate layer element data, the layer element data identifying a set of retinal layer elements; using the image input and a second neural network to generate initial pathology element data, the initial pathology element data identifying a set of retinal pathology elements; and using the layer element data to refine the initial pathology element data to generate refined pathology element data, the refined pathology element data more accurately identifying the set of retinal pathology elements compared to the initial pathology element data.

[0210] Embodiment 23. A method according to embodiment 22, wherein the initial pathology element data includes an initial pathology element image, and the initial pathology element image visually identifies the set of retinal pathology elements using a set of pathology element indicators, and the set of pathology element indicators assigns different pixel groups in the initial pathology element image to each retinal pathology element in the set of retinal pathology elements.

[0211] Embodiment 24. A method according to Embodiment 23, wherein the refined pathology element data includes a refined pathology element image, which visually identifies the set of retinal pathology elements using the set of pathology element indicators, and the set of pathology element indicators assigns an updated pixel group to at least one retinal pathology element in the set of retinal pathology elements.

[0212] Embodiment 25. The method of any one of Embodiments 22 to 24, wherein the layer element data comprises a layer element image that identifies a set of retinal layer elements using a set of layer element indicators.

[0213] Embodiment 26. A method according to any one of Embodiments 22 to 25, wherein the image input comprises an SD-OCT image.

[0214] Embodiment 27. A method according to any one of embodiments 22 to 26, wherein the retinal layer element in the set of retinal layer elements is a retinal layer or a boundary associated with the retinal layer, and wherein the retinal layer is selected from the group consisting of: an inner limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, an RPE detachment layer, a Bruch's membrane (BM) layer, and an ellipsoid zone (EZ).

[0215] Embodiment 28. A method according to any one of embodiments 22 to 27, wherein the set of retinal pathological elements includes intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), highly reflective material (HRM), subretinal highly reflective material (SHRM), intraretinal highly reflective material (IHRM), highly reflective foci (HRF), retinal fluid pockets, destruction, or characteristics or subtypes of fluid, material or destruction.

[0216] Embodiment 29. A system comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed in Embodiments 1 to 20 and 22 to 28.

[0217] Embodiment 30. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed in Embodiments 1 to 20 and 22 to 28.

Claims

1. A method for performing retinal segmentation, the method comprising: receiving optical coherence tomography (OCT) images of the retina; generating a layer element image using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indicators; generating an initial pathological element image using the OCT image and a second neural network, the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indicators that assign a different set of pixels to each retinal pathological element in the set of retinal pathological elements; as well as The initial pathology element image is refined using the layer element image to generate a refined pathology element image, the refined pathology element image visually identifying the set of retinal pathology elements using the set of pathology element indicators that assign an updated group of pixels to at least one retinal pathology element in the set of retinal pathology elements.

2. The method of claim 1 , wherein a group of pixels in the initial pathology element image is assigned to a first retinal pathology element in the set of retinal pathology elements using a pathology element indicator in the set of pathology element indicators, and wherein the refining comprises: reallocating a portion of the set of pixels in the initial pathological element image based on whether the anatomical features of the first retinal pathological element identified by the pathological element indicator are anatomically feasible, The anatomical feature of the first retinal pathological element comprises at least one of the position, size, shape, length, width, thickness or volume of the retinal pathological element.

3. The method of claim 2, wherein the reallocation comprises at least one of: reallocating a first pixel in the group of pixels from the first retinal pathology element to a second retinal pathology element in the group of retinal pathology elements based on the layer element image; or Based on the layer element image, a second pixel in the group of pixels is reallocated from the first retinal pathology element to background.

4. The method according to any one of claims 1 to 3, wherein the refinement comprises: A pixel group in the initial pathology element image assigned to the retinal pathology element is updated by constraining an allowed area for the retinal pathology element in the set of retinal pathology elements based on the layer element image to form the updated pixel group for the retinal pathology element in the refined pathology element image, wherein the updated pixel group includes fewer pixels than the pixel group.

5. The method according to any one of claims 1 to 4, wherein generating the layer element image comprises: The OCT image is used via the first neural network to generate a multi-channel map, wherein the multi-channel map includes a plurality of segmented images, wherein each segmented image of the plurality of segmented images identifies a corresponding target retinal layer.

6. The method of claim 5, wherein generating the layer element image further comprises: The multi-channel map is converted into an initial layer element image, wherein the initial layer element image identifies the set of retinal layer elements using the set of layer element indicators, wherein the set of layer element indicators assigns a different group of pixels in the initial layer element image to each retinal layer element in the set of retinal layer elements.

7. The method of claim 6, wherein the converting comprises: A piecewise logistic curve approximation is applied to the multi-channel map to generate the initial layer element image.

8. The method according to claim 6 or claim 7, wherein generating the layer element image further comprises: Smoothing is applied to the initial layer element image to generate the layer element image. 9 . The method of claim 8 , wherein applying smoothing to the initial layer element image comprises applying Gaussian smoothing to the initial layer element image to generate the layer element image.

10. The method according to any one of claims 1 to 9, wherein a retinal layer element in the set of retinal layer elements is a retinal layer or a boundary associated with the retinal layer.

11. The method of claim 10, wherein the retinal layer is selected from the group consisting of an inner limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, an RPE detachment layer, a Bruch's membrane (BM) layer, and an ellipsoid zone (EZ).

12. The method of any one of claims 1 to 11, wherein the set of retinal pathological elements comprises at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), highly reflective material (HRM), subretinal highly reflective material (SHRM), intraretinal highly reflective material (IHRM), highly reflective foci (HRF), retinal fluid pockets or disruptions.

13. The method of any one of claims 1 to 12, wherein each of the set of layer element indicators and the set of pathology element indicators comprises at least one of a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a logo, a mark, a label, or text.

14. The method of any one of claims 1 to 13, wherein the first neural network comprises a first U-Net and the second neural network comprises a second U-Net.

15. The method according to any one of claims 1 to 14, wherein: training the first neural network using a first training data set, the first training data set comprising a first plurality of training OCT images and a plurality of training layer element images; and The second neural network is trained using a second training data set, the second training data set comprising a second plurality of training OCT images and a plurality of training pathological element images. 16 . The method of claim 15 , wherein at least a portion of the first plurality of training OCT images is included in the second plurality of training OCT images.

17. A method for performing retinal segmentation, the method comprising: receiving optical coherence tomography (OCT) images of the retina; generating a multi-channel map using the OCT image via a neural network, the multi-channel map comprising a plurality of segmented images, wherein each segmented image of the plurality of segmented images identifies a corresponding target retinal layer; generating a layer element image using the multi-channel map, the layer element image identifying a set of retinal layer elements using a set of layer element indicators; as well as The layer element image is used to refine an initial pathology element image to generate a refined pathology element image, the refined pathology element image visually identifying a set of retinal pathology elements using a set of pathology element indicators, wherein the refined pathology element image more accurately identifies at least one retinal pathology element in the set of retinal pathology elements than the initial pathology element image.

18. The method of claim 17, wherein generating the layer element image comprises: The multi-channel map is converted into an initial layer element image using piecewise logistic curve approximation.

19. The method of claim 18, wherein generating the layer element image further comprises: Smoothing is applied to the initial layer element image to generate the layer element image, which is then used to refine the initial pathological element image.

20. The method according to any one of claims 17 to 19, wherein refining the initial pathological element image comprises at least one of the following: reallocating a first portion of pixels in the initial pathology element image from one retinal pathology element to a different retinal pathology element in the refined pathology element image based on the layer element image; or Based on the layer element image, a second portion of the pixels in the initial pathological element image is reallocated to a background in the refined pathological element image.

21. A system for performing automatic retinal segmentation, the system comprising: non-transitory memory; as well as a data processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations including: receiving optical coherence tomography (OCT) images of the retina; generating a layer element image using the OCT image and a first neural network, the layer element image identifying a set of retinal layer elements using a set of layer element indicators; generating an initial pathological element image using the OCT image and a second neural network, the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indicators that assign a different set of pixels to each retinal pathological element in the set of retinal pathological elements; and The initial pathology element image is refined using the layer element image to generate a refined pathology element image, the refined pathology element image visually identifying the set of retinal pathology elements using the set of pathology element indicators that assign an updated group of pixels to at least one retinal pathology element in the set of retinal pathology elements.

22. A method for performing automatic retinal segmentation, the method comprising: receiving an image input directed to a retina of a subject; generating layer element data using the image input and a first neural network, the layer element data identifying a set of retinal layer elements; generating initial pathology element data using the image input and a second neural network, the initial pathology element data identifying a set of retinal pathology elements; as well as The initial pathology element data is refined using the layer element data to generate refined pathology element data that more accurately identifies the set of retinal pathology elements than the initial pathology element data.

23. The method of claim 22, wherein the initial pathology element data comprises an initial pathology element image that visually identifies the set of retinal pathology elements using a set of pathology element indicators that assign different groups of pixels in the initial pathology element image to each retinal pathology element in the set of retinal pathology elements.

24. The method of claim 23, wherein the refined pathology element data comprises a refined pathology element image that visually identifies the set of retinal pathology elements using the set of pathology element indicators that assign an updated group of pixels to at least one retinal pathology element in the set of retinal pathology elements.

25. The method of any one of claims 22 to 14, wherein the layer element data comprises a layer element image that identifies a set of retinal layer elements using a set of layer element indicators.

26. The method of any one of claims 22 to 25, wherein the image input comprises an SD-OCT image.

27. The method of any one of claims 22 to 26, wherein the retinal layer element in the set of retinal layer elements is a retinal layer or a boundary associated with the retinal layer, and wherein the retinal layer is selected from the group consisting of an inner limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelium (RPE) layer, an RPE detachment layer, a Bruch's membrane (BM) layer, and an ellipsoid zone (EZ).

28. The method of any one of claims 22 to 27, wherein the set of retinal pathological elements comprises at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), highly reflective material (HRM), subretinal highly reflective material (SHRM), intraretinal highly reflective material (IHRM), highly reflective foci (HRF), retinal fluid pockets, or disruptions.

29. A system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed in claims 1 to 20 and 22 to 28.

30. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed in claims 1 to 20 and 22 to 28.

Images