Machine learning enabled localization of fovea centers in spectral domain optical coherence tomography volumetric scans

Abstract

A method and system for locating a foveal center of a retina. An optical coherence tomography (OCT) volume is received for a retina of a subject. The OCT volume includes a plurality of OCT B scans of the retina. A three-dimensional image input is generated for a model using the OCT volume. The model comprises a three-dimensional convolutional neural network. The model is used to generate a foveal center position comprising three-dimensional coordinates for a foveal center of the retina based on the three-dimensional image input. The foveal center location may be used to generate an output that may be used to screen for, diagnose, predict therapeutic response, and / or manage treatment of retinal disease.

Description

[0001] Inventors: Sharif Amit Kamran, Michael Greg Kaucinski, Siva Balasubramanian, Andreas Maoz

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is related to and claims the benefit of the priority date of U.S. Provisional Application No. 63 / 371,297, filed on August 12, 2022, entitled “Machine Learning Enabled Localization of Foveal Center in Spectral Domain Optical Coherence Tomography Volume Scans,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The subject matter described herein relates generally to optical coherence tomography (OCT), and more specifically to analysis methods and systems using deep learning techniques to detect and locate the center of the fovea in an OCT volume. Background Art

[0005] Various imaging techniques have been developed to capture medical images of tissues, which can then be analyzed to determine the presence or progression of a disease. For example, optical coherence tomography (OCT) refers to a technique that uses light waves to capture two-dimensional slice images (e.g., OCT B-scans) and three-dimensional volumetric images (e.g., OCT volumes) of a patient's tissue (such as the retina). OCT images can be analyzed and used to calculate quantitative measurements for retinal disease screening, diagnosis, and treatment management. Such quantitative measurements can include measurements associated with the foveal center of the retina, such as, for example, central retinal thickness (CST) and other retinal thickness measurements (e.g., thickness measurements on the Early Treatment Diabetic Retinopathy Study (ETDRS) grid). Therefore, it may be desirable to have methods and systems that improve the accuracy and reliability of quantitative measurements calculated based on the foveal center of the retina using OCT images. Summary of the invention

[0006] In one or more embodiments, a method is provided in which an optical coherence tomography (OCT) volume of a retina of a subject is received. The OCT volume includes multiple OCT B-scans of the retina. A three-dimensional image input is generated for a model using the OCT volume. The model includes a three-dimensional convolutional neural network. The model is used to generate a foveal center location including three-dimensional coordinates for a foveal center of the retina based on the three-dimensional image input.

[0007] In one or more embodiments, a method for training a model is provided. The method includes receiving a training data set, the training data set including multiple optical coherence tomography (OCT) volumes for multiple retinas, wherein each of the multiple OCT volumes includes multiple OCT B-scans. Using the multiple OCT volumes in the training data set to generate a training three-dimensional image input for a model, the model including a three-dimensional convolutional neural network and a regression layer. The model is trained to generate a foveal center location based on the training three-dimensional image input, the foveal center location including the three-dimensional coordinates of the foveal center of the retina in the selected OCT volume.

[0008] In one or more embodiments, a system is provided, comprising one or more data processors and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium containing instructions, which when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein. For example, the one or more data processors can be caused to: receive an optical coherence tomography (OCT) volume for a subject's retina, the optical coherence tomography (OCT) volume comprising a plurality of OCT B-scans of the retina; use the OCT volume to generate a three-dimensional image input for a model, the model comprising a three-dimensional convolutional neural network; and generate, via the model, a foveal center location comprising three-dimensional coordinates for a foveal center of the retina based on the three-dimensional image input.

[0009] In one or more, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform a portion or all of one or more methods disclosed herein. For example, the one or more data processors can be caused to: receive an optical coherence tomography (OCT) volume for a subject's retina, the optical coherence tomography (OCT) volume including a plurality of OCT B-scans of the retina; use the OCT volume to generate a three-dimensional image input for a model, the model including a three-dimensional convolutional neural network; and generate, via the model, a foveal center location including three-dimensional coordinates for a foveal center of the retina based on the three-dimensional image input.

[0010] The terms and expressions that have been adopted are used as terms of description and not of limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some principles associated with the disclosed implementations.

[0012] Figure 1 is a block diagram of an image processing system according to one or more exemplary embodiments.

[0013] Figure 2 is a flow chart of a process for processing an optical coherence tomography (OCT) volumetric image of a subject's retina in accordance with one or more exemplary embodiments.

[0014] Figure 3 is a flow diagram of a process for training a model to generate a foveal center location in accordance with one or more embodiments.

[0015] Figure 4 is an illustration of an example workflow for processing an OCT volume in accordance with one or more exemplary embodiments.

[0016] Figure 5 is a block diagram illustrating an example of a computing system according to one or more exemplary embodiments.

[0017] It should be understood that the drawings are not necessarily drawn to scale, nor are the objects in the drawings necessarily drawn to scale relative to each other. The drawings are depictions of various embodiments intended for clarity and understanding of the devices, systems, and methods disclosed herein. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. In addition, it should be understood that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION

[0018] I. Overview

[0019] Medical imaging techniques are powerful tools that can be used to produce medical images that enable healthcare practitioners to better visualize and understand their patients' medical problems, and therefore provide more accurate diagnostic and treatment options. These medical imaging techniques can be used to image certain parts of the retina, such as the fovea, in order to accurately diagnose and treat retinal diseases, and to monitor the retina more generally.

[0020] The fovea (or fovea centralis) is a small depression at the center of the macula of the retina. The fovea is formed by densely packed cones and is surrounded by the parafoveal band, the outer region around the fovea, and the larger peripheral region. The center of the fovea (or foveal center) is filled with the highest density of cones found in the retina, while the density of cones around the fovea is significantly lower. For example, the density of cones in the outer region around the fovea is about 12 cones per 100 microns, while there are about 50 cones per 100 microns in the most central fovea. Although the fovea occupies only a relatively small part of the retina, it is responsible for sharp central vision (e.g., foveal vision), which is indispensable for activities that rely on visual details. For example, pointing the fovea in a certain direction may focus sensory processing resources on the most relevant information sources.

[0021] The center of the fovea (or foveal center) is an important retinal feature for understanding disease states and vision loss. For example, as previously described, the foveal center is filled with the highest density of cones found in the retina. Retinal thickening, including swelling in the center or around the fovea, is often associated with loss of visual acuity.

[0022] Therefore, the center of the fovea or the foveal center is a key marker for further analysis of retinal features. Positioning the foveal center may be crucial for measuring biomarkers related to diagnosing retinal diseases, assessing disease burden, monitoring retinal disease progression, and predicting treatment responses. For example, central retinal thickness (CST) is an important quantitative measurement for disease monitoring and treatment response, which can be determined based on the foveal center of the retina. CST is usually measured by calculating the average retinal thickness on a circular area (e.g., 1 mm ring) centered on the foveal center. Similarly, measurements related to retinal segmentation can be based on the foveal center, such as but not limited to generating an early treatment study of diabetic retinopathy (ETDRS) grid, or calculating inner limiting membrane (ILM) and / or Bruch's membrane (BM) boundary segmentation.

[0023] Optical coherence tomography (OCT) is a non-invasive imaging technique that is particularly popular for capturing images of the retina. OCT can be described as an ultrasound scanning technique that scatters light waves from tissue to generate OCT images in the form of two-dimensional (2D) images and / or three-dimensional (3D) images of tissue, similar to ultrasound scanning that scans tissue using sound waves. 2D OCT images may also be referred to as OCT slices, OCT cross-sectional images, or OCT scans (e.g., OCT B scans). 3D OCT images may be referred to as OCT volume images, and may be composed of many OCT slice images.

[0024] For example, the patient has a retinal disease, such as neovascular age-related macular degeneration, diabetic macular edema, or some other type of retinal disease, which may cause the misalignment between the patient's foveal center and the geometric center of the resulting OCT volume. Similarly, poor subject fixation, eye movement, head tilt, subject age, or a combination of these factors may cause the subject's foveal center to shift relative to the geometric center of the resulting OCT volume. When the subject's fixation on the target and the foveal center do not overlap, poor fixation will occur. This poor fixation may be caused by, for example, neurological disease, eye tilt, subject age, etc. In some cases, due to the lack of training or experience of the clinician performing the OCT scan, there may be a foveal center misalignment. For example, compared with experienced OCT clinicians, medical students at teaching hospitals may be less successful in aligning the subject's foveal center and the OCT geometric center.

[0025] Currently, human scorers perform manual correction of the foveal center to prevent further problems when using the foveal center to obtain quantitative measurements and identify biomarkers. However, manual correction can be time-consuming, undesirable, and even infeasible in large datasets. Moreover, manual correction may not achieve the desired level of accuracy. For example, there may be significant variability between and within clinicians in subsequent realignment efforts. In addition, these manual-based methods for locating the foveal center rely on expert annotation of optical coherence tomography (OCT) B-scans, which can be resource-intensive and error-prone.

[0026] When an expert or manual grader attempts to detect the foveal center, the grader isolates and identifies the OCT B scan of the OCT volume that is most likely to show the foveal center, which is usually the most middle or central OCT B scan relative to the transverse axis (e.g., each OCT B scan of the OCT volume may be located at a different position on the transverse axis). On this selected OCT B scan, the grader manually looks for the fovea on the OCT B scan. For example, the grader can mark the lateral position of the foveal center and then infer the axial position of the foveal center. However, due to misalignment, the manually identified OCT B scan may not be the correct OCT B scan containing the foveal center. The grader must then view multiple OCT B scans around the central OCT B scan (relative to the transverse axis) to find the foveal center. Trying to find the foveal center in a large data set containing hundreds or thousands of OCT volumes may be difficult. In addition, in the OCT images capturing the retina, the presence of pathology (or abnormality) may change the expected appearance of the fovea, making the detection of the fovea challenging even for experienced human graders.

[0027] Some currently available methods for locating the foveal center use a two-dimensional machine learning model (e.g., a two-dimensional convolutional neural network) to automatically detect the foveal center. Such a model is iteratively trained using a single OCT B scan of the OCT volume. The OCT B scan is either the geometric center of the OCT volume or an OCT B scan selected by a human scorer to correct for misalignment. However, as described above, this correction may not be accurate. Therefore, training the model based on such an OCT B scan may result in a reduction in the accuracy of the model in detecting the foveal center. In addition, these currently existing two-dimensional machine learning models use pixel-by-pixel classification to detect the foveal center. Each pixel of the input OCT B scan is assigned a classification (or probability) indicating whether the pixel may be the foveal center. Therefore, these models require post-processing resources to convert the pixel-by-pixel classification of various OCT B scans for the OCT volume into simple coordinates for the foveal center. Therefore, these types of techniques may be undesirable in many scenarios.

[0028] Therefore, the embodiments described herein recognize that it is important to have improved methods and systems for accurately, precisely and reliably automatically detecting and locating the fovea center in an OCT volume (e.g., identifying coordinates for the fovea center). The embodiments described herein provide methods and systems for accurately, precisely and reliably automatically detecting and generating three-dimensional coordinates for the fovea center. In one or more embodiments, an optical coherence tomography (OCT) volume for the retina of a subject is received. The OCT volume includes multiple OCT B scans of the retina. A three-dimensional image input is generated for a model using the OCT volume. The model includes a three-dimensional convolutional neural network and a regression layer. The regression layer can be considered as part of the three-dimensional convolutional neural network or separate from the three-dimensional convolutional neural network. Using the model, a fovea center position is generated. The fovea center position includes a three-dimensional coordinate of the fovea center of the retina based on the three-dimensional image input. In other embodiments, the fovea center position includes two coordinates, such as, for example, a lateral coordinate and a lateral coordinate. This fovea center position for the fovea center is generated with a higher level of accuracy than a manually identified fovea center. Furthermore, this foveal center location may be more accurate and reliable than the foveal center identified based on pixel-by-pixel classification of OCT B-scans.

[0029] Using the foveal center location generated by the model, multiple outputs can be generated. For example, quantitative measurements can be automatically calculated from the OCT volume more accurately and reliably. Such quantitative measurements include, for example, but not limited to, central retinal thickness (CST) measurements and various retinal thickness measurements calculated using a retinal grid. The retinal grid can be, for example, but not limited to, an ETDRS grid.

[0030] In some embodiments, a model system is used that includes a method for automatically identifying a two-dimensional or three-dimensional foveal center location and additionally automatically identifying a plurality of quantitative measurements (e.g., CST measurements, one or more retinal thickness measurements for the retinal grid, etc.). The model system can also be used to automatically improve segmentation outputs, such as segmentation of various retinal layers or retinal features in an OCT volume (or in various OCT B-scans of the OCT volume). These segmentation outputs can have improved accuracy due to the more accurately identified foveal center location.

[0031] Thus, the embodiments described herein enable accurate and reliable foveal center localization in at least two dimensions. Because the embodiments described herein use a model to automatically and directly calculate and output the coordinates of the foveal center in the OCT volume without further processing, the amount of computing resources used can be reduced and the entire process can be made less labor intensive. Foveal center localization further enables the generation of more accurate and reliable quantitative measurements and segmentation outputs, which in turn can improve retinal disease screening, diagnosis, management, treatment selection, treatment response prediction, treatment management, or a combination thereof.

[0032] II. Example System for Foveal Center Detection

[0033] Figure 1 is a block diagram of an image processing system 100 according to one or more exemplary embodiments. The image processing system 100 may be used to process ophthalmic images to extract features from such images, correct or otherwise adjust one or more features extracted from such images, segment such images, generate one or more outputs related to diagnosis, screening and / or treatment of ophthalmic diseases, or a combination thereof.

[0034] Image processing system 100 includes analysis system 101. Analysis system 101 can be implemented using hardware, software, firmware or a combination thereof. In one or more embodiments, analysis system 101 may include computing platform 102, data storage 104 (e.g., database, server, storage module, cloud storage, etc.) and display system 106. Computing platform 102 can take various forms. In one or more embodiments, computing platform 102 includes a single computer (or computer system) or multiple computers communicating with each other. In other examples, computing platform 102 adopts a cloud computing platform, a mobile computing platform (e.g., a laptop computer, a smart phone, a tablet computer, etc.), another processor-based device (e.g., a workstation or desktop computer) or a wearable computing device (e.g., a smart watch), etc. or a combination thereof.

[0035] Data storage 104 and display system 106 are each in communication with computing platform 102. In some examples, data storage 104, display system 106, or both may be considered part of or otherwise integral to computing platform 102. Thus, in some examples, computing platform 102, data storage 104, and display system 106 may be separate components that communicate with each other, but in other examples, some combination of these components may be integrated together.

[0036] The computing platform 102 may be or may be a part of a client device, which is a processor-based device including, for example, a workstation, a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable device, or the like.

[0037] The image processing system 100 may further include an OCT imaging system 110, which may also be referred to as an OCT scanner. The OCT imaging system 110 may generate a spectral domain (SD) OCT image. The OCT imaging system 110 may generate OCT imaging data 108. The OCT imaging data 108 may include any number of three-dimensional, two-dimensional, or one-dimensional OCT images. The three-dimensional OCT image may be referred to as an OCT volume. The two-dimensional OCT image may take the form of, for example, but not limited to, an OCT B-scan.

[0038] In one or more embodiments, the OCT imaging data 108 includes an OCT volume 114 (e.g., an SD-OCT volume) for the subject's retina. The OCT volume 114 may be composed of a plurality of OCT B-scans 115 of the subject's retina. The plurality of OCT B-scans 115 may include, for example, but not limited to, tens, hundreds, thousands, tens of thousands, or some other number of OCT B-scans. An OCT B-scan may also be referred to as an OCT slice image or a cross-sectional OCT image.

[0039] In some embodiments, the retina is a healthy retina. In other embodiments, the retina is a retina that has been diagnosed with a retinal disease. For example, the diagnosis can be one of age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema, geographic atrophy, or some other type of retinal disease.

[0040] In one or more embodiments, the OCT imaging system 110 includes an optical coherence tomography (OCT) system (e.g., an OCT scanner or machine) configured to generate OCT imaging data 108 for tissue of a patient. For example, the OCT imaging system 110 can be used to generate OCT imaging data 108 for a retina of a patient. In some cases, the OCT imaging system 110 can be a large desktop configuration used in a clinical setting, a portable or handheld dedicated system, or a "smart" OCT system incorporated into a user's personal device (such as a smartphone). In some cases, the OCT imaging system 110 can include an image denoiser configured to remove noise and other artifacts from a raw OCT volume image to generate an OCT volume 114.

[0041] The analysis system 101 can communicate with the OCT imaging system 110 via a network 112. The network 112 can be implemented using a single network or a combination of multiple networks. The network 112 can be implemented using any number of wired communication links, wireless communication links, optical communication links, or a combination thereof. For example, in various embodiments, the network 112 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other appropriate types of networks. In another example, the network 112 may include a wireless telecommunications network (e.g., a cellular telephone network) suitable for communicating with other communication networks (such as the Internet). In some cases, the network 112 includes at least one of a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, or another type of network.

[0042] The OCT imaging system 110 and the analysis system 101 may each include one or more electronic processors, electronic memory, and other appropriate electronic components for executing instructions, such as program code and / or data stored on one or more computer-readable media, to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media such as memory or data storage devices (e.g., data storage 104) that are internal and / or external to various components of the image processing system 100 and / or accessible via the network 112.

[0043] Although only one of each of the OCT imaging system 110 and the analysis system 101 is shown, in other embodiments there may be more than one of each. Figure 1 The OCT imaging system 110 and the analysis system 101 are shown as two separate components, but in some embodiments, the OCT imaging system 110 and the analysis system 101 may be part of the same system (e.g., and maintained by the same entity, such as a healthcare provider or clinical trial manager). In some cases, a portion of the analysis system 101 may be implemented as a portion of the OCT imaging system 110. For example, the analysis system 101 may be configured to operate as a module implemented using a processor, microprocessor, or some other hardware component of the OCT imaging system 110. In yet other embodiments, the analysis system 101 may be implemented within a cloud computing system that may be accessed by or otherwise communicate with the OCT imaging system 110.

[0044] The analysis system 101 may include an image processor 116 configured to receive the OCT imaging data 108 from the OCT imaging system 110. The image processor 116 may be implemented using hardware, firmware, software, or a combination thereof. In one or more embodiments, the image processor 116 may be implemented within the computing platform 102. In some cases, at least a portion of the image processor 116 (e.g., its modules) is implemented within the OCT imaging system 110.

[0045] In one or more embodiments, the image processor 116 may generate a three-dimensional (3D) image input 118 using the OCT imaging data 108. For example, the OCT volume 114 may be preprocessed using a set of preprocessing operations to form the 3D image input 118. The set of preprocessing operations may include, for example, but not limited to, at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, a noise filtering operation, or some other type of preprocessing operation. The normalization operation may be performed to normalize the coordinates of a coordinate system for the OCT volume 114. In some cases, the pixel values ​​may be normalized (e.g., normalized to a value between 0-1). The scaling operation may include, for example, scaling a coordinate system associated with the OCT volume 114. The resizing operation may include changing the size of each of the plurality of OCT B-scans 115. The preprocessing operations in the set of preprocessing operations may be performed on one or more of the plurality of OCT B-scans 115 of the OCT volume 114.

[0046] The image processor 116 may include a model 120, which may also be referred to as a foveal center model. The model 120 is a deep learning model including a three-dimensional convolutional neural network (CNN) 122 and an output layer group 124. The three-dimensional CNN 122 may include a fully convolutional neural network. The output layer group 124 includes at least one regression layer. For example, using the 3D image input 118 as its input, the three-dimensional CNN 122 may output spatial information. The spatial information may take the form of an output image indicating the location of the center of the fovea. The regression layer is used to convert this output image into n-dimensional coordinates. The regression layer may be considered as part of the three-dimensional convolutional neural network or separate from the three-dimensional convolutional neural network.

[0047] In one or more embodiments, model 120 is a lightweight model. For example, 3D CNN 122 can be a lightweight convolutional neural network. A lightweight convolutional neural network is a network that reduces computational complexity. This reduction can be done in various ways, including, for example, by reducing or removing dense layers.

[0048] The model 120 is used to detect and generate a foveal center location 126 based on the 3D image input 118. The foveal center location 122 includes a three-dimensional coordinate 128 for the foveal center of the retina (i.e., three coordinates for three different axes of the coordinate system). In one or more embodiments, the three-dimensional coordinate 128 can be for a selected coordinate system corresponding to the OCT volume 114. For example, the three-dimensional coordinate 128 can include a transverse coordinate on a transverse axis, a lateral coordinate on a lateral axis, and an axial coordinate on an axial axis. The transverse axis can be an axis along which each of the plurality of OCT B scans 115 is scanned. The lateral axis and the axial axis can be axes for pixels of each of the plurality of OCT B scans 115. In some cases, each of the plurality of OCT B scans can be indexed by a transverse axis (e.g., having an index corresponding to a value on the transverse axis). The lateral axis can be, for example, a horizontal axis for each OCT B scan. The axial axis can be a vertical axis for each OCT B scan. In other embodiments, the foveal center location 126 includes two coordinates for the foveal center. For example, the foveal center location 126 may include a transverse coordinate and a lateral coordinate for the foveal center.

[0049] In some embodiments, the model 120 may include a layer or module for rounding a lateral coordinate in the three-dimensional coordinates 128 of the foveal center location 126 to a value that corresponds to an index associated with a particular OCT B scan in the plurality of OCT B scans 115. For example, an initial value for the lateral coordinate of the foveal center location 126 may be rounded to a rounded value that corresponds to an index associated with a particular OCT B scan in the plurality of OCT B scans 115. The rounded value forms one of the three-dimensional coordinates 128 of the foveal center location 126.

[0050] In one or more embodiments, the model 120 can be trained with a training data set 142 to generate the foveal center location 126. The training data set 142 includes a plurality of OCT volumes. One or more OCT volumes of the training data set 142 can capture a healthy retina. One or more OCT volumes of the training data set 142 can capture a retina diagnosed with a retinal disease (e.g., AMD, nAMD, diabetic retinopathy, macular edema, geographic atrophy, or some other type of retinal disease). An example of a method for training the model 120 (e.g., training the three-dimensional CNN 122) can be described below in Figure 3 Further described in .

[0051] The output generator 130 receives and processes the foveal center location 126 to generate an output 132 based on the foveal center location 126. The output 132 can take various forms. For example, the output 132 can include at least one of the following: a modified foveal center location 134; a central retinal thickness (CST) measurement 136; a retinal grid 138; a report 140 identifying any one or more of the foveal center location 126, the modified foveal center location 134, the central retinal thickness (CST) measurement 136, the retinal grid 138; or a combination thereof.

[0052] The output generator 130 can process the foveal center position 126 to generate a modified foveal center position 134, which can include, for example, coordinates for the foveal center that have been adjusted after the transformation. For example, the three-dimensional coordinates 128 can be transformed from a first selected coordinate system associated with the OCT volume 114 to a second selected coordinate system. The second selected coordinate system can be associated with, for example, a retina or a subject (e.g., an anatomical coordinate system). In some cases, the modified foveal center position 134 can be generated by rounding one or more of the three-dimensional coordinates 128 of the foveal center position 126 to an integer or a desired decimal place. In some cases, the lateral coordinates in the three-dimensional coordinates 128 are rounded to a value that corresponds to an index associated with a particular OCT B scan in the plurality of OCT B scans 115 to form the modified foveal center position 134.

[0053] The output generator 130 can process the foveal center location 126 to generate a CST measurement 136, which can be a measure or indication of foveal thickness. The central retina is a circular area with a diameter of 1 mm centered at the center of the fovea. The CST measurement 136 is a measurement of the thickness of the macula in the central retina. The thickness can be an average thickness (e.g., relative to a selected number of OCT B scans around the center of the fovea). The CST may also be referred to as central macular thickness or average macular thickness.

[0054] The output generator 130 can process the foveal center location 126 to determine (or otherwise define) a retinal grid 138. The retinal grid 138 is a grid that divides the retina into regions based on the three-dimensional coordinates 128 of the foveal center location 126. In one or more embodiments, the retinal grid 138 takes the form of a retinal thickness grid. Such a retinal thickness grid allows for quantitative measurements that may be important for screening for retinal disease, diagnosing retinal disease, monitoring disease progression, monitoring treatment response, selecting a treatment or treatment regimen, or a combination thereof. Such a retinal thickness grid allows for these quantitative measurements to be used as biomarkers.

[0055] As an example, the retinal grid 138 can be an Early Treatment Diabetic Retinopathy Study (ETDRS) grid. The grid divides the retina into nine regions centered on the center of the fovea-a central region, a group of four intermediate regions, and a center consisting of four outer regions. Specifically, the central region is defined as a volume within 1 mm of the diameter of the center of the fovea. This group of four intermediate regions is defined as four quadrants within a volume between the central region and the 3 mm diameter around the center of the fovea. This group of four outer regions is defined as four quadrants within a volume between this group of intermediate regions and the 6 mm diameter around the center of the fovea. The output generator 130 can use the accurately generated foveal center position 126 to accurately define the size of the ETDRS grid. The quantitative measurements generated using the accurately defined ETDRS grid can improve the ability to screen, diagnose, monitor disease progression, select treatments, predict treatment responses, manage treatments, or a combination thereof.

[0056] In one or more embodiments, the output generator 130 can generate a report 140 that includes any of the above outputs and / or other information. For example, the report 140 can include a rendition or modified version of a particular OCT B-scan that includes the foveal center and a graphical annotation or graphical label indicating the foveal center location 126. For example, the report 140 can modify the OCT B-scan by at least one of resizing, flipping (horizontally, vertically, or both), cropping, rotating, reducing noise, adding graphical features (e.g., adding one or more labels, colors, text, etc.), or other ways of modifying the OCT B-scan. In some embodiments, the foveal center location 126 and / or the modified foveal center location 134 can be identified on the rendition or modified OCT B-scan via text identifying the corresponding coordinates. In other cases, the foveal center location 126 and / or the modified foveal center location 134 can be identified by a pointer, a point, a mark, or some other type of graphical feature.

[0057] In one or more embodiments, the analysis system 101 stores the OCT imaging data 108 obtained from the OCT imaging system 110, the 3D image input 118, the foveal center position 126, the output 132, other data generated during the processing of the OCT imaging data 108, or a combination thereof in the data storage 104. In some embodiments, the portion of the data storage 104 storing such information can be configured to comply with the security requirements of the Health Insurance Portability and Accountability (HIPAA) that enforce certain security procedures when processing patient data (e.g., such as OCT images of patient tissue) (i.e., the data storage 104 can comply with HIPAA). For example, the stored information can be encrypted and anonymized. For example, the OCT volume 114 can be encrypted and processed to remove and / or obfuscate personal identifying information of the subject from which the OCT volume 114 was obtained. In some cases, the communication link between the OCT imaging system 110 and the analysis system 101 using the network 112 can also comply with HIPAA. For example, at least a portion of network 112 may be an end-to-end encrypted virtual private network (VPN) and configured to anonymize personally identifiable information data transmitted therein.

[0058] The image processing system 100 may include any number or combination of servers and / or software components that operate to perform various processes associated with the capture and processing of the OCT volume of the retina. Examples of servers may include, for example, stand-alone servers and enterprise-level servers. In one or more embodiments, the image processing system 100 may be operated and / or maintained by one or more different entities.

[0059] In some embodiments, the OCT imaging system 110 can be maintained by an entity that is assigned responsibility for obtaining OCT imaging data 108 for a tissue sample of a subject for disease screening, diagnosis, disease monitoring, disease treatment, research, clinical trial management, or a combination thereof. For example, the entity can be a healthcare provider (e.g., an ophthalmic healthcare provider) that seeks to obtain OCT imaging data 108 for a subject's retina for diagnosis of retinal disease and / or other types of eye conditions. As another example, the entity can be an administrator of a clinical trial that is assigned responsibility for collecting OCT imaging data 108 for a subject's retina to monitor retinal changes during a disease process, monitor treatment response, or both. The analysis system 101 can be maintained by the same or different entity (or entities) as the OCT imaging system 110. For example, the analysis system 101 can be maintained by an entity that is assigned responsibility for identifying or discovering biomarkers of retinal disease from OCT images.

[0060] III. Example Method for Foveal Center Localization

[0061] Figure 2 is a flow chart of a process for processing an OCT volume of a subject's retina according to one or more exemplary embodiments. Figure 2 In process 200 you can use Figure 1 In one or more embodiments, at least some steps of process 200 may be performed by a processor of a computer or server implemented as part of analysis system 101. It should be understood that additional steps may be performed before, during, or after the steps of process 200 discussed below. In addition, in some embodiments, one or more of these steps may also be omitted or performed in a different order.

[0062] Process 200 may optionally include a step 201 of training a model comprising a three-dimensional convolutional neural network (CNN). The model may be, for example, Figure 1 Model 120 in . A 3D CNN can be used, for example Figure 1 This is achieved by using the three-dimensional CNN 122 in

[14] .

[0063] Step 202 of process 200 includes receiving an optical coherence tomography (OCT) volume for a subject's retina, the OCT volume comprising a plurality of OCT B-scans of the retina. The OCT volume may be, for example, Figure 1 The OCT volume 114 in FIG. 1 may be a plurality of OCT B scans, for example Figure 1 115. Each of the multiple OCT B scans is a cross-sectional view of the retina taken at a specific position relative to a selected axis (which may be referred to as a transverse axis). Each OCT B scan may have a horizontal axis (lateral axis) and a vertical axis (axial axis).

[0064] The retina can be a healthy retina. Alternatively, the retina is a retina that has been diagnosed with or suspected of having a retinal disease. The retinal disease can be, for example, age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema, geographic atrophy, or some other type of retinal disease.

[0065] Step 204 of process 200 includes generating a three-dimensional image input for the model using the OCT volume. The 3D image input may be, for example, Figure 1 The 3D image input 118 in. The model can be, for example Figure 1The model 120 in . The model includes a three-dimensional convolutional neural network and a regression layer at the output of the three-dimensional convolutional neural network. The regression layer can be considered as part of the three-dimensional convolutional neural network or separate from the three-dimensional convolutional neural network. Step 204 can be performed in various ways. In one or more embodiments, generating the 3D image input includes performing a set of pre-processing operations on the OCT volume. The set of pre-processing operations may include, for example, at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, a noise filtering operation, or some other type of pre-processing operation.

[0066] Step 206 includes generating a foveal center location including three-dimensional coordinates for the foveal center of the retina based on the three-dimensional image input via the model. The foveal center location can be, for example, Figure 1 In one or more embodiments, the three-dimensional coordinates of the foveal center position 126 are composed of the three-dimensional coordinates 128 in the OCT volume. In one or more embodiments, the three-dimensional coordinates of the foveal center position include transverse coordinates, axial coordinates, and lateral coordinates relative to a selected coordinate system for the OCT volume. In some cases, the model generates three-dimensional coordinates with values ​​at least up to a selected decimal place. One or more of the three-dimensional coordinates can be rounded up or down to a selected decimal place. In some cases, one or more of the three-dimensional coordinates are rounded up or down to the nearest integer. In some embodiments, the initial value of the transverse coordinate for the foveal center position is rounded to a rounded value that corresponds to a rounded value of an index associated with a specific OCT B scan in a plurality of OCT B scans. The rounded value becomes the transverse coordinate for the foveal center position.

[0067] Process 200 may optionally include step 208. Step 208 includes generating an output using the foveal center location. Step 208 may be performed using one or more sub-steps performed simultaneously, sequentially, or in some other type of combination. The output may be, for example, Figure 1 Output 132 in. In one or more embodiments, the output includes a central retinal thickness measurement generated using the three-dimensional coordinates of the foveal center location. In some embodiments, the output includes a retinal grid that divides the retina into regions based on the three-dimensional coordinates of the foveal center location. The retinal grid can be, for example, an ETDRS grid that divides the retina into nine regions centered on the three-dimensional coordinates of the foveal center location.

[0068] In some embodiments, the output includes a modified foveal center position. For example, the foveal center position generated by the model in step 206 can be modified to form a modified foveal center position. In some cases, the three-dimensional coordinates of the foveal center position are transformed from a first selected coordinate system associated with the OCT volume to a second selected coordinate system associated with the retina or the subject (e.g., an anatomical coordinate system). In some cases, one or more of the three-dimensional coordinates are rounded up or down to a selected decimal place. In some cases, one or more of the three-dimensional coordinates are rounded up or down to the nearest integer. In an exemplary embodiment, the lateral coordinates in the three-dimensional coordinates of the foveal center position are rounded to a value that corresponds to an index associated with a particular OCT B scan in a plurality of OCT B scans of the retina.

[0069] In one or more embodiments, the output includes a transformation required to shift the geometric center of the OCT volume to the foveal center location. For example, the transformation can be a three-dimensional (or two-dimensional) shift that can be applied to the geometric center of the OCT volume to reach the foveal center location. The transformation can be applied to the foveal center identified by an OCT imaging system (e.g., OCT imaging system 110) that is located at the geometric center of the OCT volume so that the new foveal center will be correct when used for further analysis or processing.

[0070] In one or more embodiments, the output generated in optional step 208 includes a modified segmentation of the OCT volume. For example, the foveal center location can be used to modify a segmentation of at least one of the plurality of OCT B-scans of the retina based on the three-dimensional coordinates of the foveal center location. In some cases, a segmented image output (e.g., a mask image output) that can be two-dimensional or three-dimensional is modified based on the foveal center location.

[0071] In one or more embodiments, the output includes a report that includes any one or more of the above outputs. In some cases, the report includes a reproduction or modified version of a particular OCT B scan that includes the foveal center and a graphical annotation or graphical label indicating the location of the foveal center. For example, the report can modify the OCTB scan by at least one of: resizing, flipping (horizontally, vertically, or both), cropping, rotating, reducing noise, adding graphical features (e.g., adding one or more labels, colors, text, etc.), or other ways of modifying the OCT B scan. In some embodiments, the foveal center location and / or the modified foveal center location can be identified on the reproduced or modified OCT B scan via text identifying the corresponding coordinates. In other cases, the foveal center location and / or the modified foveal center location can be identified by a pointer, a point, a mark, or some other type of graphical feature.

[0072] Can be used Figure 1 The image processing system 100 described in or at least Figure 1 The process 200 implemented by the analysis system 101 in provides improvements for retinal disease screening, diagnosis, and treatment management. For example, by improving the accuracy, precision, and reliability of foveal center positioning, the process 200 thereby improves the accuracy, precision, and reliability of quantitative measurements taken based on the foveal center position, improves the identification of biomarkers and / or treatment responses for retinal disease, and improves the segmentation of OCT images. These improvements can be achieved regardless of the type of OCT imaging system used to generate the OCT volume, the type of scanning protocol used to generate the OCT volume, the quality of the OCT volume, or the type of destructive pathology or abnormality present in the retina. Therefore, the process 200 can facilitate improved automated analysis of large data sets of OCT volumes even in the presence of varying conditions associated with the OCT volume.

[0073] Figure 3 is a flow diagram of a process for training a model to generate a foveal center location in accordance with one or more embodiments. Figure 3 In process 300 you can use Figure 1 The process 300 may be implemented by the analysis system 101 in FIG. Figure 2 201 of the process 200 in the embodiment of the present invention. In addition, it should be understood that additional steps can be performed before, during, or after the steps of the process 300 discussed below. In addition, in some embodiments, one or more of these steps can also be omitted or performed in a different order.

[0074] Step 302 of process 300 includes receiving a training data set that includes a plurality of optical coherence tomography (OCT) volumes for a plurality of retinas. Each of the plurality of OCT volumes includes a plurality of OCT B-scans. The training data set may be, for example, Figure 1The training data set 142 in . The training data set may include OCT volumes for the retina in different health conditions. In one or more embodiments, the training data set may include one or more OCT volumes for a healthy retina. In one or more embodiments, the training data set may include OCT volumes for a retina diagnosed with a retinal disease (such as AMD, nAMD, diabetic retinopathy, macular edema, geographic atrophy, or some other type of retinal disease). In some cases, the training data may include one or more OCT volumes for a damaged retina. The training data set may include OCT volumes for the same type of retina (e.g., healthy or diseased or damaged) or different types of retinas. In some cases, multiple OCT volumes may be generated by more than one OCT imaging system (or type of OCT imaging system).

[0075] Step 304 includes generating a training three-dimensional image input for a model using a plurality of OCT volumes in a training data set, the model including a three-dimensional convolutional neural network and a regression layer. The regression layer may be considered as part of the three-dimensional convolutional neural network or separate from the three-dimensional convolutional neural network. For example, step 304 may include performing a set of preprocessing operations on the plurality of OCT volumes to form the training three-dimensional image input. The set of preprocessing operations may include at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, a noise filtering operation, or some other type of preprocessing operation.

[0076] Step 306 includes training the model to generate a foveal center location based on the training three-dimensional image input, the foveal center location including the three-dimensional coordinates of the center of the fovea for the retina in the selected OCT volume. The foveal center location may include a transverse coordinate, a lateral coordinate, and an axial coordinate for the center of the fovea in the OCT volume. The transverse coordinate may correspond to a specific OCT B-scan of the selected OCT volume. In some cases, the transverse coordinate may be a rounded value corresponding to the specific OCT B-scan. In other cases, the transverse coordinate may be a value that may be further processed and rounded up or down to correspond directly to the specific OCT B-scan.

[0077] The trained model formed after step 306 may be, for example, Figure 1 In addition, the trained model can be, for example, Figure 2 The trained model can be used to accurately and reliably locate the fovea center so that other quantitative measurements can be accurately and reliably calculated based on the fovea center location generated by the trained model. For example, the fovea center location can be used to generate outputs such as information about Figure 1Output 132 is described. The output may include CST measurements and / or various retinal thickness measurements associated with a retinal grid (e.g., an ETDRS grid). In one or more embodiments, the foveal center location may be used to improve retinal layer or fluid feature segmentation.

[0078] As described above, the training data set used to train the model can include various types of OCT volumes. In one or more embodiments, the training data set can include OCT volumes for all retinas diagnosed with a specific retinal disease (e.g., nAMD or diabetic macular edema). In some embodiments, the training data set includes an OCT volume segmented into a training OCT volume, a validation OCT volume, and a test OCT volume. In some embodiments, the OCT volume of the training data set is segmented into only a training OCT volume and a validation OCT volume.

[0079] IV. Example Workflow for Processing OCT Volumes

[0080] Figure 4 is an illustration of an example workflow for processing an OCT volume according to one or more exemplary embodiments. The OCT volume 400 may be Figure 1 1. The OCT volume 400 has an x, y, and z coordinate system, where the x-axis is the lateral axis, the y-axis is the y-axis, and the z-axis is the transverse axis. Each of the OCT B-scans that make up the OCT volume 400 falls on a different coordinate on the transverse axis, corresponds to a different coordinate on the transverse axis, or can be indexed by a different coordinate on the transverse axis. The OCT volume 400 can have a coordinate system similar to Figure 4 In other embodiments, some other type of grayscale or black and white may be used.

[0081] The OCT volume 400 may be preprocessed to form a modified OCT volume 402. The preprocessing may include, for example, but not limited to, at least one of normalization, scaling, resizing, a horizontal flip operation, a vertical flip, cropping, rotation, noise filtering, or some other type of preprocessing operation. In some examples, the modified OCT volume 402 is created to ensure that the input sent into the model 404 matches or is substantially similar in size, proportion, and / or orientation to the type of OCT volume on which the model 404 was trained.

[0082] Model 404 can be Figure 1 The model 404 includes a three-dimensional convolutional neural network 406 and a regression layer 408. The three-dimensional convolutional neural network 406 may be Figure 1 An example of a specific implementation of the three-dimensional convolutional neural network 122 in FIG. The regression layer 408 may be Figure 1 4. The regression layer 408 may be considered as a part of the three-dimensional convolutional neural network 406 or separate from the three-dimensional convolutional neural network 406. The three-dimensional convolutional neural network 406 of the model 404 may be a fully convolutional neural network including convolutional layers. In one or more embodiments, the model 404 includes one or more convolutional layers, one or more subsampling layers, and one or more fully connected layers.

[0083] In one or more embodiments, the hyperparameters of model 404 include a batch size of 1 and an epoch of 60. Model 404 may be implemented using an AdamW optimizer. In one or more embodiments, model 404 may be specifically designed so that the total number of parameters used in model 404 is less than about 500,000 parameters. Model 404 may be trained using a loss function. The loss function may be, for example, but not limited to, mean absolute error (MAE), mean square error (MSE), or some other type of loss function. Various metrics may be calculated during training and / or regular use of model 404. Such metrics include, but are not limited to, flat R 2 , uniform R 2 , variance-weighted R 2 , the original R 2 And / or other types of indicators. In some embodiments, the model 404 can be trained so that the output of the three-dimensional convolutional neural network 406 is normalized (for example, the pixel values ​​are normalized to values ​​between 0-1, etc.).

[0084] Model 404 processes modified OCT volume 402 to generate foveal center location 410. Foveal center location 410 includes three coordinates (x, y, and z coordinates for the foveal center). In other examples, foveal center location 410 includes two coordinates for the foveal center - x and z coordinates. Foveal center location 410 may be about Figure 1 An example of a specific implementation of the foveal center location 126 described and / or related Figure 2 Process 200, Figure 3 An example of a specific implementation of the foveal center location as described in process 300 or both.

[0085] In one or more embodiments, foveal center location 410 is used to generate CST measurement 412. The CST measurement may be about Figure 1 One example of a specific implementation of CST measurement 134 is described.

[0086] V. Example Computing System

[0087] Figure 5is a block diagram illustrating an example of a computing system according to one or more exemplary embodiments. The computing system 500 may be used to implement Figure 1 The computing platform 102 and / or any components therein.

[0088] like Figure 5 As shown, the computing system 500 may include a processor 510, a memory 520, a storage device 530, and an input / output device 540. The computing system 500 may be Figure 1 The processor 510, the memory 520, the storage device 530 and the input / output device 540 may be interconnected via a system bus 550. The processor 510 is capable of processing instructions for execution within the computing system 500. Such executed instructions may implement Figure 5 One or more components of the system, such as analysis engine 520, client device 530, etc. In some exemplary embodiments, processor 3810 may be a single-threaded processor. Alternatively, processor 510 may be a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 and / or on storage device 530 to display a user interface such as Figure 1 Graphic information of the display system 106 in.

[0089] Memory 520 is a computer-readable medium, such as a volatile or non-volatile computer-readable medium, that stores information within computing system 500. For example, memory 520 may store a data structure representing a configuration object database. Storage device 530 can provide persistent storage for computing system 500. Storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, a magnetic tape device, or other suitable persistent storage device. Storage device 530 may be Figure 1 The input / output device 540 provides input / output operations for the computing system 500. In some exemplary embodiments, the input / output device 540 includes a keyboard and / or a pointing device. In various implementations, the input / output device 540 includes a display unit for displaying a graphical user interface.

[0090] According to some exemplary embodiments, the input / output device 540 can provide input / output operations for network devices. For example, the input / output device 540 can include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0091] In some exemplary embodiments, the computing system 500 can be used to execute various interactive computer software applications that can be used to organize, analyze and / or store data in various formats. Alternatively, the computing system 500 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents and / or any other objects, etc.), computing functions, communication functions, etc. The application can include various additional functions or can be an independent computing product and / or function. After activation within the application, the function can be used to generate a user interface provided via the input / output device 540. The user interface can be generated by the computing system 500 and presented to the user (e.g., on a computer screen monitor, etc.).

[0092] One or more aspects or features of the subject matter described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features may include a specific implementation in one or more computer programs, which are executable and / or interpretable on a programmable system, which includes at least one programmable processor (which may be dedicated or general, coupled to receive data and instructions from it and send data and instructions to it), a storage system, at least one input device, and at least one output device. A programmable system or computing system may include a client and a server. Typically, the client and the server are remotely arranged from each other, and generally interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on each computer and a client-server relationship between each other.

[0093] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer product, device and / or equipment (such as, for example, disks, optical disks, memories and programmable logic devices (PLDs)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily (such as, for example, a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium). A machine-readable medium may store such machine instructions in a temporary manner (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores) alternatively or additionally.

[0094] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and a pointing device (such as, for example, a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive tracking pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0095] VI. Example Definition and Context

[0096] The present disclosure is not limited to these exemplary embodiments and applications, nor is it limited to the operation of the exemplary embodiments and applications or the methods described herein.In addition, the drawings may show simplified or partial views, and the sizes of elements in the drawings may be exaggerated or not proportional.

[0097] Unless otherwise defined, scientific and technical terms used in conjunction with the present teachings described herein shall have the meanings commonly understood by those of ordinary skill in the art. In addition, unless the context otherwise requires, singular terms shall include the plural and plural terms shall include the singular. In general, nomenclature and techniques used in conjunction with chemistry, biochemistry, molecular biology, pharmacology, and toxicology are described herein, and these nomenclatures and techniques are those well known and commonly used in the art.

[0098] As used herein, "substantially" means sufficient to achieve the intended purpose. Thus, the term "substantially" allows for minor, inconsequential variations relative to an absolute or ideal state, dimension, measurement, result, etc., as would be expected by one of ordinary skill in the art, but without significantly affecting the overall performance. When used relative to a numerical value or a parameter or characteristic that can be expressed as a numerical value, "substantially" means within ten percent.

[0099] As used herein, the term "about" used with respect to a numerical value or a parameter or feature that can be expressed as a numerical value means within ten percent of the numerical value. For example, "about 50" means a value within the range of 45 to 55, including the end values.

[0100] The term "ones" means more than one.

[0101] As used herein, the term "plurality" may be 2, 3, 4, 5, 6, 7, 8, 9, 10 or more.

[0102] As used herein, the term "a set" refers to one or more. For example, a set of items includes one or more items.

[0103] As used herein, the phrase "at least one of..." when used with a list of items means that different combinations of one or more of the listed items can be used, and only one item in the list may be required. An item can be a specific object, thing, step, operation, process, or category. In other words, "at least one of..." refers to any combination of items or number of items that can be used in the list, but not all items in the list are required. For example, but not limiting, "at least one of item A, item B, or item C" refers to item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and C. In some cases, "at least one of item A, item B, or item C" refers to, but is not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.

[0104] Where reference is made to a list of elements (eg, elements a, b, c), such reference is intended to include any one of the elements listed individually, any combination of less than all of the listed elements, and / or combinations of all of the listed elements.

[0105] The terms "one or more of A and / or B" and "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradictory to the context in which it is used, the phrase is intended to represent any element or feature listed individually, or any other described element or feature combined with any other described element or feature. For example, the phrases "one or more of A and B" and "A and / or B" are each intended to represent "A alone, B alone, or A and B together". Similar explanations apply to lists that include three or more items. For example, the phrases "one or more of A, B, and C" and "A, B, and / or C" are each intended to represent "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together".

[0106] The term “based on” is intended to mean “based at least in part on” such that unrecited features or elements are also permissible.

[0107] As used herein, a "model" includes at least one of an algorithm, a formula, a mathematical technique, a machine algorithm, a probability distribution or model, a model layer, a machine learning algorithm, or another type of mathematical or statistical representation.

[0108] The term "subject" may refer to a subject of a clinical trial, a human or animal being treated, a human or animal being treated for cancer, a human or animal being monitored for remission or recovery, a human or animal being monitored for preventive health (e.g., because of their medical history), or any other human or patient or animal of interest. In each case, "subject" and "patient" are used interchangeably herein.

[0109] The term "OCT image" may refer to an image of a tissue, organ, etc. (such as a retina) scanned or captured using optical coherence tomography (OCT) imaging technology. The term may refer to one or both of a 2D "slice" image and a 3D "volume" image. When not explicitly indicated, the term may be understood to include an OCT volume image.

[0110] As used herein, "machine learning" can include the practice of using algorithms to parse data, learn from it, and then make a determination or prediction about something in the world. Machine learning uses algorithms that can learn from data without relying on rule-based programming.

[0111] As used herein, "artificial neural network" or "neural network" may refer to a mathematical algorithm or computational model that simulates a set of interconnected artificial neurons that processes information based on a connectionist computational approach. A neural network (which may also be referred to as a neural network) may use one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input for the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. In various embodiments, references to "neural network" may be references to one or more neural networks.

[0112] Neural networks can process information in two ways, for example; when the neural network is being trained (e.g., using a training data set), the neural network is in training mode, and when the neural network puts what it has learned into practice (e.g., using a test data set), the neural network is in inference (or prediction) mode. Neural networks can learn through a feedback process (e.g., backpropagation) that allows the network to adjust the weight factors of various nodes in the intermediate hidden layers (modify their behavior) so that the output matches the output of the training data. In other words, a neural network can learn by being fed training data (learning examples) and ultimately learn how to obtain the correct output even if it is presented with a new range or set of inputs.

[0113] VII. Description of Exemplary Embodiments

[0114] Embodiment 1. A method comprising: receiving an optical coherence tomography (OCT) volume of a subject's retina, the optical coherence tomography (OCT) volume comprising multiple OCT B-scans of the retina; using the OCT volume to generate a three-dimensional image input for a model, the model comprising a three-dimensional convolutional neural network; and generating, via the model, a foveal center position comprising three-dimensional coordinates for the fovea center of the retina based on the three-dimensional image input.

[0115] Example 2. A method according to Example 1, wherein the three-dimensional coordinates of the foveal center position include transverse coordinates, axial coordinates and lateral coordinates relative to a selected coordinate system for the OCT volume.

[0116] Example 3. The method according to Example 2 or Example 3 also includes: using the three-dimensional coordinates of the foveal center position to generate a central retinal thickness measurement value.

[0117] Embodiment 4. The method according to any one of embodiments 1 to 3 further includes: determining a retinal grid that divides the retina into regions based on the three-dimensional coordinates of the central position of the fovea.

[0118] Embodiment 5. The method according to embodiment 4, wherein the retinal grid is an Early Treatment Diabetic Retinopathy Study (ETDRS) grid that divides the retina into nine areas centered on the three-dimensional coordinates of the central position of the fovea.

[0119] Example 6. The method according to any one of Examples 1 to 5 further includes: modifying the segmentation of at least one OCT B scan of the multiple OCT B scans of the retina based on the three-dimensional coordinates of the fovea center position.

[0120] Embodiment 7. A method according to any one of embodiments 1 to 6, wherein the model further comprises a regression layer, and the regression layer is used to convert the output of the three-dimensional convolutional neural network into the three-dimensional coordinates.

[0121] Embodiment 8. A method according to any one of embodiments 1 to 7, wherein generating the three-dimensional image input comprises: performing a set of preprocessing operations on the OCT volume to form the three-dimensional image input, the set of preprocessing operations comprising at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, or a noise filtering operation.

[0122] Example 9. The method according to any one of Examples 1 to 8 further includes: transforming the three-dimensional coordinates of the foveal center position from a first selected coordinate system associated with the OCT volume to a second selected coordinate system associated with the retina or the subject.

[0123] Embodiment 10. The method according to any one of embodiments 1 to 9, wherein the retina of the subject is a healthy retina.

[0124] Embodiment 11. A method according to any one of embodiments 1 to 9, wherein the retina of the subject is diagnosed with age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema or geographic atrophy.

[0125] Embodiment 12. A method according to any one of embodiments 1 to 11, wherein one of the three-dimensional coordinates of the foveal center position corresponds to a specific B-scan among the multiple OCT B-scans of the retina.

[0126] Example 13. The method according to any one of Examples 1 to 12 further includes: rounding the lateral coordinate in the three-dimensional coordinates of the foveal center position to a certain value, which corresponds to an index associated with a specific OCT B scan among the multiple OCT B scans of the retina.

[0127] Embodiment 14. A method according to any one of embodiments 1 to 12, wherein generating the foveal center position via the model including the three-dimensional convolutional neural network includes: rounding an initial value for a lateral coordinate of the foveal center position to a rounded value, the rounded value corresponding to an index associated with a specific OCT B scan among the multiple OCT B scans of the retina, wherein the rounded value is one of the three-dimensional coordinates of the foveal center position.

[0128] Embodiment 15. A method for training a model, the method comprising: receiving a training data set comprising multiple optical coherence tomography (OCT) volumes for multiple retinas, wherein each of the multiple OCT volumes comprises multiple OCT B-scans; using the multiple OCT volumes in the training data set to generate training three-dimensional image inputs for a model, the model comprising a three-dimensional convolutional neural network and a regression layer; and training the model to generate a foveal center position based on the training three-dimensional image input, the foveal center position comprising the three-dimensional coordinates of the foveal center of the retina in a selected OCT volume.

[0129] Example 16. The method of Example 15, wherein the foveal center location comprises a transverse coordinate, a lateral coordinate, and an axial coordinate for the foveal center in the OCT volume.

[0130] Example 17. A method according to Example 15 or Example 16, wherein generating the three-dimensional image input includes: performing a set of preprocessing operations on the multiple OCT volumes to form the training three-dimensional image input, the set of preprocessing operations including at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, or a noise filtering operation.

[0131] Embodiment 18. The method of any one of Embodiments 15 to 17, wherein the plurality of retinas includes at least one healthy retina.

[0132] Example 19. A method according to any one of Examples 15 to 18, wherein the multiple retinas include at least one retina diagnosed with a retinal disease, the retinal disease being age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema, or geographic atrophy.

[0133] Embodiment 20. A method according to any one of embodiments 15 to 19, wherein one of the three-dimensional coordinates of the foveal center position corresponds to a specific B-scan among the multiple OCT B-scans of the retina.

[0134] Embodiment 21. A system comprising: one or more data processors; and a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium contains instructions, which, when executed on the one or more data processors, cause the one or more data processors to: receive an optical coherence tomography (OCT) volume for a subject's retina, the optical coherence tomography (OCT) volume comprising multiple OCT B-scans of the retina; use the OCT volume to generate a three-dimensional image input for a model, the model comprising a three-dimensional convolutional neural network; and generate a foveal center position comprising three-dimensional coordinates for the fovea center of the retina based on the three-dimensional image input via the model.

[0135] Embodiment 22. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to: receive an optical coherence tomography (OCT) volume of a subject's retina, the optical coherence tomography (OCT) volume comprising multiple OCTB scans of the retina; generate a three-dimensional image input for a model using the OCT volume, the model comprising a three-dimensional convolutional neural network; and generate, via the model, a foveal center location comprising three-dimensional coordinates for a foveal center of the retina based on the three-dimensional image input.

[0136] Embodiment 23. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform the method according to any one or more of embodiments 1 to 20.

[0137] Embodiment 24. A system comprising: one or more data processors; and a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium contains instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform any one or more of the methods described in Embodiments 1 to 20.

[0138] VIII. Other considerations

[0139] Although the present teaching is described in conjunction with various embodiments, the present teaching is not intended to be limited to such embodiments. On the contrary, the present teaching encompasses various alternatives, modifications and equivalents that will be understood by those skilled in the art.

[0140] In describing various embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, if the method or process does not rely on the specific order of steps described herein, the method or process should not be limited to the specific sequence of steps listed, and those skilled in the art can readily understand that these sequences can be different and still remain within the spirit and scope of the various embodiments.

[0141] In addition, depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The specific implementations set forth in the description herein do not represent all specific implementations consistent with the described subject matter. Instead, they are only some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to those features and / or variations set forth herein, other features and / or variations may also be provided. For example, the specific implementations described above may be for various combinations and sub-combinations of the disclosed features and / or for combinations and sub-combinations of several further features disclosed above. In addition, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other specific implementations may be within the scope of the following claims.

Claims

1. A method comprising: receiving an optical coherence tomography (OCT) volume of a retina of a subject, the optical coherence tomography (OCT) volume comprising a plurality of OCT B-scans of the retina; generating a three-dimensional image input for a model using the OCT volume, the model comprising a three-dimensional convolutional neural network; as well as A foveal center location including three-dimensional coordinates for a foveal center of the retina is generated based on the three-dimensional image input via the model.

2. The method of claim 1, wherein the three-dimensional coordinates of the foveal center position include transverse, axial, and lateral coordinates relative to a selected coordinate system for the OCT volume.

3. The method according to claim 2 or claim 3, further comprising: A central retinal thickness measurement is generated using the three-dimensional coordinates of the foveal center location.

4. The method according to any one of claims 1 to 3, further comprising: A retinal grid that divides the retina into regions is determined based on the three-dimensional coordinates of the foveal center position.

5. The method of claim 4, wherein the retinal grid is an Early Treatment Diabetic Retinopathy Study (ETDRS) grid that divides the retina into nine areas centered around the three-dimensional coordinates of the foveal center position.

6. The method according to any one of claims 1 to 5, further comprising: Segmentation of at least one OCT B scan of the plurality of OCT B scans of the retina is modified based on the three-dimensional coordinates of the foveal center location.

7. The method according to any one of claims 1 to 6, wherein the model further comprises a regression layer, which is used to convert the output of the three-dimensional convolutional neural network into the three-dimensional coordinates.

8. The method according to any one of claims 1 to 7, wherein generating the three-dimensional image input comprises: A set of pre-processing operations are performed on the OCT volume to form the three-dimensional image input, the set of pre-processing operations including at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, or a noise filtering operation.

9. The method according to any one of claims 1 to 8, further comprising: The three-dimensional coordinates of the foveal center location are transformed from a first selected coordinate system associated with the OCT volume to a second selected coordinate system associated with the retina or the subject.

10. The method according to any one of claims 1 to 9, wherein the retina of the subject is a healthy retina.

11. according to the method described in any one of claims 1 to 9, the described retina of wherein said experimenter is diagnosed with age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema or geographic atrophy.

12. The method of any one of claims 1 to 11, wherein one of the three-dimensional coordinates of the foveal center position corresponds to a specific B-scan of the plurality of OCT B-scans of the retina.

13. The method according to any one of claims 1 to 12, further comprising: A lateral coordinate in the three-dimensional coordinates of the foveal center location is rounded to a value corresponding to an index associated with a particular OCT B scan in the plurality of OCT B scans of the retina.

14. The method of any one of claims 1 to 12, wherein generating the foveal center location via the model including the three-dimensional convolutional neural network comprises: An initial value for the lateral coordinate of the foveal center position is rounded to a rounded value corresponding to an index associated with a particular OCT B scan of the plurality of OCT B scans of the retina, wherein the rounded value is one of the three-dimensional coordinates of the foveal center position.

15. A method for training a model, the method comprising: receiving a training dataset comprising a plurality of optical coherence tomography (OCT) volumes for a plurality of retinas, wherein each of the plurality of OCT volumes comprises a plurality of OCTB scans; generating a training three-dimensional image input for a model using the plurality of OCT volumes in the training dataset, the model comprising a three-dimensional convolutional neural network and a regression layer; as well as The model is trained to generate a foveal center location comprising three-dimensional coordinates of the foveal center of the retina in a selected OCT volume based on the training three-dimensional image input.

16. The method of claim 15, wherein the foveal center location comprises a transverse coordinate, a lateral coordinate, and an axial coordinate for the foveal center in the OCT volume.

17. The method according to claim 15 or claim 16, wherein generating the training 3D image input comprises: A set of preprocessing operations are performed on the multiple OCT volumes to form the training three-dimensional image input, and the set of preprocessing operations includes at least one of a normalization operation, a scaling operation, a resizing operation, a horizontal flip operation, a vertical flip operation, a cropping operation, a rotation operation, or a noise filtering operation.

18. The method of any one of claims 15 to 17, wherein the plurality of retinas includes at least one healthy retina.

19. The method of any one of claims 15 to 18, wherein the plurality of retinas include at least one retina diagnosed with a retinal disease, the retinal disease being age-related macular degeneration (AMD), neovascular age-related macular degeneration (nAMD), diabetic retinopathy, macular edema, or geographic atrophy.

20. The method of any one of claims 15 to 19, wherein one of the three-dimensional coordinates of the foveal center position corresponds to a specific B-scan of the plurality of OCT B-scans of the retina.

21. A system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to: receiving an optical coherence tomography (OCT) volume of a retina of a subject, the optical coherence tomography (OCT) volume comprising a plurality of OCT B-scans of the retina; generating a three-dimensional image input for a model using the OCT volume, the model comprising a three-dimensional convolutional neural network; as well as A foveal center location including three-dimensional coordinates for a foveal center of the retina is generated based on the three-dimensional image input via the model.

22. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to: receiving an optical coherence tomography (OCT) volume of a retina of a subject, the optical coherence tomography (OCT) volume comprising a plurality of OCT B-scans of the retina; generating a three-dimensional image input for a model using the OCT volume, the model comprising a three-dimensional convolutional neural network; and A foveal center location including three-dimensional coordinates for a foveal center of the retina is generated based on the three-dimensional image input via the model.

Images