Systems and methods for segmentation of images for optical coherence tomography angiography

Abstract

Systems and methods for segmenting an image provided by OCTA using a trained model are provided. Methods include processing a plurality of paired training image sets using a first neural network to separate, from each of the paired training image sets, a respective anatomy component from a respective contrast component of a respective image subject, and processing second training images using a second neural network to generate, from the second training images, a representation of a respective delineated image. The first neural network can include a conditioning network for receiving one of the paired training images, and an encoder a decoder for receiving and processing the other of the paired training images. The second neural network can include the conditioning network, configured to receive the second training images, and a segmenter configured to produce a representation of the delineated image.

Description

SYSTEMS AND METHODS FOR SEGMENTATION OF IMAGES FOR OPTICAL COHERENCE TOMOGRAPHY ANGIOGRAPHYCross-Reference to Related Applications

[0001] This application claims the priority and benefit of U.S. Provisional Application No. 63 / 514,901, filed on July 21, 2023, which is hereby incorporated by reference in its entirety.Statement Regarding Federally Sponsored Research or Development

[0002] This invention was made with Government support under grant no. NS082347 awarded by the National Institute of Neurological Disorders and Stroke / NIH / DHHS, and grant no. EY032284 awarded by the National Institutes of Health / NIH / DHHS. The Government has certain rights in the invention.DESCRIPTIONField

[0003] This disclosure relates to systems and methods directed to segmentation of images for optical coherence tomography angiography.Background

[0004] Optical coherence tomography (OCT) angiography (OCTA) is a noninvasive imaging modality with applications in various retinal and neurological diseases. It provides detailed visualizations of the retina’s vascular structure [l]-[7] and foveal avascular zone (FAZ) [8], [9], Although OCTA data are acquired as volumes, for a variety of reasons including the presence of so-called projection artifacts

[0010] ,

[0011] , it is common to project selected slabs into en face images for analyzing macular OCTA images [5]-[7], The resulting two standard OCTA en face images depict the superficial vascular plexus (SVP), which incorporates the ganglion cell layer (or lies within the ganglion cell layer and nerve fiber layers), and the deep vascular plexus (DVP), which incorporates the inner nuclear layer

[0012] -

[0015] .

[0005] Quantitative studies of en face OCTA images usually require segmentation of the retinal vessels and capillaries. Different methods for segmenting OCTA images have been explored in recent years

[0016] —

[0024] . In particular, supervised trained deep learning methods have drawn a lot of attention, because of their success in many related applications. However,manually segmenting capillaries in OCTA images is extremely time-consuming and obtaining a sufficient number of such pixel level annotations for supervised training of deep networks is impractical, especially when it must be done for different scanners. From our experience, for example, manually delineating l / 64th the area of one scan and verification with repeated scans requires at least 30 minutes. Therefore, we estimate that 40 hours is needed to delineate an entire scan and complete an independent review by a second manual rater. Thus, manually delineating 39 scans (the number of scans in the ROSE-1 dataset

[0025] ) would take more than 1,500 hours. Moreover, for many scans identifying capillaries is impossible due to the limited image resolution and the presence of noise and artifacts. For this reason, existing publicly available datasets mainly focus on large vessels

[0023] with limited capillary delineation

[0025] , An example scan 100 with its manual delineation 110 from the ROSE dataset

[0025] is given in FIG. 1. The Optovue SVP scan 100 is shown at the bottom of FIG. 1, with its pixel -level annotation 101 shown at the top of FIG. 1 For each image (100 and 101), a zoomed-in view for the region inside the gray box is shown in the bottom right comer (image 105 and 110, respectively). Although, we can see numerous capillaries in the zoomed-in view of the image 105, most of them are not included in the manual delineation 110.SUMMARY

[0006] While vessel and capillary delineations require a great deal of manual labor and time to acquire as described above, it is relatively easy to acquire multiple unlabeled OCTA images of the same subject’s eye from the same or different scanners [8],

[0018] ,

[0019] ,

[0021] ,

[0026] , Consistent with this disclosure, we consider those repeated scans as paired data. Although there are inherent scanner differences, those repeated scans from the same eye should have similar anatomy but different artifacts and are corrupted by independent noise. When examining such paired scans, we can identify vessels and capillaries from their shared intensity pattern; conversely, inconsistent patterns are usually associated with noise and artifacts. For example, FIGS. 2A-C show two Optovue scans (FIGS. 2A and 2B) and one Heidelberg scan (FIG. 2C) from the same eye. The boxed portion of the three scans (box 200, box 220, and box 240, respectively, for FIGS. 2A, 2B, and 2C), also shown zoomed up (image 205, image 225, and image 245, respectively), cover the same region of the macula. That is, in the upper left comer of each of FIGS. 2A, 2B, and 2C, a zoomed view is shown (image 205, image 225, and image 245, respectively). A manual tracing for some capillaries are provided in the upper right comer of each of FIGS. 2 A, 2B, and 2C (image 210, image230, and image 250, respectively); not every recognizable capillary is highlighted for clarity in the figures. The three scans are not registered.

[0007] Although different noise and artifacts are present, we can readily identify very similar capillary structures in the zoomed-up views; some of these are highlighted for clarity. Although we could not practically require such multiple scans to be available for each future subject, consistent with this disclosure, one can take advantage of such paired images, available for many past subjects, to help train a segmentation algorithm that requires only one scan as input and tends to ignore artifacts, noise, and contrast variations.

[0008] Embodiments consistent with the present disclosure include methods and systems for forming a trained model. In an embodiment, one or more processors or systems perform operations including receiving a plurality of paired training image sets, each paired training image set in the plurality of paired training image sets including at least a first image and a second image, each of the first image and the second image associated with an image subject of a plurality of image subjects and each image subject comprising an anatomy component and a contrast component. Consistent with an aspect, operations can further include processing the plurality of paired training image sets using a first neural network, with a plurality of first training weights, to separate, from each of the plurality of paired training image sets, the respective anatomy component from the respective contrast component of the respective image subject. Furthermore, in an aspect, operations can include receiving a plurality of second training images, each second training image in the plurality of second training images having a respective delineated image of a plurality of delineated images, and processing the plurality of second training images using a second neural network, with a plurality of second training weights, to generate, from each of the plurality of second training images, a representation of the respective delineated image. In an aspect, the first neural network can be a conditional variational autoencoder including a conditioning network configured to receive the at least first image, an encoder configured to receive the at least second image, and a decoder configured to generate a representation of the at least second image such that, when the first neural network is trained, the conditioning network extracts the respective contrast component from the at least second image. Moreover, in an aspect, the second neural network can include the conditioning network, configured to receive each of the plurality of second training images, and a segmenter configured to produce the representation of the respective delineated image. Consistent with an aspect of an embodiment, when the first neural network and the second neural network are trained, thefirst set of training weights and the second set of training weights can be fixed, thereby producing the trained model.

[0009] In a further aspect, a method and system can include the above embodiments where the image subject is an eye, and the respective anatomy component is a vascular structure of the eye. In another aspect, a method and system can include the above embodiments where the conditioning network uses a U-Net architecture or a CS-Net architecture. Further still, in another aspect, a method and system can include the above embodiments where each of the encoder and the decoder comprise a plurality of max-pooling layers.

[0010] In another aspect, methods and systems for segmenting an image generated by optical coherence tomography angiography can include forming the trained model according to any of the previous embodiments, and processing the image using the conditioning network of the second neural network to produce a segmented representation of the image based upon said trained model.

[0011] Additional features and embodiments of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments and together with the description, serve to explain the principles of the disclosure. In the figures:

[0013] FIG. 1 depicts exemplary scans and pixel-level annotation from the ROSE dataset

[0025] ;

[0014] FIGS. 2A-C depict exemplary scans and manual tracings of capillaries;

[0015] FIGS. 3 and 4 depict an overview of the training process consistent with this disclosure (FIG. 3(a)), detailed architecture of the encoder and decoder (FIG. 4), an example of input (FIG. 3(b)), and segmentation results (FIG. 3(c));

[0016] FIG. 5 depict exemplary inputs and outputs consistent with this disclosure;

[0017] FIGS. 6A and 6B depict a Table providing a comparison of segmentation results, and also depicts box plots of the area under the ROC curve (AUC) and Dice coefficient (DSC) for the segmentation results produced by CS-Net

[0029] , U-net

[0027] , and ACRROSS(U- net) when trained using different amounts of training data;

[0018] FIG. 7 depicts exemplary test results from the OCTA-500 dataset when training with a different number of samples;

[0019] FIGS. 8, 9A, and 9B depict results of the ablation study using OCTA-500 dataset (FIG. 8), and the results of the reproducibility test for CS-Net, U-net, and ACRROSS when trained with different amounts of training data from the OCTA-500 dataset (FIGS. 9A and 9B);

[0020] FIG. 10 depicts a Table providing a further comparison of segmentation results;

[0021] FIG. 11 provides exemplary results of models consistent with this disclosure; and

[0022] FIG. 12 depict examples of FAZ segmentations for a Topcon scan fromOCTAGON dataset

[0046] and an Optovue scan from OCTA-500 dataset

[0024] consistent with this disclosure.DESCRIPTION OF THE EMBODIMENTS

[0023] Methods consistent with this disclosure, named Artifacts and Contrast Robust Representation for OCTA Semi-supervised Segmentation (ACRROSS), disentangle the anatomy and contrast in OCTA images for accurate segmentation of vessels and capillaries. ACRROSS is trained using two datasets: one dataset with unlabeled registered paired OCTA images and the other with a very limited set of manual labels. ACRROSS learns to disentangle an OCTA image into separate contrast and anatomy components by identifying shared structures in the paired OCTA images. While learning to do this, it also learns to segment capillaries and vessels from the anatomy component using a limited set of manual delineations. In experiments, we first use two publicly available datasets of OCTA images, ROSE

[0025] and OCTA-500

[0023] ; the manual delineations in both datasets focused on large vessels with limited capillary delineations. We show that ACRROSS trained with only patches of manual delinations (total area used was less than a single scan) can achieve comparable or even better accuracy than the comparison methods that trained on the entire dataset. We also built an in-house dataset with capillary level delineations and show quantitatively that detailed capillary segmentation can be achieved without a large amount ofmanual delineations. Since FAZ segmentation is closely related to the segmentation of capillaries, we also show that based on our segmentation results, a simple post-processing approach can segment the FAZ with close to state-of-the-art performance.

[0024] II. RELATED WORK

[0025] Several works have explored the use of deep learning methods for OCTA vessel segmentation. A U-net

[0027] architecture was used by Morgan et al.

[0028] for vessel and FAZ segmentation in SVP images from two scanners. Mou et al.

[0022] ,

[0029] proposed an attention module for vessel segmentation and applied it to OCTA images. Pissas et al.

[0030] proposed an iterative approach for 8 * 8 mm SVP scans. Li et al.

[0023] ,

[0024] and

[0031] proposed to directly output 2D vessel maps and FAZ segmentations from 3D OCTA images. Hu et al.

[0032] investigated segmenting 3D vessels from the 3D OCTA volumes; Yu et al.

[0033] proposed a method for segmenting vessels from 2D OCTA images and estimated the depth information for the segmented vessels to facilitate 3D vessel analysis. All of the previous supervised methods require a significant amount of training data with corresponding manual delineations. More recently, Xu et al.

[0034] proposed a partially-supervised method that used 3% to 5% of the training data compared with other supervised methods. In contrast, our method needs less than 2% of manually delineated scans used by supervised methods to achieve similar performance. Unlike our previous work

[0021] , which uses a dedicated encoder and decoder structure for each scanner, the majority of the network weights in ACRROSS are shared across different scanners. Thus, ACRROSS requires less computational resources and can be easily extended to multiple scanners without additional computational overhead.

[0026] The design of a semi -supervised method usually depends on the availability of weak labels. For brain lesion segmentation, unsupervised image translation between healthy and disease subjects was used

[0035] ,

[0036] , Zhou et al.

[0037] used disease severity grading to learn lesion attention maps for semi-supervised segmentation. Semi-supervised learning based on disentanglement has been previously used for both classification and segmentation. Robert et al.

[0038] designed an autoencoder structure with two encoders to separate a class specific component and a complementary component. The two components were combined by a decoder to reconstruct the unlabeled input image, and classification is learned from the class-specific component. For segmenting cardiac cine magnetic resonance (MR) images, a similar structure was used by Chartsias et al.

[0039] , where the outputs of two encoders are interpreted as spatially and non-spatially dependent components. In both methods, a selfreconstruction loss is used for learning the disentanglement. However, self-reconstruction and segmentation may play contradictory roles in feature extraction

[0038] , This is crucial for segmentation tasks, because a self-reconstruction loss reinforces the noise and allows artifacts to be learned as part of the anatomy representation. Although our method also reconstructs the encoder’s input image, we avoid this problem by learning the disentanglement from paired images, which have different noise and artifacts.

[0027] Several methods for disentangled representation learning use autoencoders

[0040] , Zhang et al.

[0041] used the encoder to learn a label -irrelevant spatial component along with a non-spatial embedding code conditioning. Dewey et al.

[0042] and related works

[0043] ,

[0044] proposed an encoder-decoder network structure for disentangling MR image modality and anatomy. Although they did not explicitly use a conditioning network, the modality vector learned from the encoder feeds into their decoder as a condition, and therefore, their method can be interpreted as a modified conditional variational autoencoder (CVAE). In fact, the two-encoder structure used in Chartsias et al.

[0039] is also a CVAE, where the spatial component branch serves as the conditioning. ACRROSS is different from the above methods in two important aspects. First, all three methods in

[0039] ,

[0042] , and

[0041] seek to disentangle a subject specific anatomy component that is spatially dependent from a scanner specific contrast component that is non-spatially dependent. Because of the spatially-varying contrast in OCTA images, we chose to disentangle two spatially dependent components. This allows us to model the contrast and artifacts regionally. Second, in

[0041] and

[0042] the anatomical component is the latent representation of the CVAE, which requires sampling during training. On the other hand, we have chosen the anatomical component to be the conditioning. With this choice, we avoid independently sampling the anatomy representation at each pixel location, which produces less noisy anatomy representations and consequently is beneficial for the segmentation task.

[0028] III. METHOD

[0029] A. Overview

[0030] FIG. 3(a) depicts a model 300, consistent with this disclosure, which includes four networks: an encoder 310, a decoder 320, a conditioning network 305 (CN 305), and a segmenter 325. The CN 305 and the segmenter 325 branch performs the segmentation. During supervised training, a cross-entropy loss computed between the output of the segmenter 325 and manual delineations can be back-propagated to both the CN 305 andsegmenter 325. However, when the number of training data samples is small, supervised training can lead to unsatisfactory results during test time. This is especially true for OCTA images, where different artifacts and contrast variations in unseen images cannot be fully covered in the training dataset. To address this problem, we introduced two additional networks: an encoder 310 and a decoder 320. The encoder 310, decoder 320, and CN 305 together form a CVAE that uses a second dataset of unlabeled, registered, paired OCTA scans for training. With the additional components, the CN 305 learns to extract a contrast- disentangled anatomy representation for the segmenter 325 to use. As a preview, a test time example result 390 from input 380, which uses only the CN 305 and the segmenter 325 to generate, is shown in FIGS. 3(b) and 3(c). In particular, FIG. 3(a) depicts an overview of the training process of the method consistent with this disclosure. The dimension of each variable is provided, with H and IFbeing the height and width of the original scan. The gray and dashed line paths indicate CVAE and supervised training flow, respectively, with the CN 305 being shared by both. As a preview, an example of test time input 380 and segmentation result 390 are provided in FIGS. 3(b) and 3(c) respectively. The model 300 was trained using the XJU-CVAE and the XJU-MD subsets (see Sec. IV below for complete details).Methodological details of an approach consistent with this disclosure are provided in the following sections.

[0031] B. Conditional variational autoencoder

[0032] Considerand x^as an unlabeled pair of registered OCTA scans acquired from the same eye. These images could be from the same or different scanners, but to avoid anatomical changes they should be acquired within a few days of each other. The CVAE framework assumes that the input x to encoder 310 can be reconstructed by the decoder 320 from a latent representation z given the conditioning c. Specifically for ACRROSS, an OCTA scan x4us reconstructed from zA^ given the conditioning c (see FIG. 3, for example). The superscript indicates that zAs learned from x^but cAs extracted from its paired scan xAIn theory, the encoder 310 and the decoder 320 approximate the posterior and likelihood distributions q zA^\xA^;cA^) and p xA^\zA^;cA^ with a parametric encoder q<> and a parametric decoder pe, respectively. Similar to a variational autoencoder (VAE)

[0045] , a CVAE can be trained using the negative variational lower bound given by

[0034] where p(z ) is assumed to be a multivariate Gaussian. The first term in (Eq. 1) is the expectation of the log posterior, which has a similar effect as a mean squared error loss that encourages reconstruction of x41. The second term in (Eq. 1) is the Kullback-Leibler divergence between the two distributions and can be thought of as a regularization term acting on the learned latent representation z.

[0035] The architectures of the encoder 310 and decoder 320 consistent with this disclosure are depicted in FIG. 4. All convolutional layers used in this work have a kernel size of 3x3 and padding of size one. The encoder 310 with four max-pooling layers compresses x^by a factor of 16 in each spatial dimension. The outputs of the encoder 310 are interpreted as the mean uA< and standard deviation <74iof a multivariate Gaussian that characterizes the distribution of the latent representation. Samples (zA1,s) of this distribution are generated by following

[0045] for training the decoder 320. The number of channels in uA' and o41is a hyper-parameter that determines the compression rate of the encoder-decoder branch. Consistent with this disclosure, 10 channels are used, but we found the result to be similar when using between 4 and 32 channels. Accordingly, each 16 x 16 block from the input is represented by a vector of length 10.

[0036] C. Anatomy-contrast disentanglement

[0037] A CVAE is generally considered to be a generative model where diverse samples for a particular class can be generated from the decoder by sampling the learned latent representation and input the samples to the decoder with the class label as conditioning. In contrast, ACRROSS uses a CVAE to learn the disentanglement of an anatomy component and a contrast component from paired scans. For an OCTA image, the anatomy component captures the vessels and capillaries whereas the contrast component captures the representation of the vessels and capillaries that makes those anatomy distinguishable from noise or background. This is achieved by the novel training strategy consistent with this disclosure: instead of manually assigning a conditioning, ACRROSS uses a feed-forward CN to learn a conditioning from42; thus, the decoder 320 uses two sources of information — zA' from the encoder 310 and c42from the CN 305 — to reconstruct x41Similar to the common autoencoder structure with a bottleneck, z4' will be a lossy compressed representation of x41given its limited capacity. With the use of cAwe can guide the encoder 310 to focus on compressing local contrast information.

[0038] Consistent with this disclosure, the conditioning variable c42is designed to have the same spatial dimension as x41. Since z4' can only encode limited information from x41, if some of the information is also contained in cAthen the full capacity of zA:can be used for information specific to x41. When we use registered paired OCTA images as input to the encoder 310 and CN 305, respectively, then this information comprises the local contrast, noise, and artifacts of x41. The encoder 310 can confidently ignore the vascular structures in producing z4' because the decoder 320 can expect that information to come from cAwhich has a much larger capacity. Therefore, the encoder 310 and CN 305 learn to cooperate for a better reconstruction of x41. In particular, the CN 305 learns to extract vessels and capillaries that x42shares with x4iso that the encoder 310 can focus on the local contrast, artifacts, and noise that are only accessible from x41. Otherwise, the redundant information in z41further limits its representation power, which would result in a higher reconstruction loss.

[0039] Importantly, the architecture is not designed such that the conditioning information is learned from x4'. If it were configured this way, then the CN 305 and decoder 320 would simply learn an identity mapping to perfectly reconstruct x41without needing z41. Methods with such an alternative design require constraints to achieve disentanglement, e.g., forced binarization for the conditioning

[0042] or an additional cycle-consistency loss

[0039] ,

[0040] Once trained, the CN 305 can extract the vascular structures that are shared between x4iand x42, but in fact it predicts the intensity patterns from x42that are useful to reconstruct x41without actually observing x41. This is beneficial at test time because the CN 305 can remove the contrast, noise, and artifacts from x42that are irrelevant for reconstructing x4i without the need of a paired scan as input to the encoder.

[0041] D. Semi-supervised segmentation

[0042] The representation learned by the CN 305 greatly reduces the contrast variations from OCTA images caused by either the OCTA algorithms or contrast-related artifacts. This allows the use of a segmenter 325 network with just two convolutional layers to segment vessels from the conditioning. For training, we use a dataset of OCTA images with manual delineations. An image xBgoes through the segmentation path (the CN 305 and segmenter 325) to produce pB, as shown in FIG. 3(a), and the cross entropy loss LCE IS computed between pBand the manual delineation. This supervised training procedure injects our preference into the segmentation model, e.g., the thickness of the vessels to be segmented orthe minimum intensity to be considered as foreground. As shown in Sec. IV below, different manual delineations lead to different results, but the same CVAE training procedure is used.

[0043] During each forward pass, both LCVAE and LCE are calculated using the two datasets, then the combination of the two losses is back-propagated to update the parameters in all subnetworks. The LCVAE has effects on the encoder 310, decoder 320, and CN 305, whereas LCE has effects on the segmenter 325 and CN 305. After training, only the CN 305 and the segmenter 325 — i.e. no paired images — are needed for segmentation of an OCTA image. Unlike other sub-networks in the proposed method, the CN 305 is the most flexible component. Because it is used for generating the conditioning, the only requirement for the CN 305 is to preserve the spatial dimension of the input. Therefore, any previously proposed dense prediction network can be used for the CN 305. In our experiments, we report the results of two versions of the proposed method using two previously proposed network structures for our CN 305: U-Net

[0027] and CS-Net

[0029] ,

[0044] IV. EXPERIMENTS

[0045] A. Datasets

[0046] The performance of vessel segmentation was evaluated on two publicly available datasets, OCTA-500

[0024] and ROSE

[0025] , both with manual delineation of vessels, and a proprietary dataset, XJU

[0018] , All training and evaluations were carried out on OCTA scans representing the superficial vascular plexus (SVP). We constructed a subset of unlabeled registered paired OCTA scans from the XJU dataset for CVAE training; this subset is termed XJU-CVAE. Taking advantage of the paired data for reliable delineation of capillaries, we built a subset of manually delineated scans from the XJU dataset; this subset is termed XJU- MD. We also use the manual delineations for FAZ from both the OCTA-500 and the OCTAGON

[0046] datasets to demonstrate how our trained vessel segmentation model can be used for FAZ segmentation. The details of each dataset are provided below.

[0047] XJU: Scans from Angiovue (RTVue XR Avanti, Optovue, Inc. Fremont, CA), Angioplex (Cirrus HD-5000, Zeiss Meditec. Dublin, CA), Triton (Topcon DRI OCT Triton, Topcon, Japan) and Spectralis OCT2 module (Heidelberg Engineering, Germany) were included

[0018] , Each eye was scanned twice on the Topcon, Zeiss, and Optovue scanners and once on the Heidelberg scanner. For each eye, all seven scans were registered to a designated Optovue scan by a deformable transformation using ANTs

[0047] , The registered scans weremanually reviewed and 138 out of 146 eyes were found to be successfully registered. The 8 failure cases were caused by major artifacts or field of view differences between the scans.

[0048] XJU-CVAE: From the 138 successfully registered scans in the XJU dataset, we randomly selected 110 eyes for CVAE training in ACRROSS. Each training sample is a pair of different OCTA images randomly selected from the seven repeats.

[0049] XJU-MD: From the remaining 28 successfully registered eyes in the XJU dataset, we randomly selected four eyes (22 scans) for manual delineation. Each scan was divided into patches of size l / 64thof the image and a set of such patches were randomly selected for delineation. To improve the delineation quality in regions with noise and artifacts, potential capillaries were verified by comparing with its repeated scans in the same location. All delineations were reviewed by a second person and corrected if necessary. In total 48 patches were delineated (13 Heidelberg, 13 Optovue, 11 Topcon, 11 Zeiss), an example of which can be seen in FIG. 11. The total area of these 48 patches is approximately equal to 3 / 4 the area of a single scan. Despite the relatively small total area finally delineated, the overall taskincluding initial labeling, verification with repeat scans, and independent review-was extraordinarily time consuming; ultimately taking eight weeks to complete with rater fatigue also being a handicap.

[0050] FIG. 11 depicts exemplary results of model 300 (trained on XJU-MD) on scans from XJU-MD (scan 1102, results 1112 and 1122), OCTA-500 (scan 1104, results 1114 and 1124), and ROSE-1 (scan 1106, results 1116 and 1126). Comparison with manual results (images 1132, 1134, and 1136, respectively) are also depicted. The supervised training does not use any examples from OCTA-500 or ROSE-1. Model 300, consistent with this disclosure, can produce detailed capillary segmentation for scans from OCTA-500 and ROSE-1 that is previously not available in the OCTA-500 and ROSE-1 manual delineations.

[0051] OCTA-500: All 200 subjects (No. 10301— No. 10500) with 3 mm x 3 mm SVP scans from the OCTA-500 dataset

[0024] are included in our experiments. The data were collected using a commercial 70 kHz SD-OCT (RTVue-XR, Optovue, CA). We use the maximum projection map between internal limiting membrane (ILM) and outer plexiform layer (OPL) because it was used for vessel delineations. We followed the same training, validation, and testing split as in

[0024] (No. 10301 — 10440 for training; No. 10441 — 10450 for validation; and No. 10451 — 10500 for testing). For each scan, a manual delineation of FAZ is also provided.

[0052] ROSE-1: All 39 scans in the ROSE-1 subset of the ROSE dataset

[0025] are included in our experiments. For each subject, we used the 3 mmx3 mm SVP scans and their corresponding manual delineations. The ROSE-2 subset is not included, because it only contains centerline-level annotations of vessels. All 39 scans were acquired on a RTVue XR Avanti SD-OCT system (Optovue, USA). As specified in

[0025] , 30 selected scans were used for training and 9 were used for testing.

[0053] OCTAGON: OCTAGON

[0046] is a publicly available dataset for FAZ segmentation; it includes 55 SVP 3 mmx3 mm scans acquired from a Topcon device (DRI OCT Triton) and their manual FAZ segmentations.

[0054] B. Illustration of disentanglement

[0055] FIG. 5 depicts some additional results for the same eye as in FIG. 3 produced by ACRROSS. In addition to the Optovue scan from FIG. 3 (scan 380), we also show its registered paired Heidelberg scan 505. To produce each contrast-removed conditioning result (cBin FIG. 3(a)), an original scan is provided as input (xBin FIG. 3(a)) to the CN 305. With the contrast largely removed, we can easily see the capillary structure in this intermediate result 510 and 515. The segmentation result for each scan (390 and 525) is produced from the conditioning (510 and 515, respectively) using the segmenter 325 followed by binarization at a 0.5 threshold. Because the CN 305 is robust to contrast variations, the loss-of-signal- strength artifact that can be seen in the Heidelberg scan 505 has minimum impact on the conditioning. This allows us to observe a very similar vascular pattern in the segmentation results 525 even in the artifact-affected region.

[0056] Specifically, FIG. 5 depicts exemplary inputs and outputs from ACRROSS. The two rows represent the processing of two different scanner manufacturers: Optovue and Heidelberg. From right to left, the columns are the original input OCTA images (380 and 505), the conditioning (510 and 515), the corresponding segmentation (390 and 525), and the reconstruction of the input (530 and 535). The two scans are registered. The XJU-MD subset is used for supervised training.

[0057] Although typically not used after training is complete, it is instructive to see a reconstructed image (V'in FIG. 3(a)) from a registered paired set of scans. The reconstructed image shown in the top left of FIG. 5 (result 530) is generated using the Optovue scan as41and the Heidelberg scan as xAi(i.e. the contrast component from the Optovue scan and the anatomy component from the Heidelberg scan.). The reconstructed image shown in thebottom left of FIG. 5 (result 535) uses the opposite assignment. It is clear from these two reconstructed images that artifacts are encoded in the variable z'4', which comes from the image .r4'. This visualization confirms the effect of disentanglement and reinforces our contention that ACRROSS segmentation results are not greatly affected by artifacts.

[0058] C. Metrics for Vessel Segmentation

[0059] To evaluate the performance of vessel segmentation algorithms, the following metrics are calculated between the manual delineation and the segmentation results produced by each algorithm:

[0060] Area under the ROC curve: AUC;

[0061] Accuracy: ACC = (TP + TN) / (TP + TN + FP + FN);

[0062] Kappa score: KAPPA = (ACC - pe) / (l - pe);

[0063] False discovery rate: FDR =FP / (FP + TP);

[0064] G-mean score: GMEAN =V(sensitivity x specificity);

[0065] Dice coefficient: DSC = 2 x TP / (FP + FN + 2 x TP),

[0066] where TP, TN, FP, FN represent the True Positives, True Negatives, False Positives, and False Negatives, respectively, and pe= ((TP+FN)(TP+FP)+(TN+FP)(TN+FN)) / (TP+ TN + FP + FN)2. Sensitivity and specificity are computed as TP / (TP + FN) and TN / (TN + FP), respectively. These metrics are also reported in

[0025] , All the / ?-values reported were computed using a paired, two-sided Wilcoxon signed rank test (null hypothesis: the difference between paired values comes from a distribution with zero median).

[0067] D. Implementation details

[0068] Model 300 was implemented using Pytorch, and all networks were trained using the Adam optimizer with a learning rate of 4 x 10~4and weight decay of 1 x 10~6. When there is a corresponding validation dataset available, the training terminates when the validation loss stops decreasing; otherwise the number of training epochs was determined empirically. Specifically, the OCTA-500 dataset has its own validation dataset, and for our semi -supervised ROSE-1 experiment the unused images serve as the validation dataset. Our other experiments use an empirically determined number of training epochs. During CVAE training, the CN 305 may take scans from different manufacturers as input. To handle thecontrast difference, the inputs of CN 305 are processed by four convolutional layers (with LeakyReLU activation), which learn a different set of weights for each scanner. It has been shown previously that such dedicated sub-networks can improve network generalizability

[0050] , The source code for this work is currently proprietary while under review for potential commercialization.

[0069] In all experiments, ACRROSS used the XJU-CVAE for CVAE training (batch size of 8) and a dataset with manual delineations (OCTA-500, ROSE-1, or XJU-MD) for supervised training (batch size of 2). Because each loss term is calculated separately before the combined loss is backpropagated, scanners used in CVAE training are not required to be used for supervised training. For example, ACRROSS can be trained using XJU-CVAE with scans from four scanners together with OCTA-500, which only has Optovue scans. For each of our three datasets, the test time procedure for new unseen images is the same. This procedure is depicted as the “dashed line” portion of the network in FIG. 3(a) (including CN 305 and segmenter 325). First, the unseen image, xB, is passed through the condition network CN 305 to generate the conditioning, cB. The conditioning is then passed to the segmentation network (segmenter 325) to generate, pB, which is then binarized to generate a segmentation (threshold at 0.5 intensity value). All the comparison methods were concatenated with the same segmenter network as in ACRROSS and trained using the same settings except for those methods that cannot use the unlabeled XJU-CVAE subset.

[0070] E. Supervised vessel segmentation on OCTA-500

[0071] We first used all 140 scans from the OCTA-500 training set to test the performance of a method consistent with this disclosure in a fully supervised setting. We provided two versions of ACRROSS using different network architectures as the conditioning network: ACRROSS(CS-Net) used CS-Net and ACRROSS(Unet) used U-net. For the comparison methods, we included U-net

[0027] , nnU-Net

[0049] , R2U-Net

[0048] and CS- Net

[0029] , For training U-net, R2U-Net, and CS-Net, we used the same hyper-parameters as our methods for fair comparison. nnUNet was originally designed for 3D images with the ability to automatically determine its hyper-parameters, we followed an example provided by the authors to make it work for 2D images. The results on the 50 test scans are summarized in Table I, depicted in FIGS. 6A and 6B. Specifically, Table I provides a comparison of the vessel segmentation results for 50 superficial vascular plexus en face test scans (mean ± std.dev.) from the octa-500 dataset, using all training images (140 scans) for supervised training.Bolded numbers in Table I indicate the best mean value.

[0072] Consistent with this disclosure, the methods (ACRROSS(CS-Net) and ACRROSS(Unet)) produce comparable or better results when measured by AUC, and ACRROSS(U-net) is significantly better than all comparison methods in terms of DSC (p- value < 0.001).

[0073] F. Semi-supervised vessel segmentation on OCTA-500

[0074] To test the semi-supervised setting when there are fewer training samples, we decreased the number of training data used for supervised training from N = 140 subjects to N = 20 and N = 4 in two additional experiments. Further reduction in the number of training samples may cause a high variance in repeated experiments where training samples with different quality and variety are selected. To address this problem, we divided the total area of each scan into 8 ^ 8 square patches and treated one patch instead of one scan as a training sample. This is implemented by only computing the cross-entropy loss inside the selected patches. However, we still input the entire image into the network to make sure the normalization layers work properly. We first randomly select 4 scans, and then 32, 16, and 8 patches are randomly selected from each of the 4 scan as training data. As a result, the total number of patches used in these three additional experiments are P = 128, P = 64, and P = 32, equivalent to N = 2, N = 1, and N = 0.5 subjects in terms of total area that is used. For example, the total area for P = 32 is (32 / (8 x 8)) subjects, which is equivalent to half a scan.

[0075] To further reduce the randomness in the result, each experiment was run three times with different random seeds and the same set of random seeds were used for all methods. We evaluated the performance of U-net

[0027] , CS-Net

[0029] , and ACRROSS(U-net) on the 50 test scans. The results of the three repeats are combined (50 x 3 data points) and reported in (a) and (b) of FIG. 6A. We can see that for AUC and DSC, ACRROSS consistently produces better results across all sets of experiments with different amounts of training data. This can also be seen from the example shown in FIG. 7, where only large vessels were segmented because the manual delineations for OCTA-500 do not include capillaries. When measured by AUC, the proposed method ACRROSS (U-net) trained with 32 patches produces comparable results to the U-net trained with 20 subjects ( -value = 0.995) and CS-Net trained with 140 subjects ( -value = 0.324). It is significantly better than CS-Net trained with 20 subjects ( -value < 0.001). We also observed that for all methods, N= 4 underperforms P = 128, although the latter case uses less area during training. This may be analogous to training using 2D slices outperforming training using 3D volumes, when the number of subjects is small

[0051] ,

[0076] In addition to Table I, FIGS. 6A and 6B depict box plots of the area under the ROC curve (AUC) and Dice coefficient (DSC) for the segmentation results produced by CS- Net

[0029] , U-net

[0027] , and ACRROSS(U-net) when trained using different amounts of training data. The amount of training data used is indicated on the horizontal axes. Each plot is divided into two parts, with the left part showing the results when trained using patches as training samples and the right part showing the results when trained using scans as training samples. The total area used for training in P =32, P =64, and P =128 are equal to A =0.5, N =1, and N=2 subjects, respectively. For training with scans, the equivalent number of patches are also shown. The results for the OCTA-500 dataset are shown in (a) and (b) of FIG. 6A and the results for the ROSE-1 dataset are shown in (c) and (d) of FIG. 6B.

[0077] FIG. 7 depicts examples of test result from the OCTA-500 dataset when training with a different number of samples. The left-most column shows the original image (505) and the zoomed-in region (515), along with its corresponding manual delineation (525). The predicted vessel probability maps of CS-Net

[0029] (row 510), U-net

[0027] (row 520), and ACRROSS(U-net) (row 530) are shown in the right columns. The variables P and N are explained below.

[0078] G. Ablation study

[0079] Ablation studies were conducted on the OCTA-500 dataset. ACRROSS was first trained without the Kullback-Leibler divergence loss in Eq. 1. The removal of this regularization reduced the CVAE to an autoencoder structure (AE). We also investigate the impact of using dedicated sub-networks as input layers for CN 305 (see Sec. IV-D below). In FIG. 8, we compared the DSC of these two methods against the original ACRROSS, where we denote the version without the use of dedicated sub-networks as “w / o DA”. In all experiments, the models were trained in the same way as described in Sec. IV-F. The results show that both the Kullback-Leibler divergence loss in Eq. 1 and the dedicated sub-networks improve the segmentation results, especially in the semi-supervised setting.

[0080] H. Reproducibility test

[0081] We test the reproducibility of ACRROSS under different contrasts in supervised and semi-supervised settings. The training follows the strategy described in Sec. IV-F. Thetrained models were then applied to the XJU-MD dataset, in which seven repeated scans from four scanners were captured for each eye (see Sec. IV-A). Because all manually delineated scans in the OCTA-500 dataset come from Optovue scanners, we separately compared the segmentation results between the two Optovue scans (i.e. intra-Optovue) and the segmentation results between the first Optovue scan and other contrast scans (i.e. inter- Optovue), which include scans from Heidelberg, Topcon and Zeiss. We calculated the DSC and reported the averaged numbers in FIGS. 9 A (inter-Optovue) and FIG. 9B (intra- Optovue). The results from U-net and CS-Net were also included for reference. Since all the scans were registered, if the segmentations are consistent we would anticipate a higher DSC; which means better reproducibility of the algorithm across the various scanners. It can be seen from FIG. 9B that the intra-Optovue experiments generally have a better consistency compared with the inter-Optovue experiments, shown in FIG. 9A. This is expected because supervised training only include Optovue scans. For CS-Net and U-Net, we observed decreasing consistency — lower DSC scores — of their results as we reduce the amount of training data. In contrast, the results produced by ACRROSS are not affected by the amount of supervised training data.

[0082] Specifically, FIGS. 9 A and 9B depict the results of the reproducibility test for CS- Net, U-net, and ACRROSS when trained with different amounts of training data from the OCTA-500 dataset. The DSC between the two Optovue scans are shown in FIG. 9B (intra- Optovue) and the DSC between the first Optovue scan and scans from other scanners are shown on in FIG. 9A (Inter-Optovue).

[0083] I. Supervised vessel segmentation on ROSE-1

[0084] The ROSE-1 subset provides delineations for large vessels as well as some capillaries. We used the provided training and testing split where all 30 training samples in the dataset were used for supervised training and test results were computed on the 9 held-out scans. We observe that our method produces comparable or better results in terms of AUC (see Table II in FIG. 10) and, when measured by the DSC, ACRROSS (U-net) results are significantly better than all comparison methods ( - value < 0.005). Table II (shown in FIG. 10), provides a comparison of the vessel segmentation results for superficial vascular plexus en face scans (mean ± std. dev.), using all thirty scans from ROSE-1 training set for supervised training. Bolded numbers in Table II indicate the best mean value.

[0085] J. Semi-supervised vessel segmentation on ROSE-1

[0086] To test the semi-supervised setting where there are insufficient training samples, we randomly selected 4 subjects for training and, to further reduce the training data, we used the same patch sampling technique as in our OCTA-500 experiments. Specifically, 4 subjects were randomly selected and then a total of P = 32, P = 64, and P = 128 patches were randomly selected from the 4 subjects (8, 16, and 32 patches from each) so that the total areas used in training are equivalent to N = 0.5, N = 1, and N = 2 subjects. The performances were evaluated on the same 9 held-out scans. We compared ACRROSS results with results from U-net and CS-Net that were trained under the same setting. Each method was trained three times with different random seeds and the same set of random seeds were used for all methods. The results of the three repeats were combined (9 x 3 data points) and averaged performances were reported in (c) and (d) of FIG. 6B. In addition, we compared with three semi -supervised methods in Table III, including: 1) Mean teacher (MT), a consistency -based semi -supervised learning approach

[0052] modified for the segmentation task

[0034] ; 2) MixMatch, a data augmentation based semi-supervised method

[0053] ; and 3) PSL, a recently proposed patch-based semi-supervised method that combined MixMatch and active learning

[0034] , These results were originally reported in

[0034] , For each method, the area of manually delineated data used for training relative to the total area of all scans in the training dataset was also reported in Table III. When using 3.3% of the manual delineation, our method outperformed MT

[0052] and MixMatch

[0053] and was comparable to PSL

[0034] , The minimum amount of training data PSL tested on was 3.3%, however, ACRROSS achieves comparable results with only 1.7% of the training data and without involving the iterative training and labeling process in PSL.

[0087] Table III

[0088] K. Semi-supervised vessel segmentation on XJU-MD

[0089] Although ROSE-1 has 39 subjects with manual delineation, in many cases the delineation does not align well with the true capillaries, as shown in FIGS. 1 and 11. Our experiments using the OCTA-500 and ROSE-1 datasets show that high accuracy results can be achieved with the proposed method using very few manual delineations for training. For accurate segmentation of OCTA images at the capillary level, the proposed method was trained on an in-house dataset with manual delineations (XJU-MD).

[0090] Given the limited number of manually labeled patches (48 patches), we used 47 patches (73.4% the area of one scan) for supervised training. We provide the qualitative result of the remaining patch in FIG. 11. To avoid potential data leakage issue, none of the other patches from this eye were used for training. Since XJU-MD contains scans from Heidelberg, Optovue, Topcon, and Zeiss, we were also able to apply the trained model on scans from both the OCTA-500 and ROSE-1 datasets without using any scans or manual delineations from those datasets during training. Qualitative results on scans from OCTA-500 and XJU-MD are shown in the two additional rows in FIG. 11. Despite the site differences between the training (XJU-MD) and testing (OCTA-500 and ROSE-1) data, ACRROSS is able to provide detailed capillary segmentation that is previously not available in the manual delineation for OCTA-500 and ROSE-1.

[0091] L. FAZ segmentation

[0092] The FAZ is the avascular region around the fovea. If the capillaries are accurately detected, a simple post-processing algorithms can be used to provide an accurate segmentation of the FAZ. Accordingly, we applied a morphological closing operation to our vessel segmentation result and found the FAZ as the largest connected component in the background (see FIG. 12). We compared our FAZ segmentation results to two FAZ segmentation methods

[0046] ,

[0054] , Despite the simplicity of our post-processing steps, we achieved similar results in terms of Jaccard Index (see Table IV, which provides a comparison of the mean Jaccard Indexes for 3 mm x 3 mm SVP scans (mean ± std. dev.).The previous methods have not reported their standard deviations. , using all thirty scans from ROSE-1 training set for supervised training. Bolded numbers in Table II indicate the best mean value. Bolded numbers indicate the best mean value in that row.

[0093] Table IV

[0094] FIG. 12 provides examples of FAZ segmentations for a Topcon scan from OCTAGON dataset

[0046] (image 1102) and an Optovue scan from OCTA-500 dataset

[0024] (image 1112). The corresponding vessel segmentations (1104 and 1114, respectively) and FAZ segmentations (1106 and 1116, respectively) are shown in the middle and right column.

[0095] The OCTA-500 dataset also contains manual delineations of the FAZ that we could compare to. Our post-processing based FAZ segmentation has a mean DSC of 0.954 ± 0.025 and a Jaccard Index of 0.912 ± 0.044, which is close to the performance of several supervised trained deep learning methods reported in

[0024] , Note that our training used the manually delineated patches from the XJU-MD, without any examples or FAZ masks from OCTAGON or OCTA-500. This result suggests that our vessel segmentation model can accurately detect the capillaries around the FAZ. We note that the current post-processing method is not suitable for many disease cases as non-perfusion areas can be falsely detected as the FAZ by our use of the largest connected component. However, it demonstrates the potential for achieving accurate FAZ segmentation in healthy controls without the need of extra manual delineations of the FAZ for supervised training.

[0096] V. DISCUSSION AND CONCLUSION

[0097] We disclose a deep network architecture called ACRROSS for disentangling local contrast and vascular structures from en face OCTA images so that retinal vessel and capillary segmentation can be learned with limited manual delineations.

[0098] ACRROSS is closely related to our previously reported method

[0021] called VICCE. The CVAE training in ACRROSS can be considered as the cross-scanner synthesis in VICCE but applied to the CN and decoder. Also, both methods can be interpreted as special cases of unsupervised representation learning

[0055] where similar anatomyrepresentations are extracted from the paired scans. Without the negative samples that are commonly used in unsupervised representation learning, VICCE forces the representation learned from one scan to be able to synthesize its paired scan in order to avoid the model collapse problem. This, however, implicitly assumes that the underlying anatomical information is identical in both scans, which is generally not true since there are inherent hardware and software differences between different scanners. Alternatively, ACRROSS uses the extra input (z) during CVAE training to encode the scan-specific information, which includes the scanspecific anatomy. For example, a layer segmentation error can cause more vessels to be included in the SVP projection map, but this will not affect the segmentation result because those extra vessels are considered scan-specific and encoded in z. In contrast, scanner-specific anatomy is encoded in c, because it is shared by repeated scans from the same scanner.

[0099] Generally, supervised training with diverse data is a preferred way to learn the variability in a real data distribution. By learning the disentanglement from the unlabeled data, ACRROSS reduces the requirement of diverse training samples in labeled data. In particular, we found when using the OCTA-500 dataset that training using 128 patches selected from 4 scans (with 32 patches per scan) is comparable to using 128 patches selected from 32 scans (with 4 patches per scan). The learned disentanglement also shows the potential for transferring the knowledge of manual delineations from one dataset to another. Our method trained using the OCTA-500 dataset and the XJU-CVAE subset can segment Topcon scans from the OCTAGON dataset without using any manual delineations from Topcon scans. Notice that this is different from the concept of domain adaption because the CVAE training uses unlabeled Topcon scans to learn the disentangled representation.

[0100] Despite these advantages, ACRROSS and similar disentangled representation learning methods are limited to segmentation tasks where all structures are labeled. Otherwise, extra manual delineations are needed to separate the labeled structures from the unlabeled structures. Essentially, the labeled and unlabeled structures determined by the manual delineation become entangled components in the conditioning. In such tasks

[0039] , the segmentation cannot take full advantage of the disentangling. Another limitation of our method is the over-segmentation problem, in which some noise is falsely recognized as capillaries. This is observed when applying our model trained on healthy subjects to disease cases. This phenomenon is related to the artifact-affected scans in the XJU-CVAE dataset. As paired scans are unlikely to be affected by the same imaging artifact, the CVAE trainingallows artifacts like loss-of-signal-strength to be disentangled from the anatomy representation and recover the capillaries in the affected area. This is undesirable, however, when the subject experiences a true loss of capillaries. Since the training dataset only consists of healthy subjects, ACRROSS may interpret missing capillaries in disease cases as an artifact. This problem can likely be solved by including paired disease cases in the CVAE training or excluding regions affected by artifacts in healthy subjects, though further investigation is needed.

[0101] Our experiments are limited to 3 mm x 3 mm SVP scans because we find it difficult to acquire consistent and reliable manual segmentations for the DVP; also 3D OCTA data with manual delineations are currently unavailable. In practice, acquiring repeated scans from one scanner is more common than repeat scans from different scanners. Therefore, we experimented with training ACRROSS using only Optovue scans. In this case, ACRROSS still outperforms the comparison methods, but including scans from another scanner can significantly improve the results. Whether the inclusion of multiple scanners helps disentangled representation learning or is simply a result of the fact that more scans prevent over-fitting is a subject for future research.

[0102] References cited herein are included in the Appendix.

[0103] Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the embodiment disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the enumerated claims.APPENDIXREFERENCES[1] M. Parravano et al. Appearance of cysts and capillary non perfusion areas in diabetic macular edema using two different OCTA devices. Scientific Reports, 10(1): 1-9, 2020.[2] M. DTez-Sotelo et al. A novel automatic method to estimate visual acuity and analyze the retinal vasculature in retinal vein occlusion using swept source optical coherence tomography angiography. Journal of Clinical Medicine, 8(10): 1515, 2019.[3] O. C. Murphy et al. Alterations in the retinal vasculature occur in multiple sclerosis and exhibit novel correlations with disability and visual function measures. Multiple Sclerosis Journal, 26(7): 815-828, 2020.[4] J. Lei et al. Distinctive analysis of macular superficial capillaries and large vessels using optical coherence tomographic angiography in healthy and diabetic eyes. Invest. Ophthalmol. Vis. Sci., 59(5): 1937-1943, 2018.[5] T. S. Hwang et al. Automated quantification of capillary nonperfusion using optical coherence tomography angiography in diabetic retinopathy. JAMA Ophthalmology, 134(4):367-373, 2016.[6] P. L. Nesper et al. Quantifying microvascular abnormalities with increasing severity of diabetic retinopathy using optical coherence tomography angiography. Invest. Ophthalmol. Vis. Sci., 58(6):BIO307- BIO315, 2017.[7] A. C. Onishi et al. Importance of considering the middle capillary plexus on oct angiography in diabetic retinopathy. Invest. Ophthalmol. Vis. Sci., 59(5):2167-2176, 2018.[8] A. Lin et al. Reliability of foveal avascular zone metrics automatically measured by cirrus optical coherence tomography angiography in healthy subjects. International Ophthalmology, 40(3):763-773, 2020.[9] C. Balaratnasingam et al. Visual acuity is correlated with the area of the foveal avascular zone in diabetic retinopathy and retinal vein occlusion. Ophthalmology, 123(11):2352-2367, 2016.

[0010] Y. Liu et al. Projection artifact suppression for inner retina in OCT angiography. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 592- 596. IEEE, 2019.

[0011] M. Zhang et al. Projection-resolved optical coherence tomographic angiography. Biomed. Opt. Express, 7(3): 816-828, 2016.

[0012] D. M. Snodderly et al. Neural -vascular relationships in central retina of macaque monkeys (macaca fascicularis). Journal of Neuroscience, 12(4): 1169-1193, 1992.

[0013] J. M. Provis. Development of the primate retinal vasculature. Progress in Retinal and Eye Research, 20(6): 799-821, 2001.

[0014] K. Kurokawa et al. Three-dimensional retinal and choroidal capillary imaging by power doppler optical coherence angiography with adaptive optics. Opt. Express, 20(20):22796-22812, 2012.

[0015] P. E. Z. Tan et al. Quantitative confocal imaging of the retinal microvasculature in the human retina. Invest. Ophthalmol. Vis. Sci., 53(9):5728-5736, 2012.

[0016] A. M. E. Engberg et al. Automated quantification of retinal microvasculature from OCT angiography using dictionary -based vessel segmentation. In Annual Conference on Medical Image Understanding and Analysis, pages 257-269. Springer, 2019.

[0017] X. Wu et al. Joint destriping and segmentation of octa images. In Annual Conference on Medical Image Understanding and Analysis, pages 423- 435. Springer, 2019.

[0018] J. Lei et al. Repeatability and reproducibility of quantification of superficial peripapillary capillaries by four different optical coherence tomography angiography devices. Scientific Reports, 8(1): 1-7, 2018.

[0019] E. S. Levine et al. Repeatability and reproducibility of vessel density measurements on optical coherence tomography angiography in diabetic retinopathy. Graefe ’s Archive for Clinical and Experimental Ophthalmology, pages 1-9, 2020.

[0020] N. Eladawi et al. Automatic blood vessels segmentation based on different retinal maps from OCTA scans. Computers in Biology andMedicine, 89: 150-161, 2017.

[0021] Y. Liu et al. Variational intensity cross channel encoder for unsupervised vessel segmentation on OCT angiography. In Medical Imaging 2020: Image Processing, volume 11313, page 113130Y. International Society for Optics and Photonics, 2020.

[0022] L. Mou et al. CS2-Net: Deep Learning Segmentation of Curvilinear Structures in Medical Imaging. Medical Image Analysis, page 101874, 2020.

[0023] M. Li et al. Image projection network: 3D to 2D image segmentation in OCTA images. IEEE Trans. Med. Imag., 39(11):3343— 3354, 2020.

[0024] M. Li et al. IPN-V2 and OCTA-500: Methodology and dataset for retinal image segmentation. arXiv preprint arXiv:2012.07261 , 2020.

[0025] Y. Ma et al. ROSE: A retinal OCT-angiography vessel segmentation dataset and new model. IEEE Trans. Med. Imag., 40(3):928-939, 2020.

[0026] P. A. Dave et al. Comparative evaluation of foveal avascular zone on two optical coherence tomography angiography devices. Optometry and Vision Science, 95( ').6Q2-6Qrl , 2018.

[0027] O. Ronneberger et al. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- assisted Intervention, pages 234-241, 2015.

[0028] P. PrentasL c et al. Segmentation of the foveal microvasculature using' deep learning networks. J. of Biomedical Optics, 21(7):075008, 2016.

[0029] L. Mou et al. CS-Net: channel and spatial attention network for curvilinear structure segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 721-730, 2019.

[0030] T. Pissas et al. Deep iterative vessel segmentation in OCT angiography. Biomed. Opt. Express, 11(5):2490-2510, 2020.

[0031] Z. Wu et al. PAENet: A progressive attention-enhanced network for 3D to 2D retinal vessel segmentation. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1579-1584. IEEE, 2021.

[0032] D. Hu et al. Life: A generalizable autodidactic pipeline for 3D OCT-A vessel segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 514-524. Springer, 2021.

[0033] S. Yu et al. 3D vessel reconstruction in oct-angiography via depth map estimation. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1609-1613. IEEE, 2021.

[0034] Y. Xu et al. Partially-supervised learning for vessel segmentation in ocular images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 271-281. Springer, 2021.

[0035] S. Andermatt et al. Pathology segmentation using distributional differences to images of healthy origin. In International MICCAI Brainlesion Workshop, pages 228-238. Springer, 2018.

[0036] E. Vorontsov et al. Boosting segmentation with weak supervision from image-to- image translation. arXiv preprint arXiv: 1904.01636, 6, 2019.

[0037] Y. Zhou et al. Collaborative learning of semi-supervised segmentation and classification for medical images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2079-2088, 2019.

[0038] T. Robert et al. Hybridnet: Classification and reconstruction cooperation for semisupervised learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-169, 2018.

[0039] A. Chartsias et al. Disentangled representation learning in cardiac image analysis. Medical Image Analysis, 58: 101535, 2019.

[0040] K. Sohn et al. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pages 3483- 3491, 2015.

[0041] Z. Zhang et al. Disentangling the spatial structure and style in conditional vae. In 2020 IEEE International Conference on Image Processing (ICIP), pages 1626-1630, 2020.

[0042] B. E. Dewey et al. A disentangled latent space for cross-site MRI harmonization. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 720-729, 2020.

[0043] L. Zuo et al. Information-based Disentangled Representation Learning for Unsupervised MR Harmonization. In 27thInf Proc, in Med. Imaging (IPMI 2021), volume 12729 of Lecture Notes in Computer Science, pages 346-359. Springer Berlin Heidelberg, 2021.

[0044] L. Zuo et al. Unsupervised MR harmonization by learning disentangled representations using information bottleneck theory. NeuroImage, 243 : 118569, 2021.

[0045] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.

[0046] M. D'laz et al. Automatic segmentation of the foveal avascular zone in ophthalmological OCT-A images. PLoS One, 14(2):e0212364, 2019.

[0047] N. J. Tustison et al. Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements. NeuroImage, 99: 166-179, 2014.

[0048] Z. Alom et al. Recurrent residual U-Net for medical image segmentation. Jrnl. of Medical Imaging, 6(l):014006, 2019.

[0049] F. Isensee et al. nnU-Net: A self-configuring method for deep learningbased biomedical image segmentation. Nature Methods, 18(2):203-211, 2021.

[0050] Q. Dou et al. PnP-AdaNet: Plug-and-play adversarial domain adaptation network with a benchmark at cross-modality cardiac segmentation. arXiv preprint arXiv: 1812.07907, 2018.

[0051] O. Bernard et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Trans. Med. Imag., 37(11):2514— 2525, 2018.

[0052] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.

[0053] D. Berthelot et al. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019.

[0054] Y. Lu et al. Evaluation of automatically quantified foveal avascular zone metrics for diagnosis of diabetic retinopathy using optical coherence tomography angiography. Invest. Ophthalmol. Vis. Sci., 59(6): 2212-2221, 2018.

[0055] Y. Taigman et al. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1701-1708, 2014.

Claims

WHAT IS CLAIMED IS:

1. A method of forming a trained model in which one or more processing devices perform operations comprising: receiving a plurality of paired training image sets, each paired training image set in the plurality of paired training image sets comprising at least a first image and a second image, each of the first image and the second image associated with an image subject of a plurality of image subjects, each image subject comprising an anatomy component and a contrast component; processing the plurality of paired training image sets using a first neural network, with a plurality of first training weights, to separate, from each of the plurality of paired training image sets, the respective anatomy component from the respective contrast component of the respective image subject; receiving a plurality of second training images, each second training image in the plurality of second training images having a respective delineated image of a plurality of delineated images; processing the plurality of second training images using a second neural network, with a plurality of second training weights, to generate, from each of the plurality of second training images, a representation of the respective delineated image; wherein the first neural network is a conditional variational autoencoder comprising a conditioning network configured to receive the at least first image, an encoder configured to receive the at least second image, and a decoder configured to generate a representation of the at least second image such that, when the first neural network is trained, the conditioning network extracts the respective contrast component from the at least second image; wherein the second neural network comprises the conditioning network, configured to receive each of the plurality of second training images, and a segmenter configured to produce the representation of the respective delineated image; andwherein, when the first neural network and the second neural network are trained, the first set of training weights and the second set of training weights are fixed, thereby producing the trained model.

2. The method of claim 1, wherein the image subject is an eye, and the respective anatomy component is a vascular structure of the eye.

3. The method of claim 1, wherein the conditioning network uses a U-Net architecture.

4. The method of claim 1, wherein the conditioning network uses a CS-Net architecture.

5. The method of claim 1, wherein each of the encoder and the decoder comprise a plurality of max-pooling layers.

6. A method of segmenting an image generated by optical coherence tomography angiography, the method comprising: forming the trained model of claim 1; processing said image using said conditioning network of said second neural network to produce a segmented representation of said image based upon said trained model.

7. The method of claim 6, wherein the conditioning network uses a CS-Net architecture.

8. A system comprising: a processor; and a non-transitory storage medium storing instructions that, when executed by the processor, cause the processor to perform a method of forming a trained model using a plurality of paired training image sets and a plurality of second training images, each paired training image set in the plurality of paired training image sets comprising at least a first image and a second image, each of the first image and the second image associated with an image subject of a plurality of image subjects, each image subject comprising an anatomy component and a contrast component and each second training image in the plurality of second training images having a respective delineated image of a plurality of delineated images, the method of forming the trained model comprising:processing the plurality of paired training image sets using a first neural network, with a plurality of first training weights, to separate, from each of the plurality of paired training image sets, the respective anatomy component from the respective contrast component of the respective image subject; and processing the plurality of second training images using a second neural network, with a plurality of second training weights, to generate, from each of the plurality of second training images, a representation of the respective delineated image; wherein the first neural network is a conditional variation autoencoder comprising a conditioning network configured to receive the at least first image, an encoder configured to receive the at least second image, and a decoder configured to generate a representation of the at least second image such that, when the first neural network is trained, the conditioning network extracts the respective contrast component from the at least second image; wherein the second neural network comprises the conditioning network, configured to receive each of the plurality of second training images, and a segmenter configured to produce the representation of the respective delineated image; and wherein, when the first neural network and the second neural network are trained, the first set of training weights and the second set of training weights are fixed, thereby producing the trained model.

9. The system of claim 8, wherein the image subject is an eye, and the respective anatomy component is a vascular structure of the eye.

10. The system of claim 8, wherein the conditioning network uses a U-Net architecture.

11. The system of claim 8, wherein the conditioning network uses a CS-Net architecture.

12. The system of claim 8, wherein each of the encoder and the decoder comprise a plurality of max-pooling layers.

13. The system of claim 8, wherein said instructions comprise additional instructions that, when executed by the processor, cause the processor to perform a method of segmenting an image generated by optical coherence tomography angiography, the method of segmenting the image generated by optical coherence tomography angiography comprising: processing said image using said conditioning network of said second neural network to produce a segmented representation of said image based upon said trained model.

14. The system of claim 13, wherein the conditioning network uses a CS-Net architecture.

15. A non-transitory computer readable medium storing instructions that when executed by a system cause the system to perform a method of forming a trained model using a plurality of paired training image sets and a plurality of second training images, each paired training image set in the plurality of paired training image sets comprising at least a first image and a second image, each of the first image and the second image associated with an image subject of a plurality of image subjects, each image subject comprising an anatomy component and a contrast component and each second training image in the plurality of second training images having a respective delineated image of a plurality of delineated images, the method of forming the trained model comprising: processing the plurality of paired training image sets using a first neural network, with a plurality of first training weights, to separate, from each of the plurality of paired training image sets, the respective anatomy component from the respective contrast component of the respective image subject; and processing the plurality of second training images using a second neural network, with a plurality of second training weights, to generate, from each of the plurality of second training images, a representation of the respective delineated image; wherein the first neural network is a conditional variational autoencoder comprising a conditioning network configured to receive the at least first image, an encoder configured to receive the at least second image, and a decoder configured togenerate a representation of the at least second image such that, when the first neural network is trained, the conditioning network extracts the respective contrast component from the at least second image; wherein the second neural network comprises the conditioning network, configured to receive each of the plurality of second training images, and a segmenter configured to produce the representation of the respective delineated image; and wherein, when the first neural network and the second neural network are trained, the first set of training weights and the second set of training weights are fixed, thereby producing the trained model.

16. The non-transitory computer readable medium of claim 15, wherein the image subject is an eye, and the respective anatomy component is a vascular structure of the eye.

17. The non-transitory computer readable medium of claim 15, wherein the conditioning network uses a U-Net architecture.

18. The non-transitory computer readable medium of claim 15, wherein the conditioning network uses a CS-Net architecture.

19. The non-transitory computer readable medium of claim 15, wherein each of the encoder and the decoder comprise a plurality of max-pooling layers.

20. The non-transitory computer readable medium of claim 15, wherein said instructions comprise additional instructions that, when executed by the system, cause the system to perform a method of segmenting an image generated by optical coherence tomography angiography, the method of segmenting the image generated by optical coherence tomography angiography comprising: processing said image using said conditioning network of said second neural network to produce a segmented representation of said image based upon said trained model.

Images