Image segmentation system and method

Abstract

This paper discloses a method for segmenting a volumetric image comprising multiple slices. The method includes: inputting a target slice of the volumetric image into a deep neural network (DNN) having a multi-task learning architecture, the multi-task learning architecture including: a segmentation DNN configured to output a segmentation of the target slice; and a reconstruction DNN configured to: receive a plurality of neighboring slices adjacent to the target slice; and output a reconstruction of the target slice based on the plurality of neighboring slices; wherein the reconstruction DNN is further configured to share spatial information with the segmentation DNN, the spatial information indicating the correlation between the neighboring slices and the target slice.

Description

Technical Field

[0001] This disclosure generally relates to methods and systems for image segmentation. It is particularly (but not exclusively) applicable to the segmentation of medical images, such as volumetric image data from optical coherence tomography. Background Technology

[0002] A fundamental task in the clinical use of medical imaging is segmentation, which involves detecting, identifying, and subsequently quantifying regions of interest in acquired scans of pathological signs. Segmentation is important because it allows for the quantitative monitoring of biomarkers, such as tissue features, for the detection and monitoring of abnormalities, which has applications in disease screening and management.

[0003] Various methods for medical image segmentation have been reported. Two-dimensional methods for medical image segmentation have been successfully demonstrated in different applications. Many architectures for volume segmentation have been developed based on the U-Net network architecture. Similar methods were proposed during the development of the V-Net network. However, these methods typically require consideration of the entire volumetric image, which can be computationally expensive and require significant memory.

[0004] Recently, recurrent networks have gained increasing popularity due to their sequential approach to volumetric medical images. A known application of recurrent networks uses a combination of fully convolutional networks (FCNs) and recurrent neural networks (RNNs). However, this still requires the entire medical scan to be available. To address this issue, several methods have been proposed that combine the sequential properties of the spatial context in medical images with 2D segmentation. However, these methods are computationally expensive and prone to memory leaks common in recurrent networks.

[0005] A network called D-UNet enables this architecture to learn the spatial context of neighboring slices during the encoding phase by using 3D convolutions and combining them with 2D segmentation that treats neighboring slices as different channels. Another approach is to learn the spatial context in the same way using the proposed Globally Guided Progressive Fusion Network (GGPF-Net). These methods successfully outperform and replace sequential and 3D FCN methods in terms of performance and computational efficiency. However, the spatial context learned by such architectures depends on the quality of the available labels.

[0006] The aim is to overcome or mitigate at least one of the aforementioned problems, or at least provide a useful alternative. Summary of the Invention

[0007] This paper discloses a method for segmenting a volumetric image comprising multiple slices, the method comprising:

[0008] The target slice of the volumetric image is input into a deep neural network (DNN) with a multi-task learning architecture, which includes:

[0009] A segmentation DNN configured to output segmentations of the target slice; and

[0010] Reconstruct the DNN, configured as follows:

[0011] Receive multiple adjacent slices that are adjacent to the target slice; and

[0012] The reconstruction of the target slice is based on multiple adjacent slices;

[0013] The reconstruction DNN is further configured to share spatial information with the segmentation DNN, and the spatial information indicates the correlation between adjacent slices and the target slice.

[0014] In some embodiments, the reconstructed DNN includes a convolutional feature extractor for generating first feature data from adjacent slices, and a reconstructed downsampling unit for generating first dimensionality-reduced feature data from the first feature data at one or more scales.

[0015] In some embodiments, reconstructing the DNN includes a reconstruction upsampler for converting the first dimensionality-reduced feature data into first upsampled data having the same dimension as the first feature data.

[0016] In some embodiments, reconstructing the DNN includes one or more dimensionality reduction layers, which are used to apply a dimensionality reduction mechanism to first feature data and / or first dimensionality-reduced feature data.

[0017] In some embodiments, the dimensionality reduction mechanism includes:

[0018] Input the first feature data and / or the first dimensionality-reduced feature data into the three-dimensional convolutional layer;

[0019] Clustering of features is applied between adjacent slices;

[0020] Batch normalization is applied to the output of the 3D convolutional layer; and

[0021] Apply the ReLU activation function to the batch normalized output.

[0022] In some embodiments, the layers of the reconstructed downsampler are connected to the layers of the reconstructed upsampler via concatenation through corresponding layers in the dimensionality reduction layer.

[0023] In some embodiments, the segmentation DNN includes a convolutional feature extractor for generating second feature data from target slices, and a segmentation downsampling unit for generating second dimensionality-reduced feature data from the second feature data at one or more scales.

[0024] In some embodiments, the segmentation DNN includes a segmentation upsampler for converting the second dimensionality-reduced feature data into second upsampled data having the same dimension as the second feature data.

[0025] In some embodiments, the layer of the segmented downsampler is connected to the layer of the segmented upsampler.

[0026] In some embodiments, the reconstruction DNN is configured to share spatial information with the segmentation DNN by adding the output of the reconstruction upsampler's layer element-wise to the output of the segmentation upsampler's layer.

[0027] In some embodiments, the loss function of the segmentation DNN is the two-dimensional intersection-union (IoU) loss function.

[0028] In some embodiments, the volumetric image is a three-dimensional medical image.

[0029] In some embodiments, the three-dimensional medical image is a three-dimensional optical coherence tomography (OCT) image.

[0030] In some embodiments, the three-dimensional OCT image is a retinal image, and the target slice corresponds to a layer of the choroid.

[0031] In some embodiments, the method is repeated for multiple target slices, and the method further includes generating a choroid thickness map from the segmentation of the multiple target slices.

[0032] This paper also discloses a system for segmenting a volumetric image comprising multiple slices, the system comprising:

[0033] At least one processor; and

[0034] A computer-readable storage device having instructions thereon for causing at least one processor to perform the disclosed method.

[0035] This document also discloses a non-transitory computer-readable storage medium having instructions stored thereon for causing at least one processor to perform the disclosed method. Attached Figure Description

[0036] Embodiments will now be described by way of non-limiting example with reference to the accompanying drawings, in which:

[0037] Figure 1 This is a flowchart illustrating a method for segmenting a volumetric image comprising multiple slices;

[0038] Figure 2 This is a flowchart illustrating another method for segmenting a volumetric image that comprises multiple slices;

[0039] Figure 3 This is a flowchart illustrating the dimensionality reduction mechanism;

[0040] Figure 4 This is a flowchart illustrating another method for segmenting a volumetric image that comprises multiple slices;

[0041] Figure 5 This is a neural network architecture diagram showing a spatial aggregation network consisting of reconstruction branches and segmentation branches;

[0042] Figure 6 The illustration shows various two-dimensional segments generated by methods consistent with this disclosure and by various prior art methods;

[0043] Figure 7 This shows the difference between the real situation and the situation from... Figure 6 The choroid thickness map generated for each of the segmentation architectures used; and

[0044] Figure 8 This is a schematic diagram illustrating components of an exemplary computer system for performing the methods described herein. Detailed Implementation

[0045] This disclosure relates to a computationally efficient and accurate segmentation method for volumetric medical images, which is robust to gap variations. A novel multi-task learning architecture for volumetric segmentation is proposed, enabling fully automated 3D segmentation of volumetric medical image data. The proposed architecture combines both reconstruction and segmentation tasks. Simultaneous reconstruction and segmentation extract features within slices, which are directly used for segmentation. Specifically, the multi-task learning architecture aggregates spatial context from adjacent cross-sectional slices to reconstruct a central slice. This architecture reconstructs the central slice by learning spatial information between adjacent slices. Soft parameter sharing between the reconstruction and segmentation tasks can be used to channelize spatial information. This soft parameter sharing more explicitly aggregates spatial features by directly learning the correlation between adjacent slices and the slice to be segmented.

[0046] The spatial context learned through the proposed reconstruction mechanism can be fused using a U-Net-based architecture. In this disclosure, the proposed U-Net-based architecture is referred to as the Spatial Aggregation Network (SA-Net network) due to the aggregation of its spatial information. The SA-Net network learns spatial information between adjacent cross-sections to reconstruct a selected cross-section. The SA-Net network is a convolutional neural network based on a fully convolutional network, and its architecture can be modified and extended to use fewer training images and produce more accurate segmentation. The main idea of ​​the proposed U-Net-based architecture is to supplement the shrinking network with successive layers, where pooling operations are replaced by upsampling operations. Therefore, these layers improve the resolution of the output. Furthermore, successive convolutional layers can then learn to aggregate accurate outputs based on this information. In the proposed SA-Net network, there are numerous feature channels in the upsampling portion, which allows the network to propagate contextual information to higher resolution layers. It should be understood that combining spatial information from corresponding adjacent slices enables the proposed SA-Net network architecture to explicitly integrate spatial correspondences. Generally, this disclosure does not require consideration of the entire volumetric image, thus avoiding costly computation and large memory requirements. At the same time, the proposed method does not require a lot of computation and is less prone to memory leaks, a common problem in recursive networks.

[0047] Figure 1 An example method 100 for segmenting a volumetric image 102 comprising multiple slices is shown. In summary, method 100 includes:

[0048] The target slice 104 of the volumetric image 102 is input into a deep neural network (DNN) (e.g., an SA-Net network) with a multi-task learning architecture 106, which includes:

[0049] Segmentation DNN 108, configured to output segmentation 110 of target slice 104; and

[0050] Reconstruct DNN 112, configured as follows:

[0051] Receive a plurality of adjacent slices 114 adjacent to the target slice 104; and

[0052] The reconstruction 116 of the target slice 104 is output based on multiple adjacent slices 114;

[0053] The reconstruction DNN 112 is further configured to share spatial information 118 with the segmentation DNN 108, the spatial information 118 indicating the correlation between adjacent slices 114 and the target slice 104.

[0054] like Figure 1As shown, the volumetric image can be a three-dimensional (3D) volume containing the target region and initially acquired by a medical imaging system. The proposed SA-Net network architecture 106 combines both reconstruction and segmentation tasks. The segmentation DNN 108 performs segmentation, while the spatial context is provided by neighboring slices in the reconstruction DNN 112. Specifically, each cross-sectional scan 114 to be segmented by the SA-Net network in the segmentation branch is processed simultaneously with its neighboring scans 114 in the reconstruction branch 112 (which provides the spatial context). In this disclosure, specifically, given a slice I_i to be segmented, the reconstruction branch will employ Λ = {I_i} i-n ,..,I i-1 I i+1 , ..., I i+n} is its input where 2n defines the number of adjacent slices to be used.

[0055] Figure 2 The detailed connection between the segmentation DNN 108 and the reconstruction DNN 112 is illustrated. This disclosure will discuss the construction of DNN 112 and the segmentation DNN 108 separately.

[0056] In the reconstruction DNN 112, explicit spatial information from neighboring slices 114 can be extracted. Specifically, explicit spatial information from neighboring slices 114 can be extracted using a series of three-dimensional convolutions. It should be understood that the reconstruction DNN 112 can be divided into downsampling and upsampling parts. Neighboring slices 114 are downsampled, and convolutions are repeated to extract multi-scale representations of the spatial context. In some embodiments, the reconstruction DNN 112 includes a convolutional feature extractor 202 for generating first feature data from neighboring slices 114. Neighboring slices 114 are then downsampled, and convolutions are repeated to extract multi-scale representations of the spatial context. During the downsampling process, rich spatial information from neighboring slices 114 is utilized by using three-dimensional convolutions and max-pooling layers. Figure 2 In one embodiment shown, the reconstruction DNN 112 further includes a reconstruction downsampler 204 for generating first dimensionality-reduced feature data from the first feature data at one or more scales. The reconstruction downsampler 204 enables the architecture 106 to learn local spatial information contained between slices, i.e., inter-slice features.

[0057] Following the downsampling stage, convolutional upsampling is then performed at different levels to ensure a consistent representation of information from different scales and concatenation with residuals at the same scale. In some embodiments, the reconstructed DNN 112 includes a reconstruction upsampling unit 206, which transforms the first dimensionality-reduced feature data generated by the reconstruction downsampling unit 204 into first upsampled data with the same dimension as the first feature data at one or more scales. After upsampling, a final two-dimensional convolution is performed, and the output is computed against the ground truth (i.e., I0). i The loss between slices. Embodiments of this disclosure use mean squared error to calculate the predicted output y. pred Compared to the actual situation y true The similar distance between them.

[0058]

[0059] Other similarity or dissimilarity measures, such as the SSIM (structural similarity) index, can also be used.

[0060] In some embodiments, the reconstructed DNN 112 may further include one or more dimensionality reduction layers for applying a dimensionality reduction mechanism (DRM) 210 to the first feature data generated by the convolutional feature extractor 202. The reconstructed DNN 112 may also include one or more dimensionality reduction layers for applying another DRM 212 to the first dimensionality-reduced feature data generated by the reconstructed downsampler 204. In this disclosure, the DRMs 210 and 212 are used to represent information more efficiently. In particular, to reduce the number of parameters given by the three-dimensional convolutional layers, in the bottleneck block, this disclosure combines the DRMs 210 and 212 to convert three-dimensional information into two-dimensional (2D) information. In some embodiments, the converted three-dimensional information generated by the DRMs 212 and 210 is then upsampled using two-dimensional convolutional layers in the reconstructed upsamplers 206 and 208, respectively.

[0061] Figure 3An example workflow for the proposed DRM 210 / 212 is shown. Generally, the DRM begins with a 3D convolutional layer, followed by batch normalization and rectified linear unit (ReLU) activation. In one embodiment, DRM 210 / 212 includes first inputting first feature data generated by convolutional feature extractor 202 and / or first dimensionality-reduced feature data generated by reconstruction downsampling unit 204 into the 3D convolutional layer (step 302). DRM 210 / 212 further includes applying feature aggregation between adjacent slices 114 (step 304). In some embodiments, during step 304, numerical features along the cross-sectional axis are summed. In step 306, batch normalization is applied to the output of the 3D convolutional layer. Batch normalization is a method for normalizing the layer input by re-centering and rescaling, thereby making the neural network faster and more stable. Finally, in step 308, a ReLU activation function is applied to the batch-normalized output. In the context of artificial neural networks, the ReLU activation function is defined as the positive part of its independent variable. In some embodiments, other activation functions, such as Leaky ReLU, Exponential Linear Unit (ELU), or Parametric Rectified Linear Unit (PReLU), can be used instead of ReLU. It should be understood that the proposed DRM ensures that the volumetric properties of adjacent slices are preserved and simultaneously scaled or normalized. Furthermore, it reduces the complexity of the reconstruction branches and improves the convergence speed.

[0062] Figure 4 Another example of the proposed multi-task learning architecture 106, including a reconstruction DNN 112, is shown. The reconstruction DNN 112 takes neighboring slices as input and extracts spatial features. The dimension is reduced along the cross-sectional axis to obtain the volumetric size. Upsampling is performed to obtain the original three-dimensional size. As will be discussed in detail, the spatial information from the reconstruction DNN 112 (see step 410) is fused into a segmentation DNN 108, which takes the corresponding slice to be segmented as input.

[0063] like Figure 4As shown, the first step (step 402) is to extract spatial features from adjacent slices 114. The reconstructed DNN 112 may include a convolutional feature extractor 202 for generating first feature data from adjacent slices 114. The reconstructed DNN 112 may further include a reconstructed downsampling unit 204 for generating first dimensionality-reduced feature data from the first feature data at one or more scales. The reconstructed DNN 112 may include one or more dimensionality-reduction layers for applying DRM 210 to the first feature data. The reconstructed DNN 112 may also include one or more dimensionality-reduction layers for applying another DRM 212 to the first dimensionality-reduced feature data generated by the reconstructed downsampling unit 204. Specifically, in step 404, DRM 210 / 212 along adjacent axes are applied to obtain the volumetric dimension. For DRM 210 / 212, the dimension is along the cross-sectional axis. In step 406, the reconstruction DNN 112 includes reconstruction upsampler 206 and reconstruction upsampler 208, which are used to convert the first dimensionality-reduced feature data generated by reconstruction downsampler 204 into first upsampled data with the same dimension as the first feature data at one or more scales. In step 406, the two-dimensional magnification along the cross-sectional axis is applied to obtain the original three-dimensional dimensions. Architecture 106 finally outputs a reconstruction 116 of the target slice 104 based on multiple adjacent slices 114.

[0064] In some embodiments, such as Figure 4 As shown, the layers of the reconstruction downsamplers 204 / 208 are connected to the layers of the reconstruction upsamplers 206 / 208 via concatenation of corresponding layers from the dimensionality reduction layers. More specifically, after performing DRM, the reconstruction DNN112 can concatenate rich spatial features with low-resolution upsampled features (see step 408). Due to this architecture, the proposed method does not require evaluating the full volumetric image, saving processing and memory time and money.

[0065] In segmentation DNN 108, explicit spatial information from target slice 104 (see...) Figure 1 and Figure 2The target slice 104, as input, refers to the corresponding slice to be segmented. Information within the target slice is extracted at multiple scales using a method similar to that used in the reconstruction DNN 112. Specifically, explicit spatial information from the target slice 104 can be extracted using a series of two-dimensional convolutions. It should be understood that the segmentation DNN 108 can be divided into downsampling and upsampling parts. The target slice 104 is downsampled, and convolutions are repeated to extract a multi-scale representation of the spatial context. The segmentation DNN 108 may include a convolutional feature extractor 214 for generating second feature data from the target slice 104. The target slice 104 is then downsampled, and convolutions are repeated to extract a multi-scale representation of the spatial context. During the downsampling process, in-slice features contained within the target slice 104 are utilized from the target slice 104 using two-dimensional convolutions and max-pooling layers. Figure 2 In one embodiment shown, the segmentation DNN 108 further includes a reconstruction downsampler 216 for generating second dimensionality-reduced feature data from the second feature data at one or more scales.

[0066] Following the downsampling stage, convolutional upsampling is then performed at different levels to ensure a consistent representation of information from different scales and concatenation with residuals of the same scale. In some embodiments, the segmentation DNN 108 includes segmentation upsampling unit 218 and segmentation upsampling unit 220 for converting the second dimensionality-reduced feature data into second upsampled data with the same dimension as the second feature data. In the upsampling portion, high-resolution features from the downsampling period are concatenated with low-resolution features. At each end of an upsampling block consisting of a two-dimensional upsampling layer and two two-dimensional convolutional layers, knowledge of inter-slice features from the reconstruction branch is fused. High-resolution two-dimensional volumetric features are added element-wise along with features extracted within the two-dimensional slices to incorporate inter-slice correlation features.

[0067] like Figure 4 As shown, the segmentation DNN 108 takes the target slice 104 as input and extracts features within the slice. Upsampling is performed to obtain the original 3D dimensions. In some embodiments, the layers of the segmentation downsampling are connected to the layers of the segmentation upsampling. Figure 2 As shown, spatial information 118 from the reconstructed DNN 112 is fused into the segmentation DNN 108, which takes the corresponding slice to be segmented as input.

[0068] like Figure 4As shown, the first step (step 412) is to extract in-slice features from the target slice 104. The segmentation DNN 108 may include a convolutional feature extractor 214 for generating second feature data from the target slice 104. The segmentation DNN 108 may also include a segmentation downsampling unit 214 for generating first dimensionality-reduced feature data from the second feature data at one or more scales. In step 414, the reconstruction DNN 112 includes a segmentation upsampling unit 218 and a segmentation upsampling unit 220 for converting the first dimensionality-reduced feature data generated by the segmentation downsampling unit 216 into first upsampling data having the same dimension as the first feature data at one or more scales. In step 406, the two-dimensional magnification along the cross-sectional axis is applied to obtain the original three-dimensional dimensions. It should be understood that the reconstruction DNN can be configured to sum the outputs of the layers of the reconstruction upsampling unit element-wise (see...). Figure 4 The output of the layer 410 in the segmentation upsampler is used to share spatial information with the segmentation DNN. Architecture 106 ultimately outputs the segmentation result 110.

[0069] In such Figure 4 In the illustrated embodiment, the layers of the segmentation downsamplers 204 / 208 are connected in series to the layers of the reconstruction upsamplers 218 / 220 via corresponding layers in the dimensionality reduction layers. More specifically, the segmentation DNN 112 can concatenate rich intra features to low-resolution upsampled features (see step 416). Due to this architecture, the proposed method does not need to consider the entire volumetric image, reducing costly processing and memory requirements.

[0070] The loss function for a segmentation DNN can be a two-dimensional intersection-over-union (IoU) loss function. Upsampling DNNs terminate with a series of two-dimensional convolutions and sigmoid activation functions. This disclosure uses a two-dimensional IoU loss function to maximize the intersection region between the prediction and the ground truth. This two-dimensional IoU loss function is defined as follows:

[0071]

[0072] In some embodiments, the volumetric image is a three-dimensional medical image. Specifically, the proposed SA-Net network can potentially be applied to the segmentation and detection of structures in medical imaging modalities that acquire three-dimensional volumetric data, including but not limited to optical coherence tomography (OCT), computed tomography (CT), and magnetic resonance imaging (MRI). The three-dimensional medical image can be a three-dimensional optical coherence tomography (OCT) image. OCT refers to a relatively new medical imaging method capable of high-resolution depth-resolved imaging of structures beneath the retinal surface. This allows for the visualization of subretinal changes that are not observable using fundus photography. The practicality of OCT imaging has led to its widespread adoption in many clinical practices and has even replaced fundus photography as the primary form of ophthalmic imaging in some practices. In this disclosure, the three-dimensional OCT image can be a retinal image, and the target slice can correspond to a layer of the choroid.

[0073] This document also discloses a system for segmenting a volumetric image comprising multiple slices, the system comprising at least one processor; and a computer-readable storage having instructions stored thereon for causing the at least one processor to perform the disclosed method.

[0074] experiment

[0075] Following preprocessing, embodiments of this disclosure employ a five-fold cross-validation strategy to train and evaluate the proposed model. To avoid the risks of training bias and overfitting, all images from the same eye are ensured to be in the same folder. This avoids situations where the test and training partitions might consist of different images from the same eye. The overall experimental results are then obtained by averaging all validation sets in each folder. The architecture was developed using Python version 3.7.4 and TensorFlow version 2.0. Experiments were conducted on a workstation equipped with an NVIDIA RTX 2080Ti GPU and 64GB of RAM.

[0076] Figure 5 The proposed SA-Net network 100 is shown, which includes reconstruction and segmentation branches (i.e., 112 and 108) for two different tasks, namely learning inter-slice features and intra-slice features, respectively. Specifically, Figure 5An implementation of the proposed SA-Net network for a specific application (i.e., OCT imaging of the choroid or RPE) is shown. During the encoding phase, spatial information aggregated in the reconstruction branch 112 is fused into the segmentation branch 108. The SA-Net network 100 extracts spatial features from neighboring slices by reconstructing the target slice. The implementation of the SA-Net network 100 for choroid segmentation leverages the volumetric properties of OCT images and the availability of datasets. OCT choroid segmentation is extremely challenging, especially in high myopia, and the results from the SA-Net network demonstrate the advantages of this method. Choroidal segmentation is important not only for myopia but also for other diseases such as age-related macular degeneration, glaucoma, diabetic retinopathy, and other types of retinal and optic nerve head diseases.

[0077] For example, such as Figure 5 As shown, feature extractors 202 / 204 can begin with a 3D convolutional layer, followed by batch normalization and ReLU activation. For example, in reconstruction branch 112, embodiments of this disclosure can begin with volume-adjacent slices. Thus, a 3D convolutional layer can then be used to encode features before DRM 202 / 212 to reduce the feature dimension from 3D to 2D. Upsamplers 206 / 208, used to convert the first dimensionality-reduced feature data generated by reconstruction downsamplers 202 / 204 into first upsampled data, can begin with a 2D convolutional layer, followed by batch normalization and ReLU activation. Feature extractors 214 / 216 for segmentation branch 108 can contain a 2D convolutional layer, followed by batch normalization and ReLU activation. Segmentation upsamplers 218 / 220, used to convert the first dimensionality-reduced feature data generated by segmentation downsamplers 214 / 216 into first upsampled data with the same dimension, can begin with a 2D convolutional layer, followed by batch normalization and ReLU activation.

[0078] The proposed SA-Net network for choroidal volume segmentation was evaluated. The choroid is the vascular layer of the eye, supplying more than 60% of the blood to the retina. Variations in the choroid are associated with many eye diseases, including age-related macular degeneration and diabetic retinopathy. Until recently, OCT imaging of the choroid has been challenging because it is obscured by highly scattered retinal pigment epithelium, and visibility of the choroid using spectral-domain OCT systems operating in the 800 nm range is very limited. However, the use of a swept-source laser operating at 1000 nm in the OCT system provides optimal timing for choroidal analysis due to reduced scattering.

[0079] The proposed SA-Net network was evaluated on two OCT datasets. The first dataset consisted of 40 highly myopic eyes obtained using the commercial scanning source OCT (SS-OCT) system – DRI OCT Triton (Topcon, Japan), with a wavelength of 1050 nm, a scan speed of 100,000 A / s, and a 7 mm × 7 mm scan scheme centered on the macula. Each eye volume in the Triton dataset contained 256 slices with dimensions 256 × 128. Another separate dataset was obtained from 9 normal eyes using the PLEX Elite 9000SS-OCT system (Carl Zeiss Meditec, Jena, Germany), operating in the wavelength range between 1040 nm and 1060 nm, with a scan speed of 100,000 A / s and a 15 mm × 9 mm scan scheme. Each eye volume in the PLEX dataset contained 834 slices with dimensions 512 × 500. Preprocessing is performed to limit the field of view of the acquired scan to the macular region, and the size is resized to 256×128. The network receives the target slice for segmentation and neighboring slices as input for reconstruction. Slices from the ends of the volume are filled by averaging the target slice with available neighboring slices.

[0080] The segmentation results are volumetrically evaluated by calculating IoU, segmentation score, and accuracy in volumetric terms regarding ground truth segmentation. Inter-slice correlation is assessed by measuring the quality of the choroid thickness map generated from the choroid segmentation. The method is repeated for multiple target slices, further including generating choroid thickness maps from the segmentation of the multiple target slices. Specifically, the choroid thickness map is obtained by superimposing the choroid thicknesses obtained from each slice. The generated map is evaluated by calculating a structural similarity index, which assesses the similarity between the predicted thickness map and the ground truth thickness map. Given two images, x and y, of the same size, the SSIM formula is given by:

[0081]

[0082] Where μ x μ y , σ x , σ y , σ xy These are the mean of x, the mean of y, the variance of x, the variance of y, and the covariance of x and y, respectively. Also, c1 = (0.001DR) 2 And c2 = (0.003DR) 2 In this disclosure, DR, or dynamic range, is given by the following formula:

[0083]

[0084] Table 1 shows a comparison of results using the proposed SA-Net network with other segmentation methods, such as the 3D U-Net network, BC U-Net network, and GGPF-Net network, on the Triton dataset. The results demonstrate that the SA-Net network architecture successfully outperforms other architectures in volume segmentation. This indicates that explicitly learning neighboring spatial features from the reconstruction can achieve more accurate 3D volume segmentation.

[0085] Table 1

[0086]

[0087] Table 2 also shows the results for the PLEX data, where the proposed architecture achieved similar results. It is also important to note that this architecture requires significantly less network complexity and computational power than the BC U-Net network, resulting in faster learning and inference times.

[0088] Table 2

[0089]

[0090]

[0091] Figure 6 From left to right, the original image and the actual situation of each architecture are shown. Figure 6 (a) and two-dimensional segmentation results ( Figure 6 (b) to 6(f) indicate the segmentation results of the various architecture implementations. It can be seen that the SA-Net network and the 3D U-Net network produce the best (i.e., the closest to the real situation) segmentation. Figure 7 The generated choroid thickness maps are shown from both the real-world data and the segmentation architecture. It was observed that the thickness maps from the 2D-based U-Net network were noisier and exhibited greater variability between slices, while the block-based 3D U-Net network resulted in less complete results. The BC U-Net, GGPF-Net, and SA-Net networks showed results closer to reality. As shown in Tables 1 and 2, the SA-Net network outperformed the other methods on both SSIM-scored datasets and was visually compared to the real-world choroid thickness maps.

[0092] Table 3 presents a detailed comparison between the proposed SA-Net network and existing networks. Spatial information can provide useful context for volumetric segmentation. In the proposed SA-Net network, incorporating spatial information from corresponding neighboring slices allows our proposed SA-Net network architecture to explicitly integrate spatial correspondences. The SA-Net network is compared with other state-of-the-art methods for segmenting the choroid from volumetric OCT images from two different commercial devices, demonstrating that the SA-Net network outperforms other methods in terms of segmentation accuracy and the quality of the generated choroid thickness map, while having lower computational requirements. The results show that the SA-Net network can be used for efficient and accurate segmentation of OCT data and potentially other volumetric medical images.

[0093] Table 3

[0094]

[0095]

[0096] This document also discloses a non-transitory computer-readable storage medium having instructions stored thereon for causing at least one processor to perform the disclosed method.

[0097] Figure 8 This is a block diagram illustrating an exemplary computer device 700, wherein the embodiment can be practiced. Computer device 700 may be a mobile computer device, such as a smartphone, wearable device, handheld computer, and cellular phone supporting multimedia internet, in-vehicle computing system, or any other computing system, mobile device (such as Apple...). TM iPhones manufactured by the company TM , or LG TM HTC TM and Samsung TM A mobile device manufactured, such as or other devices.

[0098] As shown in the figure, the mobile computer device 700 includes the following components that communicate electronically via a bus 706:

[0099] (a) Display 302;

[0100] (b) Non-volatile (non-temporal) memory 704;

[0101] (c) Random access memory (“RAM”) 708;

[0102] (d) N processing components 710;

[0103] (e) a transceiver assembly 712, comprising N transceivers; and

[0104] (f) User control 714.

[0105] although Figure 8 The components described in the text represent physical components, but Figure 8 This is not a hardware diagram. Therefore, Figure 8 Many of the components described can be implemented through a common construction or distributed across additional physical components. Furthermore, it is certainly conceivable that other existing and yet-to-be-developed physical components and architectures can be utilized to implement the reference. Figure 8 The described functional components.

[0106] Display 302 typically operates to present content to a user and can be achieved through any of a variety of displays, such as CRT, LCD, HDMI, micro-projectors, and OLED displays.

[0107] Typically, non-volatile data memory 704 (also known as non-volatile memory) is used to store (e.g., persistent storage) data and executable code. System architecture can be implemented in memory 704, or through instructions stored in memory 704.

[0108] For example, in some embodiments, the non-volatile memory 704 includes bootloader code, modem software, operating system code, file system code, and code that facilitates implementation components, which are well known to those skilled in the art and are not depicted or described for simplicity.

[0109] In many implementations, the non-volatile memory 704 is implemented using flash memory (e.g., NAND or ONENAND memory), but it is certainly expected that other memory types may also be utilized. Although code can be executed from the non-volatile memory 704, the executable code in the non-volatile memory 704 is typically loaded into RAM 708 and executed by one or more of the N processing components 710.

[0110] The N processing components 710 associated with RAM 708 typically operate to execute instructions stored in non-volatile memory 704. As will be understood by those skilled in the art, the N processing components 710 may include a video processor, a modem processor, a DSP, a graphics processing unit (GPU), and other processing components.

[0111] Transceiver assembly 712 includes N transceiver chains that can be used to communicate with external devices via a wireless network. Each of the N transceiver chains can represent a transceiver associated with a specific communication scheme. For example, each transceiver can correspond to a protocol specific to a local area network, a cellular network (e.g., a CDMA network, a GPRS network, a UMTS network), and other types of communication networks.

[0112] It should be recognized that Figure 8 The functions described herein are merely exemplary, and in one or more exemplary embodiments, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on a non-transitory computer-readable medium 704 or transmitted as one or more instructions or code encoded on the non-transitory computer-readable medium 704. The non-transitory computer-readable medium 704 includes both computer storage media and communication media, wherein the communication media includes any medium that facilitates the transfer of a computer program from one place to another. The storage medium may be any available medium that is accessible to a computer.

[0113] It should be understood that embodiments of this disclosure provide a novel segmentation architecture capable of fully automated 3D segmentation of volumetric medical image data. This architecture includes the following key novel aspects. First, soft parameter sharing more explicitly aggregates spatial features by directly learning the correlation between adjacent slices and the slice to be segmented. Furthermore, simultaneous reconstruction and segmentation extract intra-slice features directly used for segmentation. Additionally, the automatic generation of a volumetric choroid representation enables 3D visualization of the choroid. Last but not least, the generation of a full-field choroid thickness map enables positive analysis of choroid thickness variations across the entire retina.

[0114] It should be understood that numerous further modifications and arrangements of the various aspects of the described embodiments are possible. Therefore, the described aspects are intended to encompass all such changes, modifications, and variations that fall within the spirit and scope of the appended claims.

[0115] Throughout this specification and the following claims, unless the context otherwise requires, the word “comprising” and variations such as “including” and “containing” shall be understood to imply inclusion of the said integer or step, or integer or group of steps, but not to exclude any other integer or step, or integer or group of steps.

[0116] Any reference in this specification to any previously disclosed (or derived) information or any known matter is not and should not be construed as an admission or acceptance or any form of implication that such previously disclosed (or derived) information or known matter constitutes part of the general common knowledge in the field of effort covered by this specification.

Claims

1. A method for segmenting a volumetric image comprising multiple slices, the method comprising: The target slice of the volumetric image is input into a deep neural network (DNN) with a multi-task learning architecture, the multi-task learning architecture comprising: A segmentation DNN configured to receive the target slice and output segments of one or more target regions of the target slice; and Reconstruct the DNN, configured as follows: Receive a plurality of adjacent slices adjacent to the target slice; and The reconstruction of the target slice is output based on the multiple adjacent slices; The reconstruction DNN is further configured to share spatial information with the segmentation DNN, the spatial information indicating the correlation between the adjacent slices and the target slice.

2. The method of claim 1, wherein the reconstructed DNN comprises a convolutional feature extractor for generating first feature data from the adjacent slices, and a reconstructed downsampling unit for generating first dimensionality-reduced feature data from the first feature data at one or more scales.

3. The method of claim 2, wherein the reconstructed DNN includes a reconstruction upsampler, the reconstruction upsampler being used to convert the first dimensionality-reduced feature data into first upsampled data having the same dimension as the first feature data.

4. The method of claim 3, wherein the reconstructed DNN includes one or more dimensionality reduction layers, the one or more dimensionality reduction layers being used to apply a dimensionality reduction mechanism to the first feature data and / or the first dimensionality-reduced feature data.

5. The method according to claim 4, wherein the dimensionality reduction mechanism comprises: The first feature data and / or the first dimensionality-reduced feature data are input into a three-dimensional convolutional layer; Clustering of features is applied between adjacent slices; Batch normalization is applied to the output of the three-dimensional convolutional layer; as well as The ReLU activation function is applied to the batch normalized output.

6. The method of claim 4, wherein the layers of the reconstructed downsampler are connected to the layers of the reconstructed upsampler via concatenation through corresponding layers in the dimensionality reduction layers.

7. The method according to any one of claims 3 to 6, wherein the segmentation DNN includes a convolutional feature extractor for generating second feature data from the target slice, and a segmentation downsampling unit for generating second dimensionality-reduced feature data from the second feature data at one or more scales.

8. The method of claim 7, wherein the segmentation DNN includes a segmentation upsampler for converting the second dimensionality-reduced feature data into second upsampled data having the same dimension as the second feature data.

9. The method of claim 8, wherein the layer of the segmented downsampler is connected to the layer of the segmented upsampler.

10. The method of claim 8, wherein the reconstructed DNN is configured to share spatial information with the segmentation DNN by adding the output of the layer of the reconstructed upsampler element-wise to the output of the layer of the segmentation upsampler.

11. The method according to any one of claims 1 to 6, wherein the loss function of the segmented DNN is a two-dimensional intersection-union (IoU) loss function.

12. The method according to any one of claims 1 to 6, wherein the volumetric image is a three-dimensional medical image.

13. The method according to claim 12, wherein the three-dimensional medical image is a three-dimensional optical coherence tomography image.

14. The method of claim 13, wherein the three-dimensional optical coherence tomography image is a retinal image, and wherein the target slice corresponds to a layer of the choroid.

15. The method of claim 14, wherein the method is repeated for a plurality of target slices, and wherein the method further comprises generating a choroid thickness map from the segmentation of the plurality of target slices.

16. A system for segmenting a volumetric image comprising multiple slices, comprising: At least one processor; as well as A computer-readable storage device having instructions thereon for causing the at least one processor to perform the method according to any one of claims 1 to 15.

17. A non-transitory computer-readable storage memory having stored thereon instructions for causing at least one processor to perform the method according to any one of claims 1 to 15.

Images