Medical Image Segmentation with Uncertainty Estimation

Abstract

A system and method for generating a segmentation mask of an anatomical structure together with an uncertainty measure of the segmentation mask is provided. According to one or more embodiments, a plurality of candidate segmentation masks of the anatomical structure are generated from an input medical image using one or more trained machine learning networks. A final segmentation mask of the anatomical structure is determined based on the plurality of candidate segmentation masks. An uncertainty measure associated with the final segmentation mask is determined based on the plurality of candidate segmentation masks. The final segmentation mask and / or the uncertainty measure are output.

Description

Technical Field

[0001] The present invention relates generally to medical image segmentation with uncertainty estimation, and more particularly to medical image segmentation with uncertainty estimation for automating clinical workflows. Background Art

[0002] Medical imaging analysis is increasingly being applied in clinical workflows. For example, medical imaging analysis may be applied to diagnose patients with diseases or other medical conditions. Typically, such medical imaging analysis is performed by first segmenting anatomical structures from medical images before applying other medical imaging analysis tasks that use the segmented anatomical structures. Conventionally, segmenting anatomical structures from medical images is performed manually by a user, using a computer based on initial points identified by a user, or using a computer and modified or verified by a user. Such conventional segmentation techniques require user input and are therefore not suitable for integration into automatically performed clinical workflows. Summary of the invention

[0003] Deep learning techniques for segmenting anatomical structures from medical images of patients do not provide uncertainty measures associated with the segmentation results. Without knowing the uncertainty associated with the segmentation results, it is difficult to integrate such deep learning segmentation techniques into an automated clinical workflow. Advantageously, according to one or more embodiments, systems and methods for generating segmentation masks of anatomical structures along with uncertainty measures for the segmentation masks are provided.

[0004] According to one or more embodiments, a plurality of candidate segmentation masks of an anatomical structure are generated from an input medical image using one or more trained machine learning networks. A final segmentation mask of the anatomical structure is determined based on the plurality of candidate segmentation masks. An uncertainty metric associated with the final segmentation mask is determined based on the plurality of candidate segmentation masks. The final segmentation mask and / or the uncertainty metric are output.

[0005] In one embodiment, a plurality of candidate segmentation masks are generated by sampling a plurality of samples from a prior distribution, and for each respective sample in the plurality of samples, a candidate segmentation mask is generated based on the respective sample. The prior distribution is a probability distribution of segmentation variations of anatomical structures in the input medical image. In one embodiment, the one or more trained machine learning networks include a plurality of different trained machine learning networks, and the plurality of candidate segmentation masks are generated by, for each respective trained machine learning network in the plurality of different trained machine learning networks, using the respective trained machine learning network to generate a candidate segmentation mask of the anatomical structure from the input medical image.

[0006] In one embodiment, the final segmentation mask is determined as a mean of a plurality of candidate segmentation masks, and the uncertainty measure is determined as a variance of the plurality of candidate segmentation masks. In one embodiment, the uncertainty measure is determined by determining a probability associated with each pixel that may represent a boundary of the final segmentation mask and averaging the probabilities associated with each pixel that may represent a boundary of the final segmentation mask.

[0007] In one embodiment, user input may be requested for a final segmentation mask based on the uncertainty metric. In one embodiment, anomalies may be detected in an input medical image based on the uncertainty metric. In one embodiment, a probability associated with each pixel that may represent a segmentation mask boundary is compared to a threshold, a color is assigned to each pixel that may represent a segmentation mask boundary based on the comparison, and the color of each pixel that may represent a segmentation mask boundary is overlaid on the input medical image.

[0008] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A workflow for performing medical image segmentation with uncertainty estimation according to one or more embodiments is shown;

[0010] Figure 2 A method for medical image segmentation with uncertainty estimation according to one or more embodiments is shown;

[0011] Figure 3 A workflow for training a priori networks and segmentation networks according to one or more embodiments is shown;

[0012] Figure 4 An image illustrating anomaly detection based on an uncertainty metric determined according to one or more embodiments is shown; and

[0013] Figure 5 A high-level block diagram of a computer is shown. DETAILED DESCRIPTION

[0014] The embodiments described herein generally relate to methods and systems for medical image segmentation with uncertainty estimation. The embodiments described herein are described to give a visual understanding of such methods and systems. A digital image is typically composed of digital representations of one or more objects (or shapes). The digital representations of objects are typically described herein in terms of identifying and manipulating the objects. Such manipulations are virtual manipulations performed in the memory or other circuits / hardware of a computer system. Therefore, it is to be understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system.

[0015] Embodiments described herein provide segmentation of anatomical structures from medical images, along with uncertainty measures associated with the segmentation. Figure 1 A workflow 100 for performing medical image segmentation with uncertainty estimation is shown in accordance with one or more embodiments. In the workflow 100, a segmentation network 108 generates a plurality of candidate segmentation masks 110 of an anatomical structure based on 1) samples 104 sampled from a prior distribution 102 and 2) an image 106 of the anatomical structure. The prior distribution 102 is a probability distribution of segmentation variations of the anatomical structure in the image 106. A final segmentation mask 114 is determined as the mean of the plurality of candidate segmentation masks 110. The final segmentation mask 114 is overlaid on the image 106 to provide an output image 112 of the segmented anatomical structure. An uncertainty metric associated with the final segmentation mask 114 is also determined as the variance of the plurality of candidate segmentation masks 110. The uncertainty metric may be determined as a probability associated with pixels that may possibly represent a boundary of the final segmentation mask 114. The uncertainty metric may be represented by one or more colors (e.g., by comparing the probability to one or more thresholds) and overlaid on the image 106 to provide an output image 116 of the uncertainty metric associated with the segmented anatomical structure. It should be understood that references herein to image pixels may also refer to voxels of the image.

[0016] Advantageously, the uncertainty metric associated with the final segmentation mask 114 allows the final segmentation mask 114 to be incorporated into a clinical workflow (e.g., for disease diagnosis) to automatically perform the clinical workflow. For example, the final segmentation mask 114 can be used to automatically perform the clinical workflow without user input, where the uncertainty metric indicates a low level of uncertainty (e.g., based on one or more thresholds), while user input (e.g., verification or modification of the final segmentation mask 114) can be requested if the uncertainty metric indicates a high level of uncertainty.

[0017] Figure 2 A method 200 for medical image segmentation with uncertainty estimation according to one or more embodiments is shown. Figure 1The method 200 is described with reference to the workflow 100. The method 200 may be performed by one or more suitable computing devices, such as Figure 5 Computer 502.

[0018] At step 202, an input medical image of a patient's anatomy is received. In one embodiment, the input medical image is Figure 1 The anatomical structure depicted in the input medical image may include one or more organs, bones, blood vessels, or any other anatomical object of interest. The input medical image may have any suitable modality, such as, for example, x-ray, magnetic resonance imaging (MRI), ultrasound (US), computed tomography (CT), single photon emission computed tomography (SPECT), positron emission tomography (PET), or any other suitable modality or combination of modalities. The input medical image may have any suitable dimension, such as, for example, two-dimensional (2D) or three-dimensional (3D). The input medical image may be directly from an image acquisition device used to acquire the input medical image (such as, for example, Figure 5 Alternatively, the input medical image may be received by loading a medical image previously stored on a memory or storage device of a computer system (eg, a picture archiving and communication system PACS), or by receiving the input medical image data from a remote computer system via a network transmission.

[0019] At step 204, a plurality of candidate segmentation masks of anatomical structures are generated from the input medical image using one or more trained machine learning based segmentation networks. Figure 1 As shown in the workflow 100 of , the plurality of candidate segmentation masks are a plurality of candidate segmentation masks 110 generated from the image 106 using the segmentation network 108. The plurality of candidate segmentation masks of the anatomical structure are "candidates" because the candidate segmentation masks are possible segmentation masks of the anatomical structure. The plurality of candidate segmentation masks may include any number of candidate segmentation masks more than one. In one example, the plurality of candidate segmentation masks include 10 to 20 candidate segmentation masks, however, the number of candidate segmentation masks may be adjusted to include fewer candidate segmentation masks (for faster processing time) or more candidate segmentation masks (for greater accuracy and reliability). In one embodiment, the one or more trained machine learning-based segmentation networks are deep learning-based networks, such as, for example, an image-to-image network with a variational autoencoder (VAE). However, it should be understood that the one or more trained machine learning-based segmentation networks may include any suitable machine learning network.

[0020] In one embodiment, based on a plurality of samples sampled from a prior distribution, one or more trained machine learning-based segmentation networks are used to generate a plurality of candidate segmentation masks of the anatomical structure. Figure 1 As shown in the workflow 100 of , a plurality of candidate segmentation masks 110 are generated using a segmentation network 108 based on samples 104 sampled from a prior distribution 102. The prior distribution is a probability distribution (e.g., a Gaussian distribution) of segmentation variations of anatomical structures in an input medical image. The prior distribution is generated using a separate machine learning-based prior network (e.g., U-Net), which inputs the input medical image and outputs the prior distribution. In order to generate a plurality of candidate segmentation masks, a plurality of samples are sampled from the prior distribution. The one or more trained machine learning-based segmentation networks receive as input the input medical image and the plurality of samples, and output a candidate segmentation mask for each of the plurality of samples. Referring to FIG. Figure 3 Training of the one or more machine learning based segmentation networks and machine learning based prior networks is described.

[0021] In one embodiment, the one or more trained machine learning-based segmentation networks include a plurality of different machine learning networks, each machine learning network is trained to segment anatomical structures from an input medical image. Each of the plurality of trained machine learning networks receives an input medical image as input and outputs a candidate segmentation mask, thereby generating a plurality of candidate segmentation masks.

[0022] Figure 3 FIG. 3 shows a workflow 300 for training a priori network 302 and segmentation network 306 according to one or more embodiments. In one embodiment, the priori network 302 is trained on Figure 2 The method 200 is used to generate a prior network based on machine learning for generating a prior distribution, and the segmentation network 306 is in Figure 2One or more machine learning-based segmentation networks in the method 200 of . The prior network 302 is learned together with the segmentation network 306 using the posterior network 304 as a "teacher" network. The posterior network 304 is similar to the prior network 302, however the posterior network 304 receives one or more training images and a ground truth segmentation mask 312 as input and outputs a posterior distribution 314. The prior network 302 receives only one or more training images 308 as input and generates a prior distribution 310 as output. The prior network 302 is trained to generate a prior distribution 310 that is the same distribution as the posterior distribution 314 using a loss function 320. The loss function 320 can be any suitable loss function for minimizing the difference between probability distributions, such as, for example, KL (Kulbach-Leibler) divergence. The segmentation network 306 is trained simultaneously with the posterior network 304, and the prior network 302 learns to emulate the posterior network 304, but using only the training images (because the ground truth is not available during the inference phase). The segmentation network 306 receives one or more training images 316 as input and outputs a predicted segmentation mask 318 by sampling the posterior distribution 314. The segmentation network 306 is trained to generate a predicted segmentation mask 318 that is the same as the ground truth segmentation mask 322 using a loss function 324. The loss function 324 can be any suitable loss function for minimizing the difference between segmentation masks. During the inference phase, the prior network replaces the posterior network.

[0023] At step 206, a final segmentation mask of the anatomical structure is determined based on the plurality of candidate segmentation masks. In one embodiment, the final segmentation mask of the anatomical structure is calculated as the mean of the plurality of candidate segmentation masks. In other embodiments, the final segmentation mask of the anatomical structure is calculated as the median or model (or any other suitable metric) of the plurality of candidate segmentation masks. For example, Figure 1 As shown in the workflow 100 , the final segmentation mask 114 is calculated as the average of the multiple segmentation masks 110 .

[0024] Each candidate segmentation mask is obtained by thresholding the final output of the segmentation network. Before thresholding, the candidate segmentation mask can be interpreted as the pixel-wise activations for each label. The mean of the activations is determined pixel-wise across all candidate segmentation masks, and each pixel is assigned the label with the highest mean activation in that pixel. In some embodiments, the median or mode of activations (or any other suitable metric) can instead be determined pixel-wise across all candidate segmentation masks.

[0025] The final segmentation mask can be represented as a matrix of size C×n×n, where C is the number of possible labels. So, in one example, for an image of size 256×256 with labels "background" and "heart" (or any other anatomical structure), the size of the matrix will be 2×256×256. For each pixel, there are two activation values: one for background and one for heart. The label with the higher value is assigned to the pixel. For N candidate segmentation masks, a matrix of size N×2×256×256 is determined. So, in the case where there are, for example, 20 candidate segmentation masks, there are 20 values ​​for the label heart for each pixel. The mean (or median or mode or any other suitable metric) is calculated from the 20 values, resulting in a final activation for the label heart for the final segmentation mask. A similar process can be followed for any other label.

[0026] At step 208, an uncertainty metric associated with a final segmentation mask of the anatomical structure is determined based on the plurality of candidate segmentation masks. In one embodiment, the uncertainty metric associated with the final segmentation mask is calculated as the variance of the plurality of candidate segmentation masks. Figure 1 As shown in the workflow 100 of , the uncertainty metric associated with the final segmentation mask 114 is calculated as the variance of the plurality of candidate segmentation masks 110 .

[0027] In one embodiment, similar to the calculation of the mean described above with respect to step 206, the variance of the activations is calculated pixel-wise across all candidate segmentation masks to obtain a variance map for each label of the candidate segmentation mask. In the example described above with respect to step 206, where there are 20 candidate segmentation masks, the variance can be calculated for each pixel across 20 values ​​of the label heart, resulting in a variance map of size 2×256×256 for each label.

[0028] In one embodiment, the uncertainty measure is determined as a probability associated with each pixel that can possibly represent the boundary of the final segmentation mask. In another embodiment, the uncertainty measure is determined as a quantification of the uncertainty associated with the final segmentation mask. For example, the uncertainty associated with the final segmentation mask can be quantified by averaging the probabilities associated with each pixel that can possibly represent the boundary of the final segmentation mask. It should be understood that the uncertainty measure can be in any other suitable format.

[0029] At step 210, the final segmentation mask and / or the uncertainty metric associated with the final segmentation mask are output. For example, the final segmentation mask and / or the uncertainty metric associated with the final segmentation mask can be output by: in a computer system (e.g., Figure 5The final segmentation mask and / or the uncertainty metric associated with the final segmentation mask are displayed on a display device of a computer 502 of the computer system, the final segmentation mask and / or the uncertainty metric associated with the final segmentation mask are stored in a memory or storage device of the computer system, or the final segmentation mask and / or the uncertainty metric associated with the final segmentation mask are transmitted to a remote computer system. The final segmentation mask and / or the uncertainty metric associated with the final segmentation mask can be output in any suitable format. In one embodiment, the final segmentation mask is output by displaying the final segmentation mask overlaid on the input medical image (e.g., Figure 1 ), and outputting the uncertainty metric associated with the final segmentation mask by displaying the uncertainty metric associated with the final segmentation mask overlaid on the input medical image (as depicted in the output image 112 in FIG. 1 ). Figure 1 116 in FIG. 11 ).

[0030] In one embodiment, an uncertainty metric associated with the final segmentation mask is overlaid on top of an input medical image by comparing a probability associated with each pixel that may potentially represent a boundary of the final segmentation mask to one or more thresholds, assigning a color to the pixel based on whether the probability associated with the pixel satisfies the one or more thresholds, and overlaying the assigned color of the pixel on the input medical image. For example, pixels associated with probabilities that satisfy the thresholds may be considered to have low uncertainty and may be assigned a green color, while pixels associated with probabilities that do not satisfy the thresholds may be considered to have high uncertainty and may be assigned a red color, thereby resulting in an uncertainty map that provides a visual estimate of uncertainty about the accuracy of the final segmentation mask.

[0031] In one embodiment, the uncertainty measure associated with the final segmentation mask is overlaid on the input medical image by averaging the probabilities of a pixel region (e.g., a 3 pixel×3 pixel region) that may possibly represent a boundary of the final segmentation mask, comparing the average probability of the region to one or more thresholds, assigning colors to pixels (e.g., all pixels in the region or a center pixel of the region) based on whether the average probability satisfies the one or more thresholds, and overlaying the colors assigned to the pixels on the input medical image to provide an uncertainty map.

[0032] In one embodiment, regions of the final segmentation mask having a high level of uncertainty (e.g., based on one or more thresholds) may be presented to a user to request user input (e.g., to verify or modify the boundaries of the final segmentation mask) or otherwise direct the user's attention to such regions. In one embodiment, the extent to which the boundaries of the final segmentation mask in certain regions may be modified is based on the level of uncertainty associated with the final segmentation mask in such regions.

[0033] In one embodiment, the uncertainty metric associated with the final segmentation mask is used to determine an uncertainty metric associated with a downstream medical imaging analysis task in a clinical workflow. For example, the final segmentation mask can be used to determine the ejection fraction of the heart, and the range of variation (or confidence interval) associated with the ejection fraction can be determined based on the uncertainty metric associated with the final segmentation mask.

[0034] In one embodiment, the uncertainty metric associated with the final segmentation mask can be used to detect anatomical abnormalities in the input medical image or algorithmic failures of the one or more trained machine learning networks. Figure 4 Images illustrating anatomical anomaly detection based on uncertainty metrics are shown. Figure 4 An original input medical image 402 of a patient's heart, an output image 404 of a final segmentation mask 408 overlaid on the input medical image 402, and an output image 406 of uncertainty measures are shown. The input medical image 402 depicts leakage of the pericardium mask to the ribs. Figure 2 The method 200 uses one or more trained machine learning networks to generate output image 404 and output image 406. Based on the high level of uncertainty in region 410, a leak of the pericardium mask is detected in region 410, as depicted in output image 406.

[0035] In one embodiment, the uncertainty metric associated with the final segmentation mask can be used to identify data on which the one or more trained machine learning networks were not trained. For example, a high level of uncertainty in an input medical image can indicate that the one or more trained machine learning networks were not trained on the scene depicted in the input medical image. Thus, the uncertainty metric provides a method to identify and selectively augment the training data to obtain a complete representation of real-life cases.

[0036] In one embodiment, the uncertainty metric associated with the final segmentation mask can be used for robust segmentation or reconstruction of anatomical structures. Given the uncertainty metric, the prior shape model can be deformed to match regions of the final segmentation mask with low levels of uncertainty (high levels of confidence) while maintaining its shape in regions of the final segmentation mask with high levels of uncertainty (low levels of confidence).

[0037] In one embodiment, the uncertainty measure map is used to automatically create a plurality of suggested segmentation masks, which can be presented to the user for selection of a final segmentation mask. In this way, user modification of the final segmentation mask is avoided.

[0038] In one embodiment, the uncertainty metric associated with the final segmentation mask can be used to reject certain input medical images, thereby providing a fully automated clinical workflow with zero error. For example, the uncertainty metric associated with the final segmentation mask of the anatomical structure in the input medical image can be compared with a threshold, and if the uncertainty level associated with the final segmentation mask does not meet the threshold, the input medical image can be rejected.

[0039] The systems, apparatus, and methods described herein (including machine learning models) can be implemented using digital circuits or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. The computer may also include or be coupled to one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0040] The systems, devices, and methods described herein (including machine learning models) can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0041] The systems, devices, and methods (including machine learning models) described herein can be implemented in a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with a server via a web browser application that resides and operates on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to the (multiple) client computers. The server can also transmit data suitable for causing the client computer to perform a specified function, such as performing a calculation, displaying specified data on a screen, etc. For example, the server can transmit data suitable for causing the client computer to perform one or more steps or functions (including Figure 1-3 Some steps or functions of the methods and workflows described herein (including Figure 1-3 One or more steps or functions of the method and workflow described herein may be performed by a server or another processor in a network-based cloud computing system. Figure 1-3 One or more steps of the method and workflow described herein may be performed by a client computer in a network-based cloud computing system. Figure 1-3 One or more steps of the process may be performed by a server and / or a client computer in any combination in a network-based cloud computing system.

[0042] The systems, apparatus, and methods (including machine learning models) described herein may be implemented using a computer program product tangibly embodied in an information carrier (e.g., embodied in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1-3 One or more steps or functions of a program) can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a certain activity or produce a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0043] Figure 5A high-level block diagram of an example computer 502 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 502 includes a processor 504 operatively coupled to a data storage device 512 and a memory 510. The processor 504 controls the overall operation of the computer 502 by executing computer program instructions that define such operations. The computer program instructions may be stored in the data storage device 512 or other computer-readable medium and loaded into the memory 510 when execution of the computer program instructions is desired. Thus, Figure 1-3 The method and workflow steps or functions of the present invention may be defined by computer program instructions stored in the memory 510 and / or the data storage device 512 and controlled by the processor 504 executing the computer program instructions. For example, the computer program instructions may be implemented as a program programmed by a person skilled in the art to execute Figure 1-3 Thus, by executing the computer program instructions, the processor 504 performs Figure 1-3 The computer 502 may also include one or more network interfaces 506 for communicating with other devices via a network. The computer 502 may also include one or more input / output devices 508 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 502.

[0044] Processor 504 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of computer 502. For example, processor 504 may include one or more central processing units (CPUs). Processor 504, data storage device 512, and / or memory 510 may include, be supplemented by, or incorporate one or more application specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs).

[0045] The data storage device 512 and the memory 510 each include a tangible, non-transitory computer-readable storage medium. The data storage device 512 and the memory 510 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM) disks, or other non-volatile solid-state storage devices.

[0046] Input / output devices 508 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 508 may include: a display device for displaying information to a user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; a keyboard; and a pointing device through which a user can provide input to computer 502, such as a mouse or trackball.

[0047] The image acquisition device 514 can be connected to the computer 502 to input image data (e.g., medical images) to the computer 502. It is possible to implement the image acquisition device 514 and the computer 502 as one device. It is also possible for the image acquisition device 514 and the computer 502 to communicate wirelessly via a network. In a possible embodiment, the computer 502 can be remotely located relative to the image acquisition device 514.

[0048] Any or all of the systems and devices discussed herein may be implemented using one or more computers, such as computer 502 .

[0049] Those skilled in the art will recognize that the actual computer or computer system may have other structures and may include other components, and for the purpose of illustration, Figure 5 is a high-level representation of some of the components of such a computer.

[0050] The foregoing detailed description is to be understood in every respect as illustrative and exemplary, rather than restrictive, and the scope of the invention disclosed herein is not determined by the detailed description, but by the claims interpreted in accordance with the full scope permitted by patent law. It is to be understood that the embodiments shown and described herein are merely illustrations of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method comprising: generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using one or more trained machine learning networks; determining a final segmentation mask of the anatomical structure by calculating the mean of activation values ​​of corresponding pixels across the plurality of candidate segmentation masks, the candidate segmentation masks being interpreted as pixel-wise activations for each label, the mean of the activations being determined pixel-wise across all candidate segmentation masks, and each pixel being assigned the label having the highest mean activation in that pixel; as well as The uncertainty metric associated with the final segmentation mask is calculated as the variance of the plurality of candidate segmentation masks by calculating the variance of the values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map.

2. The method of claim 1 , wherein generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using one or more trained machine learning networks comprises: Sampling a plurality of samples from a prior distribution, wherein the prior distribution is a probability distribution of segmentation variations of anatomical structures in an input medical image; and For each corresponding sample in the plurality of samples, a candidate segmentation mask is generated based on the corresponding sample.

3. The method of claim 1 , wherein the one or more trained machine learning networks comprises a plurality of different trained machine learning networks, and generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using the one or more trained machine learning networks comprises: For each respective trained machine learning network in the plurality of different trained machine learning networks, a candidate segmentation mask of the anatomical structure is generated from the input medical image using the respective trained machine learning network.

4. The method of claim 1 , wherein determining an uncertainty metric associated with a final segmentation mask by computing a variance of values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map comprises: A probability associated with each pixel that may represent a boundary of the final segmentation mask is determined.

5. The method of claim 4, wherein determining an uncertainty metric associated with a final segmentation mask by computing a variance of values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map comprises: The probabilities associated with each pixel that may represent the boundary of the final segmentation mask are averaged.

6. The method according to claim 4, further comprising: comparing a probability associated with each pixel that may represent a boundary of a final segmentation mask to a threshold; Based on the comparison, assigning a color to each pixel that may represent a boundary of a final segmentation mask; and Each pixel that may represent the boundary of the final segmentation mask on the input medical image is covered with the color.

7. The method according to claim 1, further comprising: Based on the uncertainty measure, user input is requested for the final segmentation mask.

8. The method according to claim 1, further comprising: Detecting anomalies in input medical images based on uncertainty measures.

9. The method according to claim 1, further comprising: Based on the uncertainty measure, training data is identified for further training the one or more trained machine learning networks.

10. An apparatus comprising: means for generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using one or more trained machine learning networks; means for determining a final segmentation mask of the anatomical structure by computing a mean of activation values ​​of corresponding pixels across the plurality of candidate segmentation masks, the candidate segmentation masks being interpreted as pixel-wise activations for each label, the mean of the activations being determined pixel-wise across all candidate segmentation masks, and each pixel being assigned the label having the highest mean activation in that pixel; as well as Means for generating a variance map by computing the variance of values ​​of corresponding pixels across the plurality of candidate segmentation masks, the uncertainty metric associated with the final segmentation mask being computed as the variance of the plurality of candidate segmentation masks.

11. The apparatus of claim 10, wherein the means for generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using one or more trained machine learning networks comprises: means for sampling a plurality of samples from a prior distribution, the prior distribution being a probability distribution of segmentation variations of anatomical structures in an input medical image; and Means for generating, for each respective sample of the plurality of samples, a candidate segmentation mask based on the respective sample.

12. The apparatus of claim 10, wherein the one or more trained machine learning networks comprises a plurality of different trained machine learning networks, and the means for generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using the one or more trained machine learning networks comprises: Means for generating, for each respective trained machine learning network among the plurality of different trained machine learning networks, a candidate segmentation mask of an anatomical structure from an input medical image using the respective trained machine learning network.

13. A non-transitory computer readable medium storing computer program instructions which, when executed by a processor, cause the processor to perform operations comprising: generating a plurality of candidate segmentation masks of anatomical structures from an input medical image using one or more trained machine learning networks; determining a final segmentation mask of the anatomical structure by calculating the mean of activation values ​​of corresponding pixels across the plurality of candidate segmentation masks, the candidate segmentation masks being interpreted as pixel-wise activations for each label, the mean of the activations being determined pixel-wise across all candidate segmentation masks, and each pixel being assigned the label having the highest mean activation in that pixel; as well as The uncertainty metric associated with the final segmentation mask is calculated as the variance of the plurality of candidate segmentation masks by calculating the variance of the values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map.

14. The non-transitory computer-readable medium of claim 13, wherein determining an uncertainty metric associated with a final segmentation mask by computing a variance of values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map comprises: A probability associated with each pixel that may represent a boundary of the final segmentation mask is determined.

15. The non-transitory computer-readable medium of claim 14, wherein determining an uncertainty metric associated with a final segmentation mask by computing a variance of values ​​of corresponding pixels across the plurality of candidate segmentation masks to generate a variance map comprises: The probabilities associated with each pixel that may represent the boundary of the final segmentation mask are averaged.

16. The non-transitory computer readable medium of claim 13, the operations further comprising: Based on the uncertainty measure, user input is requested for the final segmentation mask.

Images