Automated organ segmentation output quality assessment

Abstract

An automated post-processing tool to assess the quality of deep learning-based organ segmentation in medical images is described. According to one example, a method includes determining, by a system including a processor, current values of defined features of respective segmentation masks generated via automatic segmentation of medical image data for different anatomical structures included in the medical image data. The method further includes determining, by the system, a respective correspondence metric between the current value and a corresponding reference value determined for the defined feature; determining one or more quality metrics for the automatic segmentation based on the respective correspondence metrics; quality assessment report data is generated for the automatic segmentation, the quality assessment report data including the one or more quality metrics in a standard format displayable by standard clinical software.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Application Serial No. 18 / 068,871, filed on December 20, 2022, entitled "AUTOMATED ORGAN SEGMENTATION OUTPUT QUALITY ASSESSMENT". The entire content of the foregoing application is incorporated herein by reference. Technical Field

[0003] This application relates to medical image processing and, more particularly, to an automated post - processing tool for assessing the quality of deep - learning - based organ segmentation in medical images. Background Art

[0004] Radiotherapy is one of the primary modalities for treating cancers in complex anatomical regions such as the head and neck. Due to advancements in adjusting radiation doses for the morphologically complex head and neck anatomy and pathologies, intensity - modulated radiotherapy (IMRT) has become the preferred radiotherapy method for head and neck cancers. In inverse optimization, the protection of organs at risk (OARs) is achieved by imposing dose penalties on the corresponding contoured volumes. Insufficient segmentation of OARs will expose them to unnecessarily high doses, but over - segmentation of OARs may render the optimization objective unachievable. Thus, the effectiveness of IMRT depends on the accuracy of OAR segmentation, which is typically performed manually by oncologists and dosimetrists. However, the manual process is not only time - consuming but also introduces inconsistencies due to both inter - patient variability and inter - observer variability.

[0005] To alleviate these problems, automated medical image segmentation has been proposed. In recent years, deep - learning - based methods, specifically convolutional neural network - based methods, have shown great promise in medical image segmentation. Applications include object or lesion classification, organ or lesion detection, organ and lesion segmentation, registration, and other tasks. However, for successful application in clinical applications such as IMRT, automated segmentation needs to address inter - patient variability and a large number of anatomical structures in relatively small regions, each of which poses specific challenges.

[0006] Although deep - learning - based segmentation techniques have outperformed statistically - based shape - appearance automatic segmentation methods, due to inter - patient variability, variations in acquisition protocols / parameters, and other factors, the accuracy of deep - learning - based segmentation results may not be sufficient for clinical applications such as IMRT. Therefore, techniques are needed to automatically evaluate the output accuracy of such segmentation models before using the segmentation results in clinical applications. Summary of the Invention

[0007] The following presents the invention content to provide a basic understanding of one or more embodiments of the present invention. The invention content is not intended to identify key or important elements, nor to depict any scope of different embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a preamble to the more detailed description presented later. In one or more embodiments described herein, systems, computer-implemented methods, devices, and / or computer program products are described that provide automated post-processing tools to evaluate the quality of deep learning-based organ segmentation in medical images.

[0008] According to one embodiment, a system is provided that includes: a memory that stores computer-executable components; and a processor that executes the computer-executable components stored in the memory. The computer-executable components include a feature evaluation component that determines a current value of a defined feature of a corresponding organ segmentation mask generated for different organs included in the medical image data via automatic segmentation of the medical image data. The computer-executable components further include a quality evaluation component that determines a corresponding correspondence metric between the current value and a corresponding reference value determined for the defined feature, and determines one or more quality metrics of the automatic segmentation based on the corresponding correspondence metric.

[0009] In some embodiments, the elements described in the disclosed systems and methods may be embodied in different forms, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An example system facilitating automated organ segmentation output quality evaluation according to one or more embodiments of the disclosed subject matter is presented.

[0011] Figure 2 An example multi-organ segmentation of magnetic resonance (MR) image data and computed tomography (CT) image data according to one or more embodiments of the disclosed subject matter is illustrated.

[0012] Figure 3 A high-level flowchart of an example process for determining reference feature values for a multi-organ segmentation model according to one or more embodiments of the disclosed subject matter is presented.

[0013] Figure 4 A high-level flowchart of an example computer-implemented process for automatically evaluating the output quality of a multi-organ segmentation model according to one or more embodiments of the disclosed subject matter is presented.

[0014] Figure 5A flow chart of an example process for determining absolute feature correspondence values and relative feature correspondence values in accordance with one or more embodiments of the disclosed subject matter is presented.

[0015] Figure 6 Depicted is a table illustrating example automatic segmentation quality assessment results in accordance with one or more embodiments of the disclosed subject matter.

[0016] Figure 7 A flow chart of an example process for evaluating output quality of multi-organ automatic segmentation in accordance with one or more embodiments of the disclosed subject matter is presented.

[0017] Figure 8 A flow chart of another example process for evaluating output quality of multi-organ automatic segmentation in accordance with one or more embodiments of the disclosed subject matter is presented.

[0018] Figure 9 Another example system that facilitates automated organ segmentation output quality assessment in accordance with one or more embodiments of the disclosed subject matter is presented.

[0019] Figure 10 A block diagram illustrating an example non-limiting operating environment in which one or more embodiments described herein may be facilitated. DETAILED DESCRIPTION

[0020] The following specific embodiments are merely illustrative and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" section or "content of the invention" section or "specific embodiments" section.

[0021] The disclosed subject matter relates to systems, computer-implemented methods, apparatus, and / or computer program products that facilitate automatic evaluation of the output quality of an organ segmentation model. The disclosed technology is advanced by using deep learning-based automatic segmentation of OARs in MR data to guide the performance of IMRT. There are many reasons why deep learning multi-organ segmentation in MR may lead to inaccurate organ contours. For example, a multi-organ segmentation model may have been trained for a specific MR sequence and may fail when the input is not acquired using the correct imaging protocol. Because it is very challenging to identify all variants of MR sequences, correct segmentation cannot be guaranteed for all variants. The disclosed technology can be used to automatically detect and identify abnormal segmentation results, and notify appropriate entities (e.g., oncologists, dosimetry scientists, etc.) accordingly before the results are used in clinical applications such as IMRT.

[0022] For this purpose, the disclosed technology provides an automated post - processing algorithm to evaluate the quality of deep - learning - based organ segmentation. This solution aims to filter out individual automatic segmentations as well as anomalies in the entire case and thus prevent inaccurate outputs from being exposed to medical professionals.

[0023] The automated post - processing algorithm uses the concept of comparing several characteristics of the structure of the segmentation to be analyzed with expected values obtained beforehand during the training process, estimating the probability of their quality and aggregating them together. The tool is generic and can be extended to additional characteristics as well as various aggregation methods for combining features to optimize different usage scenarios.

[0024] The disclosed solution is also modality - independent. Thus, although it was developed for MR segmentation, it can be applied to any other type of medical image. In this regard, the types of medical images processed / analyzed using the techniques described herein can include images captured using various types of image - capture modalities. For example, medical images can include (but are not limited to): radiotherapy (RT) images, X - ray (XR) images, digital radiography (DX) X - ray images, X - ray angiography (XA) images, panoramic X - ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis devices), magnetic resonance imaging (MRI or simply MR) images (including T1 - weighted images and T2 - weighted images), ultrasound (US) images, color - flow Doppler (CD) images, positron emission tomography (PET) images, single - photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, optical images, and DWI, etc. Medical images can also include synthetic versions of native medical images, such as synthetic X - ray (SXR) images, modified or enhanced versions of native medical images, enhanced versions of native medical images, and similar versions produced using one or more image - processing techniques. The types of medical - image data processed / analyzed herein can include two - dimensional (2D) image data, three - dimensional image data (3D) (e.g., volume representations of anatomical regions of the body), and combinations thereof.

[0025] In an example implementation, a system is provided that includes a memory storing computer-executable components and a processor that executes the computer-executable components stored in the memory. The computer-executable components include a feature evaluation component that determines a current value of a defined feature of a corresponding segmentation mask generated for different anatomical structures included in medical image data via automatic segmentation of the medical image data. The computer-executable components further include a quality evaluation component that determines a corresponding correspondence metric between the current value and a corresponding reference value determined for the defined feature, and determines one or more quality metrics for the automatic segmentation based on the corresponding correspondence metric. The computer-executable components further include a reporting component that generates quality assessment report data for the automatic segmentation that includes the one or more quality metrics. In various implementations, the one or more quality metrics include quality metrics for a particular structure or set of structures represented by the segmentation mask or set of organ segmentation masks, and wherein the reporting component integrates the quality metrics with the particular structure or set of structures in a standard clinical data format (such as the Digital Imaging and Communications in Medicine (DICOM) format) or another standard clinical reporting format.

[0026] In some specific implementations, the one or more quality metrics include structure-specific quality metrics for each of the corresponding segmentation masks, the structure-specific quality metrics reflecting the quality metrics of each of the corresponding segmentation masks. In some aspects, the quality evaluation component determines the structure-specific quality metrics for each of the corresponding segmentation masks based on an aggregated correspondence value for a plurality of defined features associated with each of the corresponding segmentation masks. The one or more quality metrics may further include an overall quality metric for the automatic segmentation determined by the evaluation component based on an aggregation of each of the organ-specific quality metrics.

[0027] In various embodiments, the defined features include absolute features representing independent characteristics of the corresponding organ segmentation masks and relative features representing relative characteristics of different pairs of the corresponding organ segmentation masks. For example, the absolute features may include one or more geometric features related to the size or shape of the corresponding organ segmentation mask (e.g., volume, surface area, centroid coordinates, etc.). The absolute features may also include latent features extracted from the segmentation output using machine learning methods (PCA, neural networks, etc.), which may not be meaningful to a human observer but can be used to characterize the correctness of the segmentation output. The relative features may include the relative distances between each of the segmentation masks included in different pairs (e.g., the corresponding mask centroid coordinates). In various embodiments, the corresponding reference values for the defined features (e.g., absolute and relative features) are based on the ground truth organ segmentation data for different organs as depicted in the training medical image data used to train the corresponding organ segmentation model for performing automatic segmentation. In some specific implementations, the feature evaluation component determines the corresponding reference values for the defined features based on an analysis of the ground truth organ segmentation data.

[0028] In one or more embodiments, the reporting component may store the quality assessment report data together with the segmentation data for the medical image data in one or more medical image data repositories, where the segmentation data includes the corresponding segmentation masks. The computer-executable component may also include a rendering component that presents the corresponding segmentation masks and the quality assessment report data in association with a clinical procedure for a patient from whom the medical image data is captured, via a device display associated with one or more clinical entities, using the corresponding organ segmentation masks. The computer-executable component may also include an alerting component that generates alert data based on one or more quality metrics failing to meet threshold quality metrics and provides the alert data to one or more devices associated with one or more clinicians. For example, the alert data may be related to a particular structure or set of structures represented by the segmentation mask or group of segmentation masks, and wherein the alerting component integrates the alert data with the particular structure or set of structures in a standard clinical data format. In this regard, the alert data can be provided to the end user without introducing any changes in the current clinical workflow (e.g., reading an additional report on the output quality). This can be addressed by including keywords (such as "error", "warning", "empty", etc.) in the structure name or set name or sequence description of the set of structures, which are standard DICOM tags and are displayed by default in any radiotherapy (RT) planning software.

[0029] In some embodiments, the computer-executable components further include a recommendation component that determines one or more clinical workflow actions to be performed based on one or more quality metrics failing to meet a threshold quality metric, and provides recommendation data identifying the one or more clinical workflow actions to one or more devices associated with one or more clinicians.

[0030] One or more embodiments will now be described with reference to the accompanying drawings, in which like reference numerals are always used to denote like elements. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a more thorough understanding of one or more embodiments. However, it is evident that in various instances, one or more embodiments may be practiced without these specific details.

[0031] Turning now to the drawings, Figure 1 A block diagram of an example non-limiting system 100 facilitating automated organ segmentation output quality assessment in accordance with one or more embodiments of the disclosed subject matter is illustrated. Embodiments of the systems described herein may include one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). When executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), such components may cause the one or more machines to perform the operations.

[0032] For example, system 100 includes a computing device 108 that includes a number of computer-executable components, including a segmentation component 110, a feature assessment component 114, a quality assessment component 116, a reporting component 118, an alert component 120, and a rendering component 132. These computer / machine-executable components (and other components described herein) may be stored in a memory associated with one or more machines. The memory is also operatively coupled to at least one processor such that the components may be executed by the at least one processor to perform the operations described. For example, in some embodiments, these computer / machine-executable components may be stored in the memory 124 of computing device 108, which may be coupled to processing unit 136 to execute these computer / machine-executable components. Examples of the memory and processor (or processing unit) and other suitable computer or computing-based elements may be found and may be incorporated in conjunction with Figure 10 found, and may be incorporated in conjunction with Figure 1or one or more of the other systems or components shown and described herein. Memory 118 may also store various information received, used, and / or generated by computing device 108 in connection with evaluating the quality of automatically segmented medical image data. In the illustrated embodiment, the information includes, but is not limited to, segmentation model 126, reference eigenvalue 128, and quality assessment guideline 130.

[0033] System 100 also includes medical image database 102 and medical imaging system 104. Medical image database 102 may correspond to any suitable database for storing medical image data 106 for processing by computing device 108, such as a medical image database associated with a picture archiving and communication system (PACS), etc. Medical imaging system 104 may correspond to any medical imaging system capable of capturing and generating (e.g., via medical image reconstruction processing techniques) medical image data 106 for processing by computing device 108. Medical image data 106 may include any type of medical image data capable of being automatically segmented via one or more segmentation models 126, including medical image data acquired via any medical imaging modality (e.g., MR, CT, PET, SPECT, US, XR, etc.) and depicting any anatomical region of the body.

[0034] In this regard, according to system 100, computing device 108 may receive medical image data 106 provided by medical image database 102 and / or medical imaging system 104, and process medical image data 106 via one or more segmentation models 126 (e.g., using segmentation component 110) to generate segmentation result data 112. Additionally or alternatively, computing device 108 may receive segmentation result data 112 generated by applying one or more segmentation models 126 to medical image data 106 by another system / device.

[0035] The segmentation result data 112 may include information defining the boundaries or contours of one or more defined anatomical objects depicted in the medical image data 106 relative to the original image data from which the segmentation result data is segmented, in 2D and / or 3D (depending on the type of image from which it is segmented). For example, the segmentation result data 112 may include one or more segmentation masks generated for one or more anatomical objects, image marker data including boundary lines, points, circles, etc. of the anatomical objects, and / or image data of the anatomical objects extracted from the input image. The segmentation result data 112 may also include the original medical image data from which the corresponding segmentation is generated. The one or more anatomical objects may include any anatomical object having consistent or substantially consistent geometric properties in all patients or a defined subgroup of patients (e.g., grouped by age or another demographic factor), such as an organ opposite a lesion. However, the anatomical objects are not limited to organs and may include any defined anatomical object or feature. It should be understood that the number and type of segmented anatomical objects may vary depending on the input image data type, the depicted anatomical structure, and the specific anatomical objects that the one or more segmentation models 126 are configured to segment. In various embodiments, the one or more segmentation models 126 may correspond to a multi-organ segmentation model configured to segment multiple different organs (and / or other types of anatomical objects) depicted in the medical image data 106. The number of different anatomical objects segmented for the medical image data 106 may vary. For example, in some specific implementations where the medical image data 106 includes MR head and neck scan data for a patient, the one or more segmentation models 126 may correspond to a multi-organ segmentation model that segments up to about 30 different defined anatomical structures included in the MR head and neck scan data. However, the disclosed techniques may also be applied to evaluate segmentation results that include only a single segmented anatomical object.

[0036] Figure 2 Examples of multi-organ segmentation of magnetic resonance (MR) image data and computed tomography (CT) image data in accordance with one or more embodiments of the disclosed subject matter are illustrated. In Figure 2In the example shown, the multi-organ segmentation data includes MR segmentation data 202 generated from MR image data 201 corresponding to an MR brain scan of a patient. The multi-organ segmentation data also includes CT segmentation data 204 generated from CT image data 203 corresponding to a CT brain scan of the patient. The MR segmentation data 202 provides segmentation data in the form of segmentation boundaries marked or contours surrounding various defined anatomical objects (e.g., tissues, blood vessels, regions of interest (ROIs), etc.) depicted in the MR image data 201. The CT segmentation data 204 provides segmentation data in the form of a segmentation mask generated on the eyeballs and optic nerves as depicted in the CT image data 203. The MR segmentation data 202 and the CT segmentation data 204 are two exemplary illustrations corresponding to the segmentation result data 112. It should be understood that Figure 2 the segmentation data depicted in

[0037] is merely exemplary, and various different types of segmentation data results for various different anatomical objects associated with all regions of the body are envisioned. Figure 1 Referring again to Figure 2 , according to the disclosed technique, the segmentation result data set 112 corresponds to anatomical object segmentations (e.g., such as

[0038] The disclosed techniques utilize the ground truth segmentation data to define the expected characteristics of each anatomical object to be segmented by a corresponding segmentation model. For example, when applied to multi-organ segmentation of medical image data, the expected characteristics may include the expected 2D and / or 3D geometric properties (e.g., size, shape, surface area, volume) of each organ segmentation of the multiple organs to be segmented. In this regard, using the ground truth (GT) segmentation data for each organ / object applied to multiple different medical images corresponding to different patients included in the training data, the disclosed techniques define one or more features (e.g., geometric properties such as volume, surface area, etc.) and the expected values of the features (e.g., the average volume observed in the GT examples for the organ / object, the average surface area observed in the GT examples for the organ / object, etc.) for each organ or object segmentation. This information is represented as reference feature values 128 in system 100. Once the features and the expected values of the features for each object segmentation have been defined in the reference feature values 128 and the corresponding segmentation model has been trained, the segmentation component 110 can apply the segmentation model to new medical image data (e.g., medical image data 106) to generate segmentation result data 112, which includes the segmentation generated by the automated segmentation model for the corresponding defined anatomical object / organs. The feature evaluation component 114 also determines the current values of the defined features of the corresponding organ / object segmentation included in the segmentation result data 112 (e.g., the segmentation mask and / or the information defining the boundary contour of the corresponding anatomical object / organs).

[0039] The quality assessment component 116 also determines the corresponding correspondence metric between the current value and the corresponding reference value determined for the defined feature as provided in the reference feature values 128 (e.g., the correspondence metric between the current volume of the segmented organ and the expected volume of the segmented organ), and determines one or more quality metrics of the automatic segmentation based on the corresponding correspondence metric (e.g., whether the organ segmentation is accurate as the current volume corresponds to the expected volume) and predefined threshold criteria for acceptable (e.g., corresponding to a correct / accurate segmentation) and unacceptable (e.g., corresponding to an incorrect / inaccurate segmentation) correspondence metrics (e.g., as provided in the quality assessment guidelines 130).

[0040] In some specific implementations, one or more quality metrics include organ-specific quality metrics for each organ segmentation mask in the corresponding organ segmentation masks, and the organ-specific quality metric reflects the quality metric of each organ segmentation mask in the corresponding organ segmentation masks. In some aspects, the quality assessment component 116 determines the organ-specific quality metric for each organ segmentation mask in the corresponding organ segmentation masks based on the aggregated correspondence values for a plurality of defined features associated with each organ segmentation mask in the corresponding organ segmentation masks. One or more quality metrics may also include an overall quality metric of the automatic segmentation determined by the quality assessment component based on the aggregation of each organ-specific quality metric in the organ-specific quality metrics.

[0041] In this regard, as described in more detail below, the quality assessment component 116 may evaluate the quality of the automatic segmentation (i.e., the segmentation result data 112) at the individual organ or object level, thereby providing one or more quality metrics (e.g., accuracy or correctness level) for each organ / object included in the segmentation result data 112 (e.g., assuming a multi-organ / object segmentation model). For each segmented organ / object, the quality assessment may be based on a plurality of different defined features, including absolute features and relative features, as described in more detail below. The quality assessment component 116 may also use various aggregation algorithms customized for different usage scenarios to evaluate the overall quality of the automatic segmentation based on the aggregated quality metrics of the individual organs / objects.

[0042] The reporting component 118 may also generate a quality assessment report (e.g., the quality assessment report 138) for the automatic segmentation that includes the results of the quality assessment performed by the quality assessment component 116. The rendering component 132 may also present the segmentation result data 112 (e.g., including the corresponding organ segmentation masks / data, such as Figure 2The corresponding organ segmentation mask / data) and quality assessment report 138 as exemplified. The reporting component 118 may also store the quality assessment report 138 together with the segmentation result data 112 for the medical image data 106 in one or more medical image data repositories (e.g., medical image database 102 and / or memory 124). In some embodiments, the alert component 120 may also generate alert data (e.g., alert 140) based on one or more quality metrics of the automatic segmentation failing to meet a threshold quality metric (e.g., as defined in the quality assessment guidelines 130), and provide the alert data to one or more devices associated with one or more clinicians. For example, the alert component 120 may generate alert data in the form of a standard DICOM tag that is integrated on or within the corresponding structure or set of structures represented by the corresponding segmentation mask. The alert tag may indicate the quality metric of the corresponding structure or set of structures. For example, the alert tag may indicate whether the segmentation of a particular structure is accurate or inaccurate, and is thus associated with the "warning" keyword.

[0043] In this regard, any information received (e.g., medical image data 106 and / or segmentation result data 112 in some specific implementations) and / or generated (e.g., segmentation result data 112, quality assessment report 138, alert 140, etc.) by computing device 108 can be presented or rendered to a user via a suitable display. The display can be included together with computing device 108 (e.g., via input / output device 134) and / or another user device (not shown) that is communicatively and operatively coupled to computing device 108 (e.g., via one or more wired or wireless communication connections). In this regard, computing device 108 can correspond to any suitable computing device that is employed by a user (e.g., a clinician, a radiologist, a technician, a machine learning (ML) model developer, etc.) to review medical image data, segmentation result data 112, and the corresponding quality assessment report 138 for the segmentation result data 112 in association with using the segmentation result data to facilitate a clinical procedure (e.g., IMRT or another clinical procedure) for a patient. For example, in various embodiments, one or more components of computing device 108 can be associated with a medical imaging application that facilitates accessing and reviewing medical images via an interactive graphical user interface (GUI) displayed at the computing device, generating and reviewing anatomical object segmentations for the medical images via the GUI, annotating medical images, running inference models on medical images, etc. In some specific implementations of these embodiments, computing device 108 can correspond to an application server that provides at least some of these features and functionality to a user device (e.g., not shown) via a network-accessible platform such as a web application. Using these embodiments, computing device 108 can be communicatively coupled to a corresponding user device (e.g., a clinician device) via one or more wired or wireless communication networks (e.g., the Internet), and the corresponding user device can use a suitable web browser to access one or more of the features and functionality of computing device 108. Additionally or alternatively, system 100 can employ a local deployment architecture (as depicted in system 100). Various other deployment architectures are envisioned for system 100 and other systems described herein.

[0044] In this regard, computing device 108 can include one or more input / output devices 134 (e.g., a keyboard, a mouse, a touch screen, a display, etc.), which can be used to receive user input associated with the use of the features and functionality of computing device 108 and present the segmentation result data 112, the quality assessment report 138, and the alert to the user. Refer to Figure 10Some example input / output devices 134 are described with respect to input device 1028 and output device 1036. Computing device 108 may also include a device bus 122 that communicatively couples the respective elements / components of the computing device to each other.

[0045] Figure 3 A high-level flowchart of an example process 300 for determining reference feature values for a multi-organ segmentation model in accordance with one or more embodiments of the disclosed subject matter is presented. Refer Figures 1 to 3 , process 300 corresponds to an example process that may be performed by feature evaluation component 114 to define reference feature values 128 for a particular segmentation model 126. Throughout several example embodiments, the disclosed techniques are described, where segmentation model 126 corresponds to a multi-organ / object segmentation model that is trained to segment multiple different anatomical organs / objects included in input medical image data. However, it should be understood that the disclosed techniques may be applied to individual organ / object segmentation models.

[0046] As noted above, reference feature values 128 may be determined by feature evaluation component 114 based on GT segmentation data associated with the training medical images used to train segmentation model 126. In Figure 3 the illustrated embodiment, this GT segmentation data may be included in training data database 302 along with the training medical images (i.e., the training data set). In this regard, according to process 300, at 304, feature evaluation component 114 may access a segmentation model training data set having GT object segmentation masks applied to the respective organs / objects that the model is trained to segment. For example, assume that the segmentation model includes multi-organ segments OAR in head and neck MR data, then the respective organs / objects may include, but are not limited to: the anterior segment of the eyeball, the posterior segment of the eyeball, the lacrimal gland, the parotid gland, the submandibular gland, the extended oral cavity, the buccal mucosa, the lips, the mandible, the cochlea, the pharyngeal constrictor muscles, the thyroid gland, the brain, the brainstem, the pituitary gland, the optic chiasm, the optic nerve, the spinal cord, the carotid artery, etc. In this regard, the segmentation model training data set includes a plurality of input medical images (e.g., captured from different patients / subjects) of the same type (e.g., modality and scanned / imaged anatomical ROI) that have GT object / organ segmentation data for the respective anatomical objects / organs 1-N (e.g., where N may include any integer greater than 1) that the segmentation model is trained to segment. The GT object / organ segmentation data may include segmentation mask data (or simply segmentation masks) that define the boundary contours of the respective objects / organs.

[0047] At 306, for each GT object segmentation mask (for each anatomical object / organ among different anatomical objects / organs to be segmented), the feature evaluation component 114 calculates absolute feature values. In this regard, as indicated above, the defined features for each object / organ segmentation may include absolute features and relative features. Absolute features represent independent characteristics of the corresponding organ segmentation mask, such as geometric features related to the size or shape of the corresponding organ segmentation mask. Some example absolute features may include, but are not limited to: object / organ volume (e.g., the number of voxels in the entire organ contour), organ / object surface area (e.g., the number of voxels in the surface of the entire organ contour), and organ / object centroid coordinates (e.g., x, y, z). Absolute features may also include one or more latent features also extracted from the segmented image data input, such as the intensity within a specific anatomical structure. For example, absolute features may include the output of any machine learning method that can reduce the information content of the input image to a lower-dimensional representation, such as PCM (principal component analysis) or a CNN (convolutional neural network)-based encoder network. It should be understood that various other geometric properties may be defined as absolute features, and the absolute features may vary depending on whether the segmented medical image data includes 3D image data or 2D image data. The number of absolute features calculated and defined for each segmented anatomical object / organ may vary. For example, in some specific implementations, for each anatomical object / organ, the absolute feature may include a single feature (e.g., volume). In other specific implementations, the absolute features may include multiple different features (e.g., volume, surface area, centroid coordinates, and other absolute features). In this regard, at 306, for each different GT organ / object segmentation mask, and for each absolute feature evaluated, the feature evaluation component 114 may calculate the distribution of the observed values.

[0048] At 308, for each different GT organ / object segmentation mask, the feature evaluation component 114 may also calculate reference values for each absolute feature, such as the mean and / or standard deviation (STD) value for each absolute feature. For example, assuming that an absolute feature includes the volume of the segmented object / organ (e.g., the brain), at 308, the feature evaluation component 114 may calculate the mean and STD of the observed volume values of the segmented object / organ based on the distribution of the values calculated for each corresponding GT segmentation mask in the corresponding GT segmentation mask. In this regard, as a result of steps 306 and 308, at 310, the feature evaluation component 114 generates absolute feature (AF) reference values for each object segmentation mask (e.g., object 1 (e.g., the brain)): AF1 (e.g., volume), reference value = (mean observed volume, STD observed volume); AF2 (e.g., surface area), reference value = (mean observed volume, STD observed volume); (object 2 (e.g., the brainstem): AF1 (e.g., volume), reference value = (mean observed volume, STD observed volume), AF2 (e.g., surface area), reference value = (mean observed volume, STD observed volume); and for each object 1-n and each AF defined for each object, and so on).

[0049] At 312, the feature evaluation component 114 may also compute the distribution of the observed relative feature values. The relative features represent relative characteristics of different pairs of corresponding organ segmentation masks, such as the relative distance between each organ segmentation mask included in different pairs (e.g., the distance between two organs using centroid coordinates). In this regard, at 312, the feature evaluation component 114 may define different pairs of GT segmentation masks. The number and distribution of different pairs may be predefined and cover all possible pair combinations or a defined subset thereof. For each pair of GT object segmentation masks, the feature evaluation component 114 may compute the distribution of the observed distances between the corresponding mask centroid coordinates and / or other relative features (e.g., relative position of the boundary lines, relative orientation, etc.). At 314, for each pair of GT object segmentation masks, the feature evaluation component 114 may also compute reference feature values for the relative features, such as the mean and standard deviation values of each relative feature. For example, assuming a pair of objects / organs includes the brainstem and the eyeball, the relative feature may include the distance between the corresponding organ centroid coordinates, and the reference value for this relative feature may include the mean observed distance value computed at 312 and / or the STD of the observed distance values. In this regard, as a result of steps 312 and 314, at 316, the feature evaluation component 114 generates relative feature (RF) reference values for each pair of object segmentation masks. The computed absolute feature reference values and relative feature reference values generated at 310 and 316 constitute the reference feature values 128 for evaluating the output quality of the segmentation results of the segmentation model at runtime (e.g., after model training and deployment in a clinical workflow).

[0050] Figure 4 FIG. 4 presents a high - level flowchart of an example computer - implemented method 400 for automatically evaluating the output quality of a multi - organ segmentation model in accordance with one or more embodiments of the disclosed subject matter. Process 400 presents a high - level method for evaluating the output quality of the segmentation results of a segmentation model for which reference feature values are defined according to process 300 (e.g., based on GT segmentation data used for model training).

[0051] Reference Figures 1 to 4, according to process 400, at 402, process 400 includes determining, by a system including a processor (e.g., system 100 and additional systems described herein), current values of defined features (e.g., one or more absolute features and / or relative features) of corresponding segmentation masks (e.g., included in segmentation result data 112) generated for different anatomical structures (e.g., and / or other defined anatomical organs and other objects / structures) included in a medical image (e.g., medical image data 106) via automatic segmentation of the medical image data (e.g., using one or more segmentation models 126 via segmentation component 110). At 404, method 400 includes determining, by the system (e.g., via quality assessment component 116), a corresponding correspondence metric between the current values and corresponding reference values (e.g., as determined according to process 300 and provided in reference feature values 128) determined for the defined features. At 406, method 400 further includes determining, by the system, one or more quality metrics of the automatic segmentation based on the corresponding correspondence metric (via quality assessment component 116). At 408, method 400 further includes generating, by the system, quality assessment report data (e.g., quality assessment report 138) for the automatic segmentation that includes one or more quality metrics. For example, the quality assessment report data may include quality information integrated on or within the segmentation result (e.g., directly as and / or as metadata of the segmentation mask) that can be rendered by a clinical reporting tool together with the segmentation result. For example, the quality information may be integrated with the segmentation result as metadata tags in a standard clinical format (e.g., DICOM, RTSS, etc.). In some embodiments, the quality information may identify or indicate the name of the structure or set of structures represented by a particular segmentation mask or set of segmentation masks, and provide a quality metric for the structure or set of structures.

[0052] Figure 5 FIG. presents a flowchart of an example process 500 for determining absolute feature correspondence values and relative feature correspondence values in accordance with one or more embodiments of the disclosed subject matter. Refer to Figures 1 to 5, Process 500 provides additional details regarding quality assessment component 116 and steps 402 and 404 of process 400. In this regard, in one or more embodiments, at 502, feature assessment component 114 may analyze the segmentation result data 112 and calculate the current absolute feature values for each object segmentation mask. For example, feature assessment component 114 may identify the absolute features (e.g., volume, surface area, centroid coordinates, etc.) defined for each object segmentation mask as provided in reference feature values 128, and calculate the corresponding current absolute feature values. At 504, for each object segmentation mask, quality assessment component 116 may compare each of the absolute feature values calculated at 502 (or one value if only a single absolute feature is defined) with the corresponding absolute feature reference value (e.g., corresponding to 310), and calculate an absolute feature correspondence value, resulting in absolute feature correspondence value 506 for each object segmentation mask. In this regard, for each organ / object segmentation mask, the absolute feature correspondence value represents a measure of the degree of correspondence (or deviation) between the current value for each absolute feature and the reference value for the corresponding absolute feature. For example, assuming the absolute features for each organ segmentation mask include volume and surface area, the absolute feature correspondence value may include a first correspondence value reflecting the degree of correspondence (or deviation) between the current volume value for a particular organ segmentation mask and the reference average volume value, and a second correspondence value reflecting the degree of correspondence (or deviation) between the current surface area value for a particular organ segmentation mask and the reference average surface area value. It should be understood that various statistical measures may be used to represent the correspondence value based on the comparison of the current feature value with the reference feature value.

[0053] At 508, for each pair of object segmentation masks defined in the reference data for the segmentation model, the feature evaluation component 114 may also calculate relative feature values based on the segmentation result data 112. For example, in a particular implementation where the relative feature encompasses the respective distances between the centroid coordinates of the corresponding organ / object segmentation masks, at 508, the feature evaluation component may calculate the respective distance values. In this regard, assuming the reference feature value 128 represents all pairs of combinations, each organ / object segmentation mask will have a set of current distance values that represent the respective distances from the centroid coordinates of the given organ / object segmentation mask to each of the centroid coordinates of all other organ / object segmentation masks. For example, for a given mask A, the set will include the distance between mask A and mask B, the distance between mask A and mask C, the distance between mask A and mask D, the distance between mask A and mask E, and so on. At 510, for each pair, the quality evaluation component 116 may also compare the relative feature value with the corresponding reference feature value and calculate a reference feature correspondence value, thereby obtaining a relative feature correspondence value 512 for each pair of object segmentation masks (or a single relative feature correspondence value if only a single relative feature such as distance is used). For example, in some particular implementations, the feature correspondence values for all current mask distances associated with mask A may include a set of correspondence values that represent the degree of correspondence (or deviation) between the current distance values and the corresponding reference distance values (e.g., the difference between the current mask A to mask B distance and the reference average mask A to mask B distance, the difference between the current mask A to mask C distance and the reference average mask A to mask C distance, the difference between the current mask A to mask D distance and the reference average mask A to mask D distance, and so on).

[0054] Reference Figure 1 and Figure 5, as a result of process 500, the quality assessment component 116 may generate a subset of correspondence values for each object / organ segmentation mask, where these correspondence values respectively reflect a measure of the correspondence between the current eigenvalue of a given organ / object segmentation mask and a reference eigenvalue for the given organ / object segmentation mask. In some embodiments, the quality assessment component 116 may determine one or more quality metrics for each organ / object segmentation mask based on an aggregated subset of the correspondence values associated with each organ / object segmentation mask and predefined quality threshold criteria defined in the quality assessment guidelines 130. For example, the quality assessment component 116 may generate an aggregated correspondence score for each organ / object segmentation mask, where the aggregated correspondence score represents an aggregated combination of each feature correspondence value in the subset of feature correspondence values. The aggregated correspondence score corresponds to an overall quality metric for the organ / object segmentation based on all evaluated features for the organ / object segmentation. The quality assessment component 116 may also determine whether the quality of a given organ / object segmentation is acceptable (e.g., meaning the segmentation is accurate enough) based on whether the aggregated correspondence score is greater than a minimum threshold correspondence score defined in the quality assessment guidelines 130. In some specific implementations, the quality assessment component 116 may characterize any organ / object segmentation that does not meet the threshold correspondence score as an "outlier" segmentation.

[0055] In some embodiments, the quality assessment component 116 may also evaluate the overall quality of the segmentation result data 112 based on the number of outlier segmentations and / or the aggregated correspondence score of the combination for all segmentation masks represented in the segmentation result data 112. For example, in some specific implementations, the quality assessment component 116 may generate an average correspondence score for the segmentation result data 112 based on the aggregated correspondence score of the combination for all segmentation masks represented in the segmentation result data 112. The quality assessment component 116 may also determine whether the overall quality of the segmentation result data 112 is acceptable based on whether the average aggregated correspondence score is greater than a minimum overall threshold correspondence score defined in the quality assessment guidelines 130. In another example, the quality assessment component 116 may determine whether the overall quality of the segmentation result data 112 is acceptable based on whether the number of outliers is less than or equal to the maximum number of outliers defined for the segmentation model in the quality assessment guidelines data 130.

[0056] It should be understood that the quality assessment component 116 can employ various additional or alternative statistical analysis algorithms to evaluate the degree of correspondence between the current absolute and / or relative eigenvalue associated with each organ segmentation mask and the corresponding reference eigenvalue (e.g., it can include the mean value and / or the STD value). The quality assessment component 116 can also employ various aggregation techniques to aggregate the set of current absolute and / or relative eigenvalues associated with each organ segmentation mask in association with comparing the set with the corresponding reference eigenvalues, to determine the measure of correspondence (or deviation) between the current eigenvalue and the reference eigenvalue for each organ / object segmentation mask and / or to determine the measure or quality of an individual organ segmentation mask and / or the overall quality of the segmentation result data 112.

[0057] For example, in some embodiments, the quality assessment component 116 can use one or more probability values to evaluate the quality of the segmentation result data 112, and the one or more probability values reflect the accuracy measure of the segmentation result data 112 according to the degree of correspondence (or deviation) between the current eigenvalue and the corresponding reference eigenvalue. In this regard, in one exemplary embodiment, it is assumed that the segmentation model applied to generate the segmentation result data 112 segments 3D medical image data (e.g., MR data, CT data, etc.) and defines the geometric contours of multiple different anatomical objects / organs (e.g., a multi-organ / object segmentation model). For each organ / object, it is assumed that the absolute features defined in the reference eigenvalue include volume and surface area, and the relative features include the distances between the centroid coordinates for all corresponding pairs of combinations of the organ / object segmentation. According to this exemplary embodiment, for each feature (e.g., volume, surface area, and each distance between the centroids of two organs), the quality assessment component 116 can approximate the probability of correct automatic segmentation using a univariate Gaussian normal distribution according to Equation 1 below, where μ represents the reference mean value of the feature, σ represents the reference STD value of the feature (both are calculated based on the training data set), y represents the feature probability value, and x represents the measured current eigenvalue based on the segmentation result data 112.

[0058]

[0059] Since the probability values of volume and surface area are single numbers, there is an entire vector of values for the Euclidean distance between the organ centroids. Therefore, the quality assessment component 116 can calculate the mean value from this vector according to Equation 2 to also obtain a single number for the Euclidean distance feature probability:

[0060]

[0061] where: Y RED represents the overall Euclidean distance feature probability, and Y IREDA vector corresponding to the individual relative Euclidean distance feature probability (i.e., an approximate probability of the correctness of the distance between the centroid of the chiasma contour and the centroids of all other organ contours).

[0062] For each organ / object segmentation (i.e., contour), the quality assessment component 116 can aggregate the probabilities of all individual features using an aggregation function such as the mean of the probabilities according to Equation 3:

[0063]

[0064]

[0065] Where: Y is the final approximate probability, Y AV is the absolute volume feature probability, and Y AS is the absolute surface area feature probability, and Y RED is the overall relative Euclidean distance feature probability. According to this example, Y corresponds to the final probability value for a given organ / object segmentation, which reflects the probability that the segmentation is accurate. In this regard, the Y value can be or can correspond to a quality metric for the corresponding organ segmentation.

[0066] The quality assessment component 116 can also evaluate the quality of the segmentation result data 112 based on the corresponding final approximate probability values (i.e., Y for each organ / object segmentation) and predefined criteria regarding acceptable and unacceptable final approximate probability values (e.g., one or more defined thresholds provided in the quality assessment guideline data 130). For example, the quality assessment component 116 can evaluate the quality of the automatically segmented result data 112 at two levels (the individual level for each organ / object segmentation and the overall or global level). As applied to the individual organ / object segmentation level, in some specific implementations, if the aggregated probability value Y for the organ / object segmentation is greater than a fixed threshold predefined in the quality assessment guideline 130 (e.g., for example, 0.2), the quality assessment component 116 can determine that the individual organ / object segmentation is correct (i.e., accurate) and "acceptable". If the aggregated probability value Y is less than or equal to the fixed threshold probability value, the quality assessment component 116 can similarly determine that the individual organ / object segmentation is incorrect (i.e., inaccurate), in which case the quality assessment component 116 can characterize the organ / object segmentation as an "outlier". If the deep learning model inference fails or the organ is not visible in the scan, the automatic segmentation of the organ can be considered "missing".

[0067] When applied jointly at the global or whole case level to represent the segmentation result data 112, in some embodiments, the quality assessment component 116 may determine whether the overall automatic segmentation is acceptable based on the number of outliers and missing individual segmentations. For example, if the number of automatically segmented organs evaluated as "acceptable" is greater than the sum of the automatically segmented ones evaluated as "outliers" and "missing", the quality assessment guideline 130 may characterize the case as acceptable; otherwise, the case may be considered unacceptable or an "outlier case". When exported in a standard format (e.g., DICOM RTSS) and / or identified / highlighted in a quality assessment report (e.g., quality assessment report 138) attached to the segmented case in a standard format (DICOM RS), the abnormality of the automatic segmentation may be indicated in the name of the structure or structure set having the segmentation result data 112.

[0068] Figure 6 Table 600 illustrates example automatic segmentation quality assessment results in accordance with one or more embodiments of the disclosed subject matter. The quality assessment results represented in Table 600 represent the quality metrics of individual organ / object segmentations as probability values (e.g., Y values) determined according to Equations 1 to 3 as discussed above. Table 600 illustrates example results of an automatic segmentation model as applied to ten different cases (i.e., ten different medical image examinations for different patients). In this example, the automatic segmentation model is configured to segment eight different anatomical objects, respectively designated as Objects 1 to 8. The values in the corresponding cells under each object correspond to the quality metrics determined for each object segmentation generated by the automatic segmentation model, where the metric corresponds to the final approximate probability value (i.e., Y value) determined according to the above Equations 1 to 3. In this example, a threshold probability of 0.2 is applied to characterize outlier segmentations (e.g., any object segmentation having a quality score or Y value less than 0.2 is considered inaccurate and thus an outlier). The criterion for characterizing a case as acceptable or overall failed (i.e., unacceptable overall segmentation) is based on the combined number of outliers and missing segmentations being less than or equal to three. The cells corresponding to outliers are indicated in gray. The cells having a zero value and filled in black represent missing segmentations (e.g., objects that the segmentation model did not identify and segment at all). In this example, Cases 1, 4, 5, and 7 to 9 have acceptable global quality results and thus pass the quality assessment check. Cases 2 to 3, 6, and 10 have unacceptable global quality results and thus fail the quality assessment check.

[0069] Refer again to Figure 1, as noted above, the reporting component 118 may generate a quality assessment report 138 that can be presented to a user (e.g., a radiologist, a clinician, etc.) in association with the segmentation result data 112 via a suitable device display (e.g., via the rendering component 132), where the quality assessment report 138 includes the results of the quality assessment performed by the quality assessment component 116. For example, the quality assessment report 138 may include information about the quality of each organ / object segmentation mask in the individual organ / object segmentation masks, such as the Y-score illustrated in Table 600 and / or another valuation of the accuracy / correctness of the corresponding segmentation mask based on the aggregated feature correspondence values for each organ / object segmentation mask. The quality assessment report 138 may also identify any organ / object segmentations that are considered to have quality defects and are thus outliers based on the quality assessment guidelines 130 and the defined outlier criteria. The quality assessment report 138 may also include information about the overall or global quality of the segmentation result data 112.

[0070] In some embodiments, the alert component 120 may generate one or more alerts 140 based on the quality assessment results meeting one or more predefined alert criteria defined in the quality assessment guidelines data 130. For example, the alert criteria may be based on individual organ / segmentation quality metrics and / or global quality metrics for the segmentation result data 112. In this regard, according to the defined alert criteria, the alert component 120 may generate an alert based on one or more individual organ / object segmentation quality metrics failing to meet a threshold quality metric and / or based on the global quality metric failing to meet a defined threshold. The alert component 120 may also include the alert 140 in the quality assessment report 138 and / or provide the alert to the clinician in a separate electronic notification message, thereby drawing the clinician's attention to any elements of the segmentation result data 112 that are considered inaccurate for clinical applications.

[0071] Figure 7 A flowchart of an example process 700 for evaluating the output quality of multi-organ automatic segmentation in accordance with one or more embodiments of the disclosed subject matter is presented. Referring to Figures 1 to 7 , process 700 provides additional details regarding the quality assessment component 116 and steps 402 and 404 of process 400. Process 700 provides an example high-level method for evaluating the quality of automatic segmentation based on the absolute feature correspondence values 506 for each object segmentation mask (determined as described with reference to process 500) and the relative feature correspondence values 512 for each pair of segmentation masks (also determined as described with reference to process 500).

[0072] According to process 700, at 702, for each object segmentation mask, the quality assessment component 116 may aggregate a subset of all absolute feature correspondence values and relative feature correspondence values. At 704, for each object segmentation mask, the quality assessment component 116 may determine an object segmentation quality metric based on the corresponding aggregated subset of absolute and / or relative feature correspondence values, thereby generating a segmentation quality metric 706 for each object segmentation. For example, the segmentation quality metric may correspond to the aggregated correspondence values of the respective absolute and relative features included in the subset for each object / organ segmentation. In another example, the segmentation quality metric may correspond to the final approximate probability value (i.e., Y value) for each segmentation determined according to Equations 1 to 3. Based on different aggregation techniques for the respective feature correspondence values, various other valuations are envisioned.

[0073] At 708, the quality assessment component 116 may determine a global automatic segmentation quality metric based on the aggregated segmentation quality metrics for each object segmentation. For example, in some embodiments, the quality assessment component 116 may determine the global quality metric as the average quality metric 706 of the respective individual segmentation quality metrics. In another example, the quality assessment component 116 may determine the global quality metric based on the number of outliers relative to the number of acceptable individual segmentations. At 710, the quality assessment component 116 determines whether the global quality metric is acceptable based on whether the global quality metric meets the defined acceptability criteria (e.g., threshold global quality value, maximum number of outliers, etc.) for the segmentation model defined in the quality assessment guidelines 130. Based on the determination at 710 that the global quality metric is acceptable, process 700 proceeds to 712, where the reporting component 118 may report the automatic segmentation as acceptable in the quality assessment report 138 (e.g., accurate enough for clinical applications). However, based on the determination at 710 that the global quality metric is unacceptable, process 700 proceeds to 714, where the reporting component 118 may report the automatic segmentation as unacceptable in the quality assessment report 138 (e.g., not accurate enough for clinical applications).

[0074] Figure 8 A flowchart of another example process 800 for evaluating the output quality of multi-organ automatic segmentation according to one or more embodiments of the disclosed subject matter is presented. Referring Figures 1 to 8 to, process 800 provides additional details regarding the quality assessment component 116 and steps 402 and 404 of process 400. Process 700 provides another example high-level method for evaluating the quality of automatic segmentation based on the absolute feature correspondence values 506 (determined as described with reference to process 500) for each object segmentation mask and the relative feature correspondence values 512 for each pair of segmentation masks (also determined as described with reference to process 500).

[0075] According to process 800, at 802, for each object segmentation mask, quality assessment component 116 may aggregate a subset of all absolute feature correspondence values and relative feature correspondence values. At 804, for each object segmentation mask, quality assessment component 116 may determine an object segmentation quality metric based on the corresponding aggregated subset of absolute and / or relative feature correspondence values, thereby generating a segmentation quality metric 806 for each object segmentation. For example, the segmentation quality metric may correspond to the aggregated correspondence values of the corresponding absolute and relative features included in the subset for each object / organ segmentation. In another example, the segmentation quality metric may correspond to the final approximate probability value (i.e., Y value) for each segmentation determined according to Equations 1 to 3. Based on different aggregation techniques for the corresponding feature correspondence values, various other valuations are envisioned.

[0076] At 808, for each object segmentation, quality assessment component 116 may compare the segmentation quality metric with a threshold quality metric and identify any outliers (e.g., outliers that fail to meet the threshold quality metric). At 810, quality assessment component 116 may evaluate whether any outliers are detected. If no outliers are detected, process 800 proceeds to 812, where reporting component 118 may report the automatic segmentation as acceptable (e.g., accurate enough for clinical applications) in quality assessment report 138. If at 810 quality assessment component 116 identifies one or more outliers, process 800 proceeds to 814, where quality assessment component 116 determines whether the number of outliers is acceptable for the segmentation model. In this regard, according to process 800, quality assessment component 116 evaluates the overall quality of segmentation result data 112 based on the number of outliers relative to a defined maximum number of acceptable outliers (e.g., as defined in quality assessment guidelines 130). However, it should be understood that various other valuations of outliers (e.g., including the corresponding values of the outliers) may be applied in connection with evaluating the overall quality of the segmentation results. If at 814 quality assessment component determines that the number of outliers is acceptable, process 800 proceeds to 818, where reporting component 118 may report the automatic segmentation as acceptable and identify any outliers in quality assessment report 138 (e.g., accurate enough for clinical applications). For example, the quality assessment report may specifically identify the particular organ / object segmentations that are considered outliers and thus draw the clinician's attention to these outliers for manual review and / or manual contouring. However, based on the determination at 814 that the number of outliers is unacceptable, process 800 proceeds to 816, where reporting component 118 may report the automatic segmentation as unacceptable (e.g., not accurate enough for clinical applications) in quality assessment report 138.

[0077] Figure 9Another example system 900 is presented that facilitates automated organ segmentation output quality assessment according to one or more embodiments of the disclosed subject matter. System 900 is similar to system 100, with the addition of a recommendation component 902, a training data curation component 904, and a training database 302. For the sake of brevity, the repeated description of similar elements employed in the corresponding embodiments is omitted.

[0078] In some embodiments, the recommendation component 902 may determine one or more recommended actions to be performed based on the quality assessment results, and provide recommendations 906 regarding the recommended actions to one or more users. For example, in some embodiments, the recommendation component 902 may recommend a manual review of the segmentation result data 112 based on one or more individual organ / object segmentation quality metrics among the global quality metric and / or the individual organ / object segmentation quality metrics failing to meet an acceptable quality metric standard. Additionally or alternatively, the recommendation component 902 may determine that the input medical image data 106 may be attributed to some defect or error based on a quality assessment result indicating that the segmentation result data 112 is significantly inaccurate (e.g., as defined in the quality assessment guidelines). For example, the quality assessment component 116 may apply a first criterion for defining the quality assessment result data as moderately inaccurate (e.g., based on the number of outliers being between 2 and 4) and a second criterion for characterizing the quality assessment result data 112 as significantly inaccurate (e.g., based on the number of outliers exceeding 4) to indicate a possible error or defect in the medical image data 106. Using these embodiments, the recommendation component 902 may recommend that the patient be rescanned (e.g., via the medical imaging system 104) to generate new medical image data for the patient for processing by the computing device 108.

[0079] The training data curation component 904 may also employ the quality assessment results to identify and collect additional cases for retraining and updating (i.e., optimizing) the performance of the corresponding segmentation model 126. For example, in some embodiments, the training data curation component 904 may be configured to collect the input medical image data of all cases having segmentation result data that is considered to have quality defects (e.g., based on the global quality metric and / or one or more individual organ / object segmentation quality metrics), and add the input medical image data to the corresponding training dataset included in the training data database 302. The training data curation component 904 may also include the segmentation result data 112 and the associated quality assessment report 138 generated for each outlier or defective case among the outlier or defective cases in the training data database 302 together with the input medical image data.

[0080] One or more embodiments may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0081] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves record instructions thereon), and any appropriate combination of the foregoing items. As used herein, a computer-readable storage medium should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable) or an electrical signal transmitted through a wire.

[0082] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the corresponding computing / processing device.

[0083] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., procedural programming languages such as the "C" programming language or similar programming languages, and machine learning programming languages such as CUDA, Python, TensorFlow, PyTorch, etc. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or a server using appropriate processing hardware. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may establish a connection with an external computer (e.g., through the Internet using an Internet service provider). In various embodiments involving machine learning programming instructions, the processing hardware may include one or more graphics processing units (GPUs), central processing units (CPUs), etc. For example, one or more of the disclosed machine learning models (e.g., the image transformation model 908, the deep learning network 1010, and / or their combination) may be written in a suitable machine learning programming language and executed via one or more GPUs, CPUs, or their combination. In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit in order to perform aspects of the present invention.

[0084] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0086] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0087] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0088] In conjunction Figure 10 with, the systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, an application specific integrated circuit (ASIC), etc. Additionally, the order in which some or all of the program blocks appear in each program should not be considered limiting. Instead, it should be understood that some program blocks can be executed in various orders, not all of which may be explicitly shown herein.

[0089] Refer to Figure 10, An example environment 1000 for implementing various aspects of the claimed subject matter includes a computer 1002. The computer 1002 includes a processing unit 1004, a system memory 1006, a codec 1035, and a system bus 1008. The system bus 1008 couples system components including, but not limited to, the system memory 1006 to the processing unit 1004. The processing unit 1004 can be any of a variety of available processors. Dual microprocessors, one or more GPUs, CPUs, and other multiprocessor architectures can also be used as the processing unit 1004.

[0090] The system bus 1008 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, or a local bus using any of a variety of available bus architectures, including but not limited to Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association Bus (PCMCIA), FireWire (IEEE 1394), and Small Computer System Interface (SCSI).

[0091] In various embodiments, system memory 1006 includes volatile memory 1010 and non-volatile memory 1012, which may employ one or more of the disclosed memory architectures. The basic input / output system (BIOS) (including basic routines for transferring information between elements within computer 1002 during startup) is stored in non-volatile memory 1012. Additionally, in accordance with the present inventive concept, codec 1035 may include at least one of an encoder or a decoder, where at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software. Although codec 1035 is depicted as a separate component, codec 1035 may be included within non-volatile memory 1012. By way of illustration and not limitation, non-volatile memory 1012 may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, 3D flash memory, or resistive memory, such as resistive random access memory (RRAM). In at least some embodiments, non-volatile memory 1012 may employ one or more of the disclosed memory devices. Additionally, non-volatile memory 1012 may be computer memory (e.g., physically integrated with computer 1002 or its motherboard) or removable memory. Examples of suitable removable memory that may be used to implement the disclosed embodiments may include secure digital (SD) cards, compact flash (CF) cards, universal serial bus (USB) memory sticks, etc. Volatile memory 1010 includes random access memory (RAM) that acts as an external cache memory, and in various embodiments may also employ one or more of the disclosed memory devices. By way of illustration and not limitation, RAM can be provided in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM), etc.

[0092] Computer 1002 may also include removable / non-removable, volatile / non-volatile computer storage media. Figure 10Exemplified is, for example, a disk storage device 1014. The disk storage device 1014 includes, but is not limited to, devices such as disk drives, solid state drives (SSDs), flash memory cards, or memory sticks. Additionally, the disk storage device 1014 may include a storage medium alone or in combination with another storage medium, which includes, but is not limited to, optical disk drives such as optical disk ROM devices (CD-ROMs), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives), or digital versatile magneto-optical ROM drives (DVD-ROMs). To facilitate connection of the disk storage device 1014 to the system bus 1008, a removable or non-removable interface, such as interface 1016, is typically used. It should be understood that the disk storage device 1014 may store user-related information. Such information may be stored at a server or provided to an application running on a server or a user device. In one embodiment, the user may be notified (e.g., via the output device 1036) of the type of information stored to the disk storage device 1014 or transmitted to the server or application. The user may be provided with the opportunity to opt in or opt out of the collection or sharing of such information by the server or application (e.g., via an input from the input device 1028).

[0093] It should be understood that Figure 10 Software is described that acts as a mediator between the user and the basic computer resources described in the suitable operating environment 1000. Such software includes an operating system 1018. The operating system 1018, which may be stored on the disk storage device 1014, is used to control and allocate the resources of the computer 1002. Application programs 1020 utilize the management of resources by the operating system 1018 through program modules 1024, and program data 1026 stored in the system memory 1006 or on the disk storage device 1014, such as power-on / power-off transaction tables and the like. It should be understood that the claimed subject matter may be implemented with various operating systems or combinations of operating systems.

[0094] A user inputs commands or information into computer 1002 through input device 1028. Input device 1028 includes, but is not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, gamepad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, etc. These and other input devices are connected to processing unit 1004 via system bus 1008 through interface port 1030. Interface port 1030 includes, for example, serial ports, parallel ports, game ports, and universal serial bus (USB). Output device 1036 uses some of the same type of ports as input device 1028. Thus, for example, a USB port can be used to provide input to computer 1002 and output information from computer 1002 to output device 1036. Output adapter 1034 is provided to illustrate some output devices 1036 such as monitors, speakers, and printers, as well as other output devices 1036 that require special adapters. By way of illustration and not limitation, output adapter 1034 includes video cards and sound cards that provide connection means between output device 1036 and system bus 1008. It should be noted that other devices or systems of devices provide input and output capabilities, such as remote computer 1038.

[0095] Computer 1002 can operate in a networking environment using a logical connection to one or more remote computers, such as remote computer 1038. Remote computer 1038 can be a personal computer, server, router, network PC, workstation, microprocessor-based appliance, peer device, smart phone, tablet, or other network node, and generally includes many of the elements described with respect to computer 1002. For the sake of brevity, only memory storage device 1040 is shown for remote computer 1038. Remote computer 1038 is logically connected to computer 1002 through network interface 1042 and then connected via communication link 1044. Network interface 1042 encompasses wired communication networks or wireless communication networks, such as local area networks (LANs) and wide area networks (WANs), as well as cellular networks. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit-switched networks such as integrated services digital network (ISDN) and its variants thereon, packet-switched networks, and digital subscriber line (DSL).

[0096] The communication connection 1044 refers to the hardware / software used to connect the network interface 1042 to the bus 1008. Although the communication connection 1044 is shown inside the computer 1002 for clarity, the communication connection can also be external to the computer 1002. For illustrative purposes only, the hardware / software required to connect to the network interface 1042 includes internal and external technologies such as modems, including conventional telephone-grade modems, cable modems, and DSL modems, ISDN adapters, and wired and wireless Ethernet cards, hubs, and routers.

[0097] Although the subject matter has been described above in the general context of computer-executable instructions of a computer program product running on one and / or more computers, those skilled in the art will recognize that the present disclosure may also be implemented in or in conjunction with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform particular tasks and / or implement particular abstract data types. In addition, those skilled in the art will appreciate that the computer-implemented methods of the present invention may be practiced using other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, and computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronics, etc. The illustrated aspects may also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0098] As used in this application, terms such as "component", "system", "platform", "interface", etc. can refer to and / or can include computer-related entities or entities related to an operating machine with one or more specific functionalities. Entities disclosed herein can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a server and the server can be components. One or more components can reside within a program and / or an execution thread, and a component can be located on one computer and / or distributed between two or more computers. In another example, corresponding components can execute according to various computer-readable media on which various data structures are stored. Components can communicate via local and / or remote processes, such as according to a signal having one or more data packets (e.g., data from one component, which interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems) via the signal). As another example, a component can be a device having specific functionality provided by mechanical parts operated by an electrical or electronic circuit, which is operated by a software or firmware application executed by a processor. In such cases, the processor can be inside or outside the device and can execute at least a portion of the software or firmware application. As yet another example, a component can be a device that provides specific functionality through electronic components rather than mechanical parts, where the electronic components can include a processor or other components for executing software or firmware that at least partially imparts functionality to the electronic components. In one aspect, a component can, for example, emulate an electronic component via a virtual machine within a cloud computing system.

[0099] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive permutation. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. Further, unless otherwise specified or clear from the context as being for the singular form, the articles "a" and "an" used in this specification and the drawings generally should be understood to mean "one or more". As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration, and are intended to be non-limiting. To avoid doubt, the subject matter disclosed herein is not limited by such examples. Further, any aspect or design described herein as "example" and / or "exemplary" need not be understood as being more preferred or advantageous than other aspects or designs, nor does it mean excluding equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0100] As used in this specification, the term "processor" can generally refer to any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capabilities; a multi-core processor; a multi-core processor with software multithreaded execution capabilities; a multi-core processor with hardware multithreaded technology; a parallel platform; and a parallel platform with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Further, a processor can utilize nanoscale architectures (such as but not limited to molecule and quantum dot-based transistors, switches, and gates) in order to optimize space usage or enhance the performance of user equipment. A processor can also be implemented as a combination of computing processing units. In the present disclosure, terms such as "repository", "storage device", "data repository", "data storage device", "database", and substantially any other information storage component related to the operation and functionality of a component are used to refer to "memory components", entities embodied in "memory", or components that include memory. It should be understood that the memory and / or memory components described herein can be volatile memory or non-volatile memory, or can include both volatile memory and non-volatile memory. By way of illustration and not limitation, non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory can include RAM that can act as an external cache memory. By way of illustration and not limitation, RAM can be provided in various forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Additionally, the disclosed memory components of the systems or computer-implemented methods herein are intended to include but not be limited to including these and any other suitable types of memory.

[0101] The foregoing description includes only examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing the present disclosure, but one of ordinary skill in the art will recognize that many other combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including," "having," "possessing," etc. are used in the detailed description, the claims, the appendices, and the drawings, such terms are intended to be inclusive in a manner similar to the term "comprising" as that term is interpreted when used as a transitional word in a claim. Descriptions of various embodiments have been given for purposes of illustration, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to one of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable other one of ordinary skill in the art to understand the embodiments disclosed herein.

[0102] Other aspects of the various embodiments described herein are provided by the subject matter of the following clauses:

[0103] 1. A system, the system comprising:

[0104] a memory that stores computer-executable components; and

[0105] a processor that executes the computer-executable components stored in the memory, wherein the computer-executable components include:

[0106] a feature evaluation component that determines a current value of a defined feature of a corresponding segmentation mask generated for different anatomical structures included in the medical image data via automatic segmentation of the medical image data;

[0107] a quality evaluation component that determines a corresponding correspondence metric between the current value and a corresponding reference value determined for the defined feature, and determines one or more quality metrics of the automatic segmentation based on the corresponding correspondence metric; and

[0108] a reporting component that generates quality evaluation report data for the automatic segmentation that includes the one or more quality metrics.

[0109] 2. The system according to any of the preceding clauses, wherein the one or more quality metrics include a quality metric for a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein the reporting component integrates the quality metric with the specific structure or set of structures in a standard clinical data format.

[0110] 3. The system according to any of the preceding clauses, wherein the one or more quality metrics include structure-specific quality metrics for each of the corresponding segmentation masks, and the structure-specific quality metrics reflect the quality metrics of each of the corresponding segmentation masks.

[0111] 4. The system according to any of the preceding clauses, wherein the quality assessment component determines the structure-specific quality metric for each of the corresponding segmentation masks based on the aggregated correspondence values for a plurality of the defined features associated with each of the corresponding segmentation masks.

[0112] 5. The system according to any of the preceding clauses, wherein the one or more quality metrics further include an overall quality metric of the automatic segmentation determined by the quality assessment component based on the aggregation of each of the structure-specific quality metrics.

[0113] 6. The system according to any of the preceding clauses, wherein the defined features include absolute features representing independent characteristics of the corresponding segmentation masks.

[0114] 7. The system according to any of the preceding clauses, wherein the absolute features include one or more geometric features related to the size or shape of the corresponding segmentation masks.

[0115] 8. The system according to any of the preceding clauses, wherein the absolute features include one or more potential features of the corresponding segmentation masks

[0116] 9. The system according to any of the preceding clauses, wherein the defined features further include relative features representing relative characteristics of different pairs of the corresponding segmentation masks.

[0117] 10. The system according to any of the preceding clauses, wherein the relative characteristics include the relative distance between each of the segmentation masks included in the different pairs.

[0118] 11. The system according to any of the preceding clauses, wherein the computer-executable component further includes a rendering component that presents the corresponding segmentation mask and the quality assessment report data in association with a clinical procedure for a patient from whom the medical image data is captured, via a device display associated with one or more clinical entities, using the corresponding segmentation mask.

[0119] 12. The system according to any of the preceding clauses, wherein the computer-executable components further include an alert component that generates alert data based on the one or more quality metrics failing to meet a threshold quality metric and provides the alert data to one or more devices associated with one or more clinicians.

[0120] 13. The system according to any of the preceding clauses, wherein the alert data is related to a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein the alert component integrates the alert data with the specific structure or set of structures in a standard clinical data format.

[0121] 14. The system according to any of the preceding clauses, wherein the computer-executable components further include a recommendation component that determines one or more clinical workflow actions to be performed based on the one or more quality metrics failing to meet a threshold quality metric and provides recommendation data identifying the one or more clinical workflow actions to one or more devices associated with one or more clinicians.

[0122] 15. The system according to any of the preceding clauses, wherein the feature evaluation component determines the corresponding reference value for the defined feature based on ground truth organ segmentation data for the different anatomical structures depicted in the training medical image data.

[0123] 16. A method, the method comprising:

[0124] determining, by a system including a processor, a current value of a defined feature of a corresponding segmentation mask generated via automatic segmentation of medical image data for different anatomical structures included in the medical image data;

[0125] determining, by the system, a corresponding correspondence metric between the current value and a corresponding reference value determined for the defined feature;

[0126] determining, by the system, one or more quality metrics of the automatic segmentation based on the corresponding correspondence metric; and

[0127] generating, by the system, quality assessment report data for the automatic segmentation including the one or more quality metrics.

[0128] 17. The method according to any of the preceding clauses, wherein determining the one or more quality metrics includes determining a quality metric for a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein the generating includes integrating the quality metric with the specific structure or set of structures in a standard clinical data format.

[0129] 18. The method according to any of the preceding clauses, the method further comprising:

[0130] generating, by the system, alert data based on the one or more quality metrics failing to meet a threshold quality metric; and

[0131] providing, by the system, the alert data to one or more devices associated with one or more clinicians.

[0132] 19. The method according to any of the preceding clauses, wherein the alert data is related to a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein generating the alert data includes integrating the alert data with the specific structure or set of structures in a standard clinical data format.

[0133] 20. A non-transitory machine-readable storage medium including executable instructions that, when executed by a processor, facilitate performance of operations, the operations including:

[0134] determining a current value of a defined feature of a respective segmentation mask generated for different anatomical structures included in the medical image data via automatic segmentation of the medical image data;

[0135] determining a respective correspondence metric between the current value and a corresponding reference value determined for the defined feature;

[0136] determining one or more quality metrics of the automatic segmentation based on the respective correspondence metric; and

[0137] generating quality assessment report data for the automatic segmentation including the one or more quality metrics.

Claims

1. A system, the system comprising: a memory that stores computer-executable components; and a processor that executes the computer-executable components stored in the memory, wherein the computer-executable components include: a feature evaluation component that determines a current value of a defined feature of a corresponding segmentation mask generated for different anatomical structures included in the medical image data via automatic segmentation of the medical image data; a quality evaluation component that determines a corresponding correspondence measure between the current value and a corresponding reference value determined for the defined feature, and determines one or more quality measures of the automatic segmentation based on the corresponding correspondence measure; and a reporting component that generates quality evaluation report data for the automatic segmentation including the one or more quality measures.

2. The system according to claim 1, wherein the one or more quality measures include quality measures for a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein the reporting component integrates the quality measures with the specific structure or set of structures in a standard clinical data format.

3. The system according to claim 1, wherein the one or more quality measures include structure-specific quality measures for each segmentation mask in the corresponding segmentation masks, the structure-specific quality measures reflecting the quality measures of each segmentation mask in the corresponding segmentation masks.

4. The system according to claim 3, wherein the quality evaluation component determines the structure-specific quality measure for each segmentation mask in the corresponding segmentation masks based on an aggregated correspondence value for a plurality of the defined features associated with each segmentation mask in the corresponding segmentation masks.

5. The system according to claim 3, wherein the one or more quality measures further include an overall quality measure of the automatic segmentation determined by the quality evaluation component based on an aggregation of each of the structure-specific quality measures.

6. The system according to claim 1, wherein the defined features include absolute features representing independent characteristics of the corresponding segmentation masks.

7. The system according to claim 6, wherein the absolute features include one or more geometric features related to the size or shape of the corresponding segmentation masks.

8. The system according to claim 6, wherein the absolute features include one or more latent features of the corresponding segmentation masks.

9. The system according to claim 6, wherein the defined features further include relative features representing relative characteristics of different pairs of the corresponding segmentation masks.

10. The system according to claim 9, wherein the relative characteristics include a relative distance between each segmentation mask included in the different pairs.

11. The system according to claim 1, wherein the computer-executable components further include: A rendering component that presents the corresponding segmentation mask and the quality assessment report data in association with a clinical procedure for a patient from whom the medical image data is captured, via a device display associated with one or more clinical entities, using the corresponding segmentation mask.

12. The system according to claim 1, wherein the computer-executable component further comprises: An alert component that generates alert data based on the one or more quality metrics failing to meet a threshold quality metric and provides the alert data to one or more devices associated with one or more clinicians.

13. The system according to claim 12, wherein the alert data is related to a specific structure or set of structures represented by a segmentation mask or a group of segmentation masks, and wherein the alert component integrates the alert data with the specific structure or set of structures in a standard clinical data format.

14. The system according to claim 1, wherein the computer-executable component further comprises: A recommendation component that determines one or more clinical workflow actions to be performed based on the one or more quality metrics failing to meet a threshold quality metric and provides recommendation data identifying the one or more clinical workflow actions to one or more devices associated with one or more clinicians.

15. The system according to claim 1, wherein the feature evaluation component determines the corresponding reference value for the defined feature based on ground truth organ segmentation data for the different anatomical structures as depicted in training medical image data.

16. A method, the method comprising: Determining, by a system including a processor, a current value of a defined feature of a corresponding segmentation mask generated for different anatomical structures included in medical image data via automatic segmentation of the medical image data; Determining, by the system, a corresponding correspondence metric between the current value and a corresponding reference value determined for the defined feature; Determining, by the system, one or more quality metrics for the automatic segmentation based on the corresponding correspondence metric; And Generating, by the system, quality assessment report data for the automatic segmentation including the one or more quality metrics.

17. The method according to claim 16, wherein determining the one or more quality metrics includes determining a quality metric for a specific structure or set of structures represented by a segmentation mask or a group of segmentation masks, and wherein the generating includes integrating the quality metric with the specific structure or set of structures in a standard clinical data format.

18. The method according to claim 16, the method further comprising: Generating, by the system, alert data based on the one or more quality metrics failing to meet a threshold quality metric; And Providing, by the system, the alert data to one or more devices associated with one or more clinicians.

19. The method according to claim 16, wherein the alert data is related to a specific structure or set of structures represented by a segmentation mask or a group of the segmentation masks, and wherein generating the alert data includes integrating the alert data with the specific structure or set of structures in a standard clinical data format.

20. A non-transitory machine-readable storage medium including executable instructions that, when executed by a processor, facilitate performance of operations including: determining a current value of a defined feature of a respective segmentation mask generated for different anatomical structures included in medical image data via automatic segmentation of the medical image data; determining a respective correspondence metric between the current value and a corresponding reference value determined for the defined feature; determining one or more quality metrics of the automatic segmentation based on the respective correspondence metric; and generating quality assessment report data for the automatic segmentation including the one or more quality metrics.

Images