Selection and Fusion of Dynamic Multimodal Segmentation

Abstract

Techniques are described for performing selection and fusion of dynamic multimodal segmentation in medical imaging. In one exemplary embodiment, a computer processing system receives a segmentation data set that includes combinations of different image segmentations of an anatomical object of interest, each segmented by a different segmentation model, from different medical images of the same acquired anatomical object. The different medical images and the different image segmentations differ in at least one of an acquisition modality, a collection protocol, and collection parameters. The system determines a ranking score for the different image segmentations that controls a relative contribution of the different image segmentations in connection with using a dynamic ranking protocol as opposed to a static ranking protocol to combine the different image segmentations to obtain a fused segmentation of the anatomical object. The system further combines the different image segmentations based on the ranking score to generate a fused image segmentation.

Description

Technical Field

[0001] This application relates to a medical imaging system, and more particularly, to techniques for the selection and fusion of dynamic multimodal segmentation.

Background Art

[0002] Multimodal imaging is a central element of current and future clinical and pre-clinical medicine. Multimodal imaging refers to the acquisition of images, preferably simultaneously or in a short period of time, using multiple complementary technical systems for the purposes of disease diagnosis, prognosis prediction, management, and continuous monitoring. For example, to observe the structural, functional, and molecular changes of cancer tissue, various state-of-the-art imaging techniques (optical imaging (by bioluminescence or fluorescence), computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), ultrasound (US), MR diffusion-weighted imaging (MR-DWI), etc.) are widely used. Each imaging modality has its own advantages and limitations (such as spatial / depth resolution and sensitivity). For example, MRI is considered the most suitable imaging modality for the characterization of soft tissue lesions, while CT is considered suitable for the analysis of bone metastases.

[0003] New research in the field of multimodal imaging has led to the publication in the literature of various clinical opinions regarding the application of the contribution degrees of different imaging modalities to the evaluation of specific types of anatomical objects (such as lesions and organs). These clinical opinions, when available, have been used to assist clinicians regarding how to utilize the contribution degrees of different imaging modalities to perform object / organ segmentation. For example, some multimodal segmentation guidelines have been developed to provide instructions on how to delineate the contours of specific anatomical objects according to the imaging modality (e.g., in scenarios where only a single modality such as CT is available), how to adjust the contours when additional imaging modalities of the object are available (e.g., MRI, T1, and / or T2), and how to adjust the contours when additional functional images are available (e.g., PET, SPECT, MR-DWI, etc.).

[0004] For example, in a certain clinical study, in order to make the contours of the esophagus and esophageal tumors clearer, in addition to CT, PET-CT was used. In another clinical study, it was found that it is beneficial to use both CT images and MRI images to depict the prostate and seminal vesicles. In another clinical study, although MRI is the gold standard for detecting lymph node metastases in most cases, it has been found that MRI-DWI, PET, and PET-CT are the most accurate in some cases, and MRI-DWI is also a technique that does not use radiation. In another example, there are no clear guidelines for laryngeal cartilage invasion (both CT and MRI are used), but clinicians feel that MRI is more accurate for a large number of patients. Based on these studies, several multi-modal segmentation guidelines have been developed and described in the literature. For example, in the case of segmenting the mandible (or lung), CT reflects the bone and / or air structures well, so no adjustment by other modalities is required, and some guidelines state that the contour can be outlined solely by CT. In another example, in the case of segmenting the prostate, some guidelines show that the contour can be outlined based on CT, but if MRI is also available, the contour can be further adjusted in a more reliable way (this is because MRI shows the boundaries of these organs well). In another example, in the case of segmenting a tumor, the contour is determined based on CT, adjusted based on MR (soft tissue discrimination is better), and if PET is available, it can be further adjusted based on PET (not only the active part but also the nearby invaded organs / lymph nodes are emphasized).

[0005] The literature defines some preliminary guidelines on how to use the contributions of different imaging modalities to perform the segmentation of several types of anatomical objects, but these guidelines are not definitive for all types of anatomical objects, let alone for all combinations of imaging modalities. Furthermore, the actual contributions obtained from each available imaging modality vary not only based on the type of object, but also for each patient and for each imaging system / image protocol in the clinical setting. Additionally, the contributions obtained from different images must take into account the temporal uncertainty and increasing uncertainty between images acquired in different imaging examinations, as well as differences in organ and body movement. Therefore, there is a need for a technology that automatically and highly intelligently takes into account all these factors and generates an optimal anatomical segmentation for a specific anatomical object and clinical context when two or more different imaging modalities are available.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] The following presents an overview for providing a basic understanding of one or more embodiments of the present invention. This overview is not intended to identify essential or important components, nor is it intended to elaborate in detail on the scope of different embodiments or the scope of the claims. The sole purpose of this overview is to present concepts in a simplified form as a prelude to the more detailed description presented later. In one or more embodiments, a system, a computer-implemented method, an apparatus, and / or a computer program product that perform the selection and fusion of dynamic multi-modal segmentation are described herein.

[0008] According to one embodiment, a system is provided. The system includes a memory storing computer-executable components and a processor executing the computer-executable components stored in the memory. The computer-executable components include a segmentation data set including combinations of different image segmentations of an anatomical object of interest segmented by different segmentation models from different medical images of the acquired anatomical object, wherein the different medical images and different image segmentations differ in at least one of an acquisition modality, a collection protocol, and collection parameters. The computer-executable components further include determining a ranking score (or ranking metric) of the different image segmentations to control a relative contribution of the different image segmentations in connection with using a dynamic ranking protocol as opposed to a static ranking protocol to combine the different image segmentations to obtain a fused segmentation of the anatomical object. The computer-executable components further include a fusion component that combines the different image segmentations based on the ranking score to generate a fused image segmentation.

[0009] In some embodiments, the elements described in connection with the disclosed system may be embodied in different forms (such as a computer-implemented method, a computer program product, or other forms). BRIEF DESCRIPTION OF THE DRAWINGS

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

DETAILED DESCRIPTION

[0011] The following forms for carrying out the invention are merely illustrative and are not intended to limit the embodiments and / or the application or use of the embodiments. Further, there is no intention to be bound by the explicit and implicit information presented in the previous "Background Art" section, "Summary of the Invention" section, or "Forms for Carrying Out the Invention" section.

[0012] As used herein, the term "multimodal segmentation data" is used to represent a segmentation data set that includes a combination of different image segmentations of an anatomical object of interest that have been segmented by different segmentation models from different medical images of an acquired anatomical object, where the different medical images and different image segmentations differ in at least one of the acquisition modality, collection protocol, and collection parameters. The disclosed subject matter is directed to a system, computer-implemented method, apparatus, and / or computer program product that performs filtering of multimodal segmentation data in connection with selecting an optimal segmentation modality for a given clinical context. The clinical context is defined at least in part based on the particular anatomical object segmented in the multimodal segmentation data and the type of imaging modality included in the multimodal segmentation data. The subject matter is also directed to a system, computer-implemented method, apparatus, and / or computer program product that fuses multimodal segmentation data into a single composite image segmentation using the relative contribution from each segmentation determined at least in part based on a given clinical use context.

[0013] To achieve this objective, in one or more embodiments, the disclosed system receives, accesses, and / or optionally generates a segmentation dataset of an anatomical object. The segmentation dataset includes image segmentations or segmentation masks of anatomical objects generated by a segmentation model from different medical images acquired and / or generated using two or more different imaging modalities (e.g., optical, CT, MRI, PET, SPECT, USI, etc.), two or more different acquisition protocols, and / or two or more different acquisition parameters. Accordingly, each of the different segmentations or segmentation masks for an object corresponds to different images having different imaging modalities and / or different image characteristics as a result of different acquisition protocols (e.g., contrast vs. non-contrast) and / or acquisition parameters used (e.g., the automatic exposure control pattern (AEC) used, the reconstruction kernel used, etc.). The segmentation or segmentation mask is generated using an existing (i.e., previously trained) trained organ / object segmentation model.

[0014] In various embodiments, the disclosed system employs a combination of both a static ranking protocol and a dynamic ranking protocol to combine different segmentations to obtain a single fused image segmentation of an anatomical object, ranking or evaluating each segmentation of the different segmentations with a ranking score that reflects the "usefulness", which is the relative contribution of each segmentation. In some implementations, the ranking score determined for each segmentation is further used to control the contribution from each segmentation in relation to generating a fused segmentation image of the anatomical object based on a combination of different segmentations (e.g., a weighted fusion procedure). Additionally, or alternatively, ranking can be used to select the "best" segmentation for clinical review and / or other processing for a given clinical use context. As described above, the clinical context is defined at least in part based on the particular anatomical object being segmented and the types of different imaging modalities. The clinical use scenario can also be defined based on one or more additional parameters regarding the intended use of the selected or fused segmentation (e.g., radiation therapy, lesion characterization, bone metastasis analysis, diagnosis, staging, use of training data, other processing by another clinical application, etc.).

[0015] The static ranking protocol involves using predefined segmentation guideline information determined for a particular anatomical object that defines or indicates the relative usefulness of different imaging modality segmentations for each anatomical object in obtaining a combined and fused segmentation by combining different imaging modality segmentations. In this regard, the predefined guideline segmentation information is based on a general understanding of how different segmentations corresponding to different imaging modalities can be combined for a given anatomical object / organ. This predefined multimodal segmentation guideline information can be determined and collected from available literature and clinical studies in the art, as described in the background art section above.

[0016] However, as described in the background art section, the literature may provide preliminary guidelines on how to use the contributions of different imaging modalities to combine different segmentations for some types of anatomical objects, but these guidelines are not definitive for all types of anatomical objects and not definitive for all combinations of imaging modalities. Furthermore, the actual contribution from each available imaging modality varies not only based on the type of object, but also on the imaging system / imaging protocol and imaging parameters used for the patient and clinical site. Additionally, the contributions from different image segmentations must take into account the temporal uncertainty and increasing uncertainty between images acquired in different imaging examinations, as well as differences in organ and body movement, which vary for each patient and clinical scenario. Moreover, the disclosed techniques are applied in situations where multimodal segmentations, each generated by one or more existing segmentation models, are provided for an anatomical object. In this context, the quality of the segmentation may vary from case to case depending on various factors (e.g., the quality of the input images, the presence of artifacts in the input images, the coverage and accuracy of the model, the presence of anatomical variations in the input images, etc.), and thus the guidelines of a static ranking protocol are insufficient or inapplicable.

[0017] To account for such a wide variety of situational variations, the disclosed system further provides a dynamic ranking protocol that individually ranks or evaluates different segmentations based on dynamic ranking factors. Unlike the guidelines of a static ranking protocol that provide general information about different anatomical objects regarding how different modalities of segmentation can be combined, the dynamic ranking factors include factors that can only be determined after imaging and organ segmentation have been completed for each patient because they include factors that cannot be predicted in advance (such as segmentation quality, presence of artifacts, deviation from standard imaging protocols, etc.). In this regard, in one or more embodiments, the dynamic ranking factors can include, but are not limited to, uncertainty measures associated with different image segmentations that reflect the uncertainty of one or more segmentation models for different image segmentations; quality measures of different image segmentations that reflect one or more measures of the quality of different image segmentations; artifact information regarding the presence or absence of artifacts; collection protocols and collection parameters respectively associated with different image segmentations; the size of the anatomical object; and the type of anatomical object.

[0018] In some embodiments, the dynamic ranking protocol includes first evaluating the quality of the multimodal segmentation in view of available static ranking multimodal segmentation guidelines. In this regard, if the guidelines of the static ranking protocol are available for a particular anatomical object and clinical context, the guidelines of the static ranking protocol are used to determine the initial evaluation, and the initial evaluation can be applied to each of the multiple segmentations that reflect the relative usefulness for the clinical context. Thereafter, according to the dynamic ranking protocol, the system can further manipulate and adjust the ranking based on multiple additional case-specific parameters or "dynamic" parameters related to the quality of the segmentation, model uncertainty, image quality, object size, object type and subtype, imaging protocol used, image acquisition timing, image registration burden, presence of bleeding, presence of artifacts, patient demographics, patient medical conditions, and clinical usage scenarios (including others). In some embodiments, the disclosed system can further employ machine learning techniques to learn and define the guidelines of the dynamic ranking protocol based on the learned patterns, correlations, and / or rules among these different dynamic parameters that affect the relative ranking of the multimodal segmentations available for a particular anatomical object. Additionally, or alternatively, the disclosed system can train and develop one or more machine learning models to automatically infer the ranking scores of the respective multimodal segmentations for a given anatomical object and clinical context based on the learned patterns, correlations, and / or rules.

[0019] In some embodiments, one or more of the disclosed systems can generate a fused segmentation image of an anatomical object based on a final ranking applied to each segmentation determined using a static ranking protocol and a dynamic ranking protocol. In these embodiments, the system can include a fusion component that projects the segmentation of the object into the same reference space in relation to an image registration process. The fusion component can further create a fused image by combining the plurality of aligned images using a ranking of the contribution of each segmentation from different modalities. In some implementations of the embodiment, the fusion component can apply a weighting scheme to different segmentation modalities based on their respective ranking scores to generate a final fused segmentation image.

[0020] Additionally, or alternatively, the fusion component can perform the fusion based at least in part on relative evaluation scores / ranking scores using one or more rule-based statistical algorithms and / or machine learning algorithms. For this purpose, one or more embodiments of the disclosed system can provide a learning-based contour selection workflow for a particular anatomical object (such as a lesion). The learning-based contour selection workflow can start with a (manual) data collection phase at multiple sites (i.e., in the data collection phase, variations are allowed as long as templates are defined so that the process is restricted and structured). In the learning phase, one or more machine learning techniques can be employed to learn the optimal way to fuse different segmentation modalities available for each type of anatomical object based on the various static and dynamic parameters described above and the results of the evaluated fusion quality. Thereafter, the system can apply the learned optimal fusion mechanism (e.g., in the form of learned rules and / or a trained fusion model) in the inference phase of the automated segmentation selection and fusion workflow.

[0021] Examples of the types of medical images processed / analyzed using the technology described in this specification include images obtained using various types of image acquisition modalities. For example, medical images can include the following images (however, they are not limited to the following images): radiotherapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis devices), magnetic resonance imaging (MRI or simply MR) images (including T1-weighted images and T2-weighted images), ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, optical images, MR-DWI, etc. In addition, medical images include images obtained by synthesizing native medical images (such as synthetic X-ray (SXR) images), images obtained by modifying native medical images or improving image quality, images obtained by expanding native medical images, and images generated using one or more image processing techniques. In this specification, the types of processed / analyzed medical image data can include two-dimensional (2D) image data, three-dimensional image data (3D) (e.g., volumetric representations of anatomical regions of the body), and combinations thereof.

[0022] As used herein, "modality" refers to a specific technical aspect in which an image or image data is captured using one or more machines or devices. In this regard, when applied to medical imaging, different acquisition modalities can include, but are not limited to, the following modalities: 2D acquisition modality, 3D acquisition modality, RT acquisition modality, XR acquisition modality, DX acquisition modality, XA acquisition modality, PX acquisition modality CT, MG acquisition modality, MR acquisition modality, MR-T1 acquisition modality, MR-T2 acquisition modality, US acquisition modality, CD acquisition modality, PET acquisition modality, SPECT acquisition modality, NM acquisition modality, MR-DWI modality, etc.

[0023] As used herein, the term "multimodal" as applied to medical image data (including segmentation data) is used to represent two or more different types of medical image data. The distinguishing factors between two or more different types of data are various. For example, the distinguishing factors can be related to the data format, data acquisition modality, data source, imaging protocol used, imaging parameters used, etc. For example, different medical images of the same anatomical object / organ corresponding to different modalities, different imaging parameters, and / or different imaging protocols can include the MR-T1 image and MR-T2 image of the object, as well as images of the object with / without using a contrast agent (however, it is not limited to these images). In another example, two different medical images corresponding to different modalities can include a diagnostic CT image of the acquired object and a CT image of the object for treatment planning.

[0024] As used herein, the terms "algorithm" and "model" are used interchangeably unless otherwise distinguished in the context. The terms "artificial intelligence (AI) model" and "machine learning (ML) model" are used interchangeably herein unless otherwise distinguished in the context.

[0025] In the present disclosure, terms such as "user" represent a human, entity, system, or combination thereof that interacts with the target medical image processing system using an appropriate computing device.

[0026] Next, one or more embodiments are described with reference to the drawings, and like numbers are used throughout the drawings to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a more thorough understanding of one or more embodiments. It will be evident, however, that in various instances, one or more embodiments may be practiced without these specific details.

[0027] Referring now to the drawings, FIG. 1 shows a block diagram of an exemplary non-limiting system 100 that performs selection and fusion of dynamic multimodal segmentation in accordance with one or more embodiments of the disclosed subject matter. Embodiments of the systems described herein can include one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), can cause the one or more machines to perform the described operations.

[0028] For example, system 100 includes computing device 101, which includes several computer-executable components (e.g., receiving component 104, ranking component 106, selection component 108, quality evaluation component 110, fusion component 112, and rendering component 116). These computer / machine-executable components (and other components described herein) can be stored in a memory associated with one or more machines. The memory is further operably coupled to at least one processor, and the components can be executed by the at least one processor such that the described operations are performed. For example, in some embodiments, these computer / machine-executable components are stored in memory 118 of computing device 101, and memory 118 can be coupled to processing unit 114 such that the components are executed. Examples of memory and processor and other suitable computer or computer-based components can be found with reference to FIG. 11 and can be used when implementing one or more of the systems or components shown and described in FIG. 1 or other figures disclosed herein.

[0029] Memory 118 can further store various information (information received by computing device 101, information used by computing device 101, and / or information generated by computing device 101 in connection with performing selection and fusion of dynamic multimodal segmentation). In the illustrated embodiment, this information includes static ranking guideline data 120, dynamic ranking protocol data 122, and fusion protocol data 124 (but is not limited thereto). In various embodiments, system 100 is configured to receive segmentation data set 102 (e.g., by receiving component 104) and process segmentation data set 102 to generate selected and / or fused segmentation data 128. Segmentation data set 102 can include a combination of different image segmentations of anatomical objects of interest, each segmented by a different segmentation model from different medical images of the acquired anatomical object, where the different medical images and different image segmentations differ in at least one of the acquisition modality, collection protocol, and collection parameters. The selected and / or fused segmentation data 128 can include a selected subset (e.g., one or more segmentation data) of segmentation data set 102 and / or a fused segmentation image generated by computing device 101 (e.g., using fusion component 112) by intelligently combining or fusing different image segmentations in a manner that optimizes the contribution from different segmentation modalities for the anatomical object to obtain a single segmentation.

[0030] Information received by computing device 101 (e.g., segmentation dataset 102) and / or information generated by computing device 101 (e.g., selected and / or fused segmentation data 128) can be presented or rendered to a user through a suitable display. In the illustrated embodiment, this display can be coupled to a user device 130 that can be communicatively and operably coupled to computing device 101 (e.g., through one or more wired or wireless communication connections) (i.e., input / output device 132 can include a display). In this regard, user device 130 can correspond to a computing device used by a user (e.g., a clinician, radiologist, technician, machine learning (ML) model developer, etc.) to interact with one or more features and functions provided by each component of computing device 101. For example, in various embodiments, one or more components of the plurality of components of computing device 101 can access and review medical images through an interactive graphical user interface (GUI) displayed on user device 130, generate and review segmentations of anatomical objects for medical images through the GUI, annotate medical images, and execute an inference model on medical images, etc., and can be associated with a medical imaging application. In some implementations of these embodiments, computing device 101 can correspond to an application server that provides at least some of these features and functions to user device 130 through a network-accessible platform (such as a web application). In these embodiments, user device 130 is communicatively coupled to computing device 101 through one or more wired or wireless communication networks (e.g., the Internet) and can access one or more of the plurality of features and functions of computing device 101 as a web application using a suitable web browser.Additionally, or alternatively, system 100 can adopt a local placement architecture in which one or more components of computing device 101 are placed locally to user device 130. Various other placement architectures for system 100 and the other systems described herein are contemplated. User device 130 can include one or more input / output devices 132 (e.g., keyboard, mouse, touch screen, display, etc.), and input / output device 132 receives user input related to the use of the features and functions of computing device 101 and displays a related GUI. Examples of some suitable input / output devices 132 are described with reference to FIG. 11 with respect to input device 1128 and output device 1136.

[0031] According to system 100, receiving component 104 can receive segmentation dataset 102 for processing by computing device 101. As described above, segmentation dataset 102 includes segmentation data of the same anatomical object depicted in two or more different medical images acquired and / or generated from the same subject using different medical imaging modalities (e.g., optical, CT, MRI, PET, SPECT, US, etc.). The number and types of different medical images and corresponding imaging modalities vary. The segmentation data of the anatomical object can include information defining the boundaries or contours of the 2D and / or 3D anatomical object (depending on the type of image on which the segmentation of the anatomical object was performed) with respect to the original image data on which the segmentation of the anatomical object was performed. For example, the segmentation data can correspond to a segmentation mask applied to the anatomical object, image markup data such as boundary lines, points, circles, etc. of the anatomical object, and / or image data of the anatomical object extracted from the input image. Segmentation dataset 102 can further include the original medical image data from which the corresponding segmentation was generated from the original medical image. In some embodiments, segmentation dataset 102 can include segmentation data of a single anatomical object of interest depicted in each multimodal image. In other embodiments, segmentation dataset 102 can include segmentation data of a plurality of different anatomical objects of interest depicted in each multimodal image (e.g., segmentation data generated using a multi-organ segmentation model).

[0032] FIG. 2 shows exemplary multimodal segmentation data according to one or more embodiments of the disclosed subject matter. In the example shown in FIG. 2, the multimodal segmentation data includes MR segmentation data 202 generated from MR image data 201 corresponding to an MR scan of a patient's brain. The multimodal segmentation data further includes CT segmentation data 204 generated from CT image data 203 corresponding to a CT scan of the same patient's brain. The MR segmentation data 202 provides segmentation data in the form of marks or contours of segmentation boundaries around anatomical objects (e.g., tissues, blood vessels, regions of interest (ROIs), etc.) whose various contours depicted in the MR image data 201 are made clear. The CT segmentation data 204 provides segmentation data in the form of a segmentation mask spanning the entire brain depicted in the CT image data 203. It should be understood that the multimodal segmentation data shown in FIG. 2 is merely exemplary, and various different types of segmentation data for various different anatomical objects related to all regions of the body are envisioned.

[0033] Referring again to FIG. 1, in accordance with the disclosed technology, the segmentation dataset 102 corresponds to an image segmentation of an anatomical object (e.g., such as the segmentation shown in FIG. 2), and the image segmentation of the anatomical object is segmented from the respective original 2D and / or 3D medical images using one or more existing segmentation models. For example, in some implementations, the segmentation model can include separate segmentation models tailored to different anatomical objects and / or different imaging modalities. In other implementations, one or more of the plurality of segmentation models can be configured to process input images from different imaging modalities to generate corresponding segmentations. Additionally, or alternatively, the segmentation model can include a multi-organ / multi-object segmentation model configured to generate segmentation data for a set of anatomical objects with well-defined contours in the input image data.

[0034] In various embodiments, each image segmentation of the plurality of anatomical object image segmentations (i.e., the output of the segmentation model) included in the segmentation dataset 102 can further include one or more uncertainty measures that reflect an estimate of the certainty or uncertainty associated with each segmentation model that generated the segmentation data. In this regard, the one or more uncertainty measures reflect a measure of certainty related to the confidence of the segmentation model in the accuracy and / or specificity of the segmentation data generated by the segmentation model. Typically, the segmentation model can be configured to output this uncertainty measure data in addition to the segmentation data. The types of uncertainty measure data are various. For example, the uncertainty measure data can include, but is not limited to, a certainty estimation score, a confidence score, a confidence interval, an accuracy score, a DICE score, and the like.

[0035] The segmentation dataset 102 can further include various rich metadata associated with each of the original image data used to generate the segmentation of anatomical objects and / or the segmentation of anatomical objects. For example, the metadata associated with each original input image data and the associated segmentation of anatomical objects can include the type of imaging modality, the imaging protocol and imaging parameters used to generate the original image data, the manufacturer / model of the imaging system used to acquire the original image data, the timestamp information describing the timing of the acquisition of the imaging data, and information identifying the source or site of the original medical image data (e.g., site name / site location). The metadata can also include information describing the characteristics of the features of the image associated with the original image data used to generate the segmentation (e.g., information regarding image resolution, luminance, pixel spacing and grid size, acquisition plane / acquisition angle, noise level, etc.). Each segmentation of a plurality of segmentations can also be associated with metadata defining or indicating the name or classification of a specific anatomical object (e.g., a specific organ, lesion, lesion type, tumor, vesicle, bone, soft tissue, etc.) according to a defined clinical ontology. In some implementations, each segmentation of a plurality of segmentations can also be associated with metadata defining or indicating the geometric characteristics of the segmentation (such as, but not limited to, the relative size, position and shape of the anatomical object (e.g., the diameter of the organ / lesion), information defining the boundary or contour of the segmentation, etc.).

[0036] In some embodiments, the metadata can also include information that identifies or indicates the quality of the segmentation of each anatomical object (i.e., segmentation quality information). For example, the segmentation quality information can include one or more quality metrics, which reflect how well the segmentation contour defines the boundary of the anatomical object clearly and correctly with respect to the original image data on which the segmentation of the anatomical object was performed. The quality information can also include a connectivity metric that represents the degree of connectivity of the contour or boundary line around the segmented anatomical object. The quality information can also include information that identifies or indicates whether artifacts (which typically degrade the image and the corresponding segmentation quality) are depicted in the original image data and / or the segmentation data. The quality information can also include one or more noise metrics that represent the level of noise in the image segmentation. Further, or alternatively, information regarding the segmentation quality of each segmentation can be determined by the quality assessment component 110 and / or information regarding the segmentation quality can be received as user feedback (such as a contrast metric, a connectivity metric of the segmentation contour, etc.) when presenting the segmentation data set 102 to one or more users (e.g., through the user device 130).

[0037] The segmentation data set 102 also includes information regarding the patient or subject represented in the segmentation data set 102. For example, the patient information can include the patient's demographic information (such as, but not limited to, age, gender, ethnicity, location, body mass index (BMI), height, weight, etc.). The patient information can also include a radiation report related to the original medical image and / or relevant pathological information of the patient derived from the patient's medical records. The patient information can also include other relevant medical history information of the patient (such as information regarding co-existing diseases, past diagnostic information, etc.).

[0038] The ranking component 106 can be configured to rank or evaluate each segmentation included in the multimodal data 102 with a ranking score that reflects the relative usefulness of each segmentation for a given clinical context. In one or more embodiments, to perform this purpose, the ranking component 106 can use a static ranking protocol, a dynamic ranking protocol, or a combination thereof. The scales used for the ranking scores are various. For example, in some embodiments, the ranking score can simply reflect the ranking order of each segmentation (e.g., first place = highest rank, second place = second highest rank, third place = third highest rank, etc.). In another example, the ranking score can reflect the relative weighted contribution of each segmentation as a contribution rate. Other scales for the ranking score are also conceivable.

[0039] In some embodiments, the ranking scores determined for each segmentation are used by the fusion component 112 to control the contribution from each segmentation when generating a fused segmentation image of an anatomical object based on combinations of different segmentations (e.g., weighted fusion). Additionally, or alternatively, the ranking is used by the selection component 108 to select the "best" segmentation for clinical review and / or other processing for a given clinical use context. As described above, the clinical context is defined at least in part based on the particular anatomical object being segmented and the distribution of corresponding different types of imaging modalities. The clinical context can also be defined based on one or more additional parameters related to the intended use of the selected or fused segmentation (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, use in training data, other processing by another clinical application, etc.). In these embodiments, information defining the intended use of the selected or fused segmentation can be included in the segmentation dataset 102, received as user feedback, and / or inferred.

[0040] The static ranking protocol involves using multimodal segmentation guideline information that is predefined and determined for specific anatomical objects, which defines or indicates the relative usefulness of different imaging modality segmentations for each anatomical object and clinical context. This predefined multimodal segmentation guideline information can be determined and collected from available literature and clinical studies in the field as described in the background art section above. In the illustrated embodiment, this predefined multimodal segmentation guideline information is represented as static ranking guideline data 120. The static ranking guideline data 120 can define ranking rules for ranking different combinations of image segmentation modalities for a specific anatomical object. For example, in some embodiments, the static ranking guideline data 120 can include an indexed data structure that identifies various known anatomical objects of interest (such as organs, lesions, bones, tissues, vesicles, etc.) that may be included in the segmentation data set 102 according to a defined clinical ontology. For each anatomical object, the static ranking guideline data 120 can further define different combinations of two or more types of imaging modalities that are considered, and each combination can receive a segmentation of the anatomical object. For each combination of imaging modalities, the static ranking guideline data 120 can further define the relative ranking score or evaluation score of each imaging modality included in that combination. For example, assume an anatomical object A and a combination of imaging modalities of modality 1, modality 2, and modality 3. The relative ranking score of this combination of the anatomical object A and the imaging modalities can indicate that the rank of modality 2 is the highest, the rank of modality 1 is the second highest, and the rank of modality 3 is the third highest.In another example, the relative ranking score can assign a ranking percentage weight to each imaging modality included in the combination (such as modality 2 is weighted 50%, modality 1 is weighted 30%, modality 3 is weighted 20%, etc.). Other ranking score metrics are also conceivable. In another implementation, a static ranking protocol or a dynamic ranking protocol can also exclusively assign the "best modality" image segmentation for use with a particular organ or anatomical object.

[0041] However, the level of granularity of the static ranking guideline data 120 is limited based on the available literature in the field, and thus it is unlikely to achieve comprehensive and definitive ranking rules for all types of anatomical objects, let alone for all combinations of imaging modality segmentation and clinical use contexts (such as radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, use of training data, other processing by other clinical applications, etc.). Moreover, the actual contribution from each available imaging modality varies not only based on the type of object but also for each patient and clinical site's imaging system / image protocol. Further, the contribution from different images must take into account the temporal uncertainty and increasing uncertainty between images acquired by different imaging examinations, as well as differences in organ and body movement, which vary for each patient and clinical scenario. Additionally, the disclosed technology is applied in situations where multimodal segmentation is provided for anatomical objects generated by one or more existing segmentation models respectively. In this situation, the quality of the segmentation may vary from case to case depending on various factors (such as the quality of the input images, the presence of artifacts in the input images, the scope and accuracy capabilities of the model, the presence of anatomical variations in the input images, etc.), and thus the guidelines of the static ranking protocol are insufficient or inapplicable in many scenarios.

[0042] Taking into account these many variations and unique defects in the static ranking guideline data 120, the ranking component 106 can additionally or alternatively use a dynamic ranking protocol to individually rank or evaluate the different multimodal segmentations of each anatomical object represented in the segmentation dataset 102. The dynamic ranking protocol considers how the actual quality and visual characteristics of the multimodal segmentations generated by different models, which are included in a given anatomical object and the subject / patient's segmentation dataset 102, affect the relative usefulness of the multimodal segmentations generated by different models for a given clinical context and / or weighted fusion process, and analyzes this. This is different from the static ranking protocol. Also, the dynamic ranking protocol can adjust the ranking scores for each combination of segmented object and modality as a function of different candidate clinical use contexts (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, use of training data, other processing by another clinical application, etc.). The dynamic ranking protocol can further consider various additional context variables as described above (e.g., in addition to simply the type of anatomical object and combination of modalities).For example, in accordance with a dynamic ranking protocol, the ranking component 106 may be based on various "dynamic" parameters related to the quality of segmentation, model uncertainty, image quality, clinical use context, object size, object type and subtype, image protocol / parameters used, image data source / site, characteristics of image features, image acquisition timing, the burden of image registration (e.g., a measure of the difference in geometric alignment between anatomical objects in each original image and / or reference space), the presence of bleeding, the presence of artifacts, patient demographics, patient pathology, and various other parameters that may be included in the metadata related to each multimodal segmentation described above, and determine a relative ranking score or evaluation score for each received multimodal segmentation of a particular anatomical object.

[0043] For example, in one or more embodiments, in accordance with the dynamic ranking protocol data 122, the ranking component 106 can determine the ranking score of each received multimodal segmentation of an anatomical object based on at least the following parameters: 1.) an uncertainty measure associated with each segmentation of a plurality of segmentations (e.g., an uncertainty measure generated by a corresponding segmentation model and included in the multimodal segmentation data), 2.) one or more segmentation quality measures associated with each segmentation of the plurality of segmentations (e.g., a segmentation quality measure determined by the quality assessment component 110 and included in the segmentation data set 102 and / or received by user feedback); and 3.) the size and / or shape of the anatomical object in each segmentation (e.g., the size and / or shape of the anatomical object determined by the quality assessment component 110 and included in the segmentation data set 102 and / or received by user feedback). In this example, the higher the measure of certainty (or the lower the measure of uncertainty), the more it correlates with a higher ranking score. Similarly, the higher the measure of segmentation quality (e.g., the stronger the measure of contour connectivity, the higher the measure of visibility of the contour boundary, the lower the noise measure, etc.), the more it correlates with a higher ranking score. The manner in which the size measure affects the ranking score varies based on the type of anatomical object. For example, in some implementations, the dynamic ranking protocol can instruct the ranking component 106 to increase the ranking score as a function of lesion size for each lesion.

[0044] It should be understood that the dynamic ranking protocol data 122 is not limited to ranking each segmentation based only on the parameters of the above example. In this regard, the dynamic ranking protocol data 122 can define multiple complex rules between different dynamic parameters for different anatomical objects and imaging modalities, and the multiple complex rules can be used by the ranking component 106 to determine the relative ranking scores of each segmentation included in different groups of multimodal segmentations. These complex rules can be based not only on the evaluation of the individual parameters associated with each segmentation, but also on the respective relative relationships between different segmentations (relative acquisition timing, relative size, relative measure of quality, relative measure of model certainty, etc.), and this relative relationship varies based on the type of each imaging modality, the combination of imaging modalities, the type of anatomical object, patient demographics, and clinical usage scenarios.

[0045] In some embodiments, the dynamic ranking protocol can include a rule-based protocol that takes into account these various dynamic parameters to determine a relative ranking score for each combination of anatomical object and segmentation modality received in the segmentation dataset 102. Additionally, or alternatively, as will be described in more detail below, the dynamic ranking protocol can incorporate the principles of artificial intelligence (AI) and machine learning (ML) to learn, define, and adapt the dynamic ranking protocol over time. The information that defines and / or controls the dynamic ranking protocol is represented as dynamic ranking protocol data 122 in the system 100.

[0046] In some embodiments, a dynamic ranking protocol (e.g., the dynamic ranking protocol defined by the dynamic ranking protocol data 122) can include first evaluating the quality of multimodal segmentation in view of the available static ranking guideline data 120. In this regard, if the ranking component 106 is available for a particular anatomical object, a combination of multimodal image segmentations, the ranking component 106 can use the static ranking guideline data 120 to determine an initial evaluation and apply the determined initial evaluation to each of a plurality of segmentations that reflect the relative usefulness of the segmentations for the clinical context. Thereafter, in accordance with the dynamic ranking protocol, the ranking component 106 can further adapt and adjust the relative ranking based on the quality of the segmentation, model uncertainty, image quality, object size, object type and subtype, the imaging protocol used, image acquisition timing, image registration burden, presence or absence of bleeding, presence or absence of artifacts, patient demographics, patient pathology, and a plurality of additional case-specific or "dynamic" parameters related to the clinical use context (including others).

[0047] As described above, the dynamic ranking protocol can be based at least in part on the quality of the segmentation of each anatomical object. The quality of each segmentation can be a function of the quality of the original image used to generate each segmentation and the quality of the segmentation itself (i.e., the output of the segmentation model). For example, quality measures related to the quality of the original image can include one or more noise measures, one or more contrast measures, one or more brightness measures, and / or one or more general quality measures related to the overall quality of each original image. Information identifying or indicating the quality of the segmentation of each anatomical object (i.e., segmentation quality information) can include one or more quality measures that reflect how well the segmentation contour defines the boundary of the anatomical object for the original image data on which the segmentation of the anatomical object was performed. The quality information can also include a connectivity measure representing the degree of connectivity of the contour or boundary line around the segmented anatomical object. Additionally, the quality information can include information identifying or indicating whether artifacts (typically, artifacts degrade the quality of the image and the corresponding segmentation) are depicted in the original image data and / or the segmentation data, and information identifying or indicating whether bleeding is detected in the original image.

[0048] In some implementations, (as described above), some or all of this quality information used by the ranking component 106 to rank each segmentation can be received along with the segmentation as metadata. Additionally, or alternatively, the quality assessment component 110 can determine one or more quality metrics for each segmentation based on metadata associated with the segmentation and / or an analysis of each segmentation using one or more image quality processing tools. For example, in some implementations, the quality assessment component 110 can integrate quality information included in the metadata associated with each segmentation to generate an overall quality score for each segmentation that takes into account different quality metrics (e.g., noise level, contrast level, brightness level, contour accuracy of the segmentation, contour connectivity, presence of artifacts, presence of bleeding, etc.). Additionally, or alternatively, the quality assessment component 110 can process each segmentation using existing image quality analysis software to determine one or more quality metrics.

[0049] In other embodiments, one or more of these quality metrics can be received as user feedback when the rendering component 116 presents each multimodal segment to one or more users. In these embodiments, prior to determining the ranking of each segmentation, the rendering component 116 presents the received multimodal segmentations to one or more users (e.g., through each user device 130) through an interactive GUI that includes means for providing quality feedback related to the quality of each segmentation. For example, through a feedback mechanism, a user can provide feedback for evaluating the overall quality of each segmentation and / or evaluate the quality of each segmentation with respect to accuracy, coverage, connectivity, visibility (e.g., depending on the lesion, the visibility of the lesion may be improved in T1, or may be improved in T2 or CT), contour boundary / background distinction, contrast, etc. The feedback mechanism can also provide a function of receiving user feedback indicating whether artifacts are present, whether bleeding is present, and whether anatomical variations are observed. The quality feedback received through the GUI can be further used by the evaluation component 106 and / or the quality evaluation component 110.

[0050] In some embodiments, after the ranking component 106 determines the relative ranking of multimodal segmentations of anatomical objects, the selection component 108 can select the most highly ranked segmentation modality. The selected segmentation can be presented to, stored / memorized for, and / or provided to one or more users for clinical review (e.g., as selected segmentation data 128) and can be used and / or processed for other uses and / or processing (e.g., uses and / or processing according to the intended clinical use context of the selected segmentation).

[0051] Further, or alternatively, the fusion component 112 can generate a fused segmentation image of the anatomical object based on the final ranking applied to each segmentation determined by the ranking component 106. In these embodiments, the fusion component 112 can project different segmentations of the anatomical object into the same reference space using conventional image registration processes that align each segmentation image relative to one another and relative to the same reference space. The fusion component 112 can further create a fused image by combining the aligned segmentation images using the ranking of the contribution of each segmentation from different modalities. In some implementations of the embodiments, the fusion component 112 can apply a weighting scheme to different segmentation modalities based on the respective ranking scores of the different segmentation modalities to generate a final fused segmentation image. The contribution of each modality segmentation is weighted relative to the ranking score of the modality, and a higher ranked segmentation modality is given a greater weight compared to a lower ranked segmentation modality. Further, or alternatively, the fusion component 112 can use one or more rule-based statistical algorithms and / or machine learning algorithms to fuse different segmentations based at least in part on the relative evaluation score / ranking score. Information defining / controlling how the fusion component 112 generates a fused segmentation of the anatomical object based on the ranking scores determined for different sets of multimodal segmentations can be defined by the fusion protocol data 124. For example, the fusion protocol data 124 can define rules and / or algorithms (e.g., weighted fusion algorithms, segmentation union algorithms, etc.).Rules and / or algorithms can be applied by the fusion component 112 to generate a fused segmentation of an anatomical object, based at least in part on the type of anatomical object, the combination of imaging modalities of the multimodal segmentation set, and the ranking scores determined for each segmentation of the multimodal segmentation set. In some embodiments, the fusion protocol data 124 can include different fusion protocols or rules tailored to different anatomical objects.

[0052] FIG. 3 shows a flowchart of an exemplary process 300 for selecting and fusing multimodal segmentation data, according to one or more embodiments of the disclosed subject matter. Process 300 corresponds to an exemplary process that can be executed by system 100 using computing device 101 and each of the components described above. For simplicity, repeated description of similar elements used in each embodiment is omitted.

[0053] Referring to FIGS. 1 and 3. In process 300, at 302, the receiving component 104 can receive a segmentation data set 102 of one or more anatomical objects. For example, in some implementations, the segmentation data set 102 can include a plurality of different medical images that depict the same anatomical region of a patient but are acquired / generated in different modalities. Each image of the plurality of different images can include segmentations of a plurality of different organs / objects (e.g., segmentation masks, segmentation image data that defines the contours of the objects superimposed on the image, etc.). In such cases, the segmentations of the individual organs / anatomical objects should be evaluated and ranked individually. In this regard, the remainder of process 300 is described with reference to a set of multimodal segmentations for a single anatomical object (e.g., a set that includes two or more multimodal segmentations).

[0054] At 304, the ranking component 106 can examine the static ranking guideline data 120 for an anatomical object to determine whether the static ranking guideline data provides conclusive static ranking information for the combination of the anatomical object and the imaging modality. In this regard, conclusive ranking information represents a clear consensus on how to rank each different segmentation modality within a set of multimodal segmentations of a particular anatomical object. At 304, if the ranking component 106 determines that no conclusive static ranking guidelines are available / defined for this particular combination of anatomical object and imaging modality, the process 300 proceeds to 308, and the ranking component 106 executes a dynamic ranking protocol to generate dynamic ranking scores for different modality segmentations. In some implementations, if some static guidelines are available for an anatomical object and one or more of the plurality of imaging modalities, the ranking component 106 can consider this "partial" static ranking guideline data for the set of multimodal segmentations in making dynamic ranking decisions. The result of the dynamic ranking at 306 includes the ranked multimodal segmentation 316 of the anatomical object.

[0055] At 304 in process 300, if the ranking component 106 determines that conclusive static ranking guidelines are defined for the combination of the anatomical object and the imaging modality, the process 300 proceeds to 308, and the ranking component 106 can generate or apply static ranking scores for different modality segmentations based on the static ranking guideline data 120.

[0056] At 310, the ranking component 106 (or the quality assessment component 110) can further evaluate the segmentation quality of each segmentation of different segmentations. In this regard, the ranking component 106 can check the segmentation quality based on static ranking (note that at 306, the segmentation quality can be incorporated into the dynamic ranking protocol in another way). In one or more embodiments, the evaluation of the segmentation quality at 310 can include the evaluation of one or more quality metrics associated with each segmentation and / or the evaluation of one or more uncertainty metrics associated with each segmentation (note that the uncertainty metric usually coincides with the quality metric). Next, at 312, the ranking component 106 (and / or the quality assessment component 110) determines whether it is possible to accept the quality of each segmentation of the set based on one or more acceptance criteria defined for one or more quality metrics and / or one or more uncertainty metrics. In some implementations, the acceptance criteria defined for the quality metric and / or the uncertainty metric can be made different according to the type of anatomical object, the type of imaging modality associated with each segmentation, and the intended clinical use context (e.g., radiotherapy, diagnosis, disease staging, etc.). In these embodiments, the information identifying or indicating the intended clinical use context can be provided in the segmentation dataset 102, received as user feedback, and / or inferred (e.g., based at least in part on the type of anatomical object). The information defining one or more acceptance criteria for one or more quality metrics and / or one or more uncertainty metrics can be included in the dynamic ranking protocol data 122.

[0057] For example, in some embodiments, at 310, the ranking component 106 can compare the uncertainty measure (e.g., confidence score, DICE score, or another model certainty / uncertainty measure) of the corresponding segmentation model associated with each segmentation of a set of multimodal segmentations to a defined tolerance criterion (such as a defined tolerance threshold) of the uncertainty measure. In these embodiments, at 312, the ranking component 106 (and / or the quality assessment component 110) can determine that the segmentation quality is unacceptable at 312 if any of the multimodal segmentations (e.g., one or more multimodal segmentations) of the plurality of multimodal segmentations have an uncertainty measure that does not meet the tolerance criterion. Next, process 300 proceeds to 306, where the ranking component 106 can determine a ranking score according to a dynamic ranking protocol. Further, or alternatively, at 310, the ranking component 106 can compare one or more quality measures associated with each segmentation of a set of multimodal segmentations to one or more defined tolerance criteria (such as defined tolerance thresholds or acceptable values) of the one or more quality measures. In these embodiments, at 312, the ranking component 106 (and / or the quality assessment component 110) can determine that the segmentation quality is not acceptable at 312 if any of the multimodal segmentations (e.g., one or more multimodal segmentations) of the plurality of multimodal segmentations have one or more quality measures that do not meet the one or more quality tolerance criteria. Next, process 300 proceeds to 306, where the ranking component 106 can determine a ranking score according to a dynamic ranking protocol.

[0058] In some implementation manners, when the ranking component 106 (or the quality evaluation component 110) determines that the segmentation quality based on only static ranking is acceptable, the process 300 proceeds to 314, and the ranking component can apply the static ranking score to different modality segmentations and bypass the dynamic ranking protocol. The result using the static ranking score at 314 also includes the ranked multimodal segmentation 316 of the anatomical object.

[0059] In some embodiments, the process 300 can proceed to 318. At 318, the selection component selects the highest-ranked segmentation for clinical use from a set of multimodal segmentations. For example, the selected segmentation 320 corresponds to the highest-ranked modality segmentation among the set of multimodal segmentations included in the multimodal segmentation data of a specific anatomical object. The rendering component 116 can render or present the selected segmentation 320 to one or more users through each user device 130 to enable clinical review, and / or provide the selected segmentation 320 to another clinical application to enable use in another clinical application. Additionally, or alternatively, the process 300 can proceed to 322. At 322, the fusion component 112 generates a fused segmentation 324 by combining a portion of different modality segmentations based on ranking as described above. The rendering component 116 can render or present the fused segmentation to one or more users through each user device 130 to enable clinical review, and / or provide the fused segmentation to another clinical application to enable use in another clinical application.

[0060] Figure 4 shows another exemplary and non - limiting system 400 for selecting and fusing dynamic multi - modal segmentation according to one or more embodiments of the disclosed subject matter. System 400 is the same as system 100 with a feedback component 406, a machine learning component 410, and machine learning data 408 added to the computing device 101, and further with initial user feedback data 402 and resultant user feedback data 404 added. For brevity, the repeated description of similar elements used in each embodiment is omitted.

[0061] As described above, in some embodiments, the rendering component 116 can present the segmentation dataset 102 to one or more users before ranking, to receive user feedback (e.g., by the feedback component 406) regarding the quality of different multimodal segmentations of each anatomically interesting object segmented in the multimodal segmentation data. In the illustrated embodiment, this user feedback on segmentation quality can include initial user feedback data 402 provided to or received by the computing device 101. In some embodiments, the initial user feedback data 402 can additionally or alternatively include information for selecting one or more specific anatomical target objects included in the multimodal segmentation data, where the multimodal segmentation data includes segmentations of a plurality of different anatomical objects (e.g., such as MR segmentation data 202), and can be used for other processing (e.g., ranking, selecting, and / or fusing using the techniques described herein). The initial user feedback data 402 can also include information for selecting a targeted clinical use context (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, use of training data, other processing by another clinical application, etc.), thereby controlling or influencing the dynamic ranking protocol applied by the ranking component 106.

[0062] The feedback component 406 can also receive result user feedback data 404 regarding the results of the ranking component 106 and / or the fusion component 112. The result user feedback data 404 can be further collected over time (e.g., as machine learning data 408), and the machine learning component 410 can improve the dynamic ranking protocol data 124, improve the fusion protocol data 124, and / or train and / or retrain the corresponding machine learning model to perform the dynamic ranking and / or segmentation fusion process as discussed in more detail below, using the result user feedback data (e.g., the corresponding ranking model / evaluation model 612 and / or the corresponding fusion model 712). In this regard, in some embodiments, the result user feedback data 404 can include information regarding the appropriateness of the evaluation scores / ranking scores applied to different multimodal segmentations for each anatomical object by the ranking component 106. In these embodiments, in addition to the selected and / or fused segmentation data 128, the rendering component 116 can render / display to the user of the user device 130 each multimodal segmentation of the received plurality of multimodal segmentations of the anatomical object, and the corresponding ranking score applied to the multimodal segmentation by the ranking component 106. This feedback information can be stored in the machine learning data 408, along with the corresponding segmentation and any associated metadata that describes the characteristics of each segmentation (i.e., the type of anatomical object and any of the dynamic parameters described herein), and the feedback information can be used by the machine learning component 410 to learn, define, improve, and / or optimize the dynamic ranking protocol data 122 over time using one or more machine learning processes.The resultant user feedback data 404 can also include user feedback regarding the quality of the fused segmentation image generated by the fusion component 112. This quality feedback information is stored in the machine learning data 408 along with the fused segmentations and the corresponding multimodal segmentations and their ranking scores (and any other information associated with the multimodal segmentations received and / or determined by the system 400), and the quality feedback information can be used by the machine learning component 410 to learn, define, improve, and / or optimize the fusion protocol data 124 over time using one or more machine learning processes.

[0063] FIG. 5 shows a flowchart of another exemplary process 500 for selecting and fusing multimodal segmentation data in accordance with one or more embodiments of the disclosed subject matter. Process 500 corresponds to an exemplary process that can be executed by system 400 using computing device 101 and each of the components described above. Process 500 is similar to process 300, and user feedback is additionally integrated into process 500. For brevity, repeated description of similar elements used in each embodiment is omitted.

[0064] Refer to FIGS. 4 and 5. In process 500, at 502, receiving component 104 can receive a segmentation data set 102 of one or more anatomical objects. At 504, the rendering component can present multimodal segmentation data to a user (e.g., a radiologist, clinician, technician, engineer, etc.) in making a request or prompt for receiving initial user feedback 402. At 506, feedback component 406 can receive initial user feedback data 402. As described above, this initial user feedback data 402 can include information for selecting a particular anatomical target object included in the multimodal segmentation data and can be used to perform other processing (e.g., ranking, selecting, and / or fusing using the techniques described herein). The initial user feedback data 402 can also include information for selecting a target clinical use context. The initial user feedback data 402 can also include user feedback regarding the quality of each segmentation associated with each modality of different modalities.

[0065] At 508, for the selected anatomical target object (if an anatomical target object is selected), or for each segmented anatomical object if no anatomical target object is selected, the ranking component 106 examines the static ranking guideline data 120 for the selected anatomical object to determine whether the static ranking guideline data provides definitive static ranking information for the combination of the anatomical object and the imaging modality. At 504, if the ranking component 106 determines that no / undefined definitive static ranking guidelines are available for this particular combination of anatomical object and imaging modality, the process 500 proceeds to 510, where the ranking component 106 executes a dynamic ranking protocol to generate dynamic ranking scores for the segmentations of different modalities. The result of the dynamic ranking at 510 includes a ranked multimodal segmentation 520 of the anatomical object.

[0066] In process 500, at 508, if the ranking component 106 determines that definitive static ranking guidelines are defined for a combination of an anatomical object and an imaging modality, the process 500 proceeds to 512, where the ranking component 106 can generate or apply static ranking scores for segmentations of different modalities based on the static ranking guideline data 120. At 514, the ranking component 106 (or the quality assessment component 110) can further evaluate the segmentation quality of each segmentation of different segmentations, as described above with reference to process 300. Then, at 516, the ranking component 106 (and / or the quality assessment component 110) determines whether the quality of each segmentation in the set is acceptable based on one or more defined acceptance criteria for one or more quality metrics and / or one or more uncertainty metrics, as described above with reference to process 300. At 516, if the ranking component determines that the segmentation quality of the set is not acceptable, the process 500 proceeds to 510, where the ranking component 106 can determine a ranking score according to a dynamic ranking protocol. In some embodiments, at 516, if the ranking component 106 (or the quality assessment component 110) determines that the segmentation quality based on static ranking only is acceptable, the process 500 can proceed to 518, and the ranking component can apply the static ranking score to different modality segmentations and bypass the dynamic ranking protocol. The result using the static ranking score at 518 also includes the ranked multi-modal segmentation 520 of the anatomical object.

[0067] In some embodiments, process 500 can proceed to 518. At 518, selection component 108 selects the highest-ranked segmentation from the set as the selected segmentation 524. Further, or alternatively, process 500 can proceed to 526. At 526, fusion component 112 generates a fused segmentation 528 by combining a portion of different modality segmentations based on the ranking as described above. At 530, the rendering component can further render or present the result, i.e., the selected segmentation 524 and / or the fused segmentation, to the user. In some embodiments, the rendered result can also include the ranked multimodal segmentation 520 of the anatomical object (e.g., together with the ranking score of the multimodal segmentation). When presenting the result at 530, the rendering component 116 can also provide the user with a prompt or request for user feedback data 404 regarding the result. At 532, the feedback component can receive the result user feedback data 404. As described above, the result user feedback data 404 can include information regarding the appropriateness of the evaluation score / ranking score applied to each multimodal segmentation by the ranking component 106 (based on a review of the ranked multimodal segment 520). The result user feedback data 404 can also include information regarding the quality of the fused segmentation 528.The feedback component 406 further stores the relevant input data and output data, along with the relevant result feedback, in the machine learning data 408, and the machine learning component 410 uses these data to improve the dynamic ranking protocol data 122, improve the fusion protocol data 124, and / or train and / or retrain a corresponding machine learning model (e.g., a corresponding ranking model / evaluation model 612 and / or a corresponding fusion model 712) to perform a dynamic ranking and / or segmentation fusion process as further discussed in more detail below.

[0068] In this regard, referring to FIG. 4, the machine learning component 410 can learn and define the dynamic ranking protocol data 122 based on learned patterns, correlations, and / or rules among various dynamic parameters that affect the relative ranking of multimodal segmentations available for a particular anatomical object, as described herein, using one or more machine learning techniques. Additionally or alternatively, the machine learning component 410 can train and develop one or more machine learning models (e.g., referred to herein as the ranking / evaluation model 612) to automatically infer the ranking scores of respective multimodal segmentations for a given anatomical object and clinical context based on the learned patterns, correlations, and / or rules. The machine learning component 410 can similarly use one or more machine learning techniques to learn and define the fusion protocol data 124 based on learned patterns, correlations, and / or rules that affect the quality of different fused segmentation images of different anatomical objects, segmentation protocols / rules applied to generate the segmentation images, the features and rankings of respective multimodal segmentations included in the set used to generate the fused segmentation images, and various additional dynamic parameters discussed herein that affect the quality of the fused segmentation images.Further, or alternatively, the machine learning component 410 trains and develops one or more machine learning models (e.g., referred to herein as the fusion model 712) to perform respective multimodal segmentations received for anatomical objects and learning patterns, and / or rules affecting the quality of different fusion segmentation images for different anatomical objects, segmentation protocols / rules applied to generate segmentation images, features and rankings of respective multimodal segmentations included in the set used to generate fusion segmentation images, and various additional dynamic parameters described herein that affect the quality of the fusion segmentation images, and can automatically generate a fused segmentation of the anatomical object based thereon.

[0069] To assist in this purpose, the machine learning component 410 can perform learning on any data received by the computing device 101 (e.g., the segmentation dataset 102, the initial user feedback data 402, and the resultant user feedback data 404), any data stored by the computing device (e.g., the static ranking guideline data 120, the dynamic ranking protocol data 122, the fusion protocol data 124, and the machine learning data 408), and any data generated by the computing device 101 (e.g., the ranking scores of respective multimodal segmentations, and the selected and / or fused segmentation data 128). In the following, the information received by the computing device 101 and the information generated by the computing device 101 are collected over time and included in the machine learning data 408. Hereinafter, the static ranking guideline data 120, the dynamic ranking protocol data 122, the fusion protocol data 124, and the machine learning data 408 are collectively referred to as the "collective machine learning data" of the machine learning component 410.

[0070] It should be understood that the machine learning component 410 can explicitly or implicitly perform learning related to collective machine learning data. Through the learning and / or inference decisions of the machine learning component 410, it is possible to identify and / or classify different patterns related to the collective machine learning data, determine one or more rules related to the collective machine learning data, and / or determine one or more relationships related to the collective machine learning data that affect the ranking and / or fusion optimization of multimodal segmentation. The machine learning component 410 can also use an automatic classification system and / or an automatic classification process to identify and / or classify different patterns associated with the collective machine learning data, determine one or more rules associated with the collective machine learning data, and / or determine one or more relationships associated with the collective machine learning data that affect the ranking and / or fusion optimization of multimodal segmentation. For example, the machine learning component 410 can use probabilistic analysis and / or statistical analysis (e.g., considering the usefulness and cost of the analysis) to learn one or more patterns related to the collective machine learning data, determine one or more rules related to the collective machine learning data, and determine one or more relationships related to the collective machine learning data that affect the ranking and / or fusion optimization of multimodal segmentation. The machine learning component 410 can, for example, use a support vector machine (SVM) classifier to perform learning of patterns related to the collective machine learning data, determine one or more rules related to the collective machine learning data, and / or determine one or more relationships related to the collective machine learning data that affect the ranking and / or fusion optimization of multimodal segmentation. Additionally or alternatively, the machine learning component 410 can use other classification techniques related to Bayesian networks, decision trees, and / or probabilistic classification models.The classifier used by the machine learning component 410 can be trained not only explicitly (e.g., trained by general training data) but also implicitly (e.g., trained by observing the user's behavior and receiving external information). For example, regarding a well-understood SVM, the SVM is configured through a learning phase or a training phase performed within a classifier constructor and a feature selection module. The classifier is a function that maps an input attribute vector x = (x1, x2, x3, x4, xn) to a confidence representing that the input belongs to a certain class, that is, f(x) = confidence(class).

[0071] In one aspect, the machine learning component 410 can learn one or more patterns related to collective machine learning data, determine one or more rules related to collective machine learning data, and / or determine one or more relationships related to collective machine learning data that affect the ranking and / or fusion optimization of multimodal segmentation, by partially utilizing a scheme using inference. The machine learning component 410 can further use appropriate techniques using machine learning, appropriate techniques using statistics, and / or appropriate techniques using probability. The machine learning component 410 can additionally or alternatively use a reduced set of factors (e.g., an optimized set of factors) to generate the ranking model / evaluation model 612 and / or the fusion model 712 described below. For example, the machine learning component 410 can use an expert system, fuzzy logic, SVM, hidden Markov model (HMM), greedy search algorithm, rule-based system, Bayesian model (e.g., Bayesian network), neural network, other non-linear training techniques, data fusion, utility-based analysis system, a system using a Bayesian model, etc. In another aspect, the machine learning component 410 can execute a series of machine learning calculations related to collective machine learning data. For example, the machine learning component 410 can execute a series of clustering machine learning calculations, a series of decision tree machine learning calculations, a series of machine learning calculations using instances, a series of regression machine learning calculations, a series of regularization machine learning calculations, a series of rule machine learning calculations, a series of Bayesian machine learning calculations, a series of deep Boltzmann machine calculations, a series of deep belief network calculations, a series of convolutional neural network calculations, a series of stacked autoencoder calculations, and / or a series of different machine learning calculations.The rules, patterns, and / or correlations learned by the machine learning component 410 for the collective machine learning data are further stored in the machine learning data 408, and the machine learning component 410 can apply the rules, patterns, and / or correlations to define and / or update / improve the dynamic ranking protocol data 122 and / or the fusion protocol data 124, and / or the machine learning component 410 can use the rules, patterns, and / or correlations to train and / or retrain the ranking evaluation model 612 and / or the fusion model 712 described below.

[0072] FIG. 6 shows an exemplary machine learning framework 600 for performing ranking of multimodal segmentation data according to one or more embodiments of the disclosed subject matter. The machine learning framework 600 provides an overview of a framework of one or more machine learning processes that can be executed by the machine learning component 410, and generates a ranking model / evaluation model 612 configured to rank or evaluate multimodal segmentations of one or more anatomical objects with an optimal dynamic ranking protocol data 610 and / or an evaluation score or ranking score reflecting the relative usefulness for a given clinical context.

[0073] Refer to FIGS. 5 and 6. In framework 600, at 604, machine learning component 410 can perform curation of training data for ranking of multimodal segmentation. In some embodiments, this training data curation process analyzes all data available to machine learning component 410 to determine how different combinations of modality segmentations for the same anatomical object contribute to the usefulness of modality segmentation (i.e., the relative evaluation score of modality segmentation) for different clinical contexts, considering various dynamic parameters (and parameter values) associated with the different combinations. Information that defines or indicates this can be identified and extracted. Machine learning component 410 can perform this curation process for all defined anatomical objects represented in received segmentation dataset 102 (e.g., all anatomical objects that can be defined in the human body, including all organs, all tissues, all vesicles, all types of lesions and tumors, etc.). As described above, training component 410 can identify and extract the above information from static ranking guideline data 120, dynamic ranking protocol data 122, fusion protocol data 124, and machine learning data 408. In this context, machine learning data 408 can include all segmentation datasets 102 previously processed by system 100 and / or system 400, and related metadata, results of processing (e.g., quality data, ranking scores, fused image segmentations that may be determined by quality assessment component 110), and (if provided) related initial user feedback data 404 and resultant user feedback data 404. In some embodiments, at 604, machine learning component 410 can index the extracted information for each defined anatomical object.

[0074] In one or more embodiments, as a result of the training data curation process, the machine learning component 410 can generate training data 606 that includes a set of multimodal segmentations of each anatomical object. Each set can include two or more different combinations of imaging modalities. For each anatomical object, there should be included a plurality of representative sets of each possible combination of the received imaging modalities of the anatomical object. Further, for each set, the training data 606 can include information that defines and / or indicates the respective relative ranking scores of the segmentations. In some embodiments, the ground truth information that defines the ranking scores can be obtained from a previously processed segmentation data set 102 and previously determined ranking scores / evaluation scores (e.g., included in the machine learning data 408). In these embodiments, the set of multimodal segmentations of each anatomical object can include a previously processed data set of multimodal segmentations. The information indicating the ranking scores can include the extracted or identified features associated with each of the segmentations of the plurality of segmentations that correlate with the ranking scores. For example, this information can include segmentation quality, model uncertainty, image quality, clinical use context, object size, object type and subtype, imaging protocol / parameters used, image data source / site, image feature properties, image acquisition timing, image registration burden (e.g., a measure of the difference in geometric alignment between each original image and / or anatomical object in a reference space), presence of bleeding, presence of artifacts, patient demographics, patient medical condition, and various other parameters (there can be many others) related to various "dynamic" parameters related to the metadata associated with each of the multimodal segmentations described above.

[0075] Additionally, or alternatively, the set of multimodal segmentations of each anatomical object can include a new (unprocessed) set of multimodal segmentation data. These new sets can be provided to and extracted from the multimodal segmentation data 602 by the machine learning component 410 during the curation process of the training data. These new sets can correspond to new instances of the segmentation data set 102 and can include or be associated with the same or similar metadata as the metadata described in the multimodal segmentation data 602 and can include information (e.g., dynamic parameters) indicating the respective relative ranking scores of the segmentations described above.

[0076] At 608, the machine learning component 410 can execute a machine learning phase of one or more machine learning processes. In this regard, in some embodiments, at 608, the machine learning component 410 uses the training data 606 to utilize one or more machine learning processes and, according to the learned correlation, learned patterns, and / or learned rules function between various dynamic parameters and their associated parameter values, can learn and define an optimal dynamic ranking protocol (e.g., optimal dynamic ranking protocol data 610) for each combination of anatomical objects and multimodal segmentation data. According to these embodiments, the optimal dynamic ranking protocol data 610 can be used to update the dynamic ranking protocol data 122. For example, in some implementations, the machine learning component 410 can adjust the dynamic ranking protocol data 122 to reflect the optimal dynamic ranking protocol data 610. Additionally, the optimal ranking protocol data 610 can be added to the memory 118.

[0077] Furthermore, or alternatively, at 608, the machine learning component 410 can use the training data 606 to train and develop an object-specific ranking evaluation model 612. For example, in some embodiments, the ranking model / evaluation model 612 can include separate models tailored to different anatomical objects. The input to each model can include segmentation data of anatomical objects of available modalities, and the input indicates the modality associated with each input image segmentation. In some embodiments, the input can also identify the clinical context. The input can also be the relevant extracted dynamic features and feature values, including the dynamic features and feature values associated with each segmentation. The output of each model can include a ranking score or an evaluation score indicating the relative usefulness for the clinical context. In these embodiments, all multi-modal segmentations within the set for an anatomical object can be processed by the same model to generate the corresponding ranking score or evaluation score. The evaluation component 106 can further apply the respective scores estimated for each segmentation within the set to rank or order the segmentations.

[0078] In this regard, the ranking model / grading model 612 can each include a machine learning model and can also correspond to a machine learning model. The ranking model / evaluation model 612 can use various types of ML algorithms (for example, deep learning models, neural network models, deep neural network models (DNN), convolutional neural network models (CNN), adversarial generative neural network models (GAN), transformers, etc., but is not limited thereto). In some implementations of these embodiments, the machine learning component 410 executes a supervised machine learning process to conform to the accuracy based on relevant result user feedback (if available), and uses the ground truth evaluation information provided for the previously processed multimodal segmentation to train each model of each object. The test set can be provided by the multimodal segmentation data 602. Once trained, the ranking model / evaluation model 612 is stored in the memory 118, and at runtime, the ranking model / evaluation model 612 can be applied by the ranking component 106 to the newly received segmentation data set 102 to estimate the ranking score. In these embodiments, executing the dynamic ranking protocol in 306 of process 300 and / or 510 of process 500 can include applying the corresponding ranking model / evaluation model 612 to generate a ranking score.

[0079] FIG. 7 shows an exemplary machine learning framework 700 for fusing multimodal segmentation data according to one or more embodiments of the disclosed subject matter. Machine learning framework 700 shows an overview of a framework for one or more machine learning processes, where one or more machine learning processes may be executed by machine learning component 410 and may generate optimal fusion protocol data 710 and / or one or more fusion models 712. Optimal fusion protocol data 710 may include learned rules and / or learned algorithms that define a method for combining different sets of multimodal segmentations of different anatomical objects to generate a fused segmentation image based on respective evaluation / ranking scores of the multimodal segmentations and / or based on different dynamic parameters / parameter values for a given clinical context. Fusion model 712 may correspond to respective machine learning models adapted to perform the same.

[0080] Refer to FIGS. 5 and 7. In framework 700, at 704, the machine learning component 410 can perform curation of training data for multimodal segmentation fusion. The curation of training data at 704 can correspond to the curation of training data at 604 described with reference to framework 600. Additionally, or alternatively, the process of training data curation at 704 can include more specifically identifying and extracting a previously processed set of multimodal segmentation data and the resulting fused segmentation images, along with relevant result user feedback that evaluates the quality of each fused image. In some implementations, additional examples of fused segmentation images for different anatomical objects and the corresponding set of multimodal segmentations can be provided in multimodal segmentation and fusion data 702. These additional examples can include information that defines or shows how different multimodal segmentations were combined to generate the corresponding fused segmentations. In some embodiments, the additional exemplification is manually generated and can be annotated.

[0081] In one or more embodiments, as a result of the training data curation process at 704, the machine learning component 410 can generate training data 706 that includes multiple sets of multimodal segmentations of each anatomical object. Each set of the multiple sets can include two or more different combinations of imaging modalities. For each anatomical object, multiple representative sets of each possible combination of imaging modalities received for the anatomical object should be included. Further, for each set, the training data 606 can include information that defines and / or indicates the relative ranking scores of each segmentation, as described with reference to architecture 600. Additionally, the training data 706 can include, for each set, a previously known and / or applied fusion protocol and, if available, a fused segmentation previously generated for each set that includes fusion quality feedback (i.e., ground truth data).

[0082] At 708, the machine learning component 410 can perform the machine learning phase of one or more machine learning processes. In this regard, in some embodiments, at 708, the machine learning component 410 can use the training data 706 to learn and define optimal fusion protocol data 710 for each combination of anatomical objects and multimodal segmentation data according to learned correlations, learned patterns, and / or learned rules between various dynamic parameters and associated parameter values using one or more machine learning processes. With these embodiments, the optimal fusion protocol data 710 can be used to update the fusion protocol data 124. For example, in some embodiments, the machine learning component 410 can adjust the dynamic fusion protocol data 124 to reflect the optimal fusion protocol data 710. Further, the optimal fusion protocol data 710 can be added to the memory 118.

[0083] Additionally, or alternatively, at 708, the machine learning component 410 can use the training data 706 to train and develop an object-specific fusion model 712. For example, in some embodiments, the fusion model 712 can include separate models tailored to different anatomical objects. The input to each model can include a set of segmentation data of the anatomical object that includes all different image segmentation modalities. In this regard, the fusion model 712 can correspond to a multi-channel input model, where the input indicates the modality associated with each input image segmentation. In some implementations, the input can identify the clinical context. In some implementations, the input can also include a ranking score or an evaluation score determined or estimated for each segmentation. In other implementations, each fusion model 712 can take into account the nature of this rating evaluation. The input can be the relevant extracted dynamic features and feature values, which can also include the dynamic features and feature values associated with each segmentation. The output of each fusion model can be a fused segmentation of the input multi-modal segmentation set of the anatomical object, which can include a fused segmentation that combines different input segmentations in an optimal fusion method. The fusion component 112 can further apply the corresponding fusion model of each anatomical object to a new input multi-modal segmentation set at runtime. In this regard, performing the fusion process at 322 of process 300 and / or at 526 of process 500 can include applying the corresponding fusion model 612 (of the particular anatomical object in question) to generate a fused segmentation. Additionally, or alternatively, once the fusion model 612 is trained and developed, the rating process or the ranking process can be completely bypassed.

[0084] In this regard, the plurality of fusion models 712 can each include a machine learning model or can correspond to a machine learning model. The fusion model 712 can use various types of ML algorithms (such as, but not limited to, deep learning models, neural network models, DNNs, CNNs, GANs, transformers, etc.). In some implementations of these embodiments, the machine learning component 410 can execute a supervised machine learning process to use previously generated fused segmentations as a ground truth exemplar to train each fusion model 612 for each object. In these embodiments, only a set having fused segmentations regarded as high-quality segmentations is used during the training of each fusion model, and the input multimodal segmentation can be converted into an optimal fused segmentation image. In these embodiments, the plurality of fusion models 712 can each include a transformation network.

[0085] FIG. 8 shows a block diagram of an exemplary and non-limiting computer-implemented method 800 for generating a fused multimodal segmentation image according to one or more embodiments of the disclosed subject matter. For brevity, repeated description of similar elements used in each embodiment is omitted.

[0086] At 802, method 800 includes a system (e.g., system 100 or system 400) including a processor receiving (e.g., by receiving component 104) a segmentation dataset (e.g., segmentation dataset 102) including combinations of different image segmentations of an anatomical object of interest segmented by different segmentation models from different medical images of the acquired anatomical object, where the different medical images and different image segmentations differ in at least one of an acquisition modality, a collection protocol, and collection parameters. At 804, method 800 includes determining (e.g., by ranking component 106) ranking scores for the different image segmentations to control a relative contribution of the different image segmentations in connection with using a dynamic ranking protocol (e.g., using dynamic ranking protocol data 122) as opposed to a static ranking protocol (e.g., using only static ranking guideline data 120) to combine the different image segmentations to obtain a fused segmentation of the anatomical object. At 806, method 800 includes combining the different image segmentations (e.g., using fusion component 112) based on the ranking scores to generate a fused image segmentation (e.g., fused segmentation 324, fused segmentation 528, etc.).

[0087] FIG. 9 shows a block diagram of another exemplary and non - limiting computer - implemented method 900 for generating a fused multimodal segmentation image according to one or more embodiments of the disclosed subject matter. For brevity, repeated description of similar elements used in each embodiment is omitted.

[0088] At 902, method 900 includes a system (e.g., system 400) including a processor receiving (e.g., by receiving component 104) a segmentation dataset (e.g., segmentation dataset 102) including combinations of different image segmentations of anatomical objects of interest each segmented by a different segmentation model from different medical images of the acquired anatomical object, where at least one of the different medical images and the different image segmentations differs in acquisition modality, acquisition protocol, and acquisition parameters. At 904, method 900 includes the system determining (e.g., by ranking component 106 and machine learning component 410) ranking scores for the different image segmentations that reflect the relative usefulness of the different image segmentations for a clinical context using one or more machine learning processes. For example, in some embodiments, the one or more machine learning processes can include learning, defining, and / or updating dynamic ranking protocol data 122 for use by ranking component 106 to perform the ranking using one or more machine learning processes (e.g., by machine learning component 110). Additionally or alternatively, the one or more machine learning processes can include training and developing a ranking model / evaluation model 612 by a machine learning component and then applying the corresponding ranking model / evaluation model 612 to the corresponding segmentation to generate ranking scores (e.g., by ranking component 106) as described with reference to machine learning framework 600. At 906, method 900 includes the system combining the different image segmentations (e.g., using fusion component 112) using a weighting scheme for the different image segmentations based on the ranking scores to generate a fused image segmentation of the anatomical object (e.g., fused segmentation 324, fused segmentation 528, etc.).

[0089] Figure 10 shows a block diagram of another exemplary and non - limiting computer - implemented method 1000 for generating a fused multimodal segmentation image according to one or more embodiments of the disclosed subject matter. For the sake of brevity, the description of similar elements used in each embodiment is omitted from being repeated.

[0090] At 1002, method 1000 includes a system (e.g., system 400) including a processor receiving (e.g., by receiving component 104) a segmentation data set (e.g., segmentation data set 102) including combinations of different image segmentations of an anatomical object of interest, each segmented by a different segmentation model, from different medical images of the acquired anatomical object, where at least one of the different medical images and different image segmentations differs in acquisition modality, collection protocol, and collection parameters. At 1004, method 1000 includes the system using a segmentation model (e.g., one or more of the plurality of fusion models 712) previously trained for the anatomical object to combine the different image segmentations (e.g., using fusion component 112) to generate a fused image segmentation of the anatomical object (e.g., fused segmentation 324, fused segmentation 528, etc.).

[0091] One or more embodiments can be a system, method, and / or computer program product that can be integrated at a technically detailed level. The computer program product can include one or more computer - readable storage media having computer - readable program instructions for causing a processor to execute one or more aspects of the present embodiments.

[0092] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. More specific examples of computer-readable storage media are listed below, and while not all possible examples can be listed, there are portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk-read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or raised structures with instructions recorded in grooves), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse propagating through an optical fiber cable), or an electrical signal transmitted through a wire.

[0093] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing devices / processing devices, or downloaded by a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or an external storage device. The network can include copper wire transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface of each computing device / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions, and the instructions are stored in a computer-readable storage medium within each computing device / processing device.

[0094] Computer-readable program instructions for carrying out the operations of the present invention can be in any combination of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages (such as object-oriented programming languages like Smalltalk, C++, procedural programming languages like the "C" programming language or similar programming languages, and machine learning programming languages like CUDA, Python, TensorFlow, PyTorch, etc.). The computer-readable program instructions can be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server using appropriate processing hardware. In the latter case, the remote computer can be connected to the user's computer through any type of network (such as a local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (e.g., through the Internet using an Internet service provider). In various embodiments including machine learning programming instructions, the processing hardware can include one or more graphics processing units (GPUs), central processing units (CPUs), etc. For example, one or more evaluation models / ranking models 612 and / or one or more fusion models 712 can be described in an appropriate machine learning programming language and executed by one or more GPUs, CPUs, or a combination thereof.In some embodiments, to implement aspects of the present invention, an electronic circuit (e.g., a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), etc.) can utilize the state information of computer-readable program instructions to execute the computer-readable program instructions and personalize the electronic circuit.

[0095] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, or other device to function in a particular manner, and the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions for implementing the aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0097] Also, computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operations to be performed on the computer, other programmable apparatus, or other devices to generate a process executable on the computer, and the instructions executed on the computer, other programmable apparatus, or other devices can implement the functions / operations specified in one or more blocks of a flowchart or block diagram.

[0098] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowchart or block diagram may represent a module, segment, or portion of instructions, which can include one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may be executed in an order different from that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the functions involved. It should also be noted that each block of the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, can be implemented by a system using dedicated hardware that executes the specified function or operation or a combination of dedicated hardware and computer instructions.

[0099] With reference to FIG. 11, the systems and processes described below can be embodied within hardware (such as a single integrated circuit (IC) chip, multiple ICs, an application-specific integrated circuit (ASIC), etc.). Furthermore, the order in which some or all of the process blocks of the multiple process blocks appear in each process should not be considered limiting. Rather, it should be understood that some process blocks can be executed in various orders, and not all of these orders can be explicitly illustrated herein.

[0100] Refer to FIG. 11. An exemplary environment 1100 for implementing various aspects of the claimed subject matter includes a computer 1102. The computer 1102 includes a processing unit 1104, a system memory 1106, a codec 1135, and a system bus 1108. The system bus 1108 couples system components (not limited to the system memory 1106) including the system memory 1106 to the processing unit 1104. The processing unit 1104 can be any of a variety of commercially available processors. Dual microprocessors and other multiprocessor architectures can also be used as the processing unit 1104.

[0101] The system bus 1108 can be any of several types of bus structures, such as a memory bus or memory controller, a peripheral bus or external bus, or a local bus using any of a variety of available bus structures. Examples of bus architectures include, but are not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association Bus (PCMCIA), FireWire (IEEE 1394), Small Computer System Interface (SCSI).

[0102] System memory 1106 includes volatile memory 1110 and non-volatile memory 1112, and in various embodiments, one or more of the disclosed memory architectures can be used. The basic input / output system (BIOS) includes basic routines for transferring information between elements within computer 1102, such as during startup, and is stored in non-volatile memory 1112. Further, according to the present technology, codec 1135 can include at least one of an encoder and a decoder, and at least one of the encoder and the decoder can be configured by hardware, software, or a combination of hardware and software. Although codec 1135 is illustrated as a separate component, codec 1135 can be included within non-volatile memory 1112. By way of example and not limitation, non-volatile memory 1112 can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, 3D flash memory, or resistive change memory (such as Resistive RAM (RRAM)). Non-volatile memory 1112 can use one or more of the disclosed memory devices in at least some embodiments. Further, non-volatile memory 1112 can be computer memory (e.g., physically integrated with computer 1102 or the main board of computer 1102) or removable memory. Examples of suitable removable memory that can be implemented in the disclosed embodiments include Secure Digital (SD) cards, CompactFlash (registered trademark) (CF) cards, Universal Serial Bus (USB) memory sticks, and the like. Volatile memory 1110 includes random access memory (RAM) that functions as an external cache memory, and one or more of the disclosed memory devices can be used in various embodiments.By way of example and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM).

[0103] Computer 1102 can also include removable / non-removable volatile / non-volatile computer storage media. FIG. 11 shows, for example, disk storage 1114. Disk storage 1114 includes, but is not limited to, devices such as magnetic disk drives, solid state disks (SSDs), flash memory cards, or memory sticks. Further, disk storage 1114 can include storage media separately from or in combination with other storage media, and this storage media can include, but is not limited to, optical disk drives (such as compact disk ROM devices (CD-ROM), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives) or digital versatile disk ROM drives (DVD-ROM)). To connect disk storage 1114 to system bus 1108, typically a removable or non-removable interface (such as interface 1116) is used. It is understood that disk storage 1114 can store information related to an entity. Such information may be stored on or supplied to a server, or supplied to an application running on an entity device. In one embodiment, an entity can be notified (e.g., by one or more output devices 1136) of the type of information stored in disk storage 1114 or transmitted to a server or application. An entity can be given the opportunity to opt in or opt out of such information being collected (e.g., by input from one or more input devices 1128) and shared with a server or application.

[0104] It should be understood that FIG. 11 illustrates software that functions as an intermediary between entities and basic computer resources in a suitable operating environment 1100. This software includes an operating system 1118. The operating system 1118 can be stored in disk storage 1114 and serves to control and allocate the resources of computer system 1102. Application 1120 utilizes the resource management by operating system 1118 through program modules 1124 and program data 1126 (such as a boot / shutdown transaction table) stored either in system memory 1106 or disk storage 1114. It should be understood that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

[0105] An entity inputs commands or information into computer 1102 through input device 1128. Input device 1128 includes, but is not limited to, pointing devices (such as mice, trackballs, styli, touch pads, keyboards, microphones, joysticks, game pads, satellite broadcast receiving antennas, scanners, TV tuner cards, digital cameras, digital video cameras, web cameras, etc.). These input devices and other input devices are connected to processing unit 1104 through interface port 1130 and system bus 1108. Interface port 1130 includes, for example, serial ports, parallel ports, game ports, and universal serial buses (USBs). Output device 1136 uses some of the ports of the same type as those of input device 1128. Thus, for example, a USB port can be used to supply input to computer 1102 and output information from computer 1102 to output device 1136. Output adapter 1134 is shown to indicate that among some of the several output devices 1136, special adapters are required for some output devices 1136 such as monitors, speakers, printers, etc. Output adapter 1134 includes video cards and sound cards that implement the connection means between output device 1136 and system bus 1108, but this is an example and is not limited thereto. It should be noted that other devices or systems of other devices provide both input and output functions (such as remote computer 1138, etc.).

[0106] Computer 1102 can operate in a network environment using a logical connection to one or more remote computers, such as remote computer 1138. Remote computer 1138 can be a personal computer, server, router, network PC, workstation, device using a microprocessor, peer device, smartphone, tablet, or other network node, and typically includes many of the elements described in relation to computer 1102. For simplicity, only memory storage device 1140 is illustrated with remote computer 1138. Remote computer 1138 is logically connected to computer 1102 through network interface 1142 and then connected through communication connection portion 1144. Network interface 1142 includes a wired or wireless communication network, such as a local area network (LAN), wide area network (WAN), and cellular network. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include point-to-point links, circuit-switched networks (such as integrated services digital network (ISDN) and ISDN-derived technologies), packet-switched networks, digital subscriber line (DSL), but are not limited to these.

[0107] Communication connection portion 1144 represents the hardware / software used to connect network interface 1142 to bus 1108. Communication connection portion 1144 is shown inside computer 1102 for illustrative clarity, but can also be provided outside computer 1102. The hardware / software required for the connection to network interface 1142 includes, for illustrative purposes only, internal and external technologies (modems (such as ordinary telephone-grade modems, cable modems, DSL modems), ISDN adapters, wired and wireless Ethernet cards, hubs, routers, etc.).

[0108] The illustrated aspects of the present disclosure can also be implemented in a distributed computing environment where certain tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0109] Referring to FIG. 12, a schematic block diagram of a computing environment 1200 according to the present disclosure is shown, in which a system (e.g., system 600, etc.), method, and computer-readable medium of the present subject matter can be arranged. The computing environment 1200 includes one or more clients 1202 (e.g., laptop, smartphone, PDA, media player, computer, portable electronic device, tablet, etc.). The client 1202 can be hardware and / or software (e.g., thread, process, computing device). The computing environment 1200 also includes one or more servers 1204. The server 1204 can be hardware, or hardware combined with software (e.g., thread, process, computing device). The server 1204 can store threads, for example, to perform conversions by using aspects of the present disclosure. In various embodiments, one or more components, devices, systems, or subsystems of the system 400 can be arranged as hardware and / or software in the client 1202 and / or as hardware and / or software arranged in the server 1204. One possible communication between the client 1202 and the server 1204 is in the form of data packets transmitted between two or more computer processes. Examples of data packets include healthcare-related data, training data, AI models, input data for AI models, encrypted output data generated by AI models, etc. The data packets can include, for example, metadata (e.g., related context information). The computing environment 1200 includes a communication framework 806 (e.g., a global communication network such as the Internet, or a mobile network) that can be used to perform communication between the client 1202 and the server 1204.

[0110] Communication can be performed by wired (e.g., optical fiber) technology and / or wireless technology. Client 122 includes, or is operably connected to, one or more client data stores 1208 that can be used to store local information of client(s) 1202 (e.g., client medical image application logic 630). Similarly, server 1204 operably includes, or is operably connected to, one or more server data stores 812 that can be used to store local information of server 1204 (e.g., server medical image application logic 612, segmentation model data 510, medical image database 518, etc.).

[0111] In one embodiment, client 1202 can transfer an encoded file to server 1204 in accordance with the disclosed subject matter. Server 1204 can save the file, decode the file, or send the file to another client 1202. It should be understood that client 1202 can transfer an uncompressed file to server 1204 and server 1204 can compress the file in accordance with the disclosed subject matter. Similarly, server 1204 can encode video information and send the information to one or more clients 1202 through communication framework 1206.

[0112] This subject matter has been described in the general context of computer-executable instructions of a computer program product executed on one or more computers, but those skilled in the art will recognize that the present disclosure can also be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Further, those skilled in the art will understand that the methods implemented by the computers of the present invention can also be implemented in other computer system configurations (single-processor or multi-processor computer systems, minicomputer devices, mainframe computers, as well as computers, handheld computer devices (e.g., PDAs, telephones), consumer or industrial electronic devices using microprocessors or programmable). Also, the illustrated aspects can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network. However, although not all aspects of the present disclosure, some aspects can be implemented on a stand-alone computer. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0113] In this application, "component", "system", "subsystem", "platform", "layer", "gateway", "interface", "service", "application", "device", etc. represent or can include one or more computer-related entities having one or more specific functions or entities related to an operating machine having one or more specific functions. The entities disclosed herein can be either hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a server and the server can be components. One or more components can exist within an execution process or execution thread, and one component can be localized to one computer or distributed among two or more computers. In another example, each component can be executed from various computer-readable media storing various data structures. A component can communicate, for example, according to a signal having one or more data packets (e.g., data from a certain component that exchanges data with another component in a local system, in a distributed system, or in other multiple systems with signals mediated on a network such as the Internet) through a local process or a remote process. As another example, a component can be a device in which a specific function is provided by a mechanical part operated by an electric circuit or an electronic circuit, and the device can be driven by software or a firmware application executed by a processor. In such a case, the processor can be inside or outside the device and can execute at least a part of the software or firmware application.As yet another example, the component can be a device that provides a particular function through an electronic component without mechanical parts, and the electronic component can include a processor or other means that executes software or firmware that provides at least a portion of the function of the electronic component. In one aspect, the component can emulate the electronic component by, for example, a virtual machine within a cloud computing system.

[0114] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" is intended to mean any inclusive and natural combination. That is, if X uses A, X uses B, or X uses both A and B, any of these examples will satisfy "X uses A or B". Additionally, the articles "a" and "an" as used herein and in the drawings should generally be construed to mean "one or more" unless otherwise specified or clear from the context that the singular form is intended. In this specification, the terms "example" and / or "exemplary" are used in the sense of serving as an example, an instance, or an illustration. To avoid misunderstanding, the subject matter disclosed herein is not limited to such examples. Additionally, aspects or designs described herein as "example" and / or "exemplary" should not necessarily be construed as more preferable or advantageous than other aspects or designs, nor are they intended to exclude exemplary equivalent structures and techniques known to those skilled in the art.

[0115] As used herein, the term "processor" can represent substantially any computing processing unit or device, including but not limited to a single-core processor, a single processor with a multi-threaded execution function by software, a multi-core processor, a multi-core processor with a multi-threaded execution function by software, a multi-core processor using hardware multi-threading technology, a parallel platform, and a parallel platform with distributed shared memory. Further, a processor can represent an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Additionally, a processor can utilize nanoscale architectures such as transistors, switches, and gates using molecules and quantum dots for the purpose of optimizing space usage or improving the performance of an entity device, but is not limited thereto. A processor can also be implemented as a combination of multiple computing processing units. In the present disclosure, terms such as "store", "storage", "data store", "data storage", "database", and substantially any other information storage element related to the operation and function of components are used to represent an entity embodied in a "memory component", "memory", or an element including a memory. It should be understood that the memory and / or memory components described herein can be volatile memory or non-volatile memory, or can include both volatile memory and non-volatile memory.As an example, the non-volatile memory can be, but is not limited to, read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). As the volatile memory, for example, there is RAM, and the RAM can function as an external cache memory. As an example, the RAM can be utilized in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct rambus RAM (DRRAM), direct rambus dynamic RAM (DRDRAM), rambus dynamic RAM (RDRAM), etc., but is not limited thereto. Further, in this specification, the disclosed memory components of the methods implemented by the system or computer are intended to include these and any other suitable types of memory, but are not limited thereto.

[0116] What has been described above is only an example of a system and a method implemented by a computer. Of course, for the purpose of explaining the present disclosure, it is impossible to describe all possible combinations of components or methods implemented by a computer. However, those skilled in the art can recognize that many further combinations and permutations of the present disclosure are possible. Further, when terms such as "include", "have", "possess", etc. are used in the embodiments for carrying out the invention, the claims, the accompanying documents, and the drawings, such terms are intended to be inclusive as interpreted in the same way as the term "comprising" when "comprising" is used as a transitional term in the claims. The descriptions of the various embodiments are shown for illustrative purposes, but are not intended to include all possible embodiments, nor are they intended to be limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, the practical applications that are superior to the technologies found in the market, or the technologies that are technically improved compared to the technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed in this specification.

Description of Reference Numerals

[0117] 100 System 101 Computing Device 104 Receiving Component 108 Selection Component 112 Fusion Component 114 Processing Unit 116 Rendering Component 118 Memory 120 Static Ranking Guideline Data 128 Segmentation Data 130 User Device 132 Output Device 202 MR segmentation data 203 CT image data 204 CT segmentation data 300 Process 316 Multimodal segmentation 320 Segmentation 324 Segmentation 400 System 406 Feedback component 408 Machine learning data 500 Process 510 Segmentation model data 518 Medical image database 524 Segmentation 528 Segmentation 602 Multimodal segmentation data 606 Training data 630 Client medical image application logic 702 Fusion data 706 Training data 710 Fusion protocol data 712 Fusion model 800 Method 806 Communication framework 812 Server data store 900 Method 1000 Method 1104 Processing unit 1110 Volatile memory 1112 Non-volatile memory 1114 Disk storage 1118 Operating system 1120 Application 1124 Program module 1126 Program data 1128 Input device 1130 Interface port 1134 Output adapter 1135 Codec 1136 Output device 1138 Remote computer 1140 Memory storage device 1142 Network interface 1144 Communication connection part 1200 Computing environment 1202 Client 1204 Server 1206 Communication framework 1208 Client data store

Claims

1. A system comprising: a memory storing computer-executable components; and a processor executing the computer-executable components stored in the memory, wherein the computer-executable components include: a receiving component (104) that receives a segmentation dataset (102) including combinations of different image segmentations of an anatomical object of interest, each segmented by a different segmentation model from different medical images of the acquired anatomical object, wherein the different medical images and the different image segmentations differ in at least one of acquisition modality, collection protocol, and collection parameters; a ranking component (106) that determines ranking scores for the different image segmentations to control relative contributions of the different image segmentations in connection with using a dynamic ranking protocol (122) as opposed to a static ranking protocol (120) to combine the different image segmentations to obtain a fused segmentation of the anatomical object; and a fusion component (112) that combines the different image segmentations based on the ranking scores to generate a fused image segmentation. A system as described above.

2. The system of claim 1, wherein the static ranking protocol includes determining the ranking scores based only on predefined guideline information (120) for the combinations and types of the anatomical objects, and the ranking component determines the ranking scores using the dynamic ranking protocol as opposed to the static ranking protocol based on a determination that the predefined guideline information for the combinations and types of the anatomical objects is not available or not definitive.

3. Each of the different image segmentations is associated with an uncertainty measure that reflects a measure of the uncertainty of the one or more segmentation models for the different image segmentations, and the ranking component determines the ranking score using the dynamic ranking protocol as opposed to the static ranking protocol based on a determination that one or more of the uncertainty measures do not meet an uncertainty measure criterion. The system according to claim 1.

4. The ranking component (106) determines the ranking score using the dynamic ranking protocol as opposed to the static ranking protocol based on a determination that one or more of the different image segmentations do not meet a quality criterion. The system according to claim 1.

5. The computer-executable component further is a quality evaluation component (110) that determines a quality measure of the different image segmentations and determines whether the different image segmentations meet the quality criterion based on the quality measure, wherein the quality measure is based on one or more factors selected from the group including the influence of artifacts, image quality, contour connectivity, and image contrast. Quality evaluation component (110) The system according to claim 4, comprising.

6. The dynamic ranking protocol includes determining the ranking score based on an uncertainty measure associated with the different image segmentations that reflects a measure of the uncertainty of the different segmentation models for the different image segmentations. The system according to claim 1.

7. The dynamic ranking protocol is an uncertainty measure associated with the different image segmentations that reflects a measure of the uncertainty of the one or more segmentation models for the different image segmentations, a quality measure of the different image segmentations that reflects one or more measures of the quality of the different image segmentations, artifact information regarding the presence or absence of artifacts, the collection protocols and collection parameters respectively associated with the different image segmentations, the size of the anatomical object, and The type of the anatomical object The system according to claim 1, comprising determining the ranking score based on a parameter selected from a group including

8. The dynamic ranking protocol further includes determining the ranking score based only on predefined guideline information for the combination and the type of the anatomical object, and the dynamic ranking protocol includes using one or more machine learning processes (600) to determine the ranking score based on the parameter and the predefined guideline information. The system according to claim 10.

9. The fusion component further uses a fusion protocol adapted to the combination and the type of the anatomical object to combine the different image segmentations, and the fusion component uses one or more machine learning processes (700) to generate a plurality of different manually generated for the combination and the type. Based on the fused segmentations, the system according to claim 1, wherein the fusion protocol for the combination and the type is learned.

10. A method comprising: A system including a processor receives a segmentation data set including a combination of different image segmentations of an anatomical object of interest segmented by different segmentation models from different medical images of the acquired anatomical object, wherein the different medical images and the different image segmentations have at least one of an acquisition modality, a collection protocol, and collection parameters different. Receiving, The system uses a dynamic ranking protocol as opposed to a static ranking protocol to determine a ranking score for the different image segmentations related to combining the different image segmentations to obtain a fused segmentation of the anatomical object, wherein the static ranking protocol includes determining the ranking score based only on predefined ranking information for the combination and the type of the anatomical object. Determining, and the system combines the different image segmentations based on the ranking score to generate the fused image segmentation A method comprising the above. **Claim 11** The dynamic ranking protocol is the system determines the ranking score based on the uncertainty measures associated with the different image segmentations that reflect the uncertainty measures of the one or more segmentation models for the different image segmentations The method according to claim 10, comprising the above. **Claim 12** The dynamic ranking protocol is the system determines the ranking score based on the quality measures of the different image segmentations that reflect one or more quality measures of the different image segmentations The method according to claim 10, comprising the above. **Claim 13** The dynamic ranking protocol is the system the uncertainty measures associated with the different image segmentations that reflect the uncertainty measures of the one or more segmentation models for the different image segmentations the quality measures of the different image segmentations that reflect one or more measures of the quality of the different image segmentations artifact information regarding the presence or absence of artifacts the collection protocols and collection parameters respectively associated with the different image segmentations the size of the anatomical object, and the type of the anatomical object The method according to claim 10, comprising determining the ranking score based on parameters selected from the group comprising the above. **Claim 14** The combining includes combining the different image segmentations using a fusion protocol adapted to the combination and the type of the anatomical object, and the method further comprises the system learns the fusion protocol for the combination and the type based on a plurality of different fused segmentations manually generated for the combination and the type using one or more machine learning processes **Claim 15** A non-transitory machine-readable storage medium containing executable instructions, which when executed by a processor Receiving a segmentation dataset including combinations of different image segmentations of an anatomical object of interest segmented by different segmentation models from different medical images of the acquired anatomical object, wherein at least one of the different medical images and the different image segmentations differs in acquisition modality, collection protocol, and collection parameters, Determining a ranking score for the different image segmentations to control a relative contribution degree of the different image segmentations in connection with combining the different image segmentations to obtain a fused segmentation of the anatomical object using a dynamic ranking protocol as opposed to a static ranking protocol, wherein the static ranking protocol includes determining the ranking score based only on predefined ranking information for the combination and the type of the anatomical object, determining the ranking score, and Combining the different image segmentations based on the ranking score to generate the fused image segmentation A non-transitory machine-readable storage medium having instructions for enabling performance of operations including the above.

Images