Dynamic multi-modal segmentation selection and fusion

Abstract

Techniques are described that facilitate dynamic multi-modal segmentation selection and fusion in medical imaging. In one example embodiment, a computer processing system receives a segmentation dataset comprising a combination of different image segments of an anatomical object of interest segmented from different medical images of the (identical) captured anatomical object via different segmentation models, respectively, the different medical images and the different image segments vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. The system employs a dynamic ranking protocol opposite to the static ranking protocol to determine ranking scores for the different image segments that control relative contributions of the different image segments in order to combine the different image segments into a fused segmentation for the anatomical object. The system also combines the different image segments based on the ranking score to generate the fused image segment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The subject application claims priority to U.S. non-provisional patent application serial number 17 / 664,702, filed on May 24, 2022, entitled “DYNAMIC MULTIMODALSEGMENTATION SELECTION AND FUSION,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to medical imaging systems, and more particularly, to techniques for dynamic multimodal segmentation selection and fusion. Background Art

[0004] Multimodal imaging is a central component of current and future clinical and preclinical medicine. Multimodal imaging refers to the use of more than one complementary technology system to obtain images, preferably simultaneously or in a short time to obtain images, so as to diagnose, prognose, manage and monitor diseases. For example, various modern imaging techniques have been widely used to monitor the structure, function and molecular changes in cancerous tissues, including optical imaging (by bioluminescence or fluorescence), computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), ultrasound (US), MR diffusion weighted imaging (MR-DWI) etc. Each imaging modality has its own unique advantages and disadvantages, such as spatial / depth resolution and sensitivity. For example, MRI is considered to be the optimal imaging modality for soft tissue lesion characterization, and CT is considered to be more suitable for bone metastasis analysis.

[0005] Emerging research in the field of multimodal imaging has resulted in a variety of clinical opinions in the literature regarding the application of contributions from different imaging modalities to assess specific types of anatomical objects, such as lesions and organs. These clinical opinions have been used to guide clinicians on how to use contributions from different imaging modalities to facilitate object / organ segmentation when available. For example, some multimodal segmentation guidelines have been developed to provide instructions on how to outline specific anatomical objects based on imaging modalities (e.g., in scenarios when a single modality is available, such as CT), how to adjust the outline when additional imaging modalities of the object are available (e.g., MRI, T1 and / or T2), and how to adjust the outline when additional functional images are available (e.g., PET, SPECT, MR-DWI, etc.).

[0006] For example, in a clinical study, PET-CT was used in addition to CT to better outline the esophagus and esophageal tumors. Another clinical study found that it was beneficial to use CT and MRI images simultaneously to depict the prostate and seminal vesicles. Another clinical study found that although MRI is the gold standard for lymph node detection in most cases, in some cases, MRI-DWI, PET or PET-CT are the most accurate, and MRI-DWI is non-radiation. In another example, although there is no clear guideline for laryngeal cartilage infiltration (using both CT and MRI), clinicians believe that MRI is more accurate for most patients. Based on these studies, some multimodal segmentation guidelines have been developed and defined in the literature. For example, with respect to mandibular (or lung) segmentation, some guidelines indicate that contour drawing can be performed only on CT, because CT can well display bone and / or air structures without adjustment based on other modalities. In another example, with respect to prostate segmentation, some guidelines indicate that contour drawing can be performed based on CT, but if MRI is also available, the contour can be further adjusted in a more reliable manner (because MRI better displays the boundaries of these organs). In another example, for tumor segmentation, contouring can be done based on CT, and contouring adjusted based on MR (with better soft tissue discrimination), and further adjusted based on PET (highlighting the active part as well as nearby infiltrating organs / lymph nodes) when available.

[0007] Although the literature has provided some preliminary guidelines on how to use the contributions of different imaging modalities to facilitate segmentation of some types of anatomical objects, these guidelines do not apply to every type of anatomical object, let alone every combination of imaging modalities. Furthermore, the actual contribution from each available imaging modality varies not only based on the type of object, but also varies for each patient and clinical site imaging system / imaging protocol. In addition, the contributions from different images must account for temporal and incremental uncertainties between images when the images were captured in different imaging studies, as well as differences between organ and body motion, which uncertainties and differences may vary for each patient and clinical scenario. Therefore, there is a need for techniques to automatically and intelligently account for all of these factors when two or more different imaging modalities are available to generate optimal anatomical segmentations for a specific anatomical object and clinical context. Summary of the invention

[0008] The following presents a summary of the invention to provide a basic understanding of one or more embodiments of the present invention. The present summary is not intended to identify key or important elements, nor is it intended to depict any scope of different embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a preface to a more detailed description presented later. In one or more embodiments, systems, computer-implemented methods, devices, and / or computer program products that facilitate dynamic multimodal segmentation selection and fusion are described herein.

[0009] According to one embodiment, a system is provided, the system comprising a memory and a processor, the memory storing a computer executable component, the processor executing the computer executable component stored in the memory. The computer executable component comprises a segmentation data set, the segmentation data set comprising a combination of different image segmentations of an anatomical object of interest, segmented from different medical images of the captured anatomical object via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. The computer executable component also comprises determining a ranking score (or ranking index) of the different image segmentations using a dynamic ranking protocol opposite to a static ranking protocol, the ranking score controlling the relative contribution of the different image segmentations so as to combine the different image segmentations into a fused segmentation for the anatomical object. The computer executable component also comprises a fusion component, which combines the different image segmentations based on the ranking score to generate a fused image segmentation.

[0010] In some embodiments, elements described in conjunction with the disclosed systems can be embodied in different forms, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An exemplary non-limiting system that facilitates dynamic multimodal segmentation selection and fusion in accordance with one or more embodiments of the disclosed subject matter is shown.

[0012] Figure 2 Exemplary multimodal segmented data according to one or more embodiments of the disclosed subject matter is presented.

[0013] Figure 3 A flowchart of an exemplary process for selecting and fusing multimodal segmentation data in accordance with one or more embodiments of the disclosed subject matter is presented.

[0014] Figure 4 Another exemplary non-limiting system that facilitates dynamic multimodal segmentation selection and fusion in accordance with one or more embodiments of the disclosed subject matter is illustrated.

[0015] Figure 5A flowchart of another exemplary process for selecting and fusing multimodal segmentation data in accordance with one or more embodiments of the disclosed subject matter is presented.

[0016] Figure 6 An exemplary machine learning framework that facilitates ranking of multimodal segmented data in accordance with one or more embodiments of the disclosed subject matter is presented.

[0017] Figure 7 An exemplary machine learning framework that facilitates multimodal segmentation data fusion in accordance with one or more embodiments of the disclosed subject matter is presented.

[0018] Figure 8 A block diagram illustrating an exemplary non-limiting computer-implemented method for generating a fused multi-modal segmented image in accordance with one or more embodiments of the disclosed subject matter is shown.

[0019] Fig. 9 A block diagram illustrating another exemplary non-limiting computer-implemented method for generating a fused multi-modal segmented image in accordance with one or more embodiments of the disclosed subject matter.

[0020] Fig.10 A block diagram illustrating another exemplary non-limiting computer-implemented method for generating a fused multi-modal segmented image in accordance with one or more embodiments of the disclosed subject matter.

[0021] Fig.11 A block diagram of an exemplary non-limiting operating environment is shown in which one or more embodiments described herein may be facilitated.

[0022] Fig.12 A block diagram is shown of another exemplary non-limiting operating environment in which one or more embodiments described herein may be facilitated. DETAILED DESCRIPTION

[0023] The following specific embodiments are merely exemplary and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" section or "content of the invention" section or "specific embodiments" section.

[0024] The term "multimodal segmentation data" is used herein to refer to a segmentation dataset that includes a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the anatomical object captured via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. The disclosed subject matter relates to systems, computer-implemented methods, apparatus, and / or computer program products that facilitate filtering multimodal segmentation data to select an optimal segmentation modality for a given clinical context. The clinical context is defined at least in part based on specific anatomical objects segmented in the multimodal segmentation data and the type of imaging modalities included in the multimodal segmentation data. The present subject matter also relates to systems, computer-implemented methods, apparatus, and / or computer program products that facilitate fusing multimodal segmentation data into a single composite image segmentation using relative contributions from each segmentation determined in part based on a given clinical use context.

[0025] To facilitate this purpose, in one or more embodiments, the disclosed system receives, accesses and / or optionally generates a segmentation data set for an anatomical object. The segmentation data set includes image segmentations or segmentation masks generated by a segmentation model of an anatomical object from different medical images captured and / or generated using two or more different imaging modalities (e.g., optical, CT, MRI, PET, SPECT, USI, etc.), two or more different acquisition protocols and / or two or more different acquisition parameters. Therefore, due to the different acquisition protocols (e.g., contrast and non-contrast) and / or acquisition parameters (e.g., the automatic exposure control mode (AEC) used, the reconstruction kernel used, etc.), each of the different segmentations or segmentation masks for the object corresponds to a different imaging modality and / or a different image with different image attributes. The segmentation or segmentation mask is generated using a previously existing (i.e., previously trained) trained organ / object segmentation model.

[0026] In various embodiments, the disclosed system adopts a combination of static sorting protocols and dynamic sorting protocols to use a sorting score that reflects the relative contribution "usefulness" of the corresponding segmentation to sort or rate each segmentation in different segmentations so that different segmentations are combined into a single fused image segmentation for anatomical objects. In some specific implementations, the sorting score determined for each segmentation is also used to control the contribution from each segmentation so as to generate a fused segmented image for anatomical objects based on a combination of different segmentations (e.g., a weighted fusion process). Additionally or alternatively, the sorting can be used to select the "optimal" segmentation for clinical viewing and / or further processing for a given clinical use context. As described above, the clinical context is defined at least in part based on the specific anatomical objects of the segmentation and the types of different imaging modalities. The clinical use context can also be defined based on one or more additional parameters (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, training data use, further processing via another clinical application, etc.) related to the intended use of the selected or fused segmentation.

[0027] The static sorting protocol involves employing predefined segmentation guideline information determined for a particular anatomical object, which defines or indicates the relative usefulness of different imaging modality segmentations for the corresponding anatomical object in order to combine the different imaging modality segmentations into a fused segmentation. In this regard, the predefined guideline segmentation information is based on a general understanding of the properties of different segmentations corresponding to different imaging modalities for a given anatomical object / organ combination. The predefined multimodal segmentation guideline information can be determined and summarized from the literature and clinical studies available in the art, as described in the background technology section above.

[0028] However, as noted in the Background section, while the literature may provide some preliminary guidance on how to use contributions from different imaging modalities to facilitate combining different segmentations of some types of anatomical objects, these guidelines are not applicable to every type of anatomical object, let alone every combination of imaging modalities. Furthermore, the actual contribution from each available imaging modality varies not only based on the type of object, but also varies for each patient and clinical site imaging system / imaging protocol and the imaging parameters used. Furthermore, the contributions from different image segmentations must account for temporal and incremental uncertainties between images when they are captured in different imaging studies, as well as differences between organ and body motion, which uncertainties and differences may vary for each patient and clinical scenario. Further, the disclosed techniques are applied in the context of providing multimodal segmentations of anatomical objects, each generated via one or more existing segmentation models. In this context, the quality of the segmentation may vary from case to case depending on a variety of factors (e.g., input image quality, presence of artifacts in the input image, model range and accuracy capabilities, presence of anatomical variants in the input image, etc.), and thus make static sorting protocol guidelines inadequate or inapplicable.

[0029] To account for these many contextual variabilities, the disclosed system also provides a dynamic ranking protocol to facilitate ranking or rating different segmentations on a case-by-case basis based on dynamic ranking factors. Unlike static ranking protocol guidelines that provide general information about how different modality segmentations can be combined for different anatomical objects, dynamic ranking factors include factors that can only be determined after imaging and organ segmentation have been completed for each patient because the dynamic factors include those factors that cannot be anticipated in advance, such as the quality of the segmentation, the presence of artifacts, deviations from standard imaging protocols, etc. In this regard, in one or more embodiments, the dynamic ranking factors may include, but are not limited to: uncertainty measures associated with different image segmentations, the uncertainty measures reflecting measures of uncertainty of one or more segmentation models relative to the different image segmentations; quality measures for different image segmentations, the quality measures reflecting one or more measures of the quality of the different image segmentations; artifact information regarding the presence or absence of artifacts; acquisition protocols and acquisition parameters, which are respectively associated with different image segmentations; the size of the anatomical object; and the type of anatomical object.

[0030] In some embodiments, the dynamic ranking protocol involves initially evaluating the quality of the multimodal segmentation according to the available static ranking multimodal segmentation guidelines. In this regard, if available for a specific anatomical object and clinical context, the static ranking protocol guidelines can be used to determine the initial rating and apply the initial rating to each segmentation, which reflects the relative usefulness of the segmentation for the clinical context. Then, according to the dynamic ranking protocol, the system can also customize and adjust the ranking based on multiple additional instance-specific parameters or "dynamic" parameters related to segmentation quality, model uncertainty, image quality, object size, object type and subtype, the imaging protocol used, image capture time, image registration burden, the presence of bleeding, the presence of artifacts, patient demographics, patient pathology and clinical use context (etc.). In some embodiments, the disclosed system can also use machine learning techniques to learn and define dynamic ranking protocol guidelines based on the learned patterns, correlations and / or rules between these different dynamic parameters, which affect the relative ranking of the available multimodal segmentation for a specific anatomical object. Additionally or alternatively, the disclosed system can train and develop one or more machine learning models to automatically infer ranking scores for corresponding multimodal segmentations for a given anatomical object and clinical context based on learned patterns, correlations, and / or rules.

[0031] In some embodiments, one or more of the disclosed systems can generate a fused segmented image for an anatomical object based on a final ranking applied to each segmentation determined using a static ranking and a dynamic ranking protocol. Using these embodiments, the system can include a fusion component that projects the segmentation of the object into the same reference space for an image registration process. The fusion component can also create a fused image by combining the registered images using a ranking of contributions for each segmentation from different modalities. In some specific implementations of the embodiments, the fusion component can apply a weighting scheme to different segmentation modalities based on their respective ranking scores to generate a final fused segmented image.

[0032] Additionally or alternatively, the fusion component may employ one or more rule-based statistical and / or machine learning algorithms to perform fusion based at least in part on relative rating / ranking scores. To this end, one or more embodiments of the disclosed system may provide a learning-based contour selection workflow for specific anatomical objects such as lesions. The learning-based contour selection workflow may initially begin with a (manual) data acquisition phase at multiple sites (i.e., variability is acceptable in the data acquisition phase as long as only templates are defined to constrain structures and processes). During the learning phase, one or more machine learning techniques may be employed to learn the optimal way to fuse different segmentation modalities available for each type of anatomical object based on the various static and dynamic parameters described above and the assessed fusion quality results. The system may then apply the learned optimal fusion mechanism (e.g., in the form of learned rules and / or trained fusion models) in the inference phase of the automatic segmentation selection and fusion workflow.

[0033] The types of medical images processed / analyzed using the techniques described herein may include images captured using various types of image capture modalities. For example, medical images may include (but are not limited to): radiotherapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis equipment), magnetic resonance imaging (MRI or simply MR) images (including T1-weighted images and T2-weighted images), ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, optical images, MR-DWI, etc. Medical images may also include synthesized versions of native medical images, such as synthesized X-ray (SXR) images, modified or enhanced versions of native medical images, extended versions of native medical images, and similar versions generated using one or more image processing techniques. The types of medical image data processed / analyzed herein may include two-dimensional (2D) image data, three-dimensional image data (3D) (eg, volumetric representations of anatomical regions of the body), and combinations thereof.

[0034] As used herein, "modality" refers to a specific technical mode of capturing images or image data using one or more machines or devices. In this regard, as applied to medical imaging, different capture modalities may include, but are not limited to: 2D capture modality, 3D capture modality, RT capture modality, XR capture modality, DX capture modality, XA capture modality, PX capture modality, CT capture modality, MG capture modality, MR capture modality, MR-T1 capture modality, MR-T2 capture modality, US capture modality, CD capture modality, PET capture modality, SPECT capture modality, NM capture modality, MR-DWI modality, etc.

[0035] The term "multimodality" applied to medical image data (including segmented data) is used herein to refer to two or more different types of medical image data. Differentiation factors between two or more different types of data can vary. For example, differentiation factors can refer to the format of the data, the capture modality of the data, the source of the data, the imaging protocol used, the imaging parameters used, etc. For example, different medical images of the same anatomical object / organ corresponding to different modalities, imaging parameters and / or imaging protocols may include (but are not limited to) MR-T1 and MR-T2 images of the object and images of the object with or without contrast agents. In another example, two different medical images corresponding to different modalities may include a diagnostic CT image of the captured object and a radiation therapy planning CT image of the object.

[0036] Unless the context warrants a specific distinction between the terms, the terms "algorithm" and "model" are used interchangeably herein. Unless the context warrants a specific distinction between the terms, the terms "artificial intelligence (AI) model" and "machine learning (ML) model" are used interchangeably herein.

[0037] As used in this disclosure, the term "user" or the like refers to a person, entity, system, or combination thereof that interfaces with the subject medical image processing system using a suitable computing device.

[0038] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0039] Now turning to the attached figure, Figure 1A block diagram of an exemplary non-limiting system 100 that facilitates dynamic multimodal segmentation selection and fusion according to one or more embodiments of the disclosed subject matter is shown. Embodiments of the system described herein may include one or more machine-executable components embodied in one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). When executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), such components may cause one or more machines to perform the operations described.

[0040] For example, system 100 includes a computing device 101, which includes several computer-executable components, including a receiving component 104, a sorting component 106, a selecting component 108, a quality assessment component 110, a fusion component 112, and a presentation component 116. These computer / machine executable components (and other components described herein) may be stored in a memory associated with one or more machines. The memory is also operably connected to at least one processor so that the components can be executed by the at least one processor to perform the described operations. For example, in some embodiments, these computer / machine executable components may be stored in a memory 118 of the computing device 101, which may be connected to a processing unit 114 to execute these computer / machine executable components. Examples of the described memory and processors and other suitable computer or computing-based elements can refer to Fig.11 Find and implement the combination Figure 1 Or one or more of the systems or components shown and described in other figures disclosed herein.

[0041] The memory 118 may also store various information received, used, and / or generated by the computing device 101 to facilitate dynamic multimodal segmentation selection and fusion. In the illustrated embodiment, the information includes (but is not limited to) static sorting guide data 120, dynamic sorting protocol data 122, and fusion protocol data 124. In various embodiments, the system 100 is configured to receive a segmentation dataset 102 (e.g., via a receiving component 104) and process the segmentation dataset 102 to generate selected and / or fused segmentation data 128. The segmentation dataset 102 may include a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the anatomical object captured via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. The selected and / or fused segmentation data 128 may include a selected subset (e.g., one or more subsets) of the segmentation dataset 102 and / or a fused segmentation image generated by the computing device 101 (e.g., using the fusion component 112) by intelligently combining or fusing different image segmentations in a manner that produces a single segmentation of the anatomical object that optimizes contributions from the different segmentation modalities.

[0042] Any information received (e.g., segmented dataset 102) and / or generated (e.g., selected and / or fused segmented data 128) by computing device 101 may be presented or provided to a user via a suitable display. In the illustrated embodiment, the display may be associated with a user device 130 (i.e., input / output device 132 may include a display) that is communicatively and operably coupled (e.g., via one or more wired or wireless communication connections) to computing device 101. In this regard, user device 130 may correspond to a computing device employed by a user (e.g., a clinician, radiologist, technician, machine learning (ML) model developer, etc.) to interface with one or more features and functions provided by corresponding components of computing device 101. For example, in various embodiments, one or more of the components of computing device 101 may be associated with a medical imaging application that facilitates accessing and viewing medical images via an interactive graphical user interface (GUI) displayed at user device 130, generating and viewing anatomical object segmentations for medical images via the GUI, annotating medical images, running inference models on medical images, and the like. In some implementations of these embodiments, computing device 101 may correspond to an application server that provides at least some of these features and functions to user device 130 via a network-accessible platform such as a web application. Using these embodiments, user device 130 may be communicatively coupled to computing device 101 via one or more wired or wireless communication networks (e.g., the Internet) and, using a suitable web browser, access one or more of the features and functions of computing device 101 as a web application. Additionally or alternatively, system 100 may employ a local deployment architecture in which one or more components of computing device 101 are locally deployed at user device 130. Various other deployment architectures are contemplated for system 100 and other systems described herein. User device 130 may include one or more input / output devices 132 (e.g., a keyboard, a mouse, a touch screen, a display, etc.) for receiving user input associated with use of the features and functions of computing device 101 and displaying an associated GUI. Reference Fig.11 Some examples of suitable input / output devices 132 are described with respect to input devices 1128 and output devices 1136 .

[0043] According to the system 100, the receiving component 104 may receive the segmentation data set 102 for processing by the computing device 101. As described above, the segmentation data set 102 includes anatomical object segmentation data of the same anatomical object depicted in two or more different medical images captured and / or generated from the same object using different medical imaging modalities (e.g., optical, CT, MRI, PET, SPECT, US, etc.). The number and type of different medical images and the corresponding imaging modalities may vary. The anatomical object segmentation data may include information defining the boundaries or contours of the anatomical object in 2D and / or 3D (depending on the type of image from which the anatomical object is segmented) relative to the original image data from which the anatomical object is segmented. For example, the segmentation data may correspond to a segmentation mask applied to the anatomical object, image marker data including boundary lines, points, circles, etc. of the anatomical object, and / or image data of the anatomical object extracted from the input image. The segmentation data set 102 may also include the original medical image data from which the corresponding segmentation is generated. In some embodiments, the segmentation data set 102 may include segmentation data for a single anatomical object of interest depicted in a corresponding multimodal image. In other embodiments, the segmentation dataset 102 may include segmentation data for a plurality of different anatomical objects of interest depicted in corresponding multimodal images (eg, generated using a multi-organ segmentation model).

[0044] Figure 2 Exemplary multimodal segmentation data according to one or more embodiments of the disclosed subject matter is presented. Figure 2 In the example shown, the multimodal segmentation data includes MR segmentation data 201 generated from MR image data 201 corresponding to an MR brain scan of a patient. The multimodal segmentation data also includes CT segmentation data 204 generated from CT image data 203 corresponding to a CT brain scan of the same patient. The MR segmentation data 202 provides segmentation data in the form of segmentation boundary markers or contours around various defined anatomical objects (e.g., tissues, vessels, regions of interest (ROIs), etc.) depicted in the MR image data 201. The CT segmentation data 204 provides segmentation data in the form of a segmentation mask over the entire brain depicted in the CT image data 203. It should be understood that Figure 2 The multimodal segmentation data depicted in is merely exemplary, and various different types of segmentation data are contemplated for various different anatomical objects associated with all regions of the body.

[0045] Reference again Figure 1 According to the disclosed technology, the segmentation dataset 102 corresponds to anatomical object image segmentations segmented from corresponding original 2D and / or 3D medical images using one or more existing segmentation models (e.g., Figure 2For example, in some implementations, the segmentation model may include separate segmentation models customized for different anatomical objects and / or different imaging modalities. In other implementations, one or more of the segmentation models may be configured to process input images from different imaging modalities to generate corresponding segmentations. Additionally or alternatively, the segmentation model may include a multi-organ / multi-object segmentation model that is configured to generate segmentation data for a defined set of anatomical objects depicted in the input image data.

[0046] In various embodiments, each image anatomical object segmentation (i.e., segmentation model output) comprising the segmentation data set 102 may also include one or more uncertainty metrics that reflect an estimated measure of certainty or uncertainty associated with the corresponding segmentation model that generates the corresponding segmentation data. In this regard, one or more uncertainty metrics reflect a certainty measure associated with the confidence of the segmentation model in the accuracy and / or specificity of the segmentation data generated thereby. Typically, the segmentation model can be configured to output the uncertainty measure data in addition to the corresponding segmentation data. The type of uncertainty measure data can be variable. For example, uncertainty measure data can include, but is not limited to, certainty estimate scores, confidence scores, confidence intervals, accuracy scores, DICE scores, and the like.

[0047] The segmentation dataset 102 may also include a variety of rich metadata associated with each anatomical object segmentation in the anatomical object segmentation and / or the raw image data from which the anatomical object segmentation was generated. For example, the metadata associated with each raw input image data and the associated anatomical object segmentation may include information identifying the type of imaging modality, the imaging protocol and parameters used to generate the raw image data, the model / model of the imaging system used to capture the raw image data, timestamp information describing the capture time of the imaging data, and information identifying the source or site of the raw medical image data (e.g., reference name / location). The metadata may also include information describing image feature attributes associated with the raw image data from which the segmentation was generated, such as information about image resolution, brightness, pixel spacing and grid size, acquisition plane / angle, noise level, etc. Each segmentation in the segmentation may also be associated with metadata defining or indicating a specific anatomical object name or classification according to a defined clinical ontology (e.g., a specific organ, lesion, lesion type, tumor, cyst, bone, soft tissue, etc.). In some embodiments, each segmentation in the segmentation may also be associated with metadata that defines or indicates geometric characteristics of the segmentation, such as, but not limited to, the relative size, position, and geometry of anatomical objects (e.g., organ / lesion diameter), information defining segmentation boundaries or contours, etc.

[0048] In some embodiments, the metadata may also include information identifying or indicating the quality of the corresponding anatomical object segmentation (i.e., segmentation quality information). For example, the segmentation quality information may include one or more quality metrics that reflect the extent to which the segmentation contour correctly and clearly defines the boundaries of the anatomical object relative to the original image data from which the anatomical object is segmented. The quality information may also include a connectivity metric that represents the degree of connectivity around the contour or boundary line of the segmented anatomical object. The quality information may also include information identifying or indicating whether any artifacts (usually degrading the image and corresponding segmentation quality) are depicted in the original image data and / or segmentation data. The quality information may also include one or more noise metrics representing the noise level in the image segmentation. Additionally or alternatively, information about the segmentation quality of each segmentation may be determined by the quality assessment component 110 and / or received as user feedback so that the segmentation data set 102 is presented to one or more users (e.g., via the user device 130). Contrast metrics, segmentation contour connectivity metrics, etc.).

[0049] The segmented dataset 102 may also include information about the patient or subject represented in the segmented dataset 102. For example, the patient information may include demographic information of the patient, such as, but not limited to, age, gender, race, location, body mass index (BMI), height, and weight. The patient information may also include relevant pathology information of the patient extracted from a radiology report associated with the original medical image and / or the patient's medical record. The patient information may also include other relevant medical history information of the patient, such as information about comorbidities, previous diagnosis information, etc.

[0050] The sorting component 106 can be configured to utilize the sorting score reflecting the relative usefulness of the corresponding segmentation for a given clinical context to sort or rate the corresponding segmentation included in the multimodal data 102. In one or more embodiments, for the convenience of this purpose, the sorting component 106 can adopt a static sorting protocol, a dynamic sorting protocol, or a combination thereof. The scale for the sorting score can be variable. For example, in some specific implementations, the sorting score can only reflect the sorting order of the corresponding segmentation (e.g., first=ranking highest, second=ranking second highest, third=ranking third highest, etc.). In another example, the relative weighted contribution of the corresponding segmentation can be reflected by the sorting score as a percentage contribution. Other sorting score scales are contemplated.

[0051] In some specific implementations, the ranking score determined for each segmentation can be used by the fusion component 112 to control the contribution from each segmentation so as to generate a fused segmented image for an anatomical object based on a combination of different segmentations (e.g., weighted fusion). Additionally or alternatively, the ranking can be used by the selection component 108 to select the "optimal" segmentation for clinical review and / or further processing for a given clinical use context. As described above, the clinical context is defined at least in part based on the distribution of the specific anatomical object of the segmentation and the corresponding imaging modalities of different types. The clinical context can also be defined based on one or more additional parameters (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, training data use, further processing via another clinical application, etc.) associated with the intended use of the selected or fused segmentation. Using these embodiments, information defining the intended use of the selected or fused segmentation can be included in the segmentation data set 102, received as user feedback, and / or inferred.

[0052] The static ranking protocol involves employing predefined multimodal segmentation guideline information determined for a particular anatomical object, which defines or indicates the relative usefulness of different imaging modality segmentations for the corresponding anatomical object and clinical context. The predefined multimodal segmentation guideline information can be determined and summarized from literature and clinical studies available in the art, as described in the background technology section above. In the illustrated embodiment, the predefined multimodal segmentation guideline information is represented as static ranking guideline data 120. The static ranking guideline data 120 can define ranking rules for ranking different combinations of image segmentation modalities for a particular anatomical object. For example, in some embodiments, the static ranking guideline data 120 can include an index data structure that identifies various known anatomical objects of interest that can be included in the segmentation data set 102, such as organs, lesions, bones, tissues, vesicles, etc., according to a defined clinical ontology. For each anatomical object, the static ranking guideline data 120 can also define different combinations of two or more types of potential types of imaging modalities, where the corresponding segmentation of the anatomical object can be received. For each imaging modality combination, the static ranking guide data 120 may further define a relative ranking or rating score for each imaging modality included in the combination. For example, assume an anatomical object A and an imaging modality combination of modality 1, modality 2, and modality 3. The relative ranking score for the anatomical object A and imaging modality combination may indicate that modality 2 is ranked highest, modality 1 is ranked second highest, and modality 3 is ranked third highest. In another example, the relative ranking score may assign a ranking percentage weight to the corresponding imaging modalities included in the combination, for example, a weight of 50% for modality 2, 30% for modality 1, and 20% for modality 3. Other ranking score scales are contemplated. In another specific implementation, the static or dynamic ranking protocol may also specifically assign an "optimal modality" image segmentation for a specific organ or anatomical object.

[0053] However, according to the literature available in the art, the granularity level of the static ranking guideline 120 is limited, and therefore it is unlikely to provide comprehensive and conclusive ranking rules for every type of anatomical object, let alone every combination of imaging modality segmentation and clinical use context (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, training data use, further processing via another clinical application, etc.). In addition, the actual contribution from each available imaging modality varies not only based on the type of object, but also varies for each patient and clinical site imaging system / imaging protocol. In addition, the contribution from different images must account for the time and incremental uncertainty between images when the images are captured in different imaging studies, as well as the differences between organ and body motion, which may vary for each patient and clinical scenario. Further, the disclosed technology is applied in the context of providing anatomical objects with multimodal segmentations generated via one or more existing segmentation models, respectively. In this case, the quality of the segmentation can vary from case to case based on a variety of factors (e.g., input image quality, the presence of artifacts in the input image, model range and accuracy capabilities, the presence of anatomical variants in the input image, etc.), and thus makes the static ranking protocol guideline insufficient or inapplicable in many scenarios.

[0054] To account for these many contextual variabilities and inherent deficiencies in the static ranking guide data 120, the ranking component 106 may additionally or alternatively employ a dynamic ranking protocol to rank or rate the different multimodal segmentations of each anatomical object represented in the segmentation dataset 102 on a case-by-case basis. The dynamic ranking protocol differs from the static ranking protocol in that the actual quality and visual characteristics of the multimodal segmentations generated by different models received in the segmentation dataset 102 for a given anatomical object and subject / patient are considered and analyzed to affect the relative usefulness of these segmentations for a given clinical context and / or for weighted fusion processing. The dynamic ranking protocol may also customize the ranking score for each segmentation object and modality combination according to different candidate clinical use contexts (e.g., radiotherapy, lesion characterization, bone metastasis analysis, diagnosis, staging, training data use, further processing via another clinical application, etc.). The dynamic ranking protocol may also account for the various additional contextual variables described above (e.g., in addition to the combination of only anatomical object type and modality). For example, according to a dynamic ranking protocol, the ranking component 106 can determine the relative ranking or ranking score of the corresponding multimodal segmentations received for a particular anatomical object based on various "dynamic" parameters related to segmentation quality, model uncertainty, image quality, clinical usage context, object size, object type and subtype, imaging protocol / parameters used, imaging data source / site, image feature attributes, image capture time, image registration burden (e.g., a measure of the difference in geometric alignment between corresponding original images and / or anatomical objects in reference space), presence of hemorrhage, presence of artifacts, patient demographics, patient pathology, and various other parameters that may be included in the metadata associated with the above-mentioned corresponding multimodal segmentations (etc.).

[0055] For example, in one or more embodiments, based on the dynamic ranking protocol data 122, the ranking component 106 can determine the ranking score for the corresponding multimodal segmentation received for the anatomical object based on at least the following parameters: 1.) an uncertainty measure associated with each segmentation in the segmentation (e.g., generated by the corresponding segmentation model and included in the multimodal segmentation data); 2.) one or more segmentation quality measures associated with each segmentation in the segmentation (e.g., included in the segmentation data set 102, determined by the quality assessment component 110, and / or received via user feedback); and 3.) the size and / or geometry of the anatomical object in the corresponding segmentation (e.g., included in the segmentation data set 102, determined by the quality assessment component 110, and / or received via user feedback). According to this example, a higher certainty measure (or a lower uncertainty measure) can be associated with a higher ranking score. Similarity, a higher segmentation quality measure (e.g., a strong contour connectivity measure, a high contour boundary visibility measure, a low noise measure, etc.) can be associated with a higher ranking score. The manner in which the size measure affects the ranking score can vary based on the type of anatomical object. For example, in some implementations, for lesions, the dynamic ranking protocol can direct the ranking component 106 to increase the ranking score according to the lesion size.

[0056] It should be understood that the dynamic ranking protocol data 122 is not limited to ranking the corresponding segmentations based only on the parameters in the above examples. In this regard, the dynamic ranking protocol data 122 can define multiple complex rules between different dynamic parameters and imaging modalities for different anatomical objects, which the ranking component 106 can use to determine the relative ranking scores for each segmentation included in a set of different multimodal segmentations. These complex rules can be based not only on the evaluation of individual parameters associated with each segmentation, but also on the relative relationship of the corresponding parameters between different segmentations, such as relative time of capture, relative size, relative quality metrics, relative model certainty metrics, etc., which can also vary based on the type of each imaging modality, the combination of imaging modalities, the type of anatomical object, patient demographics, and clinical use context.

[0057] In some embodiments, the dynamic ranking protocol may include a rule-based protocol that accounts for these various dynamic parameters to determine a relative ranking score for each anatomical object and segmentation modality combination received in the segmentation data set 102. Additionally or alternatively, as described in more detail below, the dynamic ranking protocol may incorporate principles of artificial intelligence (AI) and machine learning (ML) to learn, define, and adjust the dynamic ranking protocol over time. Information defining and / or controlling the dynamic ranking protocol is represented in the system 100 as dynamic ranking protocol data 122.

[0058] In some embodiments, a dynamic ranking protocol (e.g., as defined by the dynamic ranking protocol data 122) may involve initially assessing the quality of the multimodal segmentations based on available static ranking guideline data 120. In this regard, if a combination of multimodal image segmentations is available for a particular anatomical object, the ranking component 106 may employ the static ranking guideline data 120 to determine and apply an initial rating to each segmentation that reflects the relative usefulness of the segmentation for the clinical context. Then, based on the dynamic ranking protocol, the ranking component 106 may also customize and adjust the relative ranking based on a plurality of additional instance-specific parameters or "dynamic" parameters related to segmentation quality, model uncertainty, image quality, object size, object type and subtype, imaging protocol used, image capture time, image registration burden, presence of hemorrhage, presence of artifacts, patient demographics, patient pathology, and clinical use context (etc.).

[0059] As described above, the dynamic sorting protocol can be based at least in part on the quality of the corresponding anatomical object segmentation. The quality of each segmentation can vary with the quality of the original image from which the corresponding segmentation is generated and the quality of the segmentation itself (i.e., the output of the segmentation model). For example, the quality metric associated with the quality of the original image can include one or more noise metrics, one or more contrast metrics, one or more brightness metrics, and / or one or more general quality metrics associated with the overall quality of the corresponding original image. Information identifying or indicating the quality of the corresponding anatomical object segmentation (i.e., segmentation quality information) can include one or more quality metrics that reflect the extent to which the segmentation contour correctly and clearly defines the boundaries of the anatomical object relative to the original image data from which the anatomical object is segmented. The quality information can also include a connectivity metric that represents the degree of connectivity around the contour or boundary line of the segmented anatomical object. The quality information can also include information identifying or indicating whether any artifacts are depicted in the original image data and / or segmentation data (usually degrading the image and the corresponding segmentation quality) and whether bleeding is detected in the original image.

[0060] In some implementations, some or all of the quality information used by the sorting component 106 to facilitate sorting the corresponding segmentations (as described above) can be received as metadata along with the segmentations. Additionally or alternatively, the quality assessment component 110 can determine one or more quality metrics for the corresponding segmentations based on metadata associated with the corresponding segmentations and / or analysis of the corresponding segmentations using one or more image quality processing tools. For example, in some implementations, the quality assessment component 110 can aggregate the quality information included in the metadata associated with the corresponding segmentations to generate an overall quality score for the corresponding segmentations that describes different quality metrics (e.g., noise level, contrast level, brightness level, segmentation contour accuracy, contour connectivity, presence of artifacts, presence of bleeding, etc.). Additionally or alternatively, the quality assessment component 110 can process the corresponding segmentations using existing image quality analysis software to determine one or more quality metrics.

[0061] In other embodiments, one or more of these quality metrics may be received as user feedback so that the corresponding multimodal segmentation is presented to one or more users by the presentation component 116. Using these embodiments, before determining the ranking of the corresponding segmentation, the presentation component 116 may present the received multimodal segmentation to one or more users (e.g., via their corresponding user devices 130) via an interactive GUI, which includes a mechanism for providing quality feedback related to the quality of the corresponding segmentation. For example, the feedback mechanism may allow the user to provide feedback that rates the overall quality of each segmentation, and / or rate the quality of the corresponding segmentation relative to accuracy, coverage, connectivity, visibility (e.g., for some lesions, sometimes visibility is better in T1, and sometimes visibility is better in T2 or CT), contour boundary / background distinction, contrast, etc. The feedback mechanism may also provide for receiving user feedback indicating whether there are artifacts, whether there is bleeding, and whether anatomical variants are observed. The rating component 106 and / or the quality assessment component 110 may further employ any quality feedback received via the GUI.

[0062] In some embodiments, after the ranking component 106 has determined the relative ranking of the multimodal segmentations for the anatomical object, the selection component 108 can be configured to select the highest ranked segmentation modality. The selected segmentation can be presented to one or more users for clinical review (e.g., as selected segmentation data 128), saved / stored, and / or provided to another clinical application for further use and / or processing (e.g., depending on the intended clinical use context of the selected segmentation).

[0063] Additionally or alternatively, the fusion component 112 can generate a fused segmented image for the anatomical object based on the final ranking applied to each segmentation determined by the ranking component 106. With these embodiments, the fusion component 112 can project different segmentations of the anatomical object onto the same reference space for conventional image registration processes to register the corresponding segmented images relative to each other and the same reference space. The fusion component 112 can also create a fused image by combining the registered segmented images using the ranking of contributions for each segmentation from different modalities. In some specific implementations of the embodiments, the fusion component 112 can apply a weighting scheme to different segmentation modalities based on their respective ranking scores to generate a final fused segmented image, wherein the weighted contribution of each modality segmentation is related to the ranking score of the modality segmentation, and wherein higher ranked segmentation modalities are given higher weights relative to lower ranked segmentation modalities. Additionally or alternatively, the fusion component 112 can employ one or more rule-based statistical and / or machine learning algorithms to perform the fusion of different segmentations based at least in part on the relative ranking / ranking scores. Information defining / controlling how the fusion component 112 generates a fused segmentation for an anatomical object based on the ranking scores determined for different multimodal segmentation sets can be defined by the fusion protocol data 124. For example, the fusion protocol data 124 can define rules and / or algorithms (e.g., a weighted fusion algorithm, a segmentation union algorithm, etc.) that can be applied by the fusion component 112 to generate a fused segmentation for an anatomical object based at least in part on the type of anatomical object, the combination of imaging modalities in the multimodal segmentation set, and the ranking scores determined for the corresponding segmentations in the set. In some embodiments, the fusion protocol data 124 can include different fusion protocols or rules customized for different anatomical objects.

[0064] Figure 3 A flowchart of an exemplary process 300 for selecting and fusing multimodal segmentation data according to one or more embodiments of the disclosed subject matter is presented. Process 300 corresponds to an exemplary process that can be performed by system 100 using computing device 101 and the corresponding components described above. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0065] refer to Figure 1 and Figure 3According to process 300, at 302, receiving component 104 can receive a segmentation dataset 102 for one or more anatomical objects. For example, in some embodiments, the segmentation dataset 102 can include multiple different medical images depicting the same anatomical region of a patient but captured / generated in different modalities. Each of the different images can include multiple different organ / object segmentations (e.g., segmentation masks, segmentation image data defining the outline of an object overlaid on the image, etc.). In these cases, each individual organ / anatomical object segmentation should be evaluated and sorted separately. In this regard, the remainder of process 300 is described with reference to a set of multimodal segmentations (e.g., including two or more multimodal segmentations) for a single anatomical object.

[0066] At 304, the ranking component 106 can check the static ranking guideline data 120 for the anatomical object to determine whether the static ranking guideline data provides conclusive static ranking information for the anatomical object and imaging modality combination. In this regard, conclusive ranking information refers to a clear consensus on how to rank each different segmentation modality in the collection for a particular anatomical object. If at 304, the ranking component 106 determines that conclusive static ranking guidelines are not available / defined for the particular anatomical object and imaging modality combination, the process 300 proceeds to 308, and the ranking component 106 continues to execute the dynamic ranking protocol to generate dynamic ranking scores for different modality segmentations. In some specific implementations, if some static guidelines are available for one or more imaging modalities in the anatomical object and imaging modality, the ranking component 106 can factor in the "partial" static ranking guideline data of the collection in the dynamic ranking determination. The result of the dynamic ranking at 306 includes a ranked multi-modal segmentation 316 for the anatomical object.

[0067] According to process 300 , if at 304 the ranking component 106 determines that conclusive static ranking guidelines are defined for the anatomical object and imaging modality combination, process 300 proceeds to 308 and the ranking component 106 may generate or apply static ranking scores for different modality segmentations based on static ranking guideline data 120 .

[0068] At 310, the sorting component 106 (or the quality assessment component 110) may further assess the segmentation quality of each segmentation in the different segmentations. In this regard, the sorting component 106 may check the segmentation quality based on the static sorting (note that at 306, the segmentation quality may be otherwise incorporated into the dynamic sorting protocol). In one or more embodiments, the assessment of the segmentation quality at 310 may include an assessment of one or more quality metrics associated with the respective segmentation and / or an assessment of one or more uncertainty metrics associated with the respective segmentation (note that the uncertainty metrics are typically consistent with the quality metrics). At 312, the sorting component 106 (and / or the quality assessment component 110) then determines whether the quality of the respective segmentation in the set is acceptable based on one or more acceptable criteria defined for the one or more quality metrics and / or the one or more uncertainty metrics. In some specific implementations, the acceptability criteria defined for the quality metrics and / or the uncertainty metrics may vary depending on the type of anatomical object, the type of imaging modality associated with each segmentation, and the expected clinical use context (e.g., radiotherapy, diagnosis, disease staging, etc.). With these embodiments, information identifying or indicating the expected clinical use context can be provided with the segmented dataset 102, received as user feedback, and / or inferred (e.g., based at least in part on the type of anatomical object). Information defining one or more acceptance criteria for one or more quality metrics and / or one or more uncertainty metrics can be included in the dynamic sorting protocol data 122.

[0069] For example, in some embodiments, at 310, the ranking component 106 may compare the corresponding segmentation model uncertainty measure (e.g., a confidence score, a DICE score, or another model certainty / uncertainty measure) associated with each segmentation in the set of multimodal segmentations with an acceptable criterion defined for the uncertainty measure (such as a defined acceptable threshold). Using these embodiments, at 312, if any of the multimodal segmentations (e.g., one or more multimodal segmentations) has an uncertainty measure that fails to meet the acceptable criterion, the ranking component 106 (and / or the quality assessment component 110) may determine at 312 that the segmentation quality is unacceptable. The process 300 may then proceed to 306, and the ranking component 106 may determine the ranking score according to the dynamic ranking protocol. Additionally or alternatively, at 310, the ranking component 106 may compare one or more quality measures associated with each segmentation in the set of multimodal segmentations with one or more acceptable criteria defined for the one or more quality measures (e.g., a defined acceptable threshold or value). With these embodiments, at 312, if any of the multimodal segmentations (e.g., one or more multimodal segmentations) has one or more quality metrics that fail to meet one or more quality acceptance criteria, the ranking component 106 (and / or the quality assessment component 110) can determine that the segmentation quality is unacceptable at 312. Process 300 can then proceed to 306, and the ranking component 106 can determine a ranking score according to a dynamic ranking protocol.

[0070] In some implementations, if the ranking component 106 (or the quality assessment component 110) determines that the segmentation quality based only on the static ranking is acceptable, the process 300 can proceed to 314, where the ranking component can apply the static ranking score to the different modality segmentations and bypass the dynamic ranking protocol. At 314, the results of the use of the static ranking score also include a ranked multi-modality segmentation 316 for the anatomical object.

[0071] In some embodiments, the process 300 may proceed to 318, wherein the selection component selects the highest ranked segmentation from the set for clinical use. For example, the selected segmentation 320 corresponds to the highest ranked modality segmentation from the multimodal segmentation set included in the multimodal segmentation data for a particular anatomical object. The presentation component 116 may present or display the selected segmentation 320 to one or more users via the corresponding user devices 130 of one or more users for clinical review, and / or provide the selected segmentation 320 to another clinical application for use. Additionally or alternatively, the process 300 may proceed to 322, wherein the fusion component 112 generates a fused segmentation 324 by combining portions of different modality segmentations based on ranking, as discussed above. The presentation component 116 may present or display the fused segmentation to one or more users via the respective user devices 130 of one or more users for clinical review, and / or provide the fused segmentation to another clinical application for use.

[0072] Figure 4 Another exemplary non-limiting system 400 is shown that facilitates dynamic multimodal segmentation selection and fusion according to one or more embodiments of the disclosed subject matter. System 400 is similar to system 100, but additionally adds a feedback component 406, a machine learning component 410, and machine learning data 408 to computing device 101, as well as initial user feedback data 402 and result user feedback data 404. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0073] As described above, in some embodiments, the presentation component 116 can present the segmentation data set 102 to one or more users before performing the sorting, so as to receive user feedback (e.g., via the feedback component 406) regarding the quality of different multimodal segmentations for each anatomical object of interest segmented in the multimodal segmentation data. In the illustrated embodiment, the segmentation quality user feedback can include initial user feedback data 402 provided to the computing device 101 or otherwise received by the computing device. In some embodiments, in specific implementations where the multimodal segmentation data includes segmentations for multiple different anatomical objects (e.g., such as the MR segmentation data 202), the initial user feedback data 402 can additionally or alternatively include information regarding the selection of one or more specific target anatomical objects included in the multimodal segmentation data for further processing (e.g., for sorting, selection, and / or fusion using the techniques described herein). The initial user feedback data 402 may also include information regarding the selection of an intended clinical use context (e.g., radiation therapy, lesion characterization, bone metastasis analysis, diagnosis, staging, training data use, further processing via another clinical application, etc.), which may control or influence the dynamic sorting protocol applied by the sorting component 106.

[0074] The feedback component 406 may also provide for receiving result user feedback data 404 regarding the results of the ranking component 106 and / or the fusion component 112. The result user feedback data 404 may also be aggregated over time (e.g., as machine learning data 408) and used by the machine learning component 410 to refine the dynamic ranking protocol information 124, refine the fusion protocol information 124, and / or train and / or retrain corresponding machine learning models to perform the dynamic ranking and / or segmentation fusion process (e.g., the corresponding ranking / ranking model 612 and / or the corresponding fusion model 712), as discussed in more detail below. In this regard, in some embodiments, the result user feedback data 404 may include information regarding the appropriateness of the ranking / ranking scores applied by the ranking component 106 to the different multimodal segmentations for each anatomical object. With these embodiments, in addition to the selected and / or fused segmentation data 128, the presentation component 116 can also present / display (to a user at the user device 130) each of the multimodal segmentations received for the anatomical object and the corresponding ranking score applied to the multimodal segmentation by the ranking component 106. This feedback information can be stored in the machine learning data 408 along with the corresponding segmentation and any related metadata describing the characteristics of the corresponding segmentation (i.e., the type of anatomical object and any dynamic parameters discussed herein), and used by the machine learning component 410 to learn, define, improve and / or optimize the dynamic ranking protocol data 122 over time using one or more machine learning processes. The result user feedback data 404 can also include user feedback on the quality of the fused segmented image generated by the fusion component 112. This quality feedback information can also be stored in the machine learning data 408 along with the fused segmentation and the corresponding multimodal segmentation with the multimodal segmentation ranking score (as well as any other information associated with the multimodal segmentation received and / or determined by the system 400), and used by the machine learning component 410 to learn, define, improve and / or optimize the fused protocol data 124 over time using one or more machine learning processes.

[0075] Figure 5 A flowchart of another exemplary process 500 for selecting and fusing multimodal segmentation data according to one or more embodiments of the disclosed subject matter is presented. Process 500 corresponds to an exemplary process that can be performed by system 400 using computing device 101 and the corresponding components described above. Process 500 is similar to process 300, but user feedback is additionally integrated into process 500. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0076] refer to Figure 4 and Figure 5According to process 500, at 502, a receiving component 104 may receive a segmentation data set 102 for one or more anatomical objects. At 504, a presentation component may present the multimodal segmentation data to a user (e.g., a radiologist, clinician, engineer, technician, etc.) to provide initial user feedback 402 by request or prompt. At 506, a feedback component 406 may receive initial user feedback data 402. As discussed above, the initial user feedback data 402 may include information about selecting a specific target anatomical object included in the multimodal segmentation data for further processing (e.g., for sorting, selecting, and / or fusion using the techniques described herein). The initial user feedback data 402 may also include information about selecting an intended clinical use context. The initial user feedback data 402 may also include user feedback about the quality of the corresponding segmentation associated with each of the different modalities.

[0077] At 508, for the selected target anatomical object (if selected) or for each segmented anatomical object individually, the ranking component 106 can check the static ranking guideline data 120 for the selected anatomical object to determine whether the static ranking guideline data provides conclusive static ranking information for the anatomical object and imaging modality combination. If at 504, the ranking component 106 determines that conclusive static ranking guidelines are not available / defined for the particular anatomical object and imaging modality combination, the process 500 proceeds to 510, and the ranking component 106 continues to execute the dynamic ranking protocol to generate dynamic ranking scores for the different modality segmentations. At 510, the results of the dynamic ranking include a ranked multi-modal segmentation 520 for the anatomical object.

[0078] According to process 500, if at 508, the ranking component 106 determines that a conclusive static ranking guideline is defined for the anatomical object and imaging modality combination, then process 500 proceeds to 512, and the ranking component 106 can generate or apply a static ranking score for the different modality segmentations based on the static ranking guideline data 120. At 514, the ranking component 106 (or the quality assessment component 110) can further assess the segmentation quality of each of the different segmentations, as discussed above with reference to process 300. At 516, the ranking component 106 (and / or the quality assessment component 110) then determines whether the quality of the corresponding segmentation in the set is acceptable based on one or more acceptance criteria defined for one or more quality metrics and / or one or more uncertainty metrics, as discussed above with reference to process 300. If at 516, the ranking component determines that the segmentation quality of the set is unacceptable, then process 500 can then proceed to 510, and the ranking component 106 can determine the ranking score according to the dynamic ranking protocol. In some implementations, if at 516, the ranking component 106 (or the quality assessment component 110) determines that the segmentation quality based solely on the static ranking is acceptable, the process 500 can proceed to 518, where the ranking component can apply the static ranking score to the different modality segmentations and bypass the dynamic ranking protocol. At 518, the results of the use of the static ranking score also include a ranked multi-modality segmentation 520 for the anatomical object.

[0079] In some embodiments, the process 500 may proceed to 518, where the selection component 108 selects the highest ranked segmentation from the set as the selected segmentation 524. Additionally or alternatively, the process 500 may proceed to 526, where the fusion component 112 generates a fused segmentation 528 by combining portions of the different modal segmentations based on the ranking, as discussed above. At 530, the presentation component may further present or display the results, i.e., the selected segmentation 524 and / or the fused segmentation, to the user. In some specific implementations, the presented results may also include a ranked multimodal segmentation 520 for the anatomical object (e.g., a ranking score associated with the multimodal segmentation). In order to present the results at 530, the presentation component 116 may also provide a prompt or request to the user to provide result user feedback data 404 about the results. At 532, the feedback component may receive the result user feedback data 404. As described above, the result user feedback data 404 can include information about the appropriateness of the rating / ranking scores applied by the ranking component 106 to the corresponding multimodal segmentation (by viewing the ranked multimodal segmentation 520). The result user feedback data 404 can also include information about the quality of the fused segmentation 528. The feedback component 406 can also store the relevant input and output data together with the associated result feedback in the machine learning data 408 for use by the machine learning component 410 to refine the dynamic ranking protocol information 122, refine the fusion protocol information 124, and / or train and / or retrain the corresponding machine learning model to perform the dynamic ranking and / or segmentation fusion process (e.g., the corresponding ranking / rating model 612 and / or the corresponding fusion model 712), as discussed in more detail below.

[0080] In this regard, reference Figure 4, the machine learning component 410 can employ one or more machine learning techniques to learn and define dynamic ranking protocol data 122 based on the learned patterns, correlations, and / or rules between various dynamic parameters discussed herein that affect the relative ranking of available multimodal segmentations for a particular anatomical object. Additionally or alternatively, the machine learning component 410 can train and develop one or more machine learning models (e.g., referred to herein as ranking / rating models 612) to automatically infer ranking scores for corresponding multimodal segmentations for a given anatomical object and clinical context based on the learned patterns, correlations, and / or rules. The machine learning component 410 can similarly employ one or more machine learning techniques to learn and define fusion protocol data 124 based on learned patterns, correlations, and / or rules that affect the quality of different fused segmentation images for different anatomical objects, segmentation protocols / rules applied to generate fused segmentation images, features and rankings of corresponding multimodal segmentations included in a set used to generate fused segmentation images, and various additional dynamic parameters that affect the quality of fused segmentation images discussed herein. Additionally or alternatively, the machine learning component 410 may train and develop one or more machine learning models (e.g., referred to herein as fusion model 712) to automatically generate fused segmentations for anatomical objects based on corresponding multimodal segmentations received for anatomical objects and learned patterns and / or rules that affect the quality of different fused segmented images for different anatomical objects, segmentation protocols / rules applied to generate segmentations, features and ordering of corresponding multimodal segmentations included in a set used to generate a fused segmented image, and various additional dynamic parameters discussed herein that affect the quality of the fused segmented image.

[0081] To facilitate this purpose, the machine learning component 410 can perform learning with respect to any and all of the data received by the computing device 101 (e.g., the segmented data set 102, the initial user feedback data 402, and the resultant user feedback data 404), stored by the computing device (e.g., the static ranking guideline data 120, the dynamic ranking protocol data 122, the fused protocol data 124, and the machine learning data 408), and generated by the computing device 101 (e.g., the ranking scores of the corresponding multimodal segmentations and the selected and / or fused segmentation data 128). Hereinafter, any information received by the computing device 101 and generated by the computing device 101 may be aggregated over time and included in the machine learning data 408. Hereinafter, the static ranking guideline data 120, the dynamic ranking protocol data 122, the fused protocol data 124, and the machine learning data 408 are collectively referred to as "collective machine learning data" for the machine learning component 410.

[0082] It should be understood that the machine learning component 410 may perform learning associated with the collective machine learning data explicitly or implicitly. The learning and / or determination inference of the machine learning component 410 may facilitate the identification and / or classification of different patterns associated with the collective machine learning data, determine one or more rules associated with the collective machine learning data, and / or determine one or more relationships associated with the collective machine learning data that affect multimodal segmentation sorting and / or fusion optimization. The machine learning component 410 may also employ an automatic classification system and / or automatic classification process to facilitate the identification and / or classification of different patterns associated with the collective machine learning data, determine one or more rules associated with the collective machine learning data, and / or determine one or more relationships associated with the collective machine learning data that affect multimodal segmentation sorting and / or fusion optimization. For example, the machine learning component 410 may employ probability and / or statistical analysis (e.g., considering analytical utility and cost) to learn one or more patterns associated with the collective machine learning data, determine one or more rules associated with the collective machine learning data, and / or determine one or more relationships associated with the collective machine learning data that affect multimodal segmentation sorting and / or fusion optimization. The machine learning component 410 may employ, for example, a support vector machine (SVM) classifier to facilitate learning patterns associated with collective machine learning data, determine one or more rules associated with collective machine learning data, and / or determine one or more relationships associated with collective machine learning data that affect multimodal segmentation sorting and / or fusion optimization. Additionally or alternatively, the machine learning component 410 may employ other classification techniques associated with Bayesian networks, decision trees, and / or probabilistic classification models. The classifier employed by the machine learning component 410 may be explicitly trained (e.g., via common training data) and implicitly trained (e.g., via observing user behavior, receiving external information). For example, relative to the well-known SVM, the SVM is configured via a learning or training phase within a classifier constructor and a feature selection module. A classifier is a function that maps an input attribute vector x = (x1, x2, x3, x4, xn) to a confidence that the input belongs to a category, i.e., f(x) = confidence (category).

[0083] In one aspect, the machine learning component 410 may partially utilize an inference-based approach to facilitate learning one or more patterns associated with the collective machine learning data, determining one or more rules associated with the collective machine learning data, and / or determining one or more relationships associated with the collective machine learning data that affect the multimodal segmentation ranking and / or fusion optimization. The machine learning component 410 may also employ any suitable machine learning-based, statistical-based, and / or probability-based techniques. The machine learning component 410 may additionally or alternatively employ a reduced set of factors (e.g., an optimized set of factors) to facilitate generating the ranking / rating model 612 and / or fusion model 712 discussed below. For example, the machine learning component 410 may employ an expert system, fuzzy logic, SVM, a hidden Markov model (HMM), a greedy search algorithm, a rule-based system, a Bayesian model (e.g., a Bayesian network), a neural network, other nonlinear training techniques, data fusion, a usage-based analysis system, a system employing a Bayesian model, etc. In another aspect, the machine learning component 410 may perform a set of machine learning calculations associated with the collective machine learning data. For example, the machine learning component 410 may perform a set of clustering machine learning calculations, a set of decision tree machine learning calculations, a set of instance-based machine learning calculations, a set of regression machine learning calculations, a set of regularization machine learning calculations, a set of rule learning machine learning calculations, a set of Bayesian machine learning calculations, a set of deep Boltzmann machine calculations, a set of deep belief network calculations, a set of convolutional neural network calculations, a set of stacked autoencoder calculations, and / or a set of different machine learning calculations. Any rules, patterns, and / or correlations learned by the machine learning component 410 with respect to the collective machine learning data may be further stored in the machine learning data 408, applied by the machine learning component 410 to define and / or update / refine the dynamic ranking protocol data 122 and / or the fusion protocol data 124, and / or employed by the machine learning component 410 to train and / or retrain the ranking / rating model 612 and / or the fusion model 712 discussed below.

[0084] Figure 6 An exemplary machine learning framework 600 for facilitating ranking of multimodal segmentation data according to one or more embodiments of the disclosed subject matter is presented. The machine learning framework 600 provides a high-level framework of one or more machine learning processes that can be performed by a machine learning component 410 to generate optimal dynamic ranking protocol data 610 and / or a ranking / ranking model 612 configured to rank or rank multimodal segmentations of one or more anatomical objects using a ranking or ranking score that reflects the relative usefulness of the multimodal segmentations for a given clinical context.

[0085] refer to Figure 5 and Figure 6, according to the architecture 600, at 604, the machine learning component 410 can perform training data curation for multi-modal segmentation ranking. In some embodiments, the training data curation process can involve parsing all data available to the machine learning component 410 to identify and extract information that defines or indicates how different combinations of modality segmentation for the same anatomical object contribute to the usefulness of the modality segmentation for different clinical contexts (i.e., the relative ranking scores of the different combinations), taking into account the multiple dynamic parameters (and parameter values) associated with the different combinations. The machine learning component 410 can perform the curation process, and all defined anatomical objects can be represented in the received segmentation data set 102 (e.g., potentially all defined human anatomical objects, including all organs, all tissues, all vesicles, all types of lesions and tumors, etc.). As described above, the training component 410 can identify and extract this information from the static ranking guide data 120, the dynamic ranking protocol data 122, the fusion protocol data 124, and the machine learning data 408. In this case, the machine learning data 408 may include all segmentation data sets 102 previously processed by the system 100 and / or system 400 and associated metadata, the results of the processing (e.g., ranking scores determined by the quality assessment component 110, fused image segmentations, and potential quality data), and associated initial user feedback data 404 and result user feedback data 404 (when provided). In some embodiments, at 604, the machine learning component 410 can index the information extracted for each defined anatomical object.

[0086] In one or more embodiments, as a result of the training data curation process, the machine learning component 410 can generate training data 606 including a multimodal segmentation set for each anatomical object. Each of these sets can include two or more different combinations of imaging modalities. For each anatomical object, multiple representative sets of each possible combination of imaging modalities received for the anatomical object should be included. In addition, for each set, the training data 606 may include information defining and / or indicating each segmentation relative to a ranking score. In some specific implementations, this real information defining the ranking score can be provided from a previously processed segmentation data set 102 and a previously determined ranking / rating score of the segmentation data set (e.g., included in the machine learning data 408). With these specific implementations, the multimodal segmentation set for each anatomical object can include a previously processed multimodal segmentation data set. The information indicating the ranking score may include extracted or identified features associated with each segmentation in the segmentation associated with the segmentation ranking score. For example, this information may include various "dynamic" parameters related to segmentation quality, model uncertainty, image quality, clinical usage context, object size, object type and subtype, imaging protocol / parameters used, imaging data source / site, image feature attributes, image capture time, image registration burden (e.g., a measure of the difference in geometric alignment between corresponding original images and / or anatomical objects in reference space), presence of hemorrhage, presence of artifacts, patient demographics, patient pathology, and various other parameters (etc.) that may be included in the metadata associated with the corresponding multimodal segmentation described above.

[0087] Additionally or alternatively, the multimodal segmentation sets for each anatomical object can include new (unprocessed) sets of multimodal segmentation data. These new sets can be provided in and extracted from the multimodal segmentation data 602 by the machine learning component 410 to train the data curation process. These new sets can correspond to new instances of the segmentation dataset 102 and include or be associated with the same or similar metadata described with reference to the new instances, including information indicating each segmentation relative to the ranking score described above (e.g., dynamic parameters).

[0088] At 608, the machine learning component 410 can perform a machine learning phase of one or more machine learning processes. In this regard, in some embodiments, at 608, using the training data 606, the machine learning component 410 can employ one or more machine learning processes to learn and define an optimal dynamic sorting protocol (e.g., optimal dynamic sorting protocol data 610) for each anatomical object and combination of multimodal segmentation data based on learned correlations, patterns, and / or rules between various dynamic parameters and parameter values ​​associated therewith. Utilizing these embodiments, the optimal dynamic sorting protocol data 610 can be used to update the dynamic sorting protocol data 122 accordingly. For example, in some specific implementations, the machine learning component 410 can adjust the dynamic sorting protocol data 122 to reflect the optimal dynamic sorting protocol data 610. The optimal sorting protocol data 610 can also be added to the memory 118.

[0089] Additionally or alternatively, at 608, the machine learning component 410 can use the training data 606 to train and develop an object-specific ranking / rating model 612. For example, in some embodiments, the ranking / rating model 612 may include separate models customized for different anatomical objects. The input to each model may include segmentation data of anatomical objects of any possible modality, wherein the input indicates the modality associated with each input image segmentation. In some specific implementations, the input may also identify the clinical context. The input may also include relevant extracted dynamic features and feature values ​​associated with each segmentation. The output of each model may include a ranking or rating score indicating the relative usefulness of the segmentation for the clinical context. Using these embodiments, all multimodal segmentations in the set for anatomical objects can be processed by the same model to generate corresponding ranking or rating scores. The rating component 106 may also apply the corresponding scores estimated for each segmentation in the set to sort or rate the segmentation accordingly.

[0090] In this regard, the ranking / rating model 612 may include or correspond to a machine learning model, respectively. The ranking / rating model 612 may employ various types of ML algorithms, including (but not limited to): a deep learning model, a neural network model, a deep neural network model (DNN), a convolutional neural network model (CNN), a generative adversarial neural network model (GAN), a transformer, etc. In some specific implementations of these embodiments, the machine learning component 410 may employ a supervised machine learning process to train each model of each object using the true ranking information provided for the previously processed multimodal segmentation with adjusted accuracy based on the associated result user feedback (e.g., when available). The test set may be provided by the multimodal segmentation data 602. Once trained, the ranking / rating model 612 may be stored in the memory 118 and applied to the newly received segmentation data set 102 by the ranking component 106 at runtime to estimate the ranking score. Using these embodiments, the execution of the dynamic ranking protocol at 306 in the process 300 and / or at 510 in the process 500 may include applying the corresponding ranking / rating model 612 to generate a ranking score.

[0091] Figure 7 An exemplary machine learning framework 700 that facilitates multimodal segmentation data fusion according to one or more embodiments of the disclosed subject matter is presented. The machine learning framework 700 provides a high-level framework of one or more machine learning processes that can be executed by the machine learning component 410 to generate optimal fusion protocol data 710 and / or one or more fusion models 712. The optimal fusion protocol data 710 can include learned rules and / or algorithms that define how to combine different multimodal segmentation sets for different anatomical objects into a fused segmented image based on their respective rating / ranking scores and / or based on different dynamic parameters / parameter values ​​for a given clinical context. The fusion model 712 can correspond to a corresponding machine learning model suitable for executing the fusion model.

[0092] refer to Figure 5 and Figure 7According to the architecture 700, at 704, the machine learning component 410 can perform training data curation for multimodal segmentation fusion. The training data curation at 704 can correspond to the training data curation at 604 described with reference to the architecture 600. Additionally or alternatively, the training data curation process at 704 can involve more specifically identifying and extracting previously processed multimodal segmentation data sets and fused segmentation images generated thereby and associated resulting user feedback evaluating the quality of the corresponding fused images. In some specific implementations, additional examples of fused segmentation images and their corresponding sets of multimodal segmentations for different anatomical objects can be provided in the multimodal segmentation and fusion data 702. These additional examples may include information that defines or indicates how to combine different multimodal segmentations to generate corresponding fused segmentations. In some specific implementations, the additional examples can be manually generated and annotated.

[0093] In one or more embodiments, as a result of the training data curation process at 704, the machine learning component 410 can generate training data 706 including a set of multimodal segmentations for each anatomical object. Each of these sets can include two or more different combinations of imaging modalities. For each anatomical object, multiple representative sets should be included for each possible combination of imaging modalities received for the anatomical object. In addition, for each set, the training data 606 can include information defining and / or indicating each segmentation relative to a ranking score, as described by the reference architecture 600. In addition, the training data 706 can include previously known and / or applied fusion protocols for each set and previously generated fused segmentations for each set, which include fusion quality feedback (i.e., real data) if available.

[0094] At 708, the machine learning component 410 can perform a machine learning phase of one or more machine learning processes. In this regard, in some embodiments, at 708, using the training data 706, the machine learning component 410 can employ one or more machine learning processes to learn and define optimal fusion protocol data 710 for each anatomical object and combination of multimodal segmentation data based on learned correlations, patterns, and / or rules between various dynamic parameters and parameter values ​​associated therewith. Using these embodiments, the optimal fusion protocol data 710 can be used to update the fusion protocol data 124 accordingly. For example, in some specific implementations, the machine learning component 410 can adjust the dynamic fusion protocol data 124 to reflect the optimal fusion protocol data 710. The optimal fusion protocol data 710 can also be added to the memory 118.

[0095] Additionally or alternatively, at 708, the machine learning component 410 can use the training data 706 to train and develop an object-specific fusion model 712. For example, in some embodiments, the fusion model 712 may include separate models customized for different anatomical objects. The input to each model may include a segmentation data set for the anatomical object, including all different imaging segmentation modalities. In this regard, the fusion model 712 may correspond to a multi-channel input model, wherein the input indicates the modality associated with each input image segmentation. In some specific implementations, the input may also identify the clinical context. In some specific implementations, the input may also include a ranking or rating score determined or estimated for each segmentation. In other specific implementations, the corresponding fusion model 712 may inherently explain the ranking evaluation. The input may also include relevant extracted dynamic features and feature values ​​associated with each segmentation. The output of each fusion model may include a fused segmentation of the input multimodal segmentation set for the anatomical object, which combines different input segmentations in an optimal fusion manner. The fusion component 112 may also apply the corresponding fusion model for the corresponding anatomical object to the new input multimodal segmentation set at runtime. In this regard, performing the fusion process at 322 in process 300 and / or 526 in process 500 may include applying the corresponding fusion model 612 (for the particular anatomical object involved) to generate a fused segmentation. Additionally or alternatively, once the fusion model 612 has been trained and developed, the sorting or ranking process may be bypassed entirely.

[0096] In this regard, the fusion model 712 may include or correspond to a machine learning model, respectively. The fusion model 712 may employ various types of ML algorithms, including (but not limited to): deep learning models, neural network models, DNN, CNN, GAN, transformers, etc. In some specific implementations of these embodiments, the machine learning component 410 may employ a supervised machine learning process to train each fusion model 612 for each object using a previously generated fusion segmentation as a real example. With these embodiments, only a collection of fusion segmentations that are considered to be high-quality segmentations can be used to train the corresponding fusion model to transform the input multimodal segmentation into an optimal fused segmentation image. With these embodiments, the fusion model 712 may include a transformer network, respectively.

[0097] Figure 8 A block diagram of an exemplary non-limiting computer-implemented method 800 for generating a fused multi-modal segmented image according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0098] At 802, method 800 includes receiving (e.g., via receiving component 104) a segmentation dataset (e.g., segmentation dataset 102) by a system (e.g., system 100 or system 400) including a processor, the segmentation dataset including a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the anatomical object captured via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. At 804, method 800 includes determining, by the system (e.g., via ranking component 106), a ranking score for the different image segmentations using a dynamic ranking protocol (e.g., using dynamic ranking protocol data 122) as opposed to a static ranking protocol (e.g., using only static ranking guide data 120), the ranking score controlling the relative contributions of the different image segmentations so as to combine the different image segmentations into a fused segmentation for the anatomical object. At 806, method 800 includes combining, by the system (e.g., using fusion component 112), the different image segmentations based on the ranking score to generate a fused image segmentation (e.g., fused segmentation 324, fused segmentation 528, etc.).

[0099] Fig. 9 A block diagram of another exemplary non-limiting computer-implemented method 900 for generating a fused multi-modal segmented image according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0100] At 902, method 900 includes receiving (e.g., via receiving component 104) by a system including a processor (e.g., system 400), a segmented dataset (e.g., segmented dataset 102), the segmented dataset including a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the anatomical object captured via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. At 904, method 900 includes determining, by the system (e.g., via ranking component 106 and machine learning component 410), using one or more machine learning processes, ranking scores of the different image segmentations reflecting the relative usefulness of the different image segmentations for a clinical context. For example, in some embodiments, the one or more machine learning processes may include (e.g., by machine learning component 110) employing one or more machine learning processes to learn, define, and / or update dynamic ranking protocol data 122, which is used by ranking component 106 to perform ranking. Additionally or alternatively, one or more machine learning processes can include training, developing, by the machine learning component, a ranking / ranking model 612 (as described with reference to the machine learning framework 600) and thereafter applying (e.g., via the ranking component 106) the corresponding ranking / ranking model 612 to the corresponding segmentation to generate a ranking score. At 906, the method 900 includes combining, by the system (e.g., using the fusion component 112), the different image segmentations based on the ranking scores using a weighting scheme for the different image segmentations to generate a fused image segmentation (e.g., the fused segmentation 324, the fused segmentation 528, etc.) for the anatomical object.

[0101] Fig.10 A block diagram of another exemplary non-limiting computer-implemented method 1000 for generating a fused multi-modal segmented image according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0102] At 1002, method 1000 includes receiving (e.g., via receiving component 104) by a system including a processor (e.g., system 400), a segmentation dataset (e.g., segmentation dataset 102), the segmentation dataset including a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the anatomical object captured via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter. At 1004, method 1000 includes combining, by the system (e.g., using fusion component 112), the different image segmentations using a previously trained segmentation model customized for the anatomical object (e.g., one or more of fusion models 712) to generate a fused image segmentation for the anatomical object (e.g., fused segmentation 324, fused segmentation 528, etc.).

[0103] One or more embodiments may be systems, methods and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform one or more aspects of the present embodiment.

[0104] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following items: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded) and any suitable combination of the foregoing items. As used herein, a computer readable storage medium should not be understood as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0105] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the corresponding computing / processing device.

[0106] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.), procedural programming languages ​​(such as "C" programming languages ​​or similar programming languages), and machine learning programming languages ​​(such as CUDA, Python, Tensorflow, PyTorch, etc.). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server using suitable processing hardware. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may establish a connection with an external computer (e.g., via the Internet using an Internet service provider). In various embodiments involving machine learning programming instructions, the processing hardware may include one or more graphics processing units (GPUs), central processing units (CPUs), and the like. For example, one or more ranking / rating models 612 and / or one or more fusion models 712 may be written in a suitable machine learning programming language and executed via one or more GPUs, CPUs, or a combination thereof. In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of the computer-readable program instructions to perform aspects of the present invention.

[0107] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It is understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to function in a specific manner, so that the computer-readable storage medium with instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0110] The flow chart and block diagram in the accompanying drawings show the possible concrete implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, fragment or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative implementations, the function indicated in the frame may not occur in the order indicated in the figure.For example, depending on the function involved, two frames shown in succession can actually be performed substantially at the same time, or these frames may be performed in reverse order sometimes.It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on special hardware that performs a specified function or action or implements a combination of special hardware and computer instructions.

[0111] Combination Fig.11 , the systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application specific integrated circuits (ASICs), etc. In addition, the order in which some or all of the program blocks appear in each program should not be considered limiting. On the contrary, it should be understood that some program blocks can be executed in various orders, not all of which can be explicitly shown in this article.

[0112] refer to Fig.11, an illustrative example environment 1100 for implementing various aspects of the claimed subject matter includes a computer 1102. The computer 1102 includes a processing unit 1104, a system memory 1106, a codec 1135, and a system bus 1108. The system bus 1108 couples system components including, but not limited to, the system memory 1106 to the processing unit 1104. The processing unit 1104 can be any of a variety of available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1104.

[0113] The system bus 1108 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, or a native bus using any of a variety of available bus architectures including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Native Bus (VLB), Peripheral Component Interconnect (PCI), card bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), FireWire (IEEE 13114), and Small Computer System Interface (SCSI).

[0114] In various embodiments, the system memory 1106 includes a volatile memory 1110 and a non-volatile memory 1112, which may adopt one or more memory architectures in the disclosed memory architecture. A basic input / output system (BIOS) (including basic routines such as transferring information between elements within the computer 1102 during startup) is stored in the non-volatile memory 1112. In addition, according to the present invention, the codec 1135 may include at least one of an encoder or a decoder, wherein at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software. Although the codec 1135 is depicted as a separate component, the codec 1135 may be included in the non-volatile memory 1112. By way of illustration and not limitation, the non-volatile memory 1112 may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a 3D flash memory, or a resistive memory, such as a resistive random access memory (RRAM). In at least some embodiments, the non-volatile memory 1112 may employ one or more of the disclosed memory devices. In addition, the non-volatile memory 1112 may be a computer memory (e.g., physically integrated with the computer 1102 or its motherboard) or a removable memory. Examples of suitable removable memories that may be used to implement the disclosed embodiments may include a secure digital (SD) card, a compact flash (CF) card, a universal serial bus (USB) memory stick, and the like. The volatile memory 1110 includes a random access memory (RAM) that acts as an external cache memory, and one or more of the disclosed memory devices may also be employed in various embodiments. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM), and the like.

[0115] The computer 1102 may also include removable / non-removable, volatile / non-volatile computer storage media. Fig.11For example, a disk storage device 1114 is shown. The disk storage device 1114 includes, but is not limited to, devices such as a disk drive, a solid state disk (SSD), a flash memory card, or a memory stick. In addition, the disk storage device 1114 may include a separate storage medium or a storage medium combined with other storage media, including but not limited to an optical drive, such as a CD-ROM device (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), or a digital versatile magneto-optical disk ROM drive (DVD-ROM). In order to facilitate connecting the disk storage device 1114 to the system bus 1108, a removable or non-removable interface, such as an interface 1116, is generally used. It should be understood that the disk storage device 1114 can store information related to the entity. Such information can be stored at a server or provided to an application running on a server or entity device. In one embodiment, the entity can be notified (for example, by means of one or more output devices 1136) of the type of information stored in the disk storage device 1114 or transmitted to the server or application. An entity may be provided the opportunity to opt-in or opt-out of having such information collected or shared with the server or application (eg, via input from input device 1128).

[0116] It should be understood that Fig.11 Software that acts as an intermediary between entities and the basic computer resources described in the appropriate operating environment 1100 is described. Such software includes an operating system 1118. The operating system 1118, which may be stored on the disk storage device 1114, is used to control and allocate the resources of the computer system 1102. Application programs 1120 utilize the management of resources by the operating system 1118 through program modules 1124, and program data 1126 stored in the system memory 1106 or on the disk storage device 1114, such as a startup / shutdown transaction table. It should be appreciated that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

[0117] Entities enter commands or information into the computer 1102 through input devices 1128. Input devices 1128 include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, webcam, etc. These and other input devices are connected to the processing unit 1104 through the system bus 1108 via interface ports 1130. Interface ports 1130 include, for example, serial ports, parallel ports, game ports, and universal serial bus (USB). Output devices 1136 use some of the same types of ports as input devices 1128. Thus, for example, a USB port can be used to provide input to the computer 1102 and output information from the computer 1102 to output devices 1136. Output adapters 1134 are provided to illustrate that there are some output devices 1136 such as monitors, speakers, and printers, as well as other output devices 1136 that require special adapters. By way of illustration and not limitation, output adapters 1134 include video and sound cards that provide a means of connection between output devices 1136 and system bus 1108. It should be noted that other devices or systems of devices provide input and output capabilities, such as remote computer 1138.

[0118] The computer 1102 can work in a networked environment using logical connections to one or more remote computers, such as remote computer 1138. The remote computer 1138 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, a smart phone, a tablet computer, or other network node, and typically includes many of the elements described relative to the computer 1102. For the purpose of simplicity, only a memory storage device 1140 is shown for the remote computer 1138. The remote computer 1138 is logically connected to the computer 1102 through a network interface 1142, and then connected via a communication connection 1144. The network interface 1142 covers a wired communication network or a wireless communication network, such as a local area network (LAN) and a wide area network (WAN) and a cellular network. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variations thereon, packet switching networks, and digital subscriber lines (DSL).

[0119] The communication connection 1144 refers to the hardware / software used to connect the network interface 1142 to the bus 1108. Although the communication connection 1144 is shown internal to the computer 1102 for clarity of illustration, the communication connection can also be external to the computer 1102. For exemplary purposes only, the hardware / software required to connect to the network interface 1142 includes internal technology and external technology, such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, as well as wired and wireless Ethernet cards, hubs and routers.

[0120] The illustrated aspects of the disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0121] refer to Fig.12 , which shows a schematic block diagram of a computing environment 1200 according to the present disclosure, in which a subject system (e.g., system 600, etc.), method, and computer-readable medium can be deployed. The computing environment 1200 includes one or more clients 1202 (e.g., laptop computers, smart phones, PDAs, media players, computers, portable electronic devices, tablet computers, etc.). The client 1202 can be hardware and / or software (e.g., threads, processes, computing devices). The computing environment 1200 also includes one or more servers 1204. The server 1204 can also be hardware or hardware combined with software (e.g., threads, processes, computing devices). For example, the server 1204 can accommodate threads to perform transformations by adopting various aspects of the present disclosure. In various embodiments, one or more components, devices, systems, or subsystems of the system 400 can be deployed as hardware and / or software at the client 1202 and / or deployed as hardware and / or software deployed at the server 1204. One possible communication between the client 1202 and the server 1204 may be in the form of data packets transmitted between two or more computer processes, where the data packets may include healthcare-related data, training data, an AI model, input data to the AI ​​model, encrypted output data generated by the AI ​​model, etc. For example, the data packets may include metadata, such as associated context information. The computing environment 1200 includes a communication framework 806 (e.g., a global communication network such as the Internet, or a mobile network) that can be used to facilitate communication between the client 1202 and the server 1204.

[0122] Communication may be facilitated by wired (including fiber optic) and / or wireless technologies. The client 122 includes or is operably connected to one or more client data repositories 1208 that can be used to store information local to the client 1202 (e.g., the client medical image application logic 630). Similarly, the server 1204 may operably include or be operably connected to one or more server data repositories 812 that can be used to store information local to the server 1204 (e.g., the server medical image application logic 612, the segmentation model data 510, the medical image database 518, etc.).

[0123] In one embodiment, the client 1202 can transmit the encoded file to the server 1204 in accordance with the disclosed subject matter. The server 1204 can store the file, decode the file, or transmit the file to another client 1202. It should be appreciated that the client 1202 can also transfer an uncompressed file to the server 1204, and the file can be compressed in accordance with the disclosed subject matter. Likewise, the server 1204 can encode the video information and transmit the information to one or more clients 1202 via the communication framework 1206.

[0124] Although the subject matter is described above in the general context of computer executable instructions of a computer program product running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that other computer system configurations may be used to practice the computer-implemented method of the present invention, including single-processor or multi-processor computer systems, small computing devices, large computers, and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects may also be practiced in a distributed computing environment, in which tasks are performed by a remote processing device linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0125] As used in this application, the terms "component", "system", "subsystem", "platform", "layer", "gateway", "interface", "service", "application", "device", etc. may refer to and / or may include one or more computer-related entities or entities related to an operating machine having one or more specific functions. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a server and a server may be a component. One or more components may reside within a program and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. As another example, a corresponding component may be executed according to various computer-readable media having various data structures stored thereon. Components may communicate via local and / or remote processes, such as according to signals having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems) via a signal). As another example, a component may be a device having specific functionality provided by mechanical parts operated by electrical or electronic circuits operated by software or firmware applications executed by a processor. In such cases, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component may be a device that provides specific functionality through electronic components rather than mechanical parts, where the electronic components may include a processor or other means for executing the software or firmware that at least partially imparts the functionality to the electronic components. In one aspect, the component may emulate the electronic component, for example, via a virtual machine within a cloud computing system.

[0126] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive permutation. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. In addition, unless otherwise specified or clear from the context that it is for a singular form, the articles "a" and "an" used in this specification and the drawings should generally be understood to mean "one or more". As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration, and are intended to be non-limiting. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as being preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0127] As used in this specification, the term "processor" may refer to substantially any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, a discrete hardware component, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize a nanoscale architecture (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of physical equipment. The processor may also be implemented as a combination of computing processing units. In the present disclosure, terms such as "repository", "storage device", "data repository", "data storage device", "database", and substantially any other information storage component related to the operation and functionality of the component are used to refer to a "memory component", an entity embodied in a "memory", or a component including a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that may act as an external cache memory. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the system or computer-implemented method herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0128] What has been described above only includes examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purpose of describing the present disclosure, but those of ordinary skill in the art will recognize that many other combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including", "having", "having", etc. are used in the specific embodiments, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "including", as interpreted when "including" is used as a transitional word in the claims. Descriptions of various embodiments have been given for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best illustrate the principles of the embodiments, the practical application or technical improvement of technologies found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system, comprising: a memory storing computer executable components; and a processor that executes the computer executable components stored in the memory, wherein the computer executable components include: a receiving component (104) receiving a segmentation dataset (102) comprising a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the captured anatomical object via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol or an acquisition parameter; a ranking component (106) that employs a dynamic ranking protocol (122) as opposed to a static ranking protocol (120) to determine ranking scores for the different image segmentations, the ranking scores controlling the relative contributions of the different image segmentations in order to combine the different image segmentations into a fused segmentation for the anatomical object; and A fusing component (112) combines the different image segmentations based on the ranking scores to generate the fused image segmentation.

2. The system according to claim 1, wherein: The static sorting protocol includes determining the sorting score based solely on predefined guide information (120) for the combination and the type of the anatomical object, wherein the sorting component determines the sorting score using the dynamic sorting protocol that is opposite to the static sorting protocol based on determining that the predefined guide information for the combination and the type of the anatomical object is unavailable or uncertain.

3. The system according to claim 1, wherein: The different image segmentations are respectively associated with uncertainty metrics reflecting measures of uncertainty of the one or more segmentation models relative to the different image segmentations, and wherein the ranking component determines the ranking score using the dynamic ranking protocol as opposed to the static ranking protocol based on determining that one or more uncertainty metrics fail to satisfy an uncertainty metric criterion.

4. The system according to claim 1, wherein: The ranking component (106) determines the ranking score using the dynamic ranking protocol as opposed to the static ranking protocol based on determining that one or more of the different image segmentations fail to meet a quality criterion.

5. The system according to claim 4, wherein: The computer executable components also include: A quality assessment component (110) is provided for determining a quality metric for the different image segmentations and determining whether the different image segmentations meet the quality criteria based on the quality metric, wherein the quality metric is based on one or more factors selected from the group consisting of: impact of artifacts, image quality, contour connectivity, and image contrast.

6. The system according to claim 1, wherein: The dynamic ranking protocol includes determining the ranking scores based on uncertainty measures associated with the different image segmentations, the uncertainty measures reflecting measures of uncertainty of the different segmentation models with respect to the different image segmentations.

7. The system according to claim 1, wherein: The dynamic ranking protocol comprises determining the ranking score based on a parameter selected from the group consisting of: an uncertainty measure associated with the different image segmentation, the uncertainty measure reflecting a measure of uncertainty of the one or more segmentation models with respect to the different image segmentation; a quality metric for the different image segmentations, the quality metric reflecting one or more measures of quality of the different image segmentations; Artifact information regarding the presence or absence of artifacts; The acquisition protocols and acquisition parameters, the acquisition protocols and acquisition parameters are respectively associated with the different image segmentations; the size of the anatomical object; and The type of the anatomical object.

8. The system according to claim 10, wherein: The dynamic ranking protocol also includes determining the ranking score based on predefined guideline information for the combination and the type of the anatomical object, and wherein the dynamic ranking protocol includes employing one or more machine learning processes (600) to determine the ranking score based on the parameters and the predefined guideline information.

9. The system according to claim 1, wherein: The fusion component also combines the different image segmentations using a fusion protocol customized for the combination and the type of anatomical object, and wherein the fusion component employs one or more machine learning processes (700) to learn the fusion protocol for the combination and the type based on a plurality of different manually generated fused segmentations for the combination and the type.

10. A method comprising: receiving, by a system including a processor, a segmentation dataset comprising a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the captured anatomical object, respectively, via different segmentation models, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter; determining, by the system, ranking scores for the different image segmentations using a dynamic ranking protocol as opposed to a static ranking protocol, the ranking scores controlling relative contributions of the different image segmentations to combine the different image segmentations into a fused segmentation for the anatomical object, wherein the static ranking protocol includes determining the ranking scores based solely on predefined ranking information for the combination and the type of the anatomical object; and The different image segmentations are combined, by the system, based on the ranking scores to generate the fused image segmentation.

11. The method according to claim 10, wherein: The dynamic sorting protocol includes: The ranking score is determined by the system based on an uncertainty measure associated with the different image segmentations, the uncertainty measure reflecting a measure of uncertainty of the one or more segmentation models with respect to the different image segmentations.

12. The method according to claim 10, wherein: The dynamic sorting protocol includes: The ranking scores are determined by the system based on quality metrics for the different image segmentations, the quality metrics reflecting one or more measures of quality of the different image segmentations.

13. The method according to claim 10, wherein: The dynamic sorting protocol includes: The ranking score is determined by the system based on a parameter selected from the group consisting of: an uncertainty measure associated with the different image segmentation, the uncertainty measure reflecting a measure of uncertainty of the one or more segmentation models with respect to the different image segmentation; a quality metric for the different image segmentations, the quality metric reflecting one or more measures of quality of the different image segmentations; Artifact information regarding the presence or absence of artifacts; The acquisition protocols and acquisition parameters, the acquisition protocols and acquisition parameters are respectively associated with the different image segmentations; The dimensions of the anatomical object; and The type of the anatomical object.

14. The method according to claim 10, wherein: The combining comprises combining the different image segmentations using a fusion protocol customized for the combination and the type of the anatomical object, and wherein the method further comprises: One or more machine learning processes are employed by the system to learn the fusion protocol for the combination and the type based on a plurality of different manually generated fusion segmentations for the combination and the type.

15. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations comprising: receiving a segmentation dataset comprising a combination of different image segmentations of an anatomical object of interest segmented from different medical images of the captured anatomical object via different segmentation models, respectively, wherein the different medical images and the different image segmentations vary with respect to at least one of a capture modality, an acquisition protocol, or an acquisition parameter; determining a ranking score for the different image segmentations using a dynamic ranking protocol as opposed to a static ranking protocol, the ranking score controlling relative contributions of the different image segmentations to combine the different image segmentations into a fused segmentation for the anatomical object, wherein the static ranking protocol includes determining the ranking score based solely on predefined ranking information for the combination and the type of the anatomical object; and The different image segmentations are combined based on the ranking scores to generate the fused image segmentation.

Images