Machine learning methods and related aspects for processing and analyzing medical images

Abstract

Examples may provide a method of selecting data as part of an active learning process. The method includes extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features. The method also includes assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters. In addition, the method also includes selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster. Examples may also provide a method of segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model. Additional methods as well as related systems and computer readable media are also provided.

Description

Attorney Docket No.0184.0277-PCT (C17670_P17670-02) MACHINE LEARNING METHODS AND RELATED ASPECTS FOR PROCESSING AND ANALYZING MEDICAL IMAGES Cross-Reference To Related Applications

[0001] This application claims priority to, and the benefit of, U.S. Provisional Patent Application Ser. No.63 / 513,053, filed July 11, 2023, the disclosure of which is incorporated herein by reference. Field

[0002] This disclosure relates generally to machine learning, e.g., in the context of medical applications, such as diagnostics. Background

[0003] Active learning promises to improve annotation efficiency by iteratively selecting the most important data to be annotated first. However, there is a striking contradiction to this promise, namely, that active learning fails to select data as efficiently as the random selection at the first few choices. This can be viewed as a cold start problem in vision active learning, caused by a biased and outlier initial query.

[0004] In addition, increasingly annotated datasets have shown significant clinical impact on assessing specific anatomical structures. However, each of the datasets is small, partially labeled, and rarely investigates severe tumor subjects. Moreover, current models are limited to segmenting fixed organs / tumors, which cannot be extended to new domains and additional classes and fails to harness the anatomical relationship.

[0005] Accordingly, there is a need for approaches to solve the cold start problem in active learning and for medical image segmentation. Summary

[0006] The present disclosure provides, in certain aspects, solutions to the cold start problem that exploit three advantages of contrastive learning: (1) no annotation is required; (2) label diversity is ensured by pseudo-labels to mitigate bias; and (3) typical data is determined by contrastive features to reduce outliers. In additionalexemplary aspects, the present disclosure also provides contrastive language-image pre-training (CLIP) embedding to a segmentation model, sometimes dubbed herein as the CLIP-Driven Universal Model. In some embodiments, the CLIP-Driven Universal Model can harness the complete semantic relationship between abdominal structures to better segment 25 organs and 6 types of tumors. These and other aspects will be apparent upon a complete review of the present disclosure, including the accompanying figures.

[0007] According to various embodiments, a method of selecting data as part of an active learning process is presented. The method includes extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster, thereby selecting the data as part of the active learning process. In some embodiments, the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets. In some embodiments, the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

[0008] According to various embodiments, a system for selecting data as part of an active learning process is presented. The system includes a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operations comprising: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

[0009] According to various embodiments, a computer readable media is presented. The computer readable media comprises non-transitory computer executable instructions which, when executed by at least one electronic processor,perform at least: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

[0010] According to various embodiments, a method of segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model is presented. The method includes inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits, thereby segmenting the image using the CLIP-driven universal model. In some embodiments, the method further comprises using back-propagation to supervise corresponding interested classes for each of the medical image datasets.

[0011] According to various embodiments, a system for segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model is presented. The system comprises a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operations comprising: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP- driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-drivenuniversal model to obtain visual logits; and, generating segmentation results using the obtained visual logits.

[0012] According to various embodiments, a computer readable media is presented. The computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into a contrastive language-image pre-training (CLIP)-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP- driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits. Drawings

[0013] The above and / or other aspects and advantages will become more apparent and more readily appreciated from the following detailed description of examples, taken in conjunction with the accompanying drawings, in which:

[0014] Fig.1A is a flow chart that schematically shows exemplary method steps of selecting data as part of an active learning process according to some aspects disclosed herein;

[0015] Fig.1B is a flow chart that schematically shows exemplary method steps of segmenting an image using a contrastive language-image pre-training (CLIP)- driven universal model according to some aspects disclosed herein; and

[0016] Fig.2 is a schematic diagram of an exemplary system suitable for use with certain aspects disclosed herein.

[0017] Figs.3A-3D: Cold start problem in vision active learning. Most existing active querying strategies (e.g. BALD, Consistency, etc.) are outperformed by random selection in selecting initial queries, since random selection is i.i.d. to the entiredataset. However, some classes are not selected by active querying strategies due to selection bias, so their results are not presented in the low budget regime.

[0018] Fig.4: Label diversity of querying criteria. Random, the leftmost strategy, denotes the class distribution of randomly queried samples, which can also reflect the approximate class distribution of the entire dataset. As seen, even with a relatively larger initial query budget (40,498 images, 45% of the dataset), most active querying strategies are biased towards certain classes in the PathMNIST dataset. For example, VAAL prefers selecting data in the muscle class, but largely ignores data in the mucus and mucosa classes. On the contrary, our querying strategy selects more data from minority classes (e.g., mucus and mucosa) while retaining the class distribution of major classes. The higher the entropy is, the more balanced the class distribution is.

[0019] Figs. 5A-5C: Active querying based on Dataset Maps. (a) Dataset overview. (b) Easy- and hard-to-learn data can be selected from the maps based on ground truths. This querying strategy has two limitations: it requires manual annotations and the data are stratified by classes in the 2D space, leading to a poor label diversity in the selected queries. (c) Easy- and hard-to-contrast data can be selected from the maps based on pseudo-labels. This querying strategy is label-free and the selected hard-to-contrast data represent the most common patterns in the entire dataset, as presented in (a). These data are more suitable for training, and thus alleviate the cold start problem.

[0020] Figs.6A-6D: Quantitative comparison of map-based querying strategies. Random selection (dot-lines) can be treated as a highly competitive baseline in cold start because it outperforms six popular active querying strategies as shown in Fig.3. In comparison with random selection and three other querying strategies, hard-to- contrast performs the best. Although easy-to-learn and hard-to-learn sometimes performs similarly to hard-to-contrast, their selection processes require ground truths, which are not available in the setting of active learning.

[0021] Figs.7A and 7B On the importance of selecting a superior initial query. Hard-to-contrast data (upper-most lines) outperform other initial queries in every cycle of active learning on OrganaMNIST. We find that the performance of the initial cycle (20 images) and the last cycle (50 images) are strongly correlated.

[0022] Figs. 8A and 8B: Performance of each active learning querying strategies with different initial query strategies on BloodMNIST. Hard-to-contrast initial query strategy (upper-most lines) outperforms other initial query strategies in every cycle of active learning. With each active learning querying strategy, the performance of the initial cycle (20 labeled images) and the last cycle (50 labeled images) are strongly correlated.

[0023] Figs. 9A and 9B: Performance of each active learning querying strategies with different initial query strategies on PathMNIST. Hard-to-contrast initial query strategy (upper-most lines) outperforms other initial query strategies in every cycle of active learning. With each active learning querying strategy, the performance of the initial cycle (20 labeled images) and the last cycle (50 labeled images) are strongly correlated.

[0024] Fig. 10. Cosine similarity between CLIP embeddings. The CLIP embedding provides the hierarchical semantics among anatomical structures and their relationships. E.g., “Liver” has a large similarity with “Liver Tumor” and “Hepatic Vessel” (Hepatic vessel returns low-oxygen blood from your liver back to the heart, which has a high anatomical relationship with the liver); “Left Kidney” has a large similarity with “Right Kidney”.

[0025] Fig. 11. Overview. We have developed a Universal Model from an assembly of 14 public datasets of 3,410 CT scans. In total, 25 organs and 6 types of tumors are partially labeled. To deal with partial labels, Universal Model consists of a text branch (upper right) and a vision branch (lower right) (§3.1). The official test set of MSD and BTCV are used to benchmark the performance of organ segmentation (§4.1) and tumor detection (§4.3). 3D-IRCADb and TotalSegmentator are used for independent, external validation of model generalizability (§5.2), and transferability (§5.3). In addition to public datasets, the Universal Model has also been evaluated on a large-scale private dataset, consisting of 5,038 CT scans with 21 annotated organs, to investigate the extensibility to new classes (§5.4).

[0026] Fig. 12. Benchmark on MSD. We compare Universal Model with Swin UNETR (previously ranked first on the MSD leaderboard) on 5-hold cross validation of the MSD dataset. Universal Model achieves overall better segmentationperformance and offers substantial improvement over Swin UNETR in the tasks of segmenting liver tumors (+14%), pancreatic tumors (+8%), and colon tumors (+11%).

[0027] Fig. 13. Pseudo labels vs. human annotations. Similar performance between pseudo labels generated by the Universal Model and ground truths annotated by two human experts. These organs,i.e. spleen (Spl), liver (Liv), kidneys (Kid), stomach (Sto), and pancreas (Pan), can be annotated by AI with a similar variance to humans.

[0028] Fig. 14. Pancreatic tumor detection. Qualitative visualizations of the proposed Universal Model and five competitive baseline methods. We review the detection results of tumors from smaller to larger sizes (Rows 1–4). When it comes a CT scan without tumor from other hospitals, the Universal Model generalize well in organ segmentation and does not generate many false positives of tumors (Row 5; §4.3; §5.2).

[0029] Fig.15. Efficiency: FLOPs vs. DSC. We plot the average DSC score on the six MSD tasks against the FLOPs (Floating-point operations per second). The FLOPs is computed based on input with spatial size 96 × 96 × 96. The size of each circle indicates the number of parameters (‘#P’). In the inference, Universal Model is faster than nnU-Net (2nd best in performance) and Swin UNETR (3rd best) by 19× and 6× measured by FLOPs, respectively. Definitions

[0030] In order for the present disclosure to be more readily understood, certain terms are first defined below. Additional definitions for the following terms and other terms may be set forth throughout the specification. If a definition of a term set forth below is inconsistent with a definition in an application or patent that is incorporated by reference, the definition set forth in this application should be used to understand the meaning of the term.

[0031] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to “a method” includes one or more methods, and / or steps of the type described herein and / or which will become apparent to those persons skilled in the art upon reading this disclosure and so forth.

[0032] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Further, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In describing and claiming the methods, systems, and computer readable media, the following terminology, and grammatical variants thereof, will be used in accordance with the definitions set forth below.

[0033] Active Learning: As used herein, “active learning” refers to a process that aims to select the most important data from the unlabeled dataset and query human experts to annotate new data. The newly annotated data is then added to improve the model. This process can be repeated until the model reaches a satisfactory performance level or the annotation budget is exhausted.

[0034] Cancer Type: As used herein, “cancer type” refers to a type or subtype of cancer defined, e.g., by histopathology. Cancer type can be defined by any conventional criterion, such as on the basis of occurrence in a given tissue (e.g., blood cancers, central nervous system (CNS), brain cancers, lung cancers (small cell and non-small cell), skin cancers, throat cancers, nose cancers, liver cancers, bone cancers, lymphomas, pancreatic cancers, thyroid cancers, bladder cancers, kidney cancers, mouth cancers, stomach cancers, breast cancers, prostate cancers, bowel cancers, rectal cancers, ovarian cancers, intestinal cancers, soft tissue cancers, neuroendocrine cancers, lung cancers, gastroesophageal cancers, urothelial cancers, solid state cancers, heterogeneous cancers, homogenous cancers), head and neck cancers, gynecological cancers, colorectal cancers, unknown primary origin and the like, and / or of the same cell lineage (e.g., carcinoma, sarcoma, lymphoma, cholangiocarcinoma, leukemia, mesothelioma, melanoma, or glioblastoma) and / or cancers exhibiting cancer markers, such as Her2, CA15-3, CA19-9, CA-125, CEA, AFP, PSA, HCG, hormone receptor and NMP-22. Cancers can also be classified by stage (e.g., stage 1, 2, 3, or 4) and whether of primary or secondary origin.

[0035] Classifier: As used herein, “classifier” generally refers to algorithm computer code that receives, as input, test data and produces, as output, a classification of the input data as belonging to one or another class.

[0036] Dataset: As used herein, “dataset” refers to a group or collection of information, values, or data points related to or associated with one or more objects, records, and / or variables. In some embodiments, a given dataset is organized as, or included as part of, a matrix or tabular data structure. In some embodiments, a dataset is encoded as a feature vector corresponding to a given object, record, and / or variable, such as a given test or reference subject. For example, a medical data set for a given subject can include one or more observed values of one or more variables associated with that subject.

[0037] Electronic neural network: As used herein, “electronic neural network” refers to a machine learning algorithm or model that includes layers of at least partially interconnected artificial neurons (e.g., perceptrons or nodes) organized as input and output layers with one or more intervening hidden layers that together form a network that is or can be trained to classify data, such as test subject medical data sets (e.g., medical images or the like).

[0038] Labeled: As used herein, “labeled” in the context of data sets or points refers to data that is classified as, or otherwise associated with, having or lacking a given characteristic or property.

[0039] Machine Learning Algorithm: As used herein, "machine learning algorithm" generally refers to an algorithm, executed by computer, that automates analytical model building, e.g., for clustering, classification or pattern recognition. Machine learning algorithms may be supervised or unsupervised. Learning algorithms include, for example, artificial neural networks (e.g., back propagation networks), discriminant analyses (e.g., Bayesian classifier or Fisher’s analysis), multiple-instance learning (MIL), support vector machines, decision trees (e.g., recursive partitioning processes such as CART -classification and regression trees, or random forests), linear classifiers (e.g., multiple linear regression (MLR), partial least squares (PLS) regression, and principal components regression), hierarchical clustering, and cluster analysis. A dataset on which a machine learning algorithm learns can be referred to as "training data." A model produced using a machine learning algorithm is generally referred to herein as a “machine learning model.”

[0040] Subject: As used herein, “subject” or “test subject” refers to an animal, such as a mammalian species (e.g., human) or avian (e.g., bird) species. Morespecifically, a subject can be a vertebrate, e.g., a mammal such as a mouse, a primate, a simian or a human. Animals include farm animals (e.g., production cattle, dairy cattle, poultry, horses, pigs, and the like), sport animals, and companion animals (e.g., pets or support animals). A subject can be a healthy individual, an individual that has or is suspected of having a disease or pathology or a predisposition to the disease or pathology, or an individual that is in need of therapy or suspected of needing therapy. The terms “individual” or “patient” are intended to be interchangeable with “subject.” A “reference subject” refers to a subject known to have or lack specific properties (e.g., a known pathology, such as melanoma and / or the like).

[0041] Value: As used herein, “value” generally refers to an entry in a dataset that can be anything that characterizes the feature to which the value refers. This includes, without limitation, numbers, words or phrases, symbols (e.g., + or -) or degrees. Description of the Embodiments

[0042] Reference will now be made in detail to example implementations. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0043] I. Description of Example Embodiments

[0044] In some aspects, the present disclosure provides a method of selecting data as part of an active learning process. To illustrate, Fig. 1A is a flow chart that schematically shows certain of these exemplary method steps. As shown, method 100 includes extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features (step 102). Method 100 also includes assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters (step 104). In addition, method 100 selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster (step 106). In some embodiments, the extracting and assigning steps comprise an inter-class criterion thatenforces the label diversity to mitigate bias in the medical image datasets. In some embodiments, the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

[0045] To further illustrate, Fig. 1B is a flow chart that schematically shows certain exemplary method steps of segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model. As shown, method 108 includes inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP-driven universal model (step 110) and constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model (step 112). Method 108 also includes feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding (step 114) and generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model (step 116). In addition, method 108 also includes feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits (step 118) and generating segmentation results using the obtained visual logits (step 120). In some embodiments, method 108 further includes using back-propagation to supervise corresponding interested classes for each of the medical image datasets.

[0046] Fig. 2 is a schematic diagram of a hardware computer system 200 suitable for implementing various embodiments. For example, Fig.2 illustrates various hardware, software, and other resources that can be used in implementations of any of methods disclosed herein, including, e.g., method 100 and / or one or more instances of an electronic neural network. System 200 includes training corpus source 202 and computer 201. Training corpus source 202 and computer 201 may be communicatively coupled by way of one or more networks 204, e.g., the internet.

[0047] Training corpus source 202 may include an electronic clinical records system, such as an LIS, a database, a compendium of clinical data, or any other source of images suitable for use as a training corpus as disclosed herein. Due to hardware volatile memory storage limitations, each constituent image of an image may be broken down into a number of tiles, which may be, e.g., 128 pixels by 128 pixels. Such tiles are examples of “components” as that term is used herein. According tosome embodiments, each component is implemented as a vector, such as a feature vector, that represents a respective tile. Thus, the term “component” refers to both a tile and a feature vector representing a tile.

[0048] Computer 201 may be implemented as any of a desktop computer, a laptop computer, can be incorporated in one or more servers, clusters, or other computers or hardware resources, or can be implemented using cloud-based resources. Computer 201 includes volatile memory 214 and persistent memory 212, the latter of which can store computer-readable instructions, that, when executed by electronic processor 210, configure computer 201 to perform any of the methods disclosed herein, including method 100, 110, or 120, and / or form or store any electronic neural network, and / or perform any classification technique as described herein. Computer 201 further includes network interface 208, which communicatively couples computer 201 to training corpus source 202 via network 204. Other configurations of system 200, associated network connections, and other hardware, software, and service resources are possible.

[0049] Certain embodiments can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes. Examples Example 1: Making Your First Choice: To Address Cold Start Problem in Vision Active Learning

[0050] 1. Introduction

[0051] The cold start problem was initially found in recommender systems when algorithms had not gathered sufficient information about users with no purchase history. It also occurred in many other fields, such as natural language processing and computer vision during the active learning procedure. Active learning promises to improve annotation efficiency by iteratively selecting the most important data to annotate. However, we uncover a striking contradiction to this promise: Active learning fails to select data as effectively as random selection at the first choice. We identify this as the cold start problem in vision active learning and illustrate the problem using three medical imaging applications (Figs. 3A-3C) as well as a natural imaging application (Fig.3D). Cold start is a crucial topic because a performant initial query can lead to noticeably improved subsequent cycle performance in the active learning procedure, evidenced in section 3.3, below. There is a lack of studies that systematically illustrate the cold start problem, investigate its causes, and provide practical solutions to address it. To this end, we ask: What causes the cold start problem and how can we select the initial query when there is no labeled data available?

[0052] Random selection is generally considered a baseline to start the active learning because the randomly sampled query is independent and identically distributed (i.i.d.) to the entire data distribution. As is known, maintaining a similar distribution between training and test data is beneficial, particularly when using limited training data. Therefore, a large body of existing work selects the initial query randomly, highlighting that active querying compromises accuracy and diversity compared to random sampling at the beginning of active learning. Why? We attribute the causes of the cold start problem to the following two aspects:

[0053] (i) Biased query: Active learning tends to select data that is biased to specific classes. Empirically, Fig.4 reveals that the class distribution in the selected query is highly unbalanced. These active querying strategies (e.g. Entropy, Margin, VAAL, etc.) can barely outperform random sampling at the beginning because some classes are simply not selected for training. It is because data of the minority classes occurs much less frequently than those of the majority classes. Moreover, datasets in practice are often highly unbalanced, particularly in medical images. This can escalate the biased sampling. We hypothesize that the label diversity of a query is an importantcriterion to determine the importance of the annotation. To evaluate this hypothesis theoretically, we explore the upper bound performance by enforcing a uniform distribution using ground truth (Table 1) To evaluate this hypothesis practically, we pursue the label diversity by exploiting the pseudo-labels generated by K-means clustering (Table 2). The label diversity can reduce the redundancy in the selection of majority classes, and increase the diversity by including data of minority classes.

[0054] (ii) Outlier query: Many active querying strategies were proposed to select typical data and eliminate outliers, but they heavily rely on a trained classifier to produce predictions or features. For example, to calculate the value of Entropy, a trained classifier is required to predict logits of the data. However, there is no such classifier at the start of active learning, at which point no labeled data is available for training. To express informative features for reliable predictions, we consider contrastive learning, which can be trained using unlabeled data only. Contrastive learning encourages models to discriminate between data augmented from the same image and data from different images. Such a learning process is called instance discrimination. We hypothesize that instance discrimination can act as an alternative to select typical data and eliminate outliers. Specifically, the data that is hard to discriminate from others could be considered as typical data. With the help of Dataset Maps, we evaluate this hypothesis and propose a novel active querying strategy that can effectively select typical data (hard-to-contrast data in our definition, see section 2.2) and reduce outliers.

[0055] Systematic ablation experiments and qualitative visualizations in section 3 confirm that (i) the level of label diversity and (ii) the inclusion of typical data are two explicit criteria for determining the annotation importance. Naturally, contrastive learning is expected to approximate these two criteria: pseudo-labels in clustering implicitly enforce label diversity in the query; instance discrimination determines typical data. Extensive results show that our initial query not only significantly outperforms existing active querying strategies, but also surpasses random selection by a large margin on three medical imaging datasets (i.e. Colon Pathology, Abdominal CT, and Blood Cell Microscope) and two natural imaging datasets (i.e. CIFAR-10 and CIFAR- 10-LT). Our active querying strategy eliminates the need for manual annotation toensure the label diversity within initial queries, and more importantly, starts the active learning procedure with the typical data.

[0056] To the best of our knowledge, we are among the first to indicate and address the cold start problem in the field of medical image analysis (and perhaps, computer vision), making at least three contributions: (1) illustrating the cold start problem in vision active learning, (2) investigating the underlying causes with rigorous empirical analysis and visualization, and (3) determining effective initial queries for the active learning procedure. Our solution to the cold start problem can be used as a strong yet simple baseline to select the initial query for image classification and other vision tasks. Table 1: Diversity is a significant add-on to most querying strategies. AUC scores of different querying strategies are compared on three medical imaging datasets. In either low budget (i.e.0.5% or 1% of MedMNIST datasets) or high budget (i.e.10% or 20% of CIFAR-10-LT) regimes, both random and active querying strategies benefit from enforcing the label diversity of the selected data. The cells are marked with an “*” when adding diversity performs no worse than the original querying strategies. Coreset works very well as its original form because this querying strategy has implicitly considered the label diversity (also verified in Table 2) by formulating a K-center problem, which selects K data points to represent the entire dataset. Some results are missing (marked as “-”) because the querying strategy fails to sample at least one data point for each class. PathMNIST OrganAMNIST BloodMNIST CIFAR-10-LT 0.5% 1% 0.5% 1% 0.5% 1% 10% 20% Unif. (499) (899) (172) (345) (59) (119) (1420) (2841) Random✓ 96.8±0.6* 97.6±0.6* 91.1±0.9* 93.3±0.4* 94.7±0.7* 96.5±0.4* 91.6±1.1* 93.196.4±1±0.6*.3 97.6±0.9 90.7±1.1 93.1±0.7 93.2±1.5 95.8±0.7 62.0±6.1 -Consistency✓ 96.496.2±±0.1* 97.90.0 97.6±±0.1* 92.30.0 91.0±±0.5* 92.80.3 94.0±±1.0 92.90.±0.9* 95.9±0.5* 91.4±1.1* 93.4±0.2*687.9±0.2 95.5±0.5 67.1±17.1 88.6±0.3VAAL✓ 92.7±0.5* 93.0±0.6* 70.6±1.9* 84.6±0.5* 89.8±1.3* 93.4±0.9* 92.6±0.2* 93.7±0.4* -- - - - - - -Margin ✓ 97.9±0.2* 96.0±0.4* 81.8±1.2* 85.8±1.4 89.7±1.9* 94.7±0.7* 91.7±0.9* 93.291.0±2.3 9±0.2*6.0±0.3 - 85.9±0.7 - - 81.9±0.8 86.3±0.3Entropy✓ 93.2±1.6* 95.2±0.2* 79.1±2.3* 86.7±0.8* 85.9±0.5* 91.8±1.0* 92.0±1.2* 91.9±1.3*-87.5±0.1 - - - - 65.6Coreset95.0±±2.2 94.8±15.6 86.4±0.2✓ 85.6 95.6 0.7 97.5±±2.50.2±0.4* 89.9±0.5* 88.5±0.6* 94.1±1.1* 91.5±0.4* 93.6±0.2* 83.8±0.6 88.5±0.4 87.3±1.6 94.0±1.2 65.9±15.9 86.9±0.1 BALD✓ 95.8±0.2* 97.0±0.1* 87.2±0.3* 89.2±0.3* 89.9±0.8* 92.792.8 93.5±±0.71.3±0.1* 90.8±2.4* 92.0±2.3 95.3±1.0 - - 83.3±2.2 64.9±14.9 84.7±0.6 Table 2: Class coverage of selected data. Compared with random selection (i.i.d. to entire data distribution), most active querying strategies contain selection bias to specific classes, so the class coverage in their selections might be poor, particularly using low budgets. As seen, using 0.002% or even smaller proportion of MedMNISTdatasets, the class coverage of active querying strategies is much lower than random selection. By integrating K-means clustering with contrastive features, our querying strategy is capable of covering 100% classes in most scenarios using low budgets (≤0.002% of MedMNIST). We also found that our querying strategy covers the most of the classes in the CIFAR-10-LT dataset, which is designatedly more imbalanced.

[0057] 2. Method

[0058] In this section, we analyze in-depth the cause of cold start problem in two perspectives, biased query as the inter-class query and outlier query as the intra- class factor. We provide a complementary method to select the initial query based on both criteria. Section 2.1 illustrates that label diversity is a favourable selection criterion, and discusses how we obtain label diversity via simple contrastive learning and K-means algorithms. Section 2.2 describes an unsupervised method to sample atypical (hard-to-contrast) queries from Dataset Maps.

[0059] 2.1 Inter-class Criterion: Enforcing Label Diversity to Mitigate Bias

[0060] K-means clustering. The selected query should cover data of diverse classes, and ideally, select similar number of data from each class. However, this requires the availability of ground truth, which are inaccessible according to the nature of active learning. Therefore, we exploit pseudo-labels generated by a simple K-means clustering algorithm and select an equal number of data from each cluster to form the initial query to facilitate label diversity. Without knowledge about the exact number of ground-truth classes, over-clustering is suggested in recent works to increase performances on the datasets with higher intra-class variance. Concretely, given 9, 11, 8 classes in the ground truth, we set K (the number of clusters) to 30 in our experiments.

[0061] Contrastive features. K-means clustering requires features of each data point. It has been suggested that for the purpose of clustering, contrastivemethods (e.g. MoCo, SimCLR, BYOL) are more suitable than generative methods (e.g. colorization, reconstruction) because the contrastive feature matrix can be naturally regarded as cluster representations. Therefore, we use MoCo v2—a popular self-supervised contrastive method—to extract image features.

[0062] K-means and MoCo v2 are certainly not the only choices for clustering and feature extraction. We employ these two well-received methods for simplicity and efficacy in addressing the cold start problem. Fig.4 shows our querying strategy can yield better label diversity than other six dominant active querying strategies.

[0063] 2.2 Intra-class Criterion: Querying Hard-to-Contrast Data to Avoid Outliers

[0064] Dataset map. Given K clusters generated from Criterion #1, we now determine which data points ought to be selected from each cluster. Intuitively, a data point can better represent a cluster distribution if it is harder to contrast itself with other data points in this cluster—we consider them typical data. To find these typical data, we modify the original Dataset Map by replacing the ground truth term with a pseudo- label term. This modification is made because ground truths are unknown in the active learning setting but pseudo-labels are readily accessible from Criterion #1. For a visual comparison, Fig.5B and Fig.5C present the Data Maps based on ground truths and pseudo-labels, respectively. Formally, the modified Data Map can be formulated as follows. Letdenote a dataset of unlabeled images. Considering a minibatch of images, for each image its two augmented views form a positive pair, denoted. The contrastive prediction task on pairs of augmented images derived from the minibatch generate 2 images, in which a true labelfor an anchor augmentation is associated with its counterpart of the positive pair. We treat the other 2( - 1) augmented images within a minibatch as negative pairs. We define the probability of positive pair in the instance discrimination task as:whereis the cosine similarity between andand denote the projection head output of a positive pair for the input in a batch; is an indicator function evaluating to 1 iff and denotes a temperature parameter.denotes the parameters at the end of the epoch. We define confidence across epochs as:The confidence is the Y-axis of the Dataset Maps (see Figs.5B and 5C).

[0065] Hard-to-contrast data. We consider the data with a low confidence value (Equation 3) as “hard-tocontrast” because they are seldom predicted correctly in the instance discrimination task. Apparently, if the model cannot distinguish a data point with others, this data point is expected to carry typical characteristics that are shared across the dataset. Visually, hard-to-contrast data gather in the bottom region of the Dataset Maps and “easy-to-contrast” data gather in the top region. As expected, hard-to-learn data are more typical, possessing the most common visual patterns as the entire dataset; whereas easy-to-learn data appear like outliers, which may not follow the majority data distribution (examples in Fig.5A and Fig.5B). Additionally, we also plot the original Dataset Map in Fig.5B, which grouped data into hard-to-learn and easy-to-learn. Although the results in section 3.2 show equally compelling performance achieved by both easy-to-learn and hard-to-contrast data (ours), the latter do not require any manual annotation, and therefore are more practical and suitable for vision active learning.

[0066] In summary, to meet the both criteria, our proposed active querying strategy includes three steps: (i) extracting features by self-supervised contrastive learning, (ii) assigning clusters by K-means algorithm for label diversity, and (iii) selecting hard-to-contrast data from dataset maps.

[0067] 3. Experimental Results

[0068] Datasets & metrics. Active querying strategies have a selection bias that is particularly harmful in long-tail distributions. Therefore, unlike most existing works, which tested on highly balanced annotated datasets, we deliberately examine our method and other baselines on longtail datasets to simulate real-world scenarios. Three medical datasets of different modalities in MedMNIST are used: PathMNIST (colorectal cancer tissue histopathological images), BloodMNIST (microscopic peripheral blood cell images), OrganAMNIST (axial view abdominal CT images of multiple organs). Area Under the ROC Curve (AUC) and Accuracy are used as theevaluation metrics. All results were based on at least three independent runs, and particularly, 100 independent runs for random selection. UMAP is used to analyze feature clustering results.

[0069] Baselines & implementations. We benchmark a total of seven querying strategies: (1) random selection, (2) Max-Entropy, (3) Margin, (4) Consistency, (5) BALD, (6) VAAL, and (7) Coreset. For contrastive learning, we trained 200 epochs with MoCo v2, following its default hyperparameter settings. We set to 0.05 in equation 2.

[0070] 3.1 Contrastive Features Enable Label Diversity to Mitigate Bias

[0071] Label coverage & diversity. Most active querying strategies have selection bias towards specific classes, thus the class coverage in their selections might be poor (see Table 2), particularly at low budgets. By simply enforcing label diversity to these querying strategies can significantly improve the performance (see Table 1), which suggests that the label diversity is one of the causes that existing active querying strategies perform poorer than random selection.

[0072] Our proposed active querying strategy, however, is capable of covering 100% classes in most low budget scenarios (≤0.002% of full dataset) by integrating K- means clustering with contrastive features.

[0073] 3.2 Pseudo-labels Query Hard-to-Contrast Data and Avoid Outliers

[0074] Hard-to-contrast data are practical for cold start problem. Fig. 6 presents the quantitative comparison of four map-based querying strategies, wherein easy- or hard-to-learn are selected by the maps based on ground truths, easy- or hard- to-contrast are selected by the maps based on pseudo-labels. Note that easy- or hard- to-learn are enforced with label diversity, due to their class-stratified distributions in the projected 2D space (illustrated in Fig.5). Results suggest that selecting easy-to- learn or hard-to-contrast data contribute to the optimal models. In any case, easy- or hard-to-learn data cannot be selected without knowing ground truths, so these querying strategies are not practical for active learning procedure. Selecting hard-to- contrast, on the other hand, is a label-free strategy and yields the highest performance amongst existing active querying strategies (reviewed in Fig. 3). More importantly, hard-to-contrast querying strategy significantly outperforms random selection by 1.8% (94.14%±1.0% vs.92.27%±2.2%), 2.6% (84.35%±0.7% vs.81.75%±2.1%), and 5.2%(88.51%±1.5% vs.83.36%±3.5%) on PathMNIST, OrganAMNIST, and BloodMNIST, respectively, by querying 0.1% of entire dataset. Similarly on CIFAR-10-LT, hard-to- contrast significantly outperforms random selection by 21.2% (87.35%±0.0% vs. 66.12%±0.9%) and 24.1% (90.59%±0.1% vs. 66.53%±0.5%) by querying 20% and 30% of entire dataset respectively. Note that easy- or hard-to-learn are not enforced with label diversity, for a more informative comparison.

[0075] 3.3 On the Importance of Selecting Superior Initial Query

[0076] A good start foresees improved active learning. We stress the importance of the cold start problem in vision active learning by conducting correlation analysis. Starting with 20 labeled images as the initial query, the training set is increased by 10 more images in each active learning cycle. Fig. 9A presents the performance along the active learning (each point in the curve accounts for 5 independent trials). The initial query is selected by a total of 9 different strategies, and subsequent queries are selected by 5 different strategies. denotes the score achieved by the model that is trained by labeled images. The Pearson correlation coefficient between (starting) and (ending) shows strong positive correlation ( = 0.79, 0.80, 0.91, 0.67, 0.92 for random selection, Entropy, Margin, BALD, and Coreset, respectively). This result is statistically significant ( < 0.05). Hard-to-contrast data (our proposal) consistently outperforms the others on OrganAMNIST (Fig.7), BloodMNIST (Fig.8), and PathMNIST (Fig.9), and steadily improves the model performances within the next active learning cycles.

[0077] The initial query is consequential regardless of model initialization. A pre-trained model can improve the performance of each active learning cycle for both random and active selection, but the cold start problem remains (evidenced in Fig.9B). This suggests that the model instability and data scarcity are two independent issues to be addressed for the cold start problem. Our “hard-to-contrast” data selection criterion only exploits contrastive learning (an improved model), but also determines the typical data to be annotated first (a better query). As a result, when fine-tuning from MoCo v2, the Pearson correlation coefficient between and remains high ( = 0.92, 0.81, 0.70, 0.82, 0.85 for random selection, Entropy, Margin, BALD, and Coreset, respectively) and statistically significant ( < 0.05).

[0078] 4. Conclusion

[0079] This example systematically examines the causes of the cold start problem in vision active learning and offers a practical and effective solution to address this problem. Analytical results indicate that (1) the level of label diversity and (2) the inclusion of hard-to-contrast data are two explicit criteria to determine the annotation importance. To this end, we devise a novel active querying strategy that can enforce label diversity and determine hard-to-contrast data. The results of three medical imaging and two natural imaging datasets show that our initial query not only significantly outperforms existing active querying strategies but also surpasses random selection by a large margin. This finding is significant because it is the first few choices that define the efficacy and efficiency of the subsequent learning procedure. We foresee our solution to the cold start problem as a simple, yet strong, baseline to sample the initial query for active learning in image classification.

[0080] Example 2: Contrastive Language–Image Pre-training (CLIP)- Driven Universal Model for Organ Segmentation and Tumor Detection

[0081] 1. Introduction

[0082] Accelerated medical image computing benefits from the ever-growing amount of annotated datasets. Lately, over 4,700 CT scans are publicly available with expert labels on abdominal structures. However, an ingrained impression is that medical imaging datasets are too small to train robust deep learning models. One of the reasons for this impression is that these public datasets are associated with imaging competitions. As per the fairness concern, these datasets must be used in isolation, i.e. no external data are allowed. Since each institute has limited time, budget, and specific clinical purposes, the types of annotated organs available in the released CT scans vary from institute to institute. Abnormal CT scans with annotated tumors are fewer - only 6 types of abdominal tumors are annotated in the public datasets, accounting for a very small proportion of entire CT scans.

[0083] This has motivated us to relax the requirement that no external data is allowed, and to explore the potential of AI models in multi-organ segmentation and tumor detection. This is achieved by exploiting the datasets that are publicly available and by demonstrating the clinical impact of AI performance, including model generalizability (i.e. robust to CT scans from various hospitals), transferability (i.e. generic image representation that is transferable to multiple downstream tasks), andextensibility (i.e. adaptable to new classes without forgetting previously learned classes). To this end, we assemble 14 publicly available datasets, including 3,672 CT scans with 25 partially annotated organs and 6 tumors. Furthermore, we anticipate that most of the medical datasets, released in the near future, will also focus on a small set of interested organs, and that some current unlabeled organs, such as vermiform appendix, would be newly annotated in future datasets. This requires us to develop new strategies to continually deal with more partially labeled datasets with upcoming organs from a variety of institutes.

[0084] However, there are major challenges with assembling partially annotated datasets. The first one is the label inconsistency, which has three aspects. (i) Index inconsistency. The same organ is labeled as a different index in various datasets. For example, the stomach is labeled ‘7’ in BTCV, but ‘5’ in WORD. (ii) Name inconsistency. Naming can be confusing if multiple labels refer to the same anatomical structure. For example, “postcava” in AMOS22 and “inferior vena cava” in BTCV. (iii) Background inconsistency. For example, when combining Pancreas-CT and MSD- Spleen to segment both the pancreas and spleen together, the pancreas in MSD- Spleen might remain in the background when it should be in the foreground. The second challenge is label orthogonality. Most segmentation methods, using one-hot encoding, eliminate the semantic relationship between classes. For example, given liver [1,0,0], liver tumor [0,1,0], and pancreas [0,0,1], there is no semantic difference between liver↔liver tumor and liver↔pancreas. A possible solution is to use few-hot labels to encode the hierarchical relationship. With that strategy, the liver, liver tumor, and pancreas can be encoded as [1,0,0], [1,1,0], and [0,0,1]. Few-hot labels can indicate that liver tumors are part of the liver, but the relationship between organs is still orthogonal and, more importantly, either one-hot or few-hot labels are not easy to extend to more classes. In the future, the upcoming organs would require model retraining and increasing the dimensionality of one-hot or few-hot labels.

[0085] To address the label inconsistency problem, we employ one vs. all manner to generate the binary segmentation results for each class and supervise the interested organ for each dataset using masked back-propagation. To overcome the label orthogonality problem, we employ the CLIP pretrained text encoder to generate the text embedding and serve as the parameter of segmentor. Note that CLIP waspre-trained on 400 million image-text pairs (some are medical images and text), exploiting the semantic relationship between images and language. We illustrate in Fig.10 that CLIP embedding presents the relationship between organs and tumors. Ablation studies in Table 1 have also shown that the introduction of CLIP embedding is one of the most salient factors to the proposed framework. Moreover, the extension of upcoming organs is easily realized through adding new text embedding and fine- tuning the framework.

[0086] In this example, we propose a CLIP-driven Universal Model that can segment 25 organs and detect 6 tumors with state-of-the-art performance, generalize to CT scans from different institutes, and can be extended to more organ / tumor classes, in contrast with existing dataset-specific models. Specifically, experimental results have demonstrated six advantages of the CLIP-driven Universal Model. (I) High abdominal organ segmentation performance. We rank first in the MSD and BTCV challenges, leading to substantial performance improvement over other counterparts (§4.1; Table 2; Fig.12). The segmentation of six organs performed by our model has proven to be of similar quality as the annotation provided by human experts (§4.2; Fig. 13). (II) A higher specificity of tumor detection in normal CT scans than existing models while maintaining compelling sensitivity in abnormal CT scans (§4.3; Table 3; Fig.14). (III) Computationally more efficient compared with dataset-specific models, accelerating the inference speed by an order of six (§5.1; Fig. 15). (IV) The performance of organ segmentation and tumor detection is generalized to CT scans from a variety of hospitals without additional tuning and adaptation (§5.2; Table 4; Fig. 14). (V) An effective Foundation Model for numerous downstream medical imaging tasks, demonstrating a strong transferability on tasks across multiple diseases, organs, and datasets (§5.3; Table 5). (VI) The extensibility to novel classes. Our Universal Model shows the capability of quickly adapting to novel classes without forgetting previously learned classes and without the need for the previous training datasets (§5.4).

[0087] The high performance of the Universal Model has also resulted in a fully annotated dataset of 3,672 CT scans—the number continues to increase as new data are released—with pseudo labels for 25 organs and 6 tumors. The curated dataset, comprising multi-center, multi-vendor, multiphase, and multi-disease cases, canprovide diverse examples and a test bed to develop high-performance AI models for organ segmentation and tumor detection. Table 1. Medical prompt templates. We compare the effectiveness of three proposed prompt templates for CLIP and the conventional one-hot embedding (DoDNet) in model training. With a proper prompt template, semantic information could be transferred from pre-trained model to a text-driven segmentor.

[0088] 2. Related Work

[0089] Multi-organ Segmentation. Deep learning-based methods, especially the variants of U-Net, have demonstrated superior performance in abdominal multi- organ segmentation.

[0090] Partial Label Problem. Numerous public abdominal CT datasets have been developed in recent years, but few of them are with carefully annotated whole abdominal organs.

[0091] CLIP in Medical Imaging. Recently, large-scale pretrained vision- language models (VLM) like CLIP have been successfully applied to some vision tasks in medical domain such as medical image segmentation, diffusion model and video segmentation.

[0092] 3. Methodology

[0093] 3.1. CLIP-driven Universal Model

[0094] Problem Definition. Formally, a dataset assembly includes publicly available datasets, which is composed of organs and tumors, and define as organ and tumor dictionary with classes, i.e., = {Liver, Liver Tumor, Left Kidney, Right Kidney, Kidney Tumor, Hepatic Vessel, ...,Here,represents the dataset that containsCT scans withwith annotation selected from a corresponding sub-dictionary . The objective is to train a model with the assembly dataset and this model is capable of extending to upcoming organs and tumors.

[0095] Overview. The overall workflow of the proposed CLIP-driven Universal Model is shown in Fig. 11. Take a CT scanfrom datasetwith the corresponding subdictionaryas input. In text branch, are used to construct text prompts based on a prompt template, which is fed into a CLIP pre-trained text encoder to generate CLIP embedding. After that, a text-based controller dynamically generates the parameter for text-driven segmentor. Then in vision branch, each CT scanwith standardized processing is fed into the vision encoder and text-driven segmentor to obtain visual logits. With obtained visual logits, we employ the one vs. all manner for generating the binary segmentation results. Finally, the masked back-propagation is adopted to supervise the corresponding interested classes for each dataset.

[0096] Text Branch. To encode the semantic relationship among organs and tumors, we introduce a text-driven segmentor with parameter generated from text information. Specifically, given sub-dictionaryand medical prompt template such as “a computerized tomography of a ”, where is a concrete class names in dictionary, a CLIP text encoder generates the CLIP embedding as . Then, a MLP layer is used as the text-based controllerto generate the parameters of text-driven segmentor with CLIP embeddingrepresents the text-based controller parameters. It is worth noting that the context of medical prompt template is critical for CLIP embedding. We propose three prompt templates as shown in Table 1. Compared with traditional one-hot embedding, we can see our CLIP-driven method performs better, validating the effectiveness of CLIP embedding in encoding the semantic relationship for organs and tumors.

[0097] Vision Branch. In vision branch, each CT scanis processed with standardized processing, e.g., isotropic spacing, uniform intensity scale, to eliminate the scan gap among various dataset. Then, the vision feature is extracted with a vision encoder. After that, the text-driven segmentor is introduced to predict the logits for each class. In detail, the text-driven segmentor contains three sequential convolutional layers with kernels. The first two layers have 8 channels, andthe last one has 1 channel, representing the interested foreground. Thus, a total of 153 parameters are required from the text-based controller output. The predicted logits for class is computed asis vision feature and * represents the convolution.

[0098] Masked Back-Propagation. To address the label inconsistency problem, we proposed the masked back-propagation technique. With the logitsfor class , we employ sigmoid function to generate the prediction results for each class using one vs. all manner, . The BCE loss function is utilized for supervision. Lastly, we mask the loss terms of these classes that are not contained in sub-dictionaryand only back-propagate the accurate supervision to update the whole framework.

[0099] 3.2. Assembly of Public Dataset

[0100] To make full use of publicly available datasets and explore the potential of proposed universal model in multi-organ segmentation and tumor detection, we further assemble partially annotated datasets. However, it is non-trivial to assemble partially labeled datasets due to several challenges.

[0101] Label Inconsistency. The first challenge includes three aspects. (1) Index inconsistency: the same organ can be labeled as different index in various datasets. (2) Name inconsistency: the same anatomical structure can be named after different terminologies. Moreover, some datasets annotate left and right organs while some do not. For example, BTCV annotates “right kidney” and “left kidney”, while KiTS only annotate “kidney”. Fortunately, (1) and (2) can be solved through careful manual checks to re-index all the classes in a consistent manner. (3) Background inconsistency: partially labeled datasets annotate some other organs as background, leading to the disability of existing softmax-based training scheme in assembly training. We have addressed this problem through the proposed masked back- propagation (§3.1).

[0102] Label Orthogonality. The second challenge lies in two aspects. (1) Semantic relation: there is semantically hierarchical relationship among medical terms. For example, the liver tumor should be child node of liver instead of pancreas in semantics since tumor are part of corresponding organ. (2) Label extensibility: manyimportant organs are not annotated in current public dataset, e.g., vermiform appendix. To encode the semantic relation, we design a novel CLIP-driven scheme in text branch (§3.1). When extending to upcoming organs, it is easily enlarge the dictionary with new organ name and keep the network architecture to finetune the whole framework.

[0103] Open Challenges. (1) Inconsistent label protocols. In AMOS, “Aorta” refers to the entire region of Aorta, but in AbdomenCT-1K, a part of upper regions annotation is missing. Aorta is considered and annotated. It is because of the inconsistent definitions in different datasets and this requires considerable manual corrections when assembling these datasets together. (2) Longtail problem. The assembly of public dataset leads to severe class imbalance problems, especially for small tumors. In this example, we utilize data augmentation to alleviate the long-tail problem.

[0104] 4. Experiments & Results

[0105] Datasets and Evaluation Metrics. A total of 14 public datasets consisting of 3410 CT scans are used for training. Other 2 public datasets and one private dataset consisting of 6162 CT scans are used for evaluation. The Dice Similarity Coefficient (DSC) and Normalized Surface Distance (NSD) are used as measurement for experiment results. These two metric measures the overlap of ground truth and prediction volumes and overlap correspondingly.

[0106] Implementation Details. The Universal Model is trained using the AdamW optimizer with a warm-up cosine scheduler of 50 epochs for Universal Model experiments. The evaluation and CLIP ablation experiments use warm-up of 5 epochs for accelerating training since pre-trained model and word embedding are employed. The segmentation experiments use batch-size of 6 per GPU with a patch size of 96 × 96 × 96. Default initial learning rate of 4e−4, momentum of 0.9 and decay of 1e−5on multi-GPU with DDP. The framework is implemented in MONAI 0.9.02. The standard five-fold cross validation strategy is used to train models for Universal Model experiments. Ablation studies are conducted on fold 0. We select the best model in each fold by evaluating the validation best metrics. Models are trained on NVIDIA RTX A5000 cards.

[0107] 4.1. Organ Segmentation on MSD and BTCV

[0108] We offer #1 solutions in both Medical Segmentation Decathlon (MSD) and BTCV, surpassing the runners-up by a considerable margin. In MSD competition, we rank first in the published leaderboard. Moreover, we present the comparison with Swin UNETR (previously ranked first on the MSD leaderboard) on 5-hold cross validation in Fig.12. Universal model surpasses Swin UNETR with 2.37%, 14.09%, 2.64%, 8.30%, 0.23%, 0.83%, 0.91% and 11.68% for liver, liver tumors, pancreas, pancreatic tumors, hepatic vessel, vessel tumor, spleen and colon tumors. In the BTCV benchmark, we achieve state-of-the-art in the validation set as shown in Table 2, offering at least 3.5% improvements over the second best methods, underlining the effectiveness of Universal Model in organ segmentation.

[0109] 4.2. Towards Human-level Organ Segmentation

[0110] Manual annotations have inter-rater and intra-rater variance, particularly in segmentation tasks, because some of the organs’ boundaries are blurry and ambiguous. We assess the quality of pseudo labels predicted by Universal Model and manual annotation performed by human experts.17 CT scans in the BTCV dataset have been annotated by two independent groups of radiologists from different institutes (not test server labels). As a result, each CT scan is associated with AI prediction, and two human annotations (GT1 and GT2). Fig.13 presents their mutual DSC score for each pair, i.e. AI↔GT1, AI↔GT2, and GT1↔GT2. We find the DSC between AI and humans is slightly larger than the DSC between humans in segmenting 6 types of organs (i.e. spleen, liver, kidney, stomach, and pancreas). With this high-quality AI prediction, we assemble a large dataset of 3,410 CT scans from a diverse set of hospitals (Fig. 11 and generate pseudo labels for 25 organs and 6 tumors4. Some refinement on the pseudo labels has been performed for a few CT scans where AI’s prediction is uncertain. This fully annotated dataset will be released. Now that these 6 organs can be segmented by AI with a similar variance to human experts, we encourage the research community to concentrate on creating annotations for harder organs and tumors. Table 2. Benchmark on BTCV. For a fair comparison, we train and implement a single model evaluation without an ensemble for each method. All experiments are under the same data splits, computing resources, and testing conditions. Evaluatedon the BTCV validation set, Universal Model achieves the overall best performance, yielding at least +3.5% DSC improvement over the state-of-the-art method.

[0111] 4.3. Tumor Detection on Five Datasets

[0112] Tumor detection is the other critical need for medical AI model. To evaluate the capability of Universal Model in tumor detection, we conduct a comprehensive evaluation in five datasets for liver, kidney and pancreas tumor, as shown in Table 3. The results show Universal Model achieves highest specificity with 95.00%, 95.00% and 91.25% and maintain a compelling sensitivity with 88.89%, 91.67% and 93.98%. This reveals the effectiveness of assembly datasets, which consists of a sufficiently large number of positive examples and an even larger number of negative examples as control. Due to the long-tail distribution after data assembling, our model presents low false positive rate of tumor detection. Moreover, Fig. 14 presents examples of pancreatic tumors that are successfully detected by our model and the results of other five competitive baseline methods are unsatisfactory. Table 3. Tumor detection performance. The CT scans in LiTS, KiTS, and MSD Pancreas contain tumors in liver, kidney and pancreas, respectively. These scans are used to compute the sensitivity of tumor detection. To perform an alternative check of specificity, we use CHAOS and Pancreas-CT. It has been confirmed that CHAOS has no liver or kidney tumor, and Pancreas-CT has no pancreatic tumor in the CT scans. Universal Model achieves a higher specificity than most baseline models, while maintaining a compelling sensitivity. High specificity is clinically important because it reveals that Universal Model does not predict many false positives.

[0113] 5. Intriguing Properties

[0114] 5.1. Efficiency: FLOPs vs. DSC Scores

[0115] It is clinically important to make AI algorithms faster. To evaluate the computational efficiency of model, we utilize the Floating-point operations per second (FLOPs) to indicate the inference speed. In Fig. 15, we compare the speed- performance trade-off among Universal Model and 6 competitive methods. Universal Model is computationally more efficient compared with other dataset-specific models (at least 6× faster). Meanwhile, Universal Model achieves higher DSC score of 74.43% on average.

[0116] 5.2. Generalizability: External Datasets Results

[0117] A key expectation of reliable medical AI model is their generalizability, i.e., performance on new data across many hospitals, rather than the performance tailored to a single dataset. Compared with dataset-specific models, Universal Model is trained on an order of magnitude more diverse CT scans (at least from 14 hospitals), and therefore demonstrate significantly better generalizability. We conduct the evaluation on a public dataset 3D-IRCADb and a private dataset JHH, which are absolutely not used in training and can be regarded as the data from other hospitals. As shown in Table 4, Universal Model achieves highest performance in 3D-IRCADb and JHH with 91.62% and 84.54% DSC. For other recently proposed SOTA method, their performance is unsatisfactory in both datasets. The abundant results validate the generalizability of Universal Model. Table 4. Generalizability: Results on external datasets. We compare the performance of Universal Model and eight models on data from two external sources without additional fine-tuning or domain adaptation. Compared with dataset-specific models, our Universal Model performs more robustly to CT scans taken from a variety of different scanners, protocols, and institutes. More external (out-of- distribution) data used to evaluate the models can guarantee robust AI performance in real world conditions (e.g. clinical settings).

[0118] 5.3. Transferability: Fine-tuning Results

[0119] Another property of Universal Model is serving as a powerful pre-training model for segmentation. Through pre-training by assembly dataset directly and fine- tuning to other datasets, the Universal Model achieves highest DSC compared with other pre-training methods with 86.49%, 89.57%, 94.43% and 88.95% for four tasks in TotalSegmentator dataset, as shown in Table 5. Since the other unrelated pre- training tasks (reconstruction, colorization, jigsaw) can not capture the fine-grained information for segmentation. Table 5. Transferability: Fine-tuning performance. Fine-tuning Universal Model significantly outperforms learning from scratch on two downstream datasets (i.e. TotalSegmentator and JHH). Moreover, Universal Model, trained by image segmentation as proxy task, can extract better visual representation—more related to segmentation tasks—than other pre-trained models developed in the medical domain.

[0120] 5.4. Extensibility: Incremental Learning Results

[0121] Unlike existing models, introducing additional classes (CLIP embeddings) would not affect the segmentation model architecture, allowing the Universal Model to perform incremental learning. As a result, Universal Model is easily extended to upcoming datasets in the future with novel anatomical structures annotated. As a demonstration, our model can be easily adapted to segment the renalveins in the JHH dataset (which are absent in the current public dataset). To segment the left and right renal veins, two new CLIP embeddings for the left and right renal veins are concatenated to the original CLIP embeddings. Another text-driven segmentor for the new classes are also added to the model, and the segmentation output for the new classes are generated by this new segmentor. The other parameters of the model remain the same. In this way, the performance of the previously learned organs and tumors will not be affected. We perform continual fine- tuning to the Universal Model (pre-trained on 14 datasets) using the AdamW optimizer with learning rate 1e−4for 100 epochs. Our model achieved DSC 28.6% and 21.5% on the left and right renal veins, respectively.

[0122] 6. Conclusion

[0123] In this example, we present a CLIP-driven Universal Model for abdominal organ segmentation and tumor detection. We represent the first attempt to explore the potential of medical AI models in an assembly dataset, which consists of 3,672 CT scans with 25 partially annotated organs and 6 tumors. To solve the label inconsistency and orthogonality problem and build up an extensible framework, this example provides a universal model with CLIP embedding to enable a flexible and powerful segmentor, allowing the model to learn from the assembly of partially labeled datasets for high-performing organ segmentation and tumor detection (ranking first in both MSD and BTCV). More importantly, we have demonstrated that CLIP embedding can establish the anatomical relationship and enables the model to be extended to novel organs and tumors. Finally, the experiment results validate several clinically important merits of the CLIP-driven Universal Model, such as compelling efficiency, generalizability, transferability, and extensibility.

[0124] Some further aspects are defined in the following clauses:

[0125] Clause 1: A method of selecting data as part of a vision active learning process, the method comprising: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to- contrast data points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data pointsin the given cluster, thereby selecting the data as part of the vision active learning process.

[0126] Clause 2: The method of Clause 1, wherein the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

[0127] Clause 3: The method of Clause 1 or Clause 2, wherein the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

[0128] Clause 4: The method of any one of the preceding Clauses 1-3, wherein the medical image datasets comprise computed tomography (CT) images.

[0129] Clause 5: The method of any one of the preceding Clauses 1-4, comprising annotating the data selected as part of the vision active learning process to produce annotated data.

[0130] Clause 6: The method of any one of the preceding Clauses 1-5, comprising using the annotated data to train a machine learning model to produce a trained machine learning model.

[0131] Clause 7: The method of any one of the preceding Clauses 1-6, comprising passing a medical image dataset obtained from a test subject through the trained machine learning model and classifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

[0132] Clause 8: The method of any one of the preceding Clauses 1-7, wherein the disease state comprises a cancer type.

[0133] Clause 9: A system for selecting data as part of an active learning process, the system comprising: a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operations comprising: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to-contrast data pointin a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

[0134] Clause 10: The system of Clause 9, wherein the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

[0135] Clause 11: The system of Clause 9 or Clause 10, wherein the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

[0136] Clause 12: The system of any one of the preceding Clauses 9-11, wherein the medical image datasets comprise computed tomography (CT) images.

[0137] Clause 13: The system of any one of the preceding Clauses 9-12, wherein the instructions which, when executed on the processor, further perform operations comprising: annotating the data selected as part of the vision active learning process to produce annotated data.

[0138] Clause 14: The system of any one of the preceding Clauses 9-13, wherein the instructions which, when executed on the processor, further perform operations comprising: using the annotated data to train a machine learning model to produce a trained machine learning model.

[0139] Clause 15: The system of any one of the preceding Clauses 9-14, wherein the instructions which, when executed on the processor, further perform operations comprising: passing a medical image dataset obtained from a test subject through the trained machine learning model and classifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

[0140] Clause 16: The system of any one of the preceding Clauses 9-15, wherein the disease state comprises a cancer type.

[0141] Clause 17: A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrastdata points from dataset maps, wherein a hard-to-contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

[0142] Clause 18: The computer readable media of Clause 17, wherein the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

[0143] Clause 19: The computer readable media of Clause 17 or Clause 18, wherein the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

[0144] Clause 20: The computer readable media of any one of the preceding Clauses 17-19, wherein the medical image datasets comprise computed tomography (CT) images.

[0145] Clause 21: The computer readable media of any one of the preceding Clauses 17-20, wherein the instructions which, when executed on the processor, further perform operations comprising: annotating the data selected as part of the vision active learning process to produce annotated data.

[0146] Clause 22: The computer readable media of any one of the preceding Clauses 17-21, wherein the instructions which, when executed on the processor, further perform operations comprising: using the annotated data to train a machine learning model to produce a trained machine learning model.

[0147] Clause 23: The computer readable media of any one of the preceding Clauses 17-22, wherein the instructions which, when executed on the processor, further perform operations comprising: passing a medical image dataset obtained from a test subject through the trained machine learning model and classifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

[0148] Clause 24: The computer readable media of any one of the preceding Clauses 17-23, wherein the disease state comprises a cancer type.

[0149] Clause 25: A method of segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model, the method comprising: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP-driven universal model; constructing textprompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text- driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits, thereby segmenting the image using the CLIP-driven universal model.

[0150] Clause 26: The method of Clause 25, further comprising using back- propagation to supervise corresponding interested classes for each of the medical image datasets.

[0151] Clause 27: The method of Clause 25 or Clause 26, wherein the segmentation results comprise one or more organs.

[0152] Clause 28: The method of any one of the preceding Clauses 25-27, wherein the segmentation results comprise one or more tumors.

[0153] Clause 29: The method of any one of the preceding Clauses 25-28, wherein the image is obtained from a test subject.

[0154] Clause 30: The method of any one of the preceding Clauses 25-29, wherein the image comprises a computed tomography (CT) image.

[0155] Clause 31: The method of any one of the preceding Clauses 25-30, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

[0156] Clause 32: The method of any one of the preceding Clauses 25-31, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than a dataset-specific model.

[0157] Clause 33: The method of any one of the preceding Clauses 25-32, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

[0158] Clause 34: A system for segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model, the system comprising: a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operationscomprising: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text- driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits.

[0159] Clause 35: The system of Clause 34, wherein the segmentation results comprise one or more organs.

[0160] Clause 36: The system of Clause 34 or Clause 35, wherein the segmentation results comprise one or more tumors.

[0161] Clause 37: The system of any one of the preceding Clauses 34-36, wherein the image is obtained from a test subject.

[0162] Clause 38: The system of any one of the preceding Clauses 34-37, wherein the image comprises a computed tomography (CT) image.

[0163] Clause 39: The system of any one of the preceding Clauses 34-38, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

[0164] Clause 40: The system of any one of the preceding Clauses 34-39, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than a dataset-specific model.

[0165] Clause 41: The system of any one of the preceding Clauses 34-40, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

[0166] Clause 42: A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into a contrastive language-image pre-training (CLIP)-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universalmodel; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text- driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits.

[0167] Clause 43: The computer readable media of Clause 42, wherein the segmentation results comprise one or more organs.

[0168] Clause 44: The computer readable media of Clause 42 or Clause 43, wherein the segmentation results comprise one or more tumors.

[0169] Clause 45: The computer readable media of any one of the preceding Clauses 42-44, wherein the image is obtained from a test subject.

[0170] Clause 46: The computer readable media of any one of the preceding Clauses 42-45, wherein the image comprises a computed tomography (CT) image.

[0171] Clause 47: The computer readable media of any one of the preceding Clauses 42-46, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

[0172] Clause 48: The computer readable media of any one of the preceding Clauses 42-47, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than a dataset-specific model.

[0173] Clause 49: The computer readable media of any one of the preceding Clauses 42-48, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

[0174] While the invention has been described with reference to the exemplary embodiments thereof, those skilled in the art will be able to make various modifications to the described embodiments without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and theirequivalents. All patents, patent applications, other publications or documents, and the like cited herein are incorporated by reference in their entirety for all purposes to the same extent as if each individual item were specifically and individually indicated to be so incorporated by reference.

Claims

What is claimed is:

1. A method of selecting data as part of a vision active learning process, the method comprising: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to- contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster, thereby selecting the data as part of the vision active learning process.

2. The method of claim 1, wherein the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

3. The method of claim 1, wherein the selecting step comprises an intra- class criterion that avoids outliers in the medical image datasets.

4. The method of claim 1, wherein the medical image datasets comprise computed tomography (CT) images.

5. The method of claim 1, comprising annotating the data selected as part of the vision active learning process to produce annotated data.

6. The method of claim 5, comprising using the annotated data to train a machine learning model to produce a trained machine learning model.

7. The method of claim 6, comprising passing a medical image dataset obtained from a test subject through the trained machine learning model andclassifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

8. The method of claim 7, wherein the disease state comprises a cancer type.

9. A system for selecting data as part of an active learning process, the system comprising: a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operations comprising: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to- contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

10. The system of claim 9, wherein the extracting and assigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

11. The system of claim 9, wherein the selecting step comprises an intra- class criterion that avoids outliers in the medical image datasets.

12. The system of claim 9, wherein the medical image datasets comprise computed tomography (CT) images.

13. The system of claim 9, wherein the instructions which, when executedon the processor, further perform operations comprising: annotating the data selected as part of the vision active learning process to produce annotated data.

14. The system of claim 13, wherein the instructions which, when executed on the processor, further perform operations comprising: using the annotated data to train a machine learning model to produce a trained machine learning model.

15. The system of claim 14, wherein the instructions which, when executed on the processor, further perform operations comprising: passing a medical image dataset obtained from a test subject through the trained machine learning model and classifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

16. The system of claim 15, wherein the disease state comprises a cancer type.

17. A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: extracting features from different classes of medical image datasets using a self-supervised contrastive learning technique to produce a set of extracted image features; assigning clusters to achieve label diversity using the set of extracted image features to produce a set of clusters; and, selecting hard-to-contrast data points from dataset maps, wherein a hard-to- contrast data point in a given cluster from the set of clusters is harder to contrast itself with other data points in the given cluster.

18. The computer readable media of claim 17, wherein the extracting andassigning steps comprise an inter-class criterion that enforces the label diversity to mitigate bias in the medical image datasets.

19. The computer readable media of claim 17, wherein the selecting step comprises an intra-class criterion that avoids outliers in the medical image datasets.

20. The computer readable media of claim 17, wherein the medical image datasets comprise computed tomography (CT) images.

21. The computer readable media of claim 17, wherein the instructions which, when executed on the processor, further perform operations comprising: annotating the data selected as part of the vision active learning process to produce annotated data.

22. The computer readable media of claim 21, wherein the instructions which, when executed on the processor, further perform operations comprising: using the annotated data to train a machine learning model to produce a trained machine learning model.

23. The computer readable media of claim 22, wherein the instructions which, when executed on the processor, further perform operations comprising: passing a medical image dataset obtained from a test subject through the trained machine learning model and classifying one or more regions in the medical image dataset obtained from the test subject as being indicative of a disease state or as not being indicative of the disease state.

24. The computer readable media of claim 23, wherein the disease state comprises a cancer type.

25. A method of segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model, the method comprising: inputting at least one image from one or more medical image datasets with acorresponding subdictionary into the CLIP-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits, thereby segmenting the image using the CLIP-driven universal model.

26. The method of claim 25, further comprising using back-propagation to supervise corresponding interested classes for each of the medical image datasets.

27. The method of claim 21, wherein the segmentation results comprise one or more organs.

28. The method of claim 25, wherein the segmentation results comprise one or more tumors.

29. The method of claim 25, wherein the image is obtained from a test subject.

30. The method of claim 25, wherein the image comprises a computed tomography (CT) image.

31. The method of claim 25, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

32. The method of claim 25, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than adataset-specific model.

33. The method of claim 25, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

34. A system for segmenting an image using a contrastive language-image pre-training (CLIP)-driven universal model, the system comprising: a processor; and a memory communicatively coupled to the processor, the memory storing instructions which, when executed on the processor, perform operations comprising: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into the CLIP-driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits.

35. The system of claim 34, wherein the segmentation results comprise one or more organs.

36. The system of claim 34, wherein the segmentation results comprise one or more tumors.

37. The system of claim 34, wherein the image is obtained from a test subject.

38. The system of claim 34, wherein the image comprises a computedtomography (CT) image.

39. The system of claim 34, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

40. The system of claim 34, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than a dataset-specific model.

41. The system of claim 34, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

42. A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: inputting at least one image from one or more medical image datasets with a corresponding subdictionary into a contrastive language-image pre-training (CLIP)- driven universal model; constructing text prompts based on a prompt template using the subdictionary in a text branch of the CLIP-driven universal model; feeding the text prompts into CLIP pre-trained text encoder of the CLIP-driven universal model to generate a CLIP embedding; generating a parameter for a text-driven segmentor using a text-based controller of the CLIP-driven universal model; feeding the image into a vision encoder and the text-driven segmentor of a vision branch of the CLIP-driven universal model to obtain visual logits; and, generating segmentation results using the obtained visual logits.

43. The system of claim 42, wherein the segmentation results comprise one or more organs.

44. The system of claim 42, wherein the segmentation results compriseone or more tumors.

45. The system of claim 42, wherein the image is obtained from a test subject.

46. The system of claim 42, wherein the image comprises a computed tomography (CT) image.

47. The system of claim 42, wherein the CLIP-driven universal model is computationally more efficient than a dataset-specific model.

48. The system of claim 42, wherein the CLIP-driven universal model generalizes better to computed tomography (CT) scans from different sources than a dataset-specific model.

49. The system of claim 42, wherein the CLIP-driven universal model has better transfer learning performance than a dataset-specific model.

Images