Dual-modality models for digital pathology

Abstract

Techniques for using combination stain types for machine learning models for digital pathology are described herein. In an example, a system accesses a first image of a sample comprising an immunohistochemistry (IHC) stain for a biomarker. The system accesses a second image of the sample comprising a hematoxylin and eosin (H&E) stain for nuclei. The system can segment tissue regions in the one or more first images and the second image, partitions the tissue regions in the one or more first images and the second image, and extracts features from the set of tiles using a feature extractor. The system can generate, by a machine-learning model, an output classification indicating a first phenotype based on the features extracted from the set of tiles. The machine-learning model can include one or more classifiers and an aggregation model that provides an aggregated output for the set of tiles.

Description

DUAL-MODALITY MODELS FOR DIGITAL PATHOLOGYCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This non-provisional application claims priority to and the benefit of U.S. Provisional Application No. 63 / 657,541, filed on June 7, 2024, and titled “COMBINATION OF STAIN TYPES IN MACHINE LEARNING MODELS FOR DIGITAL PATHOLOGY,” the content of which is herein incorporated in its entirety for all purposes.BACKGROUND

[0002] Digital pathology comprises converting traditional glass slides with tissue samples into high-resolution digital images that can be viewed, analyzed, and managed on computer systems. Pathology slides are digitized using specialized scanners, thereby enabling the use of software to view, annotate, and analyze the digital images. This technology can be used to apply artificial intelligence (Al) and machine learning algorithms for image analysis. In cancer care, digital pathology facilitates quantitative analysis of tumor characteristics.

[0003] Recent advancements in immunotherapy, notably the application of pembrolizumab — an anti-PD-1 antibody — have demonstrated efficacy in treating various cancer ty pes, including microsatellite instability-high (MSI-H) / mismatch repair deficient (MMRd) metastatic CRC and PD-L1 -positive TNBC, significantly prolonging progression-free and overall survival1,2

[0004] The conventional detection of MSI often relies on polymerase chain reaction (PCR) based methods3although next generation sequencing (NGS)4is increasingly used for comprehensive profiling, and MMRd and PD-L1 are routinely assessed by immunohistochemistry (IHC)5. IHC, while extensive in its application, remains labor- intensive, requiring significant time investment from highly trained pathologists for interpretation. The process is not only time-consuming but also subject to inter-observer variability, which can affect diagnostic consistency and accuracy6Similarly, NGS offers a high-resolution view of the genetic landscape of tumors but also demands complicated workflows and expert interpretation7

[0005] Therefore, improvements in digital pathology are desirable.SUMMARY

[0006] Embodiments provided herein involve accessing one or more first images of a biological sample from a subject having an immunohistochemistry (IHC) stain for a respective biomarker and a second image of the biological sample comprising a hematoxylin and eosin (H&E) stain. Features extracted from the images can be processed using a machine-learning model that includes one or more classifiers and an aggregation model to generate an output classification indicating a phenotype of the subject.

[0007] In some embodiments, a method can involve accessing one or more first images of a biological sample from a subject comprising IHC stain for a respective biomarker. The method can further involve accessing a second image of the biological sample comprising a hematoxylin and eosin (H&E) stain. The method can also involve segmenting tissue regions in the one or more first images and the second image, partitioning the tissue regions in the one or more first images and the second image, thereby generating a set of tiles, and extracting features from the set of tiles using a feature extractor trained on at least some unlabeled H&E and / or at least some IHC images. The method can also involve generating, by a machine-learning model, an output classification indicating a first phenotype based on the features extracted from the set of tiles. The machine-learning model can include one or more classifiers and an aggregation model that provides an aggregated output for the set of tiles.

[0008] These and other embodiments of the disclosure are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.

[0009] A better understanding of the nature and advantages of embodiments of the present disclosure may be gained with reference to the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0010] FIGS. 1A-1F illustrate an overview of machine learning in digital pathology. FIG. 1A: Tissue acquisition, segmentation, and patching. FIG. IB: Patch-level supervision. FIG. 1C: Multi-instance learning model (MIL). FIG. ID: Local context MIL. FIG. IE: Global context MIL. FIG. IF: Transformer-based model.

[0011] FIGS. 2A-2B illustrate an overview of data pre-processing. FIG. 2A: The preprocessing pipeline initiates with the digitization of whole slide images (WSIs), followed bytissue segmentation and tessellation of the WSIs into patches for analysis. FIG. 2B: Illustration of the model architecture showcasing the pre-trained feature extractor, CTransPath, which processes the initial input data.

[0012] FIGS. 3A-3C illustrate examples of model architectures. FIG. 3A: The transformerbased feature aggregation module where both hematoxylin & eosin (H&E) and immunohistochemistry (IHC) features are included in a single input. FIG. 3B: The transformer-based feature aggregation module where H&E and IHC features are processed by separate projection layers. FIG. 3C: The transformer-based feature aggregation module where H&E and IHC features are processed by a multi-branch model with shared parameters.

[0013] FIGS. 4A-4E illustrate an assessment of biomarker prediction performance for microsatellite instability and mismatch repair deficiency in colorectal cancer. FIG. 4A: Aggregated Area Under the Receiver Operating Characteristic (AUROC) scores for predictions of microsatellite instability -high (MSI-H) and mismatch repair deficiency (MMRd) across three model types: H&E only, IHC only, H&E / IHC dual -modality. Each value is the mean from a 5-fold cross-validation ± 95% confidence interval (95% CI). FIG. 4B: AUROC curves for MMRd and MSI prediction performance of the H&E / IHC dualmodality model. FIG. 4C: Stratified analysis of MMRd and MSI predictions, differentiated by specimen site and scanner type. FIG. 4D: Cost-benefit analysis of dual-modality model predictions. The X-axis indicates the False Negative Percentage (False Negatives / Total Cases). The left Y-axis represents the True Negative Percentage (True Negatives / Total Cases), and the right Y-axis denotes Sensitivity. FIG. 4E: Visualization of attention and classification scores for deficient MMR specimens. The attention heatmap illustrates the perpatch attention rollout of our trained transformer-based feature aggregation duet model, with larger values (yellow) indicating higher contribution to the model’s prediction and smaller values (purple) indicating lower contribution. The classification heatmap displays the perpatch MMRd classification scores, with deficient MMR as the positive class and proficient MMR as the negative class. The attention x classification heatmap highlighted tiles that provide final weighted classification score.

[0014] FIGS. 5A-5D illustrate an evaluation of biomarker prediction for PD-L1 status in breast cancer. FIG. 5 A: AUROC curves for the prediction of PD-L1 status (where CPS > 10 denotes PD-L1 positivity), comparing the performance of the hematoxylin & eosin (H&E) only, immunohistochemistry (IHC) only, and H&E / IHC duet models. FIG. 5B: Stratifiedanalysis of PD-L1 status predictions by specimen site and scanner type, elucidating the model’s performance across different conditions. FIG. 5C: Cost-benefit analysis of the dualmodality model’s predictions, with the False Negative Percentage (False Negatives / Total Cases) on the X-axis, the True Negative Percentage (True Negatives / Total Cases) on the left Y-axis, and Sensitivity on the right Y-axis. FIG. 5D: Visualization of attention and classification scores for PD-L1 -positive specimens. The attention heatmap conveys the perpatch significance through our trained transformer-based feature aggregation duet model, where yellow indicates a high contribution and purple a low contribution to the model’s output. The classification heatmap portrays the per-patch PD-L1 classification scores. The attention x classification heatmap highlighted tiles that provide final weighted classification score.

[0015] FIGS. 6A-6H illustrate a comparative survival analysis by evaluating the impact of ground truth and predicted MMRd status in CRC patients. FIGS. 6A-D show Kaplan-Meier survival curves for patients categorized by mismatch repair deficiency (MMRd) status, determined through pathological assessment (FIG. 6A), prediction from hematoxylin & eosin (H&E WSI) whole slide images (WSIs) (FIG. 6B), prediction from immunohistochemistry (IHC) WSI (FIG. 6C), and prediction from a combined H&E / IHC WSI approach (FIG. 6D). The hazard ratio (HR) for the MMRd group is provided, with the mismatch repair proficient (MMRp) group serving as the reference. The shaded regions indicate 95% confidence intervals. (CI). The p-values were derived using the log-rank test to compare each MMRd group with the respective MMRp group.

[0016] FIGS. 6E-6H show Kaplan-Meier survival curves for patients classified by MSI status, determined through pathological assessment (FIG. 6E), prediction from H&E WSI (FIG. 6F), prediction from IHC WSI (FIG. 6G), and prediction from a combined H&E / IHC WSI method (FIG. 6H). The HR for the MSI group is provided, using the microsatellite stable (MSS) group as the reference. Shaded areas delineate 95% confidence intervals, p- values were calculated using the log-rank test to contrast each MSI group with the corresponding MSS group.

[0017] FIGS. 7A-7D illustrate a comparative survival analysis by evaluating the influence of actual and predicted programmed death-ligand 1 status in breast cancer patients. The figures show Kaplan-Meier curves comparing patient groups classified by PD-L1 status, which is determined through different methods: pathological evaluation (FIG. 7A), predictionusing hematoxylin & eosin (H&E WSI) whole slide images (WSIs) (FIG. 7B), prediction using immunohistochemistry (IHC) WSI (FIG. 7C), and prediction using a combined H&E / IHC WSI model (FIG. 7D). The hazard ratio (HR) is reported for the PD-L1 -positive group with the PD-L1 -negative group as the baseline for comparison. Confidence intervals (CI) at 95% are shown as shaded regions around the curves. P-values were computed by employing the log-rank test to compare the survival rates of each PD-L1 -positive group against their PD-L1 -negative counterparts.

[0018] FIG. 8 illustrates performances of a single input model architecture.

[0019] FIG. 9 illustrates performances of a separate projection layer per input model architecture.

[0020] FIG. 10 illustrates performances of a multi -branch with shared parameters model architecture.

[0021] FIGS. 11A-11B illustrate additional exemplary combinations of multiple immunohistochemistry images that could be combined with a single hematoxylin & eosin image according to the dual-modality WSI model provided herein.

[0022] FIG. 12 illustrates an example of a flow of a process for using a dual-modality model in digital pathology , according to embodiments of the present disclosure.

[0023] FIG. 13 illustrates a measurement system according to an embodiment of the present disclosure.

[0024] FIG. 14 shows a block diagram of an example computer system usable with systems and methods according to embodiments of the present disclosure.TERMS

[0025] A "'machine learning moder (ML model) can refer to a software module configured to be run on one or more processors to provide a classification or numerical value of a property of one or more samples. An ML model can include various parameters (e.g., for coefficients, weights, thresholds, functional properties of function, such as activation functions). As examples, an ML model can include at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or one million parameters. An ML model can be generated using sample data (e.g., training samples) to make predictions on test data. Various number of training samples can be used, e.g., at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or at least200,000 training samples. One example is an unsupervised learning model. Another example type of model is supervised learning that can be used with embodiments of the present disclosure. Example supervised learning models may include different approaches and algorithms including analytical learning, statistical models, artificial neural network (e.g. including convolutional and / or transformer layers), boosting (meta-algorithm), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), random forests, ensembles of classifiers, ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn, a multicnteria classification algorithm. The model may include linear regression, logistic regression, deep recurrent neural network (e.g., long short term memory, LSTM), hidden Markov model (HMM), linear discriminant analysis (LDA), k-means clustering, density-based spatial clustering of applications with noise (DBSCAN), random forest algorithm, support vector machine (SVM), or any model described herein. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques.

[0026] The term “phenotype” as used herein can mean any trait or characteristic that can be identified in part or in whole by using the systems and / or methods provided herein. In some embodiments, the sy stems can include one or more computer programs on one or more computers in one or more locations, e.g., configured for use in a method described herein. Phenotypes may be determined by analyzing a biological sample obtained from a subject. Phenotypes to be characterized can be any phenotype of interest, including without limitation a tissue, anatomical origin, medical condition, ailment, disease, disorder, or useful combinations thereof. A phenotype can be any observable characteristic or trait of, such as a disease or disorder, a stage of a disease or disorder, susceptibility' to a disease or disorder, prognosis of a disease stage or disorder, a physiological state, or response / potential response(or lack thereof) to interventions such as therapeutics. A phenotype can result from a subject’s genetic makeup as well as the influence of environmental factors and the interactions between the two, as well as from epigenetic modifications to nucleic acid sequences. In various embodiments, a phenotype in a subject is characterized by obtaining a biological sample from a subject and analyzing the sample using the systems and / or methods provided herein. For example, characterizing a phenotype for a subject or individual can include detecting a disease or disorder (including pre-symptomatic early -stage detection), determining a prognosis, diagnosis, or theranosis of a disease or disorder, or determining the stage or progression of a disease or disorder. Characterizing a phenotype can include identifying appropriate treatments or treatment efficacy for specific diseases, conditions, disease stages and condition stages, predictions and likelihood analysis of disease progression, particularly disease recurrence, metastatic spread or disease relapse. A phenotype can also be a clinically distinct type or subty pe of a condition or disease, such as a cancer or tumor. Phenotype determination can also be a determination of a physiological condition, or an assessment of organ distress or organ rejection, such as post-transplantation. The compositions and methods described herein allow assessment of a subject on an individual basis, which can provide benefits of more efficient and economical decisions in treatment.

[0027] A medical condition as used herein can refer to a disease or disorder of a subject, as well as a stage or severity of the medical condition.

[0028] A subject (individual, patient, or the like) can be any animal which may benefit from the methods described herein. A subject or individual can be any animal which may benefit from the methods described herein, including, e.g., humans and non-human mammals, such as primates, rodents, horses, dogs and cats. Subjects include without limitation a eukaryotic organisms, most preferably a mammal such as a primate, e.g., chimpanzee or human, cow; dog; cat; a rodent, e.g., guinea pig, rat, mouse; rabbit; or a bird; reptile; or fish. Subjects specifically intended for treatment using the methods described herein include humans. A subject may also be referred to herein as an individual or a patient. The subject can have a pre-existing disease or disorder, including without limitation cancer. Alternatively, the subject may not have any known pre-existing condition. The subject may also be non-responsive to an existing or past treatment, such as a treatment for cancer.

[0029] Theranostics as used herein includes therapy-related diagnostic testing that provides the ability to affect therapy or treatment of a medical condition such as a disease or disease state. Theranostics testing provides a theranosis in a similar manner that diagnostics or prognostic testing provides a diagnosis or prognosis, respectively. As used herein, theranostics encompasses any desired form of therapy related testing, including predictive medicine, personalized medicine, precision medicine, integrated medicine, pharmacodiagnostics and Dx / Rx partnering. Therapy related tests can be used to predict and assess drug response in individual subjects, thereby providing personalized medical recommendations. Predicting a likelihood of response can be determining whether a subject is a likely responder or a likely non-responder to a candidate therapeutic agent, e.g., before the subject has been exposed or otherwise treated with the treatment. Assessing a therapeutic response can be monitoring a response to a treatment, e.g., monitoring the subject’s improvement or lack thereof over a time course after initiating the treatment. Therapy related tests are useful to select a subject for treatment who is particularly likely to benefit or lack benefit from the treatment or to provide an early and objective indication of treatment efficacy in an individual subject. Characterization using the systems and methods provided herein may indicate that treatment should be altered to select a more promising treatment, thereby avoiding the expense of delaying beneficial treatment and avoiding the financial and morbidity costs of less efficacious or ineffective treatment(s).

[0030] Theranosis can comprise predicting a treatment efficacy or lack thereof, classifying a patient as a responder or non-responder to treatment. A predicted “responder” can refer to a patient likely to receive a benefit from a treatment whereas a predicted “non-responder” can be a patient unlikely to receive a benefit from the treatment. Unless specified otherwise, a benefit can be any clinical benefit of interest, including without limitation cure in whole or in part, remission, or any improvement, reduction or decline in progression of the condition or symptoms. The theranosis can be directed to any appropriate treatment, e.g., the treatment may comprise at least one of chemotherapy, immunotherapy, targeted cancer therapy, a monoclonal antibody, small molecule, surgery, radiation, or any useful combinations thereof.

[0031] A classification of a medical condition can include a diagnosis, prognosis, or theranosis of the subject. The term “c / a yzcofio / r” as used herein refers to any number(s) or other characters(s) that are associated with a particular property of a sample, e.g., a medicinal condition of a subject from whom the sample was obtained. For example, a “+” symbol (or the word “positive”) could signify that a sample is classified as having deletions oramplifications. The classification can be binary (e.g., positive or negative) or have more levels of classification (e.g., a scale from 1 to 10 or 0 to 1), including probabilities.

[0032] A ‘‘biomarker” or “marker” can be any physical characteristic of the human body that can be measured. As non-limiting examples, blood pressure and hemoglobin levels are biomarkers. In oncology , common biomarkers include the DNA, RNA and protein originating from a tumor or tumor microenvironment. Such biomarkers can be used to characterize a medical condition, including without limitation making a diagnosis, predicting cancer aggressiveness or predicting that a particular therapy will be effective. Biomarkers or sets of biomarkers can be used to train and test machine learning models and classify samples. Particular biomarkers may be used, such as particular nucleic acids (e.g., DNA or RNA, such as mRNA and microRNA), proteins, cell types (e.g., different T-cell types), lipids, carbohydrates and metabolites and optionally also include a state of such molecules. Examples of the state of a biomarker include various aspects that can be queried such as presence, level (quantity, concentration, etc., such as an expression level), sequence, location, activity, structure, modifications, covalent or non-covalent binding partners, and the like. As non-limiting examples, a biomarker can be an individual gene or gene product, such as the CD274 gene or its gene product Programmed Cell Death 1 Ligand 1 (PD-L1), and also biomarker / genomic signatures, e.g., MSI, tumor mutational burden (TMB), loss of heterozygosity (gLOH), homologous recombination deficiency (HRD), and / or human leukocyte antigen (HLA) genotyping. Various numbers of biomarkers can be used to generate an input data structure (also referred to as an input feature vector) in machine learning applications, e.g., at least 10, 20, 50, 100, 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, or 100,000 biomarkers can be used.

[0033] An “actionable biomarker” can refer to a biomarker that can be targeted directly or indirectly by treatments to either prevent or treat medical conditions. Thus, an actionable biomarker can be used to direct patient management. As a non-limiting example, an actionable biomarker may drive or define a cancer and can be targeted by chemotherapies and / or biologic agents. As another non-limiting example, an actionable biomarker may carry a mutation that indicates treatment with a therapy targeted to a related biomarker, such as a biomarker in a related biological pathway. As still another non-limiting example, an actionable biomarker can be a biomarker involved in an immune response to a cancer, such as checkpoint inhibitor therapies. Cancer biomarkers are molecules or indicators (e.g., proteins,genes, gene changes, etc.) that can be detected in biological samples and that provide insights into the presence, progression, or response to treatment of cancer.DETAILED DESCRIPTION

[0034] This disclosure provides an efficient dual-modality transformer22-based artificial intelligence (Al) framework that integrates both hematoxylin & eosin (H&E) and immunohistochemistry (IHC) stained images for enhanced prediction of biomarkers. The term “biomarker” is used generally herein to refer to individual genes or gene products, e.g., the CD274 gene or its gene product Programmed Cell Death 1 Ligand 1 (PD-L1), and also biomarker / genomic signatures, e.g., MSI, tumor mutational burden (TMB), loss of heterozygosity (gLOH), homologous recombination deficiency (HRD), and / or human leukocyte antigen (HLA) genotyping. In an exemplary embodiment presented herein, the dual-modality approach exhibited prognostic stratification that was comparable to, or even surpassed, that of the actual biomarker status detected through conventional methods. The system provides a flexible and comprehensive framework that can adapt to the available data types, e.g., H&E, IHC, or both. It also features the capability to customize predictive thresholds to fit a range of clinical scenarios. The model has strong potential for clinical application, promising to aid decision-making in oncology and to contribute to more personalized patient care. With its robust performance and adaptability, the model may be an invaluable resource in clinical settings, enhancing patient outcomes through precise, biomarker-driven therapies.

[0035] In some embodiments, instead of using only H&E images as input to a model to predict biomarker labels, H&E image(s) and IHC image(s) can be used. Such images may be combined by concatenating the whole-slide images or tiles therefrom, or no concatenation may be performed. Additionally or alternatively features from both types of images (e.g., extracted from tiles of both types of images) can be concatenated, which may occur before or after aggregation. In addition, the H&E images for a sample with known state for a first biomarker (e.g., MSI) can be combined with an image generated from a sample stained for a second biomarker (e.g., MMR) to develop a model that improves prediction of the first biomarker when there is a correlated relationship between these two biomarkers.

[0036] A tissue mask generating pipeline can be used to generate a tissue mask (or other mechanism), which can segment the tissues of both the H&E image and the IHC image or from a concatenated image. The identified tissue regions can be partitioned into tiles fromwhich features are extracted. A model can receive the extracted features (possibly concatenated or separate features for each tile or image) or feature embeddings of the images (also possibly concatenated or separate features for each tile or image) and predict one or more biomarker labels for the images. This label may be more accurate and more consistent than predictions based solely on H&E images or IHC images.

[0037] A feature extractor that is used to generate embeddings can be applied for each IHC image and each H&E image. The feature extractor may be trained on H&E images but may be used to extract features from both IHC images and H&E images. Alternatively, a feature extractor may be developed that is trained using IHC images to further the model’s performance. The feature extractor may generate embeddings for each tile of the IHC image and the H&E image, where each tile is associated with an (x,y) coordinate in their respective whole-slide image. As an example, to concatenate the images, the H&E image tiles can be represented as their (x,y) coordinates, and the whole-slide image width of the H&E image can be added to the x-coordinate for each of the IHC image tiles. In this way, the images can be concatenated in the x direction.

[0038] In some instances, IHC images for multiple biomarkers can be combined with an H&E image. As a non-limiting example, consider mismatch repair (MMR). There are four mismatch repair proteins that are typically assessed as biomarkers for MMRd using IHC: MLH1, PMS2, MSH2 and MSH6. Deficiencies in any of these proteins can result in accumulation of mismatch defects and MMRd, Microsatellites are particularly susceptible to MMRd errors, thus MMRd can lead to MSI and the terms are sometimes used interchangeably. Continuing with this example, an IHC image can be generated for 1, 2, 3 or 4 of these proteins. See FIG. 11 A. The H&E image can then be concatenated with some or all of the IHC images. In some embodiments, the concatenation is used by the model to predict the MMRd label.

[0039] As a second non-limiting example, breast cancer is routinely assessed using IHC for the proteins ER, PR, HER2 and Ki67. Growth of cancer cells with estrogen receptors (ER+) and progesterone receptors (PR+) are promoted by the hormones estrogen and progesterone, respectively. Hormone therapy is used to treat ER+ and / or PR+ cancers. HER2 also promotes cancer cell growth. Targeted therapies such as the anti-HER2 monoclonal antibodies are used to treat HER2+ cancers. Triple negative breast cancers (i.e., ER-, PR-, HER2-) are aggressive and lack targeted therapies, thus requiring chemotherapy. Unlike hormone receptors (ER / PR)or HER2, Ki67 specifically measures cell proliferation rates. Ki67 is used to identify patients who might benefit from more aggressive treatment approaches, an IHC image can be generated for 1, 2, 3 or 4 of these proteins. See FIG. 1 IB. The H&E image can then be concatenated with some or all of the IHC images. In some embodiments, the output of the model is used to aid in treatment planning.I. EXAMPLE ML TECHNIQUES FOR DIGITAL PATHOLOGY

[0040] Machine learning models can be used in digital pathology to provide predictions related to biological samples. For example, a biological sample from a subject can be stained for one or more biomarkers. Images of the biological sample can be generated and segmented. Features can be extracted from the images, and the extracted features can be input into a machine learning model that can generate an output classification indicating a phenotype based on the features.A. Generation of patches (tiles) and whole slide classifications

[0041] FIG. 1 A illustrates an example of tissue acquisition, segmentation, and patching according to embodiments of the present disclosure. To generate the predictions and classifications, an image of a biological sample of a subject can be received. The biological sample can be imaged with a pathology stain to provide visual contrast to the sample and / or highlight certain features, thereby facilitating histologic image analysis. In some implementations, the biological sample can be imaged, e.g., using radiological instruments. The biological sample may be a tissue sample, including without limitation a tumor sample. In some implementations, the tissue sample is fixed on a glass slide 101 to preserve structure. Fixation can be performed using chemical fixatives such as crosslinking agents.Formaldehyde is the most common fixative used in histology applications. The tissue can also be embedded into a medium to allow sectioning. Examples of such mediums include epoxy, acrylic, agar, gelatin, celloidin, and waxes. Paraffin wax is the most common embedding material in light microscopy. In preferred embodiments, the biological sample is a formalin-fixed paraffin-embedded (FFPE) tissue. The FFPE tissue can be sectioned into slides that can be stained prior to image analysis.

[0042] The pathology stain can be any desired stain, dye or other imaging agent, or any useful combination thereof. In some implementations, the stain comprises a nuclear stain, cytoplasmic stain, and / or stain for the extracellular matrix. The most commonly used stain forhistopathology is a combination of hematoxylin and eosin (“H&E”). Generally, hematoxylin stains the cell nuclei whereas eosin stains the cytoplasm and extracellular matrix. Such staining shows the general layout and distribution of cells and general overview of the tissue sample’s structure, which can be used to provide histological information. The stain may alternatively be an immunohistochemistry (IHC) for a biomarker. IHC stains antibodies to detect specific antigens in cells or tissues. Other stains are also possible. For review of tissue imaging including alternative imaging agents, see, e.g., Alturkistani, H et al, Histological Stains: A Literature Review and Case Study, Glob J Health Sci. 2015 Jun 25;8(3):72-79; Javaeed, A et al., Histological Stains in the Past, Present, and Future, Cureus. 2021 Oct 4; 13(10): el 8486; Veuthey, T, et al, Dyes and stains: from molecular structure to histological application, Front Biosci (Landmark Ed) 2014 Jan 1 ; 19(1 ):91 -112. See also Biological Stain Commission, available at biologicalstaincommission.org.

[0043] The glass slide 101 can be imaged using a slide scanner 102 to generate a wholeslide image 103. The slide scanner 102 may be a Philips scanner, a Leica scanner, or any other suitable slide scanner. As illustrated, the whole-slide image 103 is 100,000 pixels x 150,000 pixels, but other sized of images are possible.

[0044] The whole-slide image 103 can be segmented and patched to generate tiles 104. For segmenting, a tissue mask can be applied to the whole-slide image 103 to identify regions of interest. The regions of interest may be regions of the whole-slide image 103 that depict tissue. In this manner, the regions of interest (tissue regions) can be separated from portions that just include background.

[0045] Tiles 104 can then be generated from the segmented regions. For the whole-slide image 103, tiles 104 can be generated that correspond to portions of the whole-slide image 103. As such, each tile can include a portion of the pixels of the whole-slide image 103. The dimensions of the tiles 104 can vary depending on the architecture of the machine learning system. The system can consider all tiles or tiles from certain regions of interest, such as from tumor tissue. The tiles 104 can then be processed to extract features and make predictions based on the features.

[0046] Each tile is a collection of pixels corresponding to a portion of a whole-slide image. In some embodiments, a tile (also referred to as a patch) is a region of a whole-slide image or an area of interest having (x,y) pixel dimensions (e.g., 256 pixels by 256 pixels). For example, a whole-slide image of 1000 pixels by 1000 pixels divided into 100 pixel by 100pixel tiles would segmented into 100 tiles (each tile containing 10,000 pixels). In other embodiments, the tiles 104 may overlap with each patch having (x,y) pixel dimensions and sharing one or more pixels with another tile.

[0047] For the whole-slide image 103, tiles 104 are generated corresponding to portions of the whole-slide image 103, and a feature vector can be generated for each tile. To determine an output classification, aggregated embeddings representing aggregations of the feature vectors can be used. The aggregation can result in an aggregated output for the whole-slide image 103. The feature vectors can be aggregated using various techniques to generate the embeddings.

[0048] FIGS. 1B-1E illustrate various aggregation techniques for feature vectors, according to embodiments of the present disclosure. FIG. IB illustrates a patch-level supervision technique in which the tiles 104, which can also be referred to as patches, are processed by a feature extractor 105 to generate feature vectors for each tile. The feature vectors can correspond to embedding 106, where there is an embedding for each tile. As examples, the feature extractor 105 may include one or more of a Swin Transformer, CNN, hybrid, CNN- transformer, self-supervised or contrastive learning-based model, transformer pretrained on histopathology data, or foundation models such as Virchow, UNI, CTransPath, CONCH, TITAN, THREADS, TANGLE, CHIEF, PRISM. A predictor 107, which can be a classifier, may be trained to predict an output classification corresponding to a phenoty pe for each tile based on feature vectors. These output classifications correspond to patch predictions 108. The patch predictions 108 may also be referred to as tile classifications. Different tiles may be associated with different patch predictions.

[0049] An aggregator 109 can then determine a whole-slide image prediction 110 based on the patch predictions 108. The aggregator 109 can be an aggregator model that aggregates the patch predictions 108 to obtain the whole-slide image prediction 110. For instance, the whole-slide image prediction 110 can correspond to a maximum output classification of the patch predictions 108, a majority output classification of the patch predictions 108, an average output classification of the patch predictions 108, etc. The whole-slide image prediction 110 corresponds to an overall output classification for the whole-slide image. For instance, if the model determines that ten tiles are predicted to have a first phenotype and seven tiles are predicted to have a second phenotype, the aggregator 109 can predict thewhole-slide image to have the first state based on the majority of the tiles being associated with the first prediction in the patch predictions 108.

[0050] In some examples, other aggregation techniques may involve determining weights for each tile, where the weights represent an importance of the association between the tile and the whole-slide image label. In these techniques, the aggregator can be multi-instance learning models that are used to output embeddings that represent a weight of a measure of the association or contribution from each tile to the final whole-slide image prediction.

[0051] FIG. 1C illustrates an example of an aggregator 111 that is a plain multi-instance learning model that is trained to leam weights for each tile. The plain multi-instance learning model can be considered attention-based, as it learns the relationship between the tile and the prediction, but it does not leam any interaction between the tiles 104. For the plain multiinstance learning model to generate a whole-slide image embedding 112, the feature extractor 105 can extract the features from each tile to generate embeddings 106. The plain multiinstance learning model can multiply the embeddings 106 by the weight for the tile and adds the features of the other tiles times the weight of the other tiles to get the whole-slide image embedding 112, which corresponds to an aggregated embedding. The whole-slide image embedding 112 is then input into the predictor 107 (e.g., a classifier), which generates the whole-slide image prediction 110 corresponding to the output classification for the wholeslide image.

[0052] FIG. ID illustrates an example of an aggregator 113 that is associated with a local context multi-instance learning model 114 that is trained to leam weights for each tile based on its local context (e.g., the target tile and surrounding tiles). The local context multiinstance learning model 114 can be implemented as a graph neural network (GNN) that models the relationships between tiles by treating tiles as nodes and defining edges based on spatial proximity or feature similarity. The GNN learns how the features of each tiles and its surrounding tiles contribute to the overall prediction. The feature extractor 105 extracts the features from each tile.

[0053] The local context multi -instance learning model 114 aggregates a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of a surrounding area of the target portion. The surrounding area can be within a pixel distance (e.g., 100 pixels) of the target portion. Each portion (e.g., the target portion and the additional portions) corresponds to a tile. To generatethe whole-slide image embedding 112, the aggregator 113 aggregates the features of all tiles by applying learned weights to each tile’s features. Specifically, the feature vector for each tile is multiplied by its learned weight, and the weighted feature vectors from all tiles are combined (e.g., summed or averaged) to form the whole-slide image embedding 112. This embedding captures both the local and global context of the whole-slide image. The resulting whole-slide image embedding 112 is then input into the predictor 107, which generates the whole-slide image prediction 110 corresponding to the output classification for the wholeslide image 401.

[0054] FIG. IE illustrates an example of an aggregator that is a global context multiinstance learning model 115 that is trained to leam weights for a global context (e.g., all of the tiles 104). As examples, the global context multi-instance learning model 115 can be a transformer neural network or a similar architecture that incorporates a self-attention mechanism. This mechanism enables the global context multi-instance learning model 115 to leam both the relationship between each tile and the whole-slide image label, as well as the relationship between each tile and all other tiles. The self-attention mechanism assigns weights based on these relationships, reflecting the importance of each tile in the context of all tiles and the label.

[0055] The feature extractor 105 extracts the features from each tile. The global context multi -instance learning model 115 aggregates a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of the image. The additional portions make up an entirety of the image. Each portion (e.g., the target portion and the additional portions) corresponds to a tile. To generate a whole-slide image embedding 112, the global context multi -instance learning model 115 computes a weighted combination of tile features, where the weight for each tile is influenced by its interaction with all other tiles and its contribution to the overall prediction. Specifically, the features from each tile are adjusted based on the learned weights and aggregated to form a holistic representation of the entire slide. The resulting whole-slide image embedding 112 is then input into the predictor 107 (e.g., a classifier), which generates the whole-slide image prediction 110 corresponding to the output classification or other target outcome for the whole-slide image.B. Feature extraction-aggregation-classification

[0056] As described above, a machine-learning model can be used to generate an output classification indicating a phenotype of a subject based on an image of a biological sample from the subject. The machine-learning model may be a transformer-based model, such as the global context multi-instance learning model 115 in FIG. IE. The machine-learning model includes one or more classifiers and an aggregator model that generate the output classification based on features extracted from tiles of the image. The machine-learning model can include one or more of: vision transformer (ViT), Swin Transformer, convolutional neural network (CNN), hybrid CNN-transformer model, graph neural network, attention-based MIL model, or foundation models pretrained on histopathology images.

[0057] FIG. IF illustrates an example of a transformer-based machine-learning model, according to embodiments of the present disclosure. The biomarker prediction framework operates through a three-stage pipeline, as shown: i) data preprocessing, ii) feature extraction using a transformer-based model, and iii) aggregation of features to produce the final wholeslide image-level prediction. The transformer-based machine-learning model includes a feature extractor 105, an aggregator 116, and a classifier 117. A whole-slide image (e.g., whole-slide image 103) can be partitioned into tiles 104. The tiles 104 may be tissue regions of the whole-slide image. The tiles 104 can then be input into the feature extractor 105, which extracts features from the tiles 104. Classifier 117 can be any machine learning model such as described herein and may be referred to as a classifier model.

[0058] In the example shown, the feature extractor 105 is illustrated as CTransPath that includes a convolutional neural network (CNN)-based patch partition layer(s), a Swin Transformer (e.g., with 4 stages), and a global pooling layer. The feature extractor 105 can be pretrained in a self-supervised manner. In this way, the feature extractor 105 may be trained on at least some unlabeled images. For example, the tiles 104 may be from an image of the biological sample including an IHC stain and / or from an image of the biological sample including an H&E stain. So, the feature extractor 105 may be trained on at least some unlabeled H&E and / or at least some IHC images. In some examples, the feature extractor 105 may be trained on only H&E images but may receive and process both H&E images and IHC images. In other examples, the feature extractor 105 may be trained on both H&E images and IHC images and can receive and process both H&E images and IHC images. As another example, a first feature extractor can be trained on only H&E images and can receive andprocess only H&E images, while a second feature extractor can be trained on only IHC images and can receive and process only IHC images.

[0059] Once the features are extracted from the tiles 104 by the feature extractor 105, the aggregator 116, which is an example of an aggregator model, receives the features. Each tile is shown being associated with a tile embedding having a fixed dimension of 768. The dimension can be based on the machine-learning model, so different dimensions are possible. The aggregator 116 includes a linear projection layer that reduces the dimensionality of each tile embedding to 512. Transformer layers of the aggregator 116 receive the reduced- dimensionality tile embeddings and generate an aggregated embedding for the tiles 104. So, the aggregator 116 provides an aggregated output for the tiles 104, where the aggregated output includes the aggregated embedding.

[0060] A multilayer perceptron (MLP) head of the classifier 117 receives the aggregated embedding and determines an output classification 118 based on the aggregated embedding. The classifier 117 is an example of the predictor 107 in FIGS. 1B-1E and the output classification 118 is an example of the whole-slide image prediction 110 in FIGS. 1B-1E. The output classification 118 indicates a phenotype based on the aggregated output. For example, the output classification 118 indicates a phenotype of microsatellite instability (MSI) I mismatch repair deficiency (MMRd) or microsatellite stable (MSS) I mismatch repair proficient (MMRp) for colorectal cancer.

[0061] For the experiments in this exemplary implementation (results of which are below), a 5-fold cross-validation scheme was employed, subdividing the dataset into training, validation, and testing sets in a 6:2:2 ratio, respectively. During training, the validation set can be used to determine the best model, which can be evaluated on the test set. Upon completion of training, the model’s performance can be evaluated across each fold’s test set as well as on a holdout dataset. The area under the receiver operator curve (AUROC) was used as the main evaluation metric in the example results below.C. Visualization and explainability

[0062] Visualization and explainability of transformer-based models can be helpful for clinicians to comprehend the decision-making process of deep learning models. As an example, to understand how the dual -modality model provided herein analyzes WSIs for decision-making, a visualization technique from Wagner’s study22can be used and wasadopted in the results below. For example, to assess the influence of each individual patch on the classification score, attention rollout can be employed. This technique can include the recursive multiplication of attention maps from preceding layers, offering a cumulative insight into patch impact. Additionally, the attention scores assigned by each transformer head through class token self-attention can be visualized. These scores can be normalized and tuned to ensure a coherent visual representation within the standardized range of [0,1], By processing each patch independently via the transformer model, their respective contributions to the classification decisions can be directly quantified, facilitating the visualization of their impact scores within the pre-defined range. The classification heatmap can be calculated by using a single tile as an input to the model. Contribution heatmaps (attention x classification score) can be derived as the product of individual tile classification score and attention score. Using the methodologies above, we generated heatmaps for models trained on H&E WSI only, IHC WSI only and dual H&E + IHC WSIs, as shown in FIG. 4A-4E and 5A-5D. The resulting visualizations serve as an intuitive guide to understanding the areas within the tissue samples that most inform the model’s predictions.IL USING FEATURES FROM H&E AND IHC

[0063] With advances in computational algorithms and tools, significant investment has been made in the development of Al tools to aid in the clinical assessment of biomarkers and clinical outcomes8'21. Given that hematoxylin & eosin (H&E) staining is the most widely used and cost-effective method in clinical settings, researchers have focused on developing tools for analyzing H&E-stained whole slide images (WSIs) to predict biomarker status22'26. For example, deep learning-based predictive algorithms have been successful in extrapolating programmed death-ligand 1 (PD-L1) status from H&E-stained images in breast cancer27. Wagner et al. introduced an end-to-end transformer-based model for colorectal cancer (CRC) biomarker prediction from H&E WSI in 202322. On the other hand, limited studies have been conducted on Al-assisted IHC interpretations28'31. For instance, a weakly supervised deep learning model on raw IHC images was utilized to predict PD-L1 status in non-small cell lung cancer (NSCLC). This model achieved area under the receiver operator curves (AUROC) scores of 0.80 - 0.88 and led to an improved association with response to immune checkpoint inhibition29. In another study, Huang et al. developed an automatic WSI features extraction pipeline called IMPRESS (Image-based Pathological Registration and Segmentation Statistics) using both H&E and multiplex IHC (CD8, CD 163, and PD-L1) stained images. The pipeline generated human-interpretable features that characterizeddifferent cellular components of the tumor immune microenvironment. These human- interpretable features, combined with clinical variables, were used to train machine learning models to predict neoadjuvant chemotherapy (NAC) outcomes in patients with HER2- positive (HER2+) breast cancer and triple-negative breast cancer (TNBC)32

[0064] In the present disclosure, a machine-learning model can receive features extracted from images with different staining methods to determine an output classification indicating a phenotype. For example, a first image of a biological sample from a subject may be stained using IHC for a specific biomarker, while a second image of the same biological sample may be stained using H&E to reveal general tissue morphology. These complementary images can be used together to improve model performance, e.g., when the biomarker targeted by the IHC stain is associated with characteristic morphologic patterns observable in the H&E image. Tissue regions in each of the images can be segmented, and the tissue regions can be partitioned to generate a set of tiles. Features are extracted from the set of tiles, which the machine-learning model then uses to generate the output classification. The machine-learning model can be an ensemble model that includes sub-models. For example, the sub-models can include an aggregator model and one or more classifiers. As such, the output classification is based on features from both the IHC image and the H&E image.

[0065] The dual-modality framework provided herein introduces advancements in integrating input from H&E and IHC images. Compared to existing methodology, the framework enables feature integration from both H&E and IHC images, offering enhanced flexibility. In embodiments, the framework can employ features from either or both image types for the inference of new cases. In embodiments, the framework implements efficient tile sampling. To optimize resources, this approach incorporates random tile selection in training and validation. Instead of processing all tiles from WSIs, the model selects a random subset of tiles, e.g., a maximum of 100, 150, 200, 250, 300, 350, 400, 450, 500, 600, 700, 800, 900 or 1000 tiles, for each training epoch. This strategy can maintain model performance without utilizing exhaustive data. In embodiments, the framework can also incorporate accelerated multi-GPU training. The training (e.g., 5-fold training), validation, and testing setup can be empowered by multi-GPU training configurations. Each fold is processed on a dedicated GPU, which can significantly reduce total training duration and facilitate a more rapid turnaround of results.A. Multiplex 1HC

[0066] Traditional IHC is based on the use of one antibody for one biomarker per tissue section. However, the systems and methods provided herein are not so limited. Techniques to multiplex IHC have been developed, such as multiplex chromogenic IHC and multiplex fluorescent IHC. See, e.g., Sheng et al., Multiplex Immunofluorescence: A Powerful Tool in Cancer Immunotherapy, Int J Mol Sci. 2023 Feb 4;24(4):3086. Thus, a single slide with multiple biomarker visualizations can be used in the systems and methods herein as if the individual biomarkers had been visualized on different slides. Each biomarker visualization (e.g., a respective dye) can correspond to a respective IHC stain, which can correspond to a respective imageB. Example iTissue processing

[0067] FIGS. 2A-2B illustrate an overview of data pre-processing for images of biological samples according to some embodiments of the present disclosure. In FIG. 2A, the preprocessing pipeline can initiate with digitization of whole slide images, followed by tissue segmentation and tessellation of the whole-slide images into tiles for analysis. Image 201A is a whole-slide image of a biological sample including an H&E stain. Image 20 IB is a wholeslide image of the biological sample including an IHC stain.

[0068] A tissue mask can be generated for each of the images 201 A-201B so that tissue regions of the images 201A-201B can be identified. The tissue masks may be generated by an object detection model. Based on the identified tissue regions, the images 201 A-201B are segmented. That is, portions of the images 201 A-201B that are determined not to include tissue may be segmented out, leaving only portions of the images 201 A-201B that are determined to include tissue. The segmented tissue regions of the images 201 A-201B can then be partitioned into tiles 204, where a first portion of the tiles 204 are from the image 201 A and a second portion of the tiles 204 are from the image 201B.

[0069] Various techniques can be used for the tissue processing. The following are some examples, but others may be used, e.g., other scanner, number of tiles, cohorts of patients, pixel classification models, object classification models, etc..C. Feature extraction

[0070] FIG. 2B illustrates an example model architecture showcasing a feature extractor 205, CTransPath, which processes the initial input data. Features for each tile are extracted utilizing the CTransPath model71, a hybrid architecture that combines the Swin Transformer72with a convolutional neural network (CNN) structure. The CTransPath is equipped with three initial convolution layers that enhance local feature detection and improve training stability, succeeded by four stages of Swin Transformer layers. These transformer layers add global contextual information via self-attention modules. The CTransPath was pretrained in a self-supervised manner on a large dataset of unlabeled histopathological images from The Cancer Genome Atlas (TCGA) and Pathology Al Platform (PAIP)73. This feature extractor generates a 768-dimensional feature vector per tile, which is then utilized in further downstream analysis.D. Examples of aggregation prior to classification

[0071] Once features are extracted from dies, aggregation can be performed for the features to generate an aggregated embedding. The aggregation can be performed prior to a classifier determining an output classification. An aggregation stage can ingest tile embeddings from a whole-slide image, e.g., employing a transformer-based aggregation module with multiheaded self-attention to process the sequence of embeddings. The aggregation model can permit each tile to interact with every other, thereby enabling a comprehensive assessment. The aggregation model can include a transformer-based aggregator, which can include one or more linear projection layers, one or more (e.g., two) transformer layers, and an MLP head.

[0072] Various architectures may be used. Examples of which are provided below.1. Concatenation of features or of images

[0073] FIG. 3A illustrates a single-input architecture for a machine-learning model. Features 306 from an H&E image and from one or more IHC images are extracted. Each IHC image can include a respective IHC stain for a different biomarker. In some examples, in addition to features 306 from H&E and IHC images, data from additional data sources may additionally be included in the input to the machine-learning model. For example, genomics data such as obtained by next-generation sequencing (NGS), radiology images, and multiple IHC stains may be included in the input. The machine-learning model can determine the output classification further based on the additional data.

[0074] Features 306 can be generated in various ways from various input images. For example, the images can be concatenated producing a concatenated image and then features extracted from the concatenated image, e.g., after partitioning the concatenated image into tiles. Alternatively, features can be extracted from individual images and their resulting tiles.

[0075] The H&E image and the IHC image(s) can be partitioned into tiles, and the features 306 can be extracted for each tile. In some instances, the features 306 may be extracted for a whole-slide image. The features 306 may be extracted by one or more feature extractors (e.g., feature extractor 105 in FIGS. 1B-1F). For instance, features of tiles from the H&E image may be extracted by a first feature extractor and features from tiles from the IHC image(s) may be extracted by a second feature extractor, or the features 306 from the H&E image and the IHC image(s) may be extracted by a same feature extractor.

[0076] The features 306 can be concatenated. For example, in FIG. 3 A, the features 306 are concatenated along a tile dimension. But the features 306 may be concatenated according to various techniques. For instance, the concatenation can involve one or more of: spatial concatenation along axes, channel-wise concatenation of feature maps, independent processing followed by feature fusion, tile-level concatenation during embedding, concatenation after positional encoding, or integration via dual-branch transformer modules. Channel-wise concatenation is shown in FIG. 3B and integration via dual-branch transformer modules is shown in FIG. 3C. In some examples, the concatenation may involve generating a concatenation image using the IHC image(s) and the H&E image. The tiles can then be partitioned from the concatenation image.

[0077] The machine-learning model can include a projection layer 320 that performs a dimensional reduction projector on the features 306 after the concatenation. For example, the projection layer 320 may reduce the features 306 from (M+N) x 768 to (M+N+l) x 512, where M is the number of H&E tiles and N is the number of IHC tiles. After the projection layer 320, an aggregator model 315 aggregates the features 306 to generate an aggregated output of an aggregated embedding. As illustrated, the aggregated embedding can have a dimensionality of 1 x 512. The aggregated embedding is then input to a classifier 317 that generates an output classification 318 indicating a phenotype.2. Use of separate projectors before concatenation

[0078] FIG. 3B illustrates a machine-learning model with a separate projection layer for each input. Features 306A from an H&E image and features 306B from one or more IHC images are extracted. Each IHC image can include a respective IHC stain for a different biomarker. The H&E image and the IHC image(s) can be partitioned into tiles, and the features 306A-306B can be extracted for each tile. In some instances, the features 306A- 306B may be extracted for whole-slide images. The features 306A-306B may be extracted by one or more feature extractors (e.g., feature extractor 105 in FIGS. IB- IF). For instance, the features 306A may be extracted by a first feature extractor and the features 306C may be extracted by a second feature extractor. Or the features 306A-306C may extracted by a same feature extractor.

[0079] The machine-learning model can include a projection layer 320A that can apply a dimensional reduction projector on the features 306A. In addition, a projection layer 320B can apply a dimensional reduction projector on the features 306B. For example, the features 306A-306B can each have a dimensionality of N x 768, where N is the max(Number of H&E tiles, Number of IHC tiles). The projection layers 320A-320B can perform the dimensional reduction projector to reduce the dimensionality to N x 512.

[0080] After performing the dimensional reduction projector, the features 306A-306B can be concatenated. For example, in FIG. 3B, the features 306A-306B are concatenated by channel-wise concatenation. But, the features 306A-306B may be concatenated according to various techniques. The machine-learning model can also include a projection layer 320C that performs a dimensional reduction projector on the features 306A-306B after the concatenation. For example, the projection layer 320C may reduce the features 306A-306B from N x 1024 to (N+l) x 512. After the projection layer 320C, an aggregator model 315 aggregates the features 306A-306C to generate an aggregated output of an aggregated embedding. As illustrated, the aggregated embedding can have a dimensionality of 1 x 512. The aggregated embedding is then input to a classifier 317 that generates an output classification 318 indicating a phenotype.

[0081] There can be an additional branch with a projection layer for each whole-slide image from which features were extracted. For example, features 306B can be extracted from a first IHC image. If there is an additional IHC image that also has extracted features, these additional features can be input to an additional projection layer (e g., a respective projectorfor each image), and the output of the additional projection layer can also be concatenated with the outputs of the projection layers 320A-320B. The approach can be applied to any number of additional IHC images, e.g., at least 1, 2, 3, 4, 6, 6, 7, 8, 9, or 10 additional images.3. Concatenation of aggregated embeddings

[0082] FIG. 3C illustrates a machine-learning model with two parameter-shared transformer branches. Features 306A from an H&E image and features 306B from one or more IHC images are extracted. Each IHC image can include a respective IHC stain for a different biomarker. The H&E image and the IHC image(s) can be partitioned into tiles, and the features 306A-306B can be extracted for each tile. In some instances, the features 306A- 306B may be extracted for whole-slide images. The features 306A-306B may be extracted by one or more feature extractors (e.g., feature extractor 105 in FIGS. IB- IF). For instance, the features 306A may be extracted by a first feature extractor and the features 306C may be extracted by a second feature extractor. Or, the features 306A-306C may extracted by a same feature extractor.

[0083] The machine-learning model can separately aggregate the features 306 A extracted from the H&E image and the features 306B extracted from the IHC image(s) to obtain respective aggregated embeddings. As shown, the features 306A may be aggregated by aggregation model 315A and the features 306B may be aggregated by aggregation model 315B. The aggregation model 315A has shared parameters with the aggregation model 315B. Each of the aggregation models 31 A-315B includes a projection layer that can apply a dimensional reduction projector on the features 306A-306B. In addition, the aggregation models 315A-315B each include two transformer layers that aggregate the features 306 A- 306B to generate the aggregated embeddings. The features 306A-306B can each have a dimensionality of N x 768, where M is the number of H&E tiles and N is the number of IHC tiles. After the aggregation models 315A-315B, the features 306A-306B can have a dimensionality to 1 x 512.

[0084] The respective aggregated embeddings from the aggregation models 315A-315B are then concatenated (e.g., along the channel dimension) and passed to the classifier 317, i.e. MLP head, for final prediction of an output classification 318. Parameter-sharing helps to reduce the number of model parameters, which subsequently results in memor -efficient model. Furthermore, a branch-dropout mechanism 322 is incorporated that omits one of the branches during model training, effectively enhancing the model’s generalization ability.E. Aggregation of Tile Classifications

[0085] In some examples, the aggregation may occur at a tile level after a classification is generated for each tile. As previously described in FIG. IB, a predictor model, or classifier, can generate a tile classification for each tile. The tiles can be from both an H&E image and one or more IHC images. The tile classification indicating a phenotype based on the features extracted from the tile. So, there can be a set of tile classifications. The set of tile classifications can be aggregated using an aggregation model (e.g., aggregator 109 in FIG. IB) to obtain an output classification. The aggregation model may use a majority voting technique, a maximum classification technique, an averaging technique, or any other suitable method to determine the output classification. As such, rather than aggregating the features from the images prior to classification, the features can be classified and then aggregated to obtain the output classification.III. EXAMPLE IMPLEMENTATIONS

[0086] The dual-modality model framework provided herein is exemplified in two distinct real -world setings.A. Example Phenotypes

[0087] A first seting is the prediction of MSI I MMRd status in colorectal cancer. Another setting is prediction of PD-L1 status in breast cancer. Each of these biomarkers is used as an indication for checkpoint inhibitor immunotherapy.

[0088] Use of the dual-modality transformer-based model provided herein is exemplified by the prediction of MSI I MMRd and PD-L1 status using both H&E and IHC stained whole slide images. As described below in Section I, the Al framework yielded enhanced predictive accuracy by integrating features from both staining methods, and exhibited superior prognostic precision compared to current biomarker assessments. The approach not only achieved clinical-grade performance with AUROC exceeding 0.97 in colorectal cancer (CRC), but also provides a customizable framework that can be applied in various clinical and research and development scenarios. Additionally, the model supports decision-making in oncology by potentially refining patient selection for immunotherapy, suggesting a reevaluation of existing PD-L1 status thresholds. These data support the integration of advanced Al tools in clinical pathology, aiming to enhance the precision and efficiency of cancer biomarker evaluation.1. Microsatellite Instability (MSI) Assay and Interpretation

[0089] MSI status was assessed by directly analyzing 2,810 known homopolymer to pentapolymer microsatellite regions within the targeted whole exome sequencing (WES) gene panel. These regions were compared against the reference genome (hg38) available from the UCSC Genome Browser database. Microsatellite loci alterations were identified through somatic insertions or deletions, counting only those changes that affected the number of tandem repeats. Detection of genomic variants at these loci utilized the same depth and frequency criteria as mutation detection protocols. A microsatellite instability -high (MSI-H) status was assigned to samples with 116 or more altered loci. Samples with 113 to 115 altered loci were considered equivocal, and those with 112 or fewer were classified as microsatellitestable (MSS).2. Mismatch Repair (MMR) assay and Interpretation

[0090] IHC analysis was conducted on formalin-fixed, paraffin-embedded (FFPE) tissue sections on glass slides per standard clinical workflows. Automated staining techniques, following the manufacturer’s instructions, were applied, and validated according to Clinical Laboratory Improvement Amendments (CLIA), College of American Pathologists (CAP) and International Organization for Standardization (ISO) standards. Board-certified pathologists independently reviewed all IHC results. MMR protein expression was determined using specific antibody clones for MLH1 (Ml antibody), MSH2 (G2191129 antibody), MSH6 (44 antibody), and PMS2 (EPR3947 antibody, Ventana Medical Systems, Inc., Tucson, AZ, USA). MMRd was indicated by the complete absence of expression for any tested protein, whereas proficient MMR (MMRp) demonstrated positive staining across all four proteins. Internal controls were utilized where possible, and every IHC slide utilized external positive and negative run controls.3. PD-L1 Assay and Interpretation

[0091] The PD-L1 immunohistochemical staining was carried out on formalin-fixed, paraffin-embedded (FFPE) sections using the PD-L1 22C3 pharmDx kit (Agilent Technologies, Santa Clara, CA, USA), according to United States Food and Drug Administration (FDA) standards. This process was subjected to the same automated staining techniques, optimization, and validation protocols as established for the MMR IHC assays, in compliance with CLIA / CAP and ISO guidelines. The Combined Positive Score (CPS) wascalculated with the formula: (number of PD-L1 positive cells (tumor cells, lymphocytes, and macrophages) I total number of viable tumor cells) * 100. Board-certified pathologists were trained according to the PD-L1 IHC 22c3 pharmDx Interpretation Manual and evaluated the results independently at a single institution (Caris Life Sciences, Irving, TX). PD-L1 subgroups were defined as negative (CPS < 10) or positive (CPS > 10).B. Clinical outcome analysis

[0092] Two different endpoints were assessed in this study. Time-on-treatment (TOT) was inferred using insurance claim data, calculated as the interval between the last and first administrations of pembrolizumab. Overall survival (OS) was defined from the initiation of pembrolizumab treatment to either the date of death in the real-world evidence (RWE) dataset or the last known contact in the insurance claims database. Patients with no claim for over 100 days were presumed deceased, while those with contact within 100 days of the last claims data refresh were considered alive but censored in the analysis76. In addition, for the survival analysis, patients in the holdout dataset with TOT of less than 21 days were excluded from the final analysis, as these cases were considered to represent poor quality data. Kaplan- Meier survival metrics were computed, with hazard ratios (HR) derived from the Cox proportional hazard model and p-values ascertained via the log-rank test.C. Results

[0093] H&E and IHC WSIs from the same patient were acquired from a cohort of breast cancer (BRCA) and colorectal cancer (CRC) patients, including samples from both primary and metastatic sites (20,820 cases for CRC MMR cohort, 20,879 cases for CRC MSI cohort, 15,173 cases for BRCA PD-L1 cohort). WSIs were scanned using either Phillips or Leica scanners at 40X resolution.1. Patient Characteristics and experimental setup

[0094] In this exemplary implementation of the dual-modality framework provided herein, cohorts of breast and colorectal cancer patients (20,820 cases for CRC MMRd cohort; 20,879 cases for CRC MSI cohort; 15,173 cases for BRCA PD-L1 cohort) were identified. Each case had tumor molecular profiling using next-generation sequencing and accompanying WSIs with available test results for MSI / MMRd or PD-L1. The detailed patient characteristics are summarized in Table 1 below.Table 1 - Patient characteristics for this study

[0095] As example implementations, paired H&E and IHC WSIs from the same patient were acquired from a cohort of breast cancer (BRCA) and colorectal cancer (CRC) patients, including samples from both primary and metastatic sites (20,820 cases for CRC MMR cohort, 20,879 cases for CRC MSI cohort, 15,173 cases for BRCA PD-L1 cohort). WSIs were scanned using either Philips or Leica scanners at 40X resolution. Initially, two QuPath69pixel classification models were trained to segment tissues from H&E and IHC WSIs respectively. An object detection model based on YOLO framework70was trained to detect control tissue (tissue microarray cores) on IHC WSIs and exclude it from the WSI tissue mask. Subsequently, WSIs were tessellated into 224 x 224-pixel tiles at lOx magnification, with a detailed resolution of ~1.0 micron per pixel, post-application of the tissue masks generated from the QuPath pixel classifiers. For Philips images in iSyntax format, a conversion to TIFF format was performed prior to processing.

[0096] A transformer model was used for these example results. Other techniques and example parameter values, number of iterations, number of epochs, batch sizes, etc. can be used. The transformer models were trained with the AdamW optimizer using weight decay of 5* 10'4and learning rate of 1* 10'6. All models were trained for 10 epochs with a batch size of one and a branch dropout probability of 0.3. The models were evaluated every 500 iterations for all cohorts. The number of tiles per WSI varied significantly, ranging from 2 to 11,000 tiles. Approximately 80% of WSIs contained fewer than 3200 tiles, and 50% had fewer than 1000 tiles. Given the wide variation in tile counts, no performance degradation was found when limiting the model to training with only 500 tiles. Consequently, a maximumof 500 tiles per epoch were randomly selected during training and validation phases, which reduced the graphic processing unit (GPU) memory usage and allowed the model to be trained on a GPU with a limited memory capacity' (16 GB). For the testing phase, all available tiles from each WSI were used and the processing was performed on a central processing unit (CPU). Using multiple GPUs, with one assigned to each fold, considerably sped up the training process.2. MMR and MSI status prediction in CRC

[0097] MMR and MSI status prediction in CRC using the dual-modality model with combined H&E and IHC WSIs outperforms single-stain models.

[0098] Pembrolizumab has been approved for treating patients with advanced CRC harboring MSI or MMRd, markers pivotal for predicting immunotherapy outcomes. Several deep-leaming-based algorithms have leveraged H&E WSIs for MSI prediction in CRC using a variety of architectures (see, e.g., references 22, 23, 34, 38). The performance of such single stain models is compared to the dual-modality model provided herein using MSI and MMRd in CRC as an exemplary model system.

[0099] In this implementation, the dual-image model, incorporating both H&E and IHC images, focused on MLH1 status, as MLH1 promoter hypermethylation and consecutive loss of MEH1 expression is the most common reason for MMRd40. For MMRd prediction, the model achieved AUROC scores of 0.922, 0.947, and 0.967 for H&E alone, IHC alone, and a combination of H&E / IHC, respectively. These data indicate the dual-modality model achieved the highest predictive accuracy. It exhibited even higher performance in MSI status prediction (0.939, 0.952 and 0.973 for H&E alone, IHC alone and combination of H&E and IHC). The standard H&E image based model achieved an AUROC score of 0.9, yet integrating IHC data further refined predictive capability, boasting an AUROC of -0.97 for both MSI and MMRd prediction (FIG. 4A and 4B). In addition, histograms displaying the distribution of prediction probabilities demonstrated a more distinct separation of probabilities for the dual-modality model compared to the models that used H&E or IHC staining images alone (data not shown). This demonstrates an enhanced predictive performance by the integrated approach provided herein.

[0100] A stratified analysis further demonstrated the model’s robustness, accommodating variances in scanner types — specifically Philips and Leica — and specimen site types,encompassing primary' and metastatic sites (FIG. 4C). Overall, minimal impact of different scanner types on the model performance was observed. The model performed slightly better for WSIs scanned by Philips scanner, with the maximum difference observed in H&E WSI- based MMRd prediction (averaged AUROC score of 0.930 and 0.905 for Phillips and Leica scanner scanned images, respectively). On the other hand, the specimen site had more impact on model performance, with the maximum difference observed in H&E WSI-based MMRd prediction (averaged AUROC score of 0.940 and 0.852 for primary and metastatic specimens, respectively). The impact of specimen type was lesser on the IHC WSI-based model (FIG. 4C).

[0101] In preferred embodiments, the digital pathology models provided herein are used in clinical practice. Accordingly, a detailed trade-off analysis was performed (FIG. 4D)to assess the effectiveness of the Al model as a pre-screening tool in medical settings. For instance, by focusing on MSI prediction, the goal was to evaluate the potential of the model to obviate the need for MSI NGS testing in certain patients, which could substantially lighten the workload for pathologists and technicians. For the dual-modality approach using H&E / IHC for MSI prediction, with a false negative percentage (False negative cases I Total cases x 100) of just 0. 1%, the model achieved a sensitivity exceeding 98%. Among the 20,410 patients in the cohort, 11,698 (57%) were correctly classified as MSS, demonstrating the model’s ability to capture a significant portion of MSS cases — bolstering use of the model for clinical application. The trade-off analysis further demonstrated the model’s ability to adapt thresholds for prediction. For instance, by lowering the false negative percentage to 0.05%, the sensitivity increased to above 99%, while still correctly classifying over 44% of the total patient cohort as MSS. These results highlight the flexibility of the model’s adjustable threshold. In embodiments, the performance metrics can be optimized to suit a variety of clinical situations and user needs.

[0102] Additionally, this exemplary implementation harnessed the interpretative power of attention heatmaps (FIG. 4E), which serve as a visual guide to the model’s decision-making process. Such heatmaps provide valuable insights that can assist pathologists in verifying known biomarker associations and potentially uncovering novel patterns.3. Comparative Analysis of H&E and IHC Inputs in PD-L1 Expression Al Models for BRCA

[0103] Pembrolizumab has received approval for treating patients with TNBD with a Combined Positive Score (CPS) of 10 or higher2’41. Unlike the straightforward evaluation of dichotomous IHC staining, the assessment of PD-L1 status is complicated by an additional scoring system, introducing greater complexity and variability. The prediction of PD-L1 expression using Al tools has become a popular research topic27,42'44With the dual-input model, AUROC scores of 0.866 were achieved for H&E-only inputs, 0.959 for IHC-only inputs, and 0.957 for combined H&E / IHC inputs (FIG. 5A).

[0104] Performing stratified analysis, it was observed that the H&E-only model exhibited better performance with primary site samples, and the type of scanner utilized had a negligible influence on the model’s effectiveness. For IHC-stained cases, neither the specimen site type nor the scanner type stratification affected the model’s performance (FIG. 5B). Additionally, a trade-off analysis was conducted to demonstrate the model’s versatile applicability in various clinical settings (FIG. 5D). Attention and classification heatmaps were also generated for interpretation (FIG. 5D). Observations indicate a more discernible separation of prediction probabilities when using the IHC or H&E / IHC-based models, as opposed to the model based solely on H&E.4. Prognostic Significance of Predicted Biomarkers in Pembrolizumab Treatment Outcomes

[0105] To assess the prognostic impact of the biomarker predictions on clinical outcomes, survival analyses were conducted utilizing two distinct endpoints: time-on-treatment (TOT) and overall survival (OS).

[0106] In the cohort of patients with CRC treated with pembrolizumab, those exhibiting MSI-H or MMRd showed notably longer TOT and OS. Specifically, patients with MMRd had a hazard ratio (HR) of 0.463 (95% confidence interval [CI]: 0.365-0.587; p <0.001) for TOT and of 0.398 (95% CI: 0.282-0.562; p <0.001) for OS. These trends were consistently observed with the predictive models for MSI and MMRd status. For example, when performing survival analysis utilizing MMRd status predicted by dual-modality model, the HR for TOT was 0.506 (95% CI: 0.400-0.641; p <0.001) and for OS was 0.38 (95% CI: 0.267-0.539; p <0.001), aligning closely with the actual status trends. Comparing all threemodels, the MMRd status predicted by the dual-modality model demonstrated stronger association with clinical outcomes than the single-stain WSI models (FIGS. 6A-6D). Similar patterns were observed when performing survival analysis based on MSI status (FIGS. 6E- 6H).

[0107] Conversely, among patients with breast cancer treated with pembrolizumab, extended TOT and OS were noted among patients with a CPS of 10 or above, but only TOT reached statistical significance. Specifically, patients with CPS of 10 or above had a HR for TOT of 0.785 (95% CI: 0.629-0.979, p <0.05) and for OS of 0.882 (95% CI: 0.626-1.241, p >0.1). When predicted CPS statuses were applied for clinical outcome analyses, the prognoses inferred from the H&E model surpassed the actual pathologist-scored CPS in significance. The HR for TOT was 0.671 (95% CI: 0.525-0.858, p <0.005) and for OS was 0.511 (95% CI: 0.358-0.729, p <0.001) (see FIGS. 7A-7D).5. Test Performance of H&E-IHC Dual-Modality Models

[0108] For 5-fold cross-validation models, an overall area under the curve (AUC) can be computed in different ways. For example: (a) an AUC per fold can be computed and the AUCs can be averaged to indicate consistency or generalizability across different subsets of a dataset, or (b) the predictions for all of the folds can be concatenated to compute a single AUC to indicate an overall performance on the whole dataset. Table 2 shows test performance for MSI for different tumor sites using the two methods. Table 3 shows test performance for MMRd for different tumor sites using the two methods.Table 2Table 3

[0109] FIGS. 8-10 illustrate the performance of different architectures of the dual-image model using receiver operating characteristic (ROC) curves. FIG. 8 corresponds to the performance of the architecture in FIG. 3A, FIG. 9 corresponds to the performance of the architecture in FIG. 3B, and FIG. 10 corresponds to the performance of the architecture in FIG. 2C. The results are summarized in Table 4.Table 4Overall AUC -> single AUC after concatenating all test predictions Mean AUC -> mean of AUCs of all folds

[0110] An additional assessment was performed of the dual-modality model using several recent pathology foundation models — Virchow53, Virchow254, UNI55, and H-Optimus-O56— as feature encoders. Specifically, feature embeddings were extracted from both H&E andIHC images and then fed into either the dual-modality model or a single-staining model. The results, summarized in Table 5, demonstrate that the dual-modality model consistently outperformed the single-staining model for most biomarkers, confirming the flexible nature of integrating morphological (i.e., H&E) and immunohistochemical information.Table 5 - Comparison of AUROC across models using different feature encodersD. Analysis of Results[OHl] The development of advanced algorithms and the improvement of computing hardware have given researchers powerful tools to apply Al in healthcare45,46. Al have been developed to tackle complex challenges such as predicting biomarker status, clinical outcomes, and classifying cancer subtypes47-3'. Recent perspectives emphasized that AI- based biomarkers derived from routine clinical data could greatly improve the accessibility of personalized medicine by providing rapid, cost-effective alternatives to traditional molecular testing. In addition, Al-powered decision support systems may reduce the workload of healthcare practitioners by automating time-consuming tasks and streamlining patient stratification50. This disclosure provides a dual-modality approach for digital pathology that significantly increases the accuracy of biomarker predictions compared to single-input models and enhances their correlation with clinical outcomes, highlighting its ability to improve patient treatments.

[0112] Immunotherapy has become a cornerstone of treatment for many cancers52. In this exemplary embodiment of the dual-modality framework provided herein, the model was trained and evaluated using a substantial cohort, encompassing over 20,000 patients with CRC and more than 15,000 patients with BRCA. Patient specimens were sourced from avariety of clinics and research facilities, with tissue samples ranging from primary to metastatic sites. Collection methods were diverse, including resection, fine needle biopsy, core needle biopsy, and others. Additionally, the pathological slides were processed using two distinct scanners from Philips and Leica, which underscores the heterogeneity of the data set. This diversity in data sources and processing methods highlights the strong adaptability and robustness of the model framework, as it is trained on a wide array of samples, ensuring better generalization to real-world clinical scenarios.

[0113] The cohort reflects the intricate diversity present in tumor specimens from patients, which is essential for developing clinically relevant models. Despite the inherent variability, the model demonstrated remarkable predictive accuracy for biomarkers, highlighting its clinical value. The stratified analysis revealed that the conditions of WSI — specifically, the site of the specimen and the type of scanner used — contributed differentially to model performance. This not only underscores the model’s robustness but also its adaptability to diverse clinical conditions.

[0114] Use of the dual-modality framework provided herein extends beyond biomarker prediction. The findings indicate that the Al-predicted biomarker status correlates with clinical outcomes as strongly as-and in some cases more strongly than-biomarker status annotated by pathologists. This indicates Al systems can be integrated into clinical decisionmaking processes. For example, the PD-L1 status, as predicted by the H&E WSI-based model, correlated with patient outcomes more effectively than the CPS scores calculated by pathologists. The precision of the model’s predictions may pave the way for refining patient selection criteria for pembrolizumab therapy. This could lead to a reassessment of the CPS thresholds currently used to evaluate PD-L1 status, potentially transforming treatment protocols and substantially improving patient care.

[0115] For BRCA-PD-L1 prediction, gain in predictive performance when adding H&E data to the IHC-only model was not significant. This finding suggests that PD-L1 status in breast cancer can often be captured effectively through a single immunostain. In contrast, we saw marked improvements in performance for CRC-MMR and CRC-MSI predictions when combining H&E and IHC data. Without being bound by theory , because the analysis used only MLH1 IHC among the four MMR proteins, the morphological features in H&E slides likely complemented the single IHC stain, capturing a broader range of tumor characteristics that boosted the model’s accuracy. It is also noted that using H&E alone produced anAUROC below 0.9 for PD-L1 prediction in breast cancer but exceeded 0.9 for MMR / MSI prediction in CRC, underscoring that the magnitude of histopathological changes associated with a given biomarker may determine how much H&E data contribute to predictive power.

[0116] The dual-modality model provided herein showed clear advantages for biomarkers that exhibit strong morphological correlates — such as MMR or MSI — where the synergy between H&E and IHC data appears to be most beneficial. But not all biomarkers may benefit equally from dual-staining approaches. Markers like PD-L1 in breast cancer may be adequately characterized by a single IHC stain, such that H&E data may not contribute statistically significant benefits. Without being bound by theory, the choice between single- or dual-staining strategies may ultimately depend on the biomarker in question, the nature of its morphological footprint, and the specific research or clinical objectives. And in any event, the dual-modality model never proved statistically inferior to the IHC only model, confirming the general applicability of the dual approach regardless of underlying biological phenomena.

[0117] This model can also be further refined to be a multimodal system capable of incorporating multiple staining features. As a non-limiting example, applications include subtyping cancers, such as BRCA, by concurrently assessing ER, PR, and HER2 IHC staining. In embodiments, the dual-modality framework can be further expanded by incorporating various additional data sources, including without limitation genomics data such as obtained by next-generation sequencing (NGS), radiology images, and multiple IHC stains. Non-limiting examples of useful NGS techniques include whole exome sequencing (WES), whole transcriptome sequencing (WTS), whole genome sequencing (WGS), targeted analysis of specific gene panels, or any useful combination thereof. For example, the NGS may comprise WES and WTS. By integrating diverse data sources that offer a more comprehensive view of patient-specific charactenstics, treatment can be better tailored to individual patient profiles.

[0118] Trade-off analysis, such as described above, can be used to evaluate the efficiency of Al-based tools in clinical settings, such as the pre-screening of patients using routine H&E or combined H&E / IHC without pathological scoring. The MSIntuit tool was shown to rule out almost half of the non-MSI population while accurately classifying over 96% of MSI patients, surpassing the current gold-standard methods (92-95%)38. In comparison, a single modality H&E WSI-based model provided herein ruled out -42.8% of the non-MSI population at a sensitivity of 98% and -62.4% at a sensitivity' of 96%. The dual-modalityH&E / IHC model provided herein further improved performance, ruling out -65.1% of the non-MSI population at a sensitivity of 98% and -78.3% at a sensitivity of 96%. The dualmodality approach provides a more effective tool for MSI pre-screening as it achieved higher specificity than MSIntuit while maintaining high sensitivity.

[0119] Biomarker testing is time-intensive and pathological scoring systems, such as CPS, are not only complex but also subject to inter-pathologist variability67The Al-based system provided herein can assist in mitigating these issues. For example, the methodology could influence oncologists’ decision-making processes and expedite the delivery of optimal treatments to patients. Additionally, it could streamline costs for pathology labs.

[0120] Provided herein is a dual-modality framework that improves upon conventional pathology slide analysis and prior AI-WSI technologies. Therefore, the framework provides an improvement in the field of medicine, including without limitation the management of medical conditions (e.g., cancer) and medical Al. In the exemplary implementation described in this Section I, the capability of the model to predict critical prognostic biomarkers for immunotherapy was demonstrated, highlighting its use as a supportive tool for pathologists.IV. METHOD

[0121] FIG. 12 illustrates an example flow of a process for using a dual -modality model in digital pathology, according to embodiments of the present disclosure. The steps may be performed by a computer system, such as computer system 10 in FIG. 9.

[0122] At block 1202, the computer system accesses one or more first images of a biological sample from a subject. The one or more first images include an IHC stain for a respective biomarker. In some instances, the one or more first images can include a plurality of images. Each member of the plurality of images can include a respective IHC stain for a different biomarker. The plurality of images can be from a same slide but stained for different biomarkers using multiplex IHC approaches. The IHC stain include use of at least one primary antibody or aptamer, secondary antibody or aptamer, and a reporter molecule. The primary antibody or aptamer and / or the secondary antibody or aptamer may be a functional fragment. The reporter molecule can include an enzyme or a dye, optionally wherein the dye comprises a fluorescent dye. The biological sample can include a formalin-fixed paraffin- embedded (FFPE) tissue sample, fixed tissue, a core needle biopsy, a fine needle aspirate, fresh frozen (FF) tissue, formalin sample, tissue comprised in a solution that preservesnucleic acid or protein molecules, a fresh sample, or any combination thereof. The respective biomarker can be a first biomarker including a cancer biomarker, an actionable biomarker, or a combination thereof Non-limiting examples of useful biomarkers can be found in International Patent Publications WO / 2007 / 137187 (Int’l Appl. No. PCT / US2007 / 069286), published November 29, 2007; WO / 2010 / 045318 (Int’l Appl. No. PCT / US2009 / 060630), published April 22, 2010; WO / 2010 / 093465 (Int’l Appl. No. PCT / US2010 / 000407), published August 19, 2010; WO / 2012 / 170715 (Int’l Appl. No. PCT / US2012 / 041393), published December 13, 2012; WO / 2014 / 089241 (Int’l Appl. No. PCT / US2013 / 073184), published June 12, 2014; WO / 2011 / 056688 (Int’l Appl. No. PCT / US2010 / 054366), published May 12, 2011; WO / 2012 / 092336 (Int’l Appl. No. PCT / US2011 / 067527), published July 5, 2012; WO / 2015 / 116868 (Int’l Appl. No. PCT / US2015 / 013618), published August 6, 2015; WO / 2017 / 053915 (Int’l Appl. No. PCT / US2016 / 053614), published March 30, 2017; WO / 2016 / 141169 (Int’l Appl. No. PCT / US2016 / 020657), published September 9, 2016; W02018175501 (Int’l Appl. No. PCT / US2018 / 023438), published September 27, 2018; WO / 2020 / 113237 (based on Int’l Patent Appl. No. PCT / US2019 / 064078, filed December 2, 2019); WO / 2020 / 146554 (based on Int’l Patent Appl. No.PCT / US2020 / 012815, filed January 8, 2020); WO / 2021 / 112918 (based on Int’l Patent Appl. No. PCT / US2020 / 035990, filed June 3, 2020); WO / 2021 / 163706 (based on Int’l Patent Appl. No. PCT / US2021 / 018263, filed February 16, 2021); WO / 2021 / 222867 (based on Int’l Patent Appl. No. PCT / US2021 / 030351, filed April 30, 2021); WO / 2022 / 056328 (based on Int’l Patent Appl. No. PCT / US2021 / 049966, filed September 10, 2021); WO / 2022 / 103809 (based on Int’l Patent Appl. No. PCT / US2021 / 058741, filed November 10, 2021); andWO / 2022 / 132964 (based on Int’l Patent Appl. No. PCT / US2021 / 063603, file December 15, 2021); each of which publications is incorporated by reference herein in its entirety.

[0123] In various examples, the biological sample can be from a tumor. The tumor can be a primary tumor or a metastatic tumor. For example, the tumor may be of any tissue, organ, or cell type including but not limited to: the myeloid, breast, bile ducts, colon, rectum, female genital tract, stomach, esophagus, gastrointestinal stromal cells, small intestine, brain, mouth, sinuses, nose, throat, blood, liver, nervous system, lung, lymph, male genital tract, pleura, skin, plasma cells, neuroendocrine cells, B-cells, T-cells, ovary, pancreas, pituitary gland, spinal cord, prostate, peritoneum, large intestine, soft tissue, connective tissue, fat tissue, thymus, thyroid, or eye. The primary tumor can be a tumor of the bladder, breast, colon,rectum, endometrium, uterus, ovary, female genital tract, kidney, blood, liver, lung, skin, lymph, pancreas, prostate, or thyroid.

[0124] In some examples, the cancer biomarker includes one or more of ABL, ABL1, ACVR1, AIP, AKT1, AKT2, AKT3, ALK, AMER1, APC, AR, ARAF, ARHGAP26, ARHGAP35, ARID1A, ARID2, AR-V7, ASXL1, ATM, ATR, ATRX, AXIN1, AXIN2, AXL, B2M, BAP1, BARD1, BCL2, BCL9, BCOR, BCR, BLM, BMPR1A, BRAF, BRCA1, BRCA2, BRD3, BRIM, BRIP1, BTK, CALR, CARD11, CASP8, CBFB, CCND1, CCND2, CCND3, CD79B, CDC73, CDH1, CDK12, CDK4, CDK6, CDKN1B, CDKN2A, CHEK1, CHEK2, CIC, CREBBP, CSF1R, CTCF, CTNNA1, CTNNB1, CXCR4, CYLD, CYP17A1, DDR2, DICER1, DNMT3A, EGFR, EGFR vIII, EGLN1, ELF3, EP300, EPHA2, ERBB2, ERBB3, ERBB4, ERCC2, ERG, ESRI, ETV1, ETV4, ETV5, ETV6, EWSR1, EXO1, EZH2, FANCA, FANCB, FANCC, FANCD2, FANCE, FANCF, FANCG, FANCI, FANCL, FANCM, FAS, FAT1, FBXW7, FGFR1, FGFR2, FGFR3, FGFR4, FGR, FH, FLCN, FLT1, FLT3, FLT4, FOLR1, FOXA1, FOXL2, FUBP1, FYN, GALNT12, GATA3, GLI2, GNA11, GNA13, GNAQ, GNAS, H3F3A, H3F3B, HDAC1, HIST1H3B, HIST1H3C, HNF1A, HOXB13, HRAS, IDH1, IDH2, INSR, IRF4, JAK1, JAK2, JAK3, KDM5C, KDM6A, KDR, KEAP1, Ki-67, KIF1B, KIT, KLF4, KMT2A, KMT2C, KMT2D, KRAS, LCK, LYN, LZTR1, MAML2, MAP2K1, MAP2K2, MAP2K4, MAP3K1, MAPK1, MAPK3, MAST1, MAST2, MAX, MED12, MEF2B, MEN1, MET, MET Exon 14 Skipping, MGA, MITF, MLH1, MLH3, MPL, MRE11, MSH2, MSH3, MSH6, MSMB, MST1R, MTOR, MUSK, MUTYH, MYB, MYC, MYCN, MYD88, NBN, NF1, NF2, NFE2L2, NFKBIA, NOTCH1, NOTCH2, NPM1, NRAS, NRG1, NSD1, NTHL1, NTRK1, NTRK2, NTRK3, NUMBL, NUTM1, PALB2, PARP1, PBRM1, PDGFRA, PDGFRB, PHOX2B, PIK3CA, PIK3CB, PIK3R1, PIK3R2, PIM1, PKN1, PMS1, PMS2, POLDI, POLD2, POLD3, POLD4, POLE, POLQ, POTI, PPARG, PPP2R1A, PPP2R2A, PRDM1, PRKACA, PRKAR1A, PRKCA, PRKCB, PRKDC, PTCHI, PTEN, PTPN11, RABL3, RAC1, RAD50, RAD51B, RAD51C, RAD51D, RAD54L, RAFI, RASA1, RBI, RELA, RET, RHOA, RNF43, ROS1, RPA1, RPA2, RPA3, RPA4, RSPO2, RSPO3, RUNX1, SDHA, SDHAF2, SDHB, SDHC, SDHD, SETD2, SF3B1, SMAD2, SMAD4, SMARCA4, SMARCB1, SMARCE1, SMO, S0CS1, SPEN, SPOP, SRC, SSBP1, STAG2, STAT3, STK11, SUFU, SUZ12, TCF7L2, TERT, TET2, TFE3, TFEB, THADA, TMEM127, TMPRSS2, TNFAIP3, TNFRSF14, TP53, TRAF7, TSC1, TSC2, U2AF1, VHL, WRN, WT1, XPO1, XRCC1, XRCC2, and YESL Additionally or alternatively, the cancer biomarker can include one or more of ALK, AR,CLDN18, ER, FGFR2b, F0LR1, Her2 / Neu, MAGE-A4, MET, MLH1, MSH2, MSH6, PMS2, pl6, PD-L1, PR, PTEN.

[0125] At block 1204, the computer system accesses a second image of the biological sample. The second image includes a H&E stain. The H&E stain may be for a same biomarker as one of the biomarkers of the one or more first images. Alternatively, the stain may be for a different biomarker.

[0126] At block 1206, the computer system segments tissue regions in the one or more first images and the second image. The computer system can generate a tissue mask to identify the tissue regions in the one or more first images and the second image.

[0127] At block 1208, the computer system partitions the tissue regions in the one or more first images and the second image. The partitioning generates a set of tiles. In some examples, the computer system can generate a concatenation image using the one or more first images and the second image, the set of tiles can be partitioned from the concatenation image. Generating the concatenation of the one or more first images and the second image can include one or more of: spatial concatenation along axes, channel-wise concatenation of feature maps, independent processing followed by feature fusion, tile-level concatenation during embedding, concatenation after positional encoding, or integration via dual-branch transformer modules.

[0128] At block 1210, the computer system extracts features from the set of tiles using a feature extractor. The feature extractor is trained on at least some unlabeled H&E and / or at least some IHC images. The feature extractor can be pretrained in a self-supervised manner on unlabeled H&E and IHC images. The feature extractor can include one or more of Swin Transformer, CNN, hybrid, CNN-transformer, self-supervised or contrastive learning-based model, transformer pretrained on histopathology data, or foundation models such as Virchow, UNI, CTransPath, CONCH, TITAN, THREADS, TANGLE, CHIEF, or PRISM.

[0129] At block 1212, the computer system generates, by a machine-learning model, an output classification indicating a first phenotype. The output classification is generated based on the features extracted from the set of tiles. The machine-learning model includes one or more classifiers and an aggregation model that provides an aggregated output for the set of tiles. The aggregation model can include one or more transformer layers. The machinelearning model comprises one or more of: vision transformer (ViT), Swin Transformer, convolutional neural network (CNN), hybrid CNN-transformer model, graph neural network,attention-based MIL model, or foundation models pretrained on histopathology images. The computer system may further generate, by the machine-learning model, at least one additional output classification indicating at least one additional phenotype.

[0130] In some examples, the first phenotype can be a state of a second biomarker. The second biomarker can include one or more genomic signature. Optionally, the genomic signature can include at least one of mismatch repair deficiency (MMRd), microsatellite instability (MSI), tumor mutational burden (TMB), loss of heterozygosity (gLOH), homologous recombination deficiency (HRD), human leukocyte antigen (HLA) genotyping, mismatch repair proficiency (MMRp), and / or microsatellite stability (MSS).

[0131] The respective biomarker can include one or more of MLH1, MSH2, MSH6, PMS2 and the second biomarker comprises MMRd and / or MSI. Optionally, the biological sample can include colorectal cancer cells. Additionally or alternatively, the respective biomarker and the second biomarker can each include programmed death ligand 1 (PD-L1). Optionally , the biological sample can include breast cancer cells. Further, the respective biomarker can include one or more of ER, PR, Ki67 and HER2, and the second biomarker can include a breast cancer subtype. The biological sample can include breast cancer cells, where the breast cancer subtype is determined based on combinations of expression levels or positivity of said biomarkers, optionally in accordance with established subtypes including but not limited to Luminal A, Luminal B, HER2-enriched, and Tripal-negative.

[0132] Generating the output classification may include aggregating, using the aggregation model, the features extracted from the set of tiles to generate an aggregated embedding, where the aggregated output includes the aggregated embedding. The features extracted from the set of tiles can be concatenated and input into the aggregation model. Prior to concatenation, a dimensional reduction projector can be performed on the features extracted from the set of tiles. Different dimensional reduction projectors can be applied to the features extracted from each of the one or more first images and the second image. The output classification indicating the first phenotype can then be generated by the one or more classifiers based on the aggregated output.

[0133] In various examples, aggregating the extracted features to generate the aggregated embedding can include separately aggregating the features extracted from each image of the one or more first images and the second image to obtain respective aggregated embeddings. The respective aggregated embeddings can be concatenated to obtain the aggregated output.

[0134] In some examples, generating the output classification includes, for each of the set of tiles, generating a tile classification indicating the first phenotype based on the features extracted from the tile. A set of tile classifications is generated as a result. The computer system can aggregate, using the aggregation model, the set of title classifications to obtain the output classification.

[0135] In some implementations, the computer system can determine a diagnosis, prognosis, and / or theranosis for a medical condition, based on the output classification. Additionally or alternatively, the computer system can determine whether a treatment for a medical condition is of likely benefit, lack of benefit, or indeterminate benefit in treating the subject, based on the output classification. The treatment can be administered to the subject based on the determining.

[0136] In some examples, the treatment can include an immunotherapy, which may involve an immune checkpoint therapy. The immune checkpoint therapy can include at least one of anti-PD-1 therapy, anti-PD-Ll therapy, anti-CTLA-4 therapy, ipilimumab, nivolumab, pembrolizumab, atezolizumab, avelumab, durvalumab, cemiplimab, and any combination thereof.

[0137] The method may be performed over a time course to track a progression of the medical condition. Metrics such as time-on-treatment, overall survival, disease recurrence, survival time, or any other suitable metric may be evaluated to track the progression of the medical condition. The medical condition may be cancer, which may include an acute lymphoblastic leukemia; acute myeloid leukemia; adrenocortical carcinoma; AIDS-related cancer; AIDS-related lymphoma; anal cancer; appendix cancer; astrocytomas; atypical teratoid / rhabdoid tumor; basal cell carcinoma; bladder cancer; brain stem glioma; brain tumor, brain stem glioma, central nervous method atypical teratoid / rhabdoid tumor, central nervous method embryonal tumors, astrocytomas, craniopharyngioma, ependymoblastoma, ependymoma, medulloblastoma, medulloepithelioma, pineal parenchymal tumors of intermediate differentiation, supratentorial primitive neuroectodermal tumors and pineoblastoma; breast cancer; bronchial tumors; Burkitt lymphoma; cancer of unknown primary site (CUP); carcinoid tumor; carcinoma of unknown primary site; central nervous method atypical teratoid / rhabdoid tumor; central nervous method embryonal tumors; cervical cancer; childhood cancers; chordoma; chronic lymphocytic leukemia; chronic myelogenous leukemia; chronic myeloproliferative disorders; colon cancer; colorectal cancer;craniopharyngioma; cutaneous T-cell lymphoma; endocnne pancreas islet cell tumors; endometrial cancer; ependymoblastoma; ependymoma; esophageal cancer; esthesioneuroblastoma; Ewing sarcoma; extracranial germ cell tumor; extragonadal germ cell tumor; extrahepatic bile duct cancer; gallbladder cancer; gastric (stomach) cancer; gastrointestinal carcinoid tumor; gastrointestinal stromal cell tumor; gastrointestinal stromal tumor (GIST); gestational trophoblastic tumor; glioma; hairy cell leukemia; head and neck cancer; heart cancer; Hodgkin lymphoma; hypopharyngeal cancer; intraocular melanoma; islet cell tumors; Kaposi sarcoma; kidney cancer; Langerhans cell histiocytosis; laryngeal cancer; lip cancer; liver cancer; malignant fibrous histiocytoma bone cancer; medulloblastoma; medulloepithehoma; melanoma; Merkel cell carcinoma; Merkel cell skin carcinoma; mesothelioma; metastatic squamous neck cancer with occult primary; mouth cancer; multiple endocrine neoplasia syndromes; multiple myeloma; multiple myeloma / plasma cell neoplasm; mycosis fungoides; myelodysplastic syndromes; myeloproliferative neoplasms; nasal cavity cancer; nasopharyngeal cancer; neuroblastoma; Non-Hodgkin lymphoma; nonmelanoma skin cancer; non-small cell lung cancer; oral cancer; oral cavity cancer; oropharyngeal cancer; osteosarcoma; other brain and spinal cord tumors; ovarian cancer; ovarian epithelial cancer; ovarian germ cell tumor; ovarian low malignant potential tumor; pancreatic cancer; papillomatosis; paranasal sinus cancer; parathyroid cancer; pelvic cancer; penile cancer; pharyngeal cancer; pineal parenchymal tumors of intermediate differentiation; pineoblastoma; pituitary tumor; plasma cell neoplasm / multiple myeloma; pleuropulmonary blastoma; primary central nervous method (CNS) lymphoma; primary hepatocellular liver cancer; prostate cancer; rectal cancer; renal cancer; renal cell (kidney) cancer; renal cell cancer; respiratory tract cancer; retinoblastoma; rhabdomyosarcoma; salivary gland cancer; Sezary syndrome; small cell lung cancer; small intestine cancer; soft tissue sarcoma; squamous cell carcinoma; squamous neck cancer; stomach (gastric) cancer; supratentorial primitive neuroectodermal tumors; T-cell lymphoma; testicular cancer; throat cancer; thymic carcinoma; thymoma; thyroid cancer; transitional cell cancer; transitional cell cancer of the renal pelvis and ureter; trophoblastic tumor; ureter cancer; urethral cancer; uterine cancer; uterine sarcoma; vaginal cancer; vulvar cancer; Waldenstrom macroglobulinemia; or Wilm’s tumor. In some instances, the cancer may include the cancer comprises an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck squamous carcinoma,leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma.

[0138] In some implementations, the computer system can provide a report including the first phenotype and / or the at least one additional phenotype, and / or information derived from the same. The information can optionally include a presence or absence, diagnosis, prognosis, and / or theranosis of a cancer in the subject.

[0139] The computer system can identify a set of images for subsequent processing based on the output classification of the machine-learning model. The set of images can include the one or more first images and the second image. An indication of the set of images for the subsequent processing can be output. As such, pathologists may receive and analyze the set of images after the determination by the machine-learning model. So, the number of images evaluated by the pathologists can be decreased, since the pathologists may only receive images indicating a particular state for a phenotype.V. USES OF PHENOTYPE

[0140] The systems and methods provided herein can be used for determining or predicting a phenotype of the subject, such as classifying a subject in relation to a medical condition. As examples, such a classification can include a diagnosis of the presence / absence of a disease or disorder, a stage of a disease or disorder, susceptibility to a disease or disorder, prognosis of a disease stage or disorder, theranosis of a disease or disorder, a physiological state, or response I potential response (or lack thereof) to interventions such as therapeutics. A phenotype can result from a subject’s genetic makeup as well as the influence of environmental factors and the interactions between the two.

[0141] The phenotype can be assessed using analysis of whole slide images (WSIs). In a non-limiting example, the WSIs comprise sections from FFPE tissue samples. In someembodiments, characterizing a phenotype comprises combining WSI analysis with alternate data sources, such as clinical data and / or molecular profiling data. In embodiments, the molecular profiling comprises next-generation sequencing. As a non-limiting example, consider that a tumor-containing formalin fixed paraffin embedded block (FFPE) from most recent surgery or biopsy is obtained for a subject. The block can be sectioned into multiple slides. Certain slides can be used for WSI analysis while nucleic acids (e.g., genomic DNA and / or mRNA) are extracted from other slides and used to perform next-generation sequencing, e.g., whole exome sequencing (WES), whole transcriptome sequencing (WTS), whole genome sequencing (WGS), analysis or targeted gene panels, or any useful combination thereof. In the example, the results of the WSI and sequencing analysis can be combined to characterize a phenotype.

[0142] The determination of the phenotype can use any information from image analysis, a molecular profile, as well as any patient specific and / or clinical information. The phenotype can be delivered to a treating physician in a molecular profiling report. A molecular profiling report can include various genetic and epigenetic information (e.g., as determined from laboratory measurements of samples, such as those described herein) as well other characteristics of a subject, such as height, weight, age, blood pressure, medical history, etc.

[0143] Embodiments can determine a phenotype (e.g., response / potential response (or lack thereof) to interventions such as therapeutics) multiple times over a time period (time course), e.g., intervals of 1, 2, 3, 4 weeks; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, or 12 months; or 1, 2, 3, 4, 5 years. For instance, a biopsy may be performed on a time course following treatment. Such monitoring over a time course can track a medical condition, such as remission, recurrence or disease progression.A. Diagnosis

[0144] The systems and methods provided by the disclosure can be used to characterize various phenotypes of interest in addition to or instead of providing treatment guidance. For example, image analysis may be used to screen for disease, monitor disease before and / or after treatment, or characterize a disease aggressiveness. For example, image analysis may be used to predict the risk that a primary tumor will metastasize. Thus, such analysis of a primary tumor may provide both personalized treatment options for the patient and in addition provide a metastatic potential for the tumor. The treating physician may consider the predicted metastatic potential when deciding a course of treatment for the patient.

[0145] As further examples, characterizing a phenotype can comprise detecting the presence of or likelihood of developing a tumor, neoplasm, or cancer (e.g., by analyzing a biopsy of a suspicious area), or characterizing the tumor, neoplasm, or cancer (e.g., stage, grade, aggressiveness, likelihood of metastasis or recurrence, etc). In some embodiments, the cancer comprises an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumors (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAG), lung non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma. The systems and methods herein can be used to characterize these and other cancers. Characterizing a phenotype can be providing a diagnosis, prognosis or theranosis of the cancer.B. Treatment selection

[0146] The systems and methods provided herein can be used for selecting treatments for an individual that could favorably change the clinical course of a medical condition, including without limitation cancer. The WSI analysis can provide a personalized approach to selecting treatments that are more or less likely to benefit a cancer patient and can be used to guide treatment in any desired setting, including without limitation the front-line I standard of care setting, for patients with poor prognosis, such as those with metastatic disease or those whose cancer has progressed on standard front-line therapies, or whose cancer has progressed on previous chemotherapeutic or hormonal regimens. The cancer can be a metastatic cancer or recurrent cancer. The treatments can be on-compendium or off-compendium treatments. The treatments may be standard of care for the type of cancer in the individual, or the treatments may be typically used for other types of cancer. Thus, image analysis may expand the choice of treatments for the individual.

[0147] The systems and methods provided herein may be used to classify patients as more or less likely to benefit or respond to various treatments, including which of two or more treatments (therapies) would a patient respond to better. Unless otherwise noted, the terms “response” or “non-response,” as used herein, refer to any appropriate indication that a treatment provides a benefit to a patient (a “responder” or “benefiter”) or has a lack of benefit to the patient (a “non-responder” or “non-benefiter”). Such an indication may be determined using accepted clinical response criteria such as the standard Response Evaluation Criteria in Solid Tumors (RECIST) criteria, or other useful patient response criteria such as progression free survival (PFS), time to progression (TTP), disease free survival (DFS), time-to-next treatment (TNT, TTNT), tumor shrinkage or disappearance, or the like. RECIST is a set of rules published by an international consortium that define when tumors improve (“respond”), stay the same (“stabilize”), or worsen (“progress”) during treatment of a cancer patient. As used herein and unless otherwise noted, a patient “benefit” from a treatment may refer to any appropriate measure of improvement, including without limitation a RECIST response or longer PFS, TTP, DFS, or TNT / TTNT. Beneficial or desired clinical results include, but are not limited to, alleviation or amelioration of one or more symptoms, diminishment of extent of disease, stabilized (i.e., not worsening) state of disease, preventing spread of disease, delay or slowing of disease progression, amelioration or palliation of the disease state, and remission (whether partial or total), whether detectable or undetectable. Benefit also includes prolonging survival as compared to expected survival if not receiving a treatment or if receiving a different treatment. Likewise, “lack of benefit” from a treatment may refer to any appropriate measure of worsening disease during treatment. Generally, disease stabilization is considered a benefit, although in certain circumstances, if so noted, stabilization may be considered a lack of benefit. A predicted or indicated benefit may be described as “indeterminate” if there is not an acceptable level of prediction of benefit or lack of benefit. In some cases, benefit is considered indeterminate if it cannot be calculated, e.g., due to lack of necessary data.

[0148] When multiple treatment options are revealed using the approaches provided herein, decision rules can be put in place to prioritize the selection of a treatment regimen. For example, treatments may be prioritized based on direct results of molecular analysis (e.g., WSI results have higher priority than sequencing results, or vice versa), anticipated efficacy of therapeutic agent, prior history with the same or other treatments, expected side effects, availability of therapeutic agent, cost of therapeutic agent, drug-drug interactions, and otherfactors. For example, if multiple therapies (e.g., two being tested) provide equal response, the therapy with the least side effects (e.g., if the patient is above a specified age and / or has other health conditions) or the therapy that is more aggressive (e.g., if patient is below a specified age and / or does not have other health conditions) may be identified as a recommended treatment. Based on the recommended and prioritized therapeutic agent targets, a treating physician can decide on the course of treatment for a particular individual.

[0149] The systems and methods described herein can be used to provide personalized treatment options for cancer patients. In some embodiments, the subject has been previously administered one or more therapeutic agents to treat the cancer. The cancer may be refractory to one of these agents, e.g., by acquiring drug resistance mutations. Such acquired mutations may be identified over a time course using liquid biopsy and / or the methods provided herein. In some embodiments, the cancer is metastatic. In some embodiments, the subject has not previously been treated with one or more therapeutic agents identified by the method. Using the systems and methods provided herein, candidate treatments can be selected regardless of the stage, anatomical location, or anatomical origin of the cancer cells. Accordingly, such methods and systems can identify treatments based on individual characteristics of diseased cells, e.g., tumor cells, and other personalized factors in a subject in need of treatment, as opposed to relying on a traditional one-size fits all approach that is conventionally used to treat individuals suffering from a disease, especially cancer. In some cases, the recommended treatments are those not typically used to treat the disease or disorder inflicting the subject. In some cases, the recommended treatments are used after standard-of-care therapies are no longer providing adequate benefit.

[0150] The treating physician can use the results of the methods provided herein to assist in planning or optimizing a treatment regimen for a patient. The candidate treatment identified by the methods as descnbed herein can be used to treat a patient; however, such treatment is not necessarily required of the methods. Final treatment decisions are made at the discretion of the treating physician.

[0151] Treatments associated with one or more of the biomarkers or sample / training / reference vectors may be determined using treatment association such as in any of International Patent Publications WO / 2007 / 137187 (Int’l Appl. No. PCT / US2007 / 069286), published November 29, 2007; WO / 2010 / 045318 (Int’l Appl. No. PCT / US2009 / 060630), published April 22, 2010; WO / 2010 / 093465 (Int’l Appl. No.PCT / US2010 / 000407), published August 19, 2010; WO / 2012 / 170715 (Int’l Appl. No. PCT / US2012 / 041393), published December 13, 2012; WO / 2014 / 089241 (Int’l Appl. No. PCT / US2013 / 073184), published June 12, 2014; WO / 2011 / 056688 (Int’l Appl. No. PCT / US2010 / 054366), published May 12, 2011; WO / 2012 / 092336 (Int’l Appl. No. PCT / US2011 / 067527), published July 5, 2012; WO / 2015 / 116868 (Int’l Appl. No. PCT / US2015 / 013618), published August 6, 2015; WO / 2017 / 053915 (Int’l Appl. No. PCT / US2016 / 053614), published March 30, 2017; WO / 2016 / 141169 (Int’l Appl. No. PCT / US2016 / 020657), published September 9, 2016; and WO2018175501 (Int’l Appl. No. PCT / US2018 / 023438), published September 27, 2018; each of which publications is incorporated by reference herein in its entirety. Such rules can be updated as new information becomes available regarding various biomarkers, biosignatures, treatments, and the relationships thereof. The indication whether each treatment is likely to benefit the patient, not benefit the patient, or has indeterminate benefit may be weighted. For example, a likely or potential benefit may be a strong potential benefit or a lesser potential benefit. Such weighting can be based on any appropriate criteria, e.g., the strength of the evidence of the biomarker-treatment association, or the results of the image analysis and / or any molecular profiling, e.g., strength of machine learning model predictions, a degree or level of over- or underexpression, mutation, or any other relevant state (e.g., wild type or altered). As the treating physician is ultimately responsible for treating their patient, such physician may use results to assist in guiding their treatment recommendations.

[0152] In some embodiments, therapy selection is based upon multiple marker signatures, including those that use machine learning and artificial intelligence. See, e.g., International Patent publications WO / 2020 / 113237 (based on Int’l Patent Appl. No. PCT / US2019 / 064078, filed December 2, 2019); WO / 2020 / 146554 (based on Int’l Patent Appl. No. PCT / US2020 / 012815, filed January 8, 2020); WO / 2021 / 112918 (based on Int’l Patent Appl. No. PCT / US2020 / 035990, filed June 3, 2020); WO / 2021 / 163706 (based on Int’l Patent Appl. No. PCT / US2021 / 018263, filed February 16, 2021); WO / 2021 / 222867 (based on Int’l Patent Appl. No. PCT / US2021 / 030351, filed April 30, 2021); WO / 2022 / 056328 (based on Int’l Patent Appl. No. PCT / US2021 / 049966, filed September 10, 2021); WO / 2022 / 103809 (based on Int’l Patent Appl. No. PCT / US2021 / 058741, filed November 10, 2021); andWO / 2022 / 132964 (based on Int’l Patent Appl. No. PCT / US2021 / 063603, file December 15, 2021); each of which publications is incorporated by reference herein in its entirety.C. Report generated using digital pathology and / or molecular profiling

[0153] The results obtained via the systems and methods provided herein can be summarized in a patient report. Such report can be delivered to the caregiver for the subject, e.g., the oncologist or other treating physician. The caregiver can use the report to guide a treatment regimen for the subject. For example, the caregiver may administer one or more treatments indicated as likely benefit in the report. Similarly, the caregiver may avoid treating the patient with one or more treatments indicated as likely lack of benefit in the report. In some embodiments, such as when the report includes analysis indicating a likely recurrence or metastasis, the treating physician may choose, for example, a more aggressive treatment regimen, more frequent monitoring, or both. Such decisions can be made by the caregiver with guidance from the report.

[0154] The patient report can comprise multiple sections of relevant information, including but not limited to: 1) description of the patient and sample; 2) a complete or partial listing of the analysis performed, including but not limited to image analysis and that of any other biomarkers (e.g., nucleic acids, proteins, or other biological matter of interest); 3) a description of the state of one or more of the biomarkers determined via the image analy sis or molecular profile as determined for the subject; 4) a description of one or more biological signatures as determined for the patient, such as microsatellite stability, tumor mutational load / burden, tissue-of-origin, recurrence predictors, treatment response predictors; and / or metastasis predictors; 5) a prediction of the tissue of origin of the tumor; 6) one or more treatment associated with one or more of the biomarkers, groups of biomarkers, and / or biological signatures determined for the patient, which can include treatments identified using the ML methods provided herein in additional to ML modeling of any additional molecular profiling results; 7) an indication whether one or more treatment is likely to benefit the patient, not benefit the patient, or has indeterminate benefit; 8) one or more clinical trials for which the patient may be eligible; 9) an indication whether the cancer is predicted to recur and / or metastasize; and / or 10) evidence relevant to the foregoing, such as literature reports and / or clinical trial results.

[0155] The report can be computer generated, and can be a printed report, a computer file or both. The report can be made accessible via a secure web portal. The report may be displayed using any desired medium. In some embodiments, the display is a printout, a computer file, including without limitation a pdf file, or may be displayed via an applicationon a computer display such as a computer monitor, laptop display, tablet, smartphone, or other mobile device.

[0156] In an aspect, the disclosure provides a system for generating a patient report such as described above, comprising: (a) at least one host server; (b) at least one interface (user, automated, or both) for accessing the at least one host server to access and input data; (c) at least one processor for processing the inputted data; (d) at least one memory coupled to the processor for storing the processed data and instructions for: i) accessing a biomarker status determined by methodology as described herein; and ii) identifying biomarkers, biosignatures and related data and any information derived using such data (treatments, clinical trials, phenotypes, predictions, etc., as described herein); and (e) at least one display for displaying results and outcomes of the analysis performed as described herein. In some embodiments, the system further comprises at least one memory coupled to the processor for storing the processed data and instructions for identifying, based on the machine learning modeling according to the methods herein, at least one therapy with potential benefit for treatment of the cancer; and at least one display for display thereof. The system may further comprise at least one database comprising references for various biomarker states, data for drug / biomarker associations, or both. The at least one display can be a report provided by the present disclosure.D. Additional operations

[0157] In addition to a determination of biomarker status, various embodiments can perform additional measurements and / or assessments (e.g., additional molecular profiling) or responsive actions (e.g., treatment selection, treatment, or screening).1. Further screening modalities

[0158] In some embodiment, the systems and methods provided herein are trained to perform a risk analysis I classification. Based on the risk classification, e.g., the risk of metastasis determined using the systems and methods provided herein, the subject can be referred for additional screening modalities, e.g. using chest X ray, ultrasound, computed tomography, magnetic resonance imaging, or positron emission tomography. Such screening may be used to investigate the presence of secondary (metastatic) lesions and may be performed for cancer.2. Treatment

[0159] Embodiments may further include treating the medical condition in the patient after determining a classification for the subject. Treatment can be provided according to a determined medical condition, a risk determined via the image analysis, and / or a tissue of origin. For example, an identified biomarker can be targeted with a particular drug or chemotherapy. The tissue of origin can be used to guide a surgery or any other form of treatment. And the determined medical condition or prognosis can be used to determine how aggressive to be with any type of treatment. Examples of such treatment can include surgery', radiation therapy, chemotherapy, immunotherapy, targeted therapy, hormone therapy, and stem cell transplant.

[0160] Treatment may include chemotherapy, which is the use of drugs to destroy cancer cells, usually by keeping the cancer cells from growing and dividing. The drugs may involve, for example but are not limited to, mitomycin-C (available as a generic drug), gemcitabine (Gemzar), and thiotepa (Tepadina) for intravesical chemotherapy . The systemic chemotherapy may involve, for example but not limited to, cisplatin gemcitabine, methotrexate (Rheumatrex, Trexall), vinblastine (Velban), doxorubicin, and cisplatin.

[0161] In some embodiments, treatment may include immunotherapy. Immunotherapy may include immune checkpoint inhibitors that block checkpoint proteins such as CTLA4, PD-1 and / or PD-L1. Inhibitors may include but are not limited to atezolizumab (Tecentriq), nivolumab (Opdivo), avelumab (Bavencio), durvalumab (Imfinzi), and pembrolizumab (Keytruda).

[0162] Treatment embodiments may also include targeted therapy. Targeted therapy is a treatment that targets the cancer’s specific genes and / or proteins that contributes to cancer growth and survival. For example, erdafitinib is a small molecule inhibitor of fibroblast growth factor receptor (FGFR) and is approved to treat people with locally advanced or metastatic urothelial carcinoma with FGFR3 or FGFR2 genetic mutations that has continued to grow or spread of cancer cells. As another example, the monoclonal antibody trastuzumab works by binding to the HER2 receptor and inhibiting cell replication.

[0163] Some treatments may include radiation therapy. Radiation therapy is the use of high-energy x-rays or other particles to destroy cancer cells. In addition to each individual treatment, combinations of these treatments described herein may be used. In someembodiments, a combination of the treatments may be used. Information on treatments in the references are incorporated herein by reference.VI. EXAMPLE SYSTEMS

[0164] FIG. 13 illustrates a measurement system 1300 according to an embodiment of the present disclosure. The system as shown includes a sample 1305, such as a tissue sample, within an imaging device 1310, where an image 1308 can be generated from the sample 1305. For example, sample 1305 can be sliced, stained, and prepared on a slide for imaging. The image 1308 is sent imaging device 1310 to logic system 1330. As an example, the image 1308 can be used to predict biomarker labels, including without limitation MMR, MSI or PD- Ll. The image 1308 may be stored in a local memory 1335, an external memory 1340, or a storage device 1345.

[0165] Logic system 1330 may be, or may include, a computer system, ASIC, microprocessor, graphics processing unit (GPU), etc. It may also include or be coupled with a display (e.g., monitor, LED display, etc.) and a user input device (e.g., mouse, keyboard, buttons, etc.). Logic system 1330 and the other components may be part of a stand-alone or network connected computer system, or they may be directly attached to or incorporated in a device (e.g., a sequencing device) that includes imaging device 1310. Logic system 1330 may also include software that executes in a processor 1350. Logic system 1330 may include a computer readable medium storing instructions for controlling measurement system 1300 to perform any of the methods described herein. For example, logic system 1330 can provide commands to a system that includes imaging device 1310 such that imaging or other physical operations are performed. Such physical operations can be performed in a particular order, e.g., with images being captured in a particular order. Such physical operations may be performed by a robotics system, e.g., including a robotic arm, as may be used to obtain a sample and perform imaging.

[0166] Measurement system 1300 may also include a treatment device 1360, which can provide a treatment to the subject. Treatment device 1360 can determine a treatment and / or be used to perform a treatment. Examples of such treatment can include surgery, radiation therapy, chemotherapy, immunotherapy, targeted therapy, hormone therapy, and stem cell transplant. Logic system 1330 may be connected to treatment device 1360, e.g., to provide results of a method described herein. The treatment device may receive inputs from otherdevices, such as an imaging device and user inputs (e.g., to control the treatment, such as controls over a robotic system).

[0167] Measurement system 1300 may also include a reporting device 1355, which can present results of any of the methods described herein, e.g., as determined using the measurement system. Reporting device 1355 can be in communication with a reporting module within logic system 1330 that can aggregate, format, and send a report to reporting device 1355. The reporting module can present information determined using any of the method described herein. The information can be presented by reporting device 1355 in any format that can be recognized and interpreted by a user of the measurement system 1300. For example, the information can be presented by reporting device 1355 in a displayed, printed, or transmitted format, or any combination thereof.

[0168] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 13 in computer system 10. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0169] The subsystems shown in FIG. 13 are interconnected via a system bus 75.Additional subsystems such as a printer 74, keyboard 78, storage device(s) 79, monitor 76 (e.g., a display screen, such as an LED), which is coupled to display adapter 82, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 71, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 77 (e.g., USB, FireWire®). For example, I / O port 77 or external interface 81 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 10 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 75 allows the central processor 73 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 72 or the storage device(s) 79 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 72 and / or the storage device(s) 79 may embody a computer readable medium. Another subsystem is a data collection device 85,such as a camera, microphone, accelerometer, and the like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0170] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 81, by an internal interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 100, 200, 500, 1,000, 10,000, 50,000, 100,000, 500,00, or one million communication messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0171] Aspects of embodiments can be implemented in the form of control logic using hardware circuitry (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software stored in a memory with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardware circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein, a processor can include a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0172] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as R, Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission. A suitable non-transitory computer readablemedium can include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk) or Blu-ray disk, flash memory, and the like. The computer readable medium may be any combination of such devices. In addition, the order of operations may be re-arranged. A process can be terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the mam function.

[0173] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0174] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term “real-time” may refer to computing operations or processes that are completed within a certain time constraint. The time constraint may be 1 minute, 1 hour, 1 day, or 7 days. Thus, embodiments can be directed to computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or at different times or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, any of the steps of any of the methods can be performed with modules, units, circuits, or other means of a system for performing these steps.

[0175] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure. However, other embodiments of the disclosure may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0176] The above description of example embodiments of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above.

[0177] A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or a “second” component does not limit the referenced component to a particular location unless expressly stated. The term “based on” is intended to mean “based at least in part on.”

[0178] The claims may be drafted to exclude any element which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0179] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art. Where a conflict exists between the instant application and a reference provided herein, the instant application shall dominate.VII. REFERENCES1. Andre, T., et al. Pembrolizumab in Microsatellite-Instability-High Advanced Colorectal Cancer. N Engl J Med 383, 2207-2218 (2020).2. Cortes, J., et al. Pembrolizumab plus Chemotherapy in Advanced Triple-Negative Breast Cancer. N Engl J Med 387, 217-226 (2022).3. Li, K., et al. Microsatellite instability: a review of what the oncologist should know. Cancer Cell International 20, 16 (2020).4. Bartley, A.N., et al. Mismatch Repair and Microsatellite Instability Testing for Immune Checkpoint Inhibitor Therapy: Guideline From the College of American Pathologists in Collaboration With the Association for Molecular Pathology and Fight Colorectal Cancer. Archives of Pathology & Laboratory Medicine 146, 1194-1210 (2022).5. Sholl, L.M., et al. Programmed Death Ligand-1 and Tumor Mutation Burden Testing of Patients With Lung Cancer for Selection of Immune Checkpoint Inhibitor Therapies: Guideline From the College of American Pathologists, Association for Molecular Pathology, International Association for the Study of Lung Cancer, Pulmonary Pathology Society, and LUNGevity Foundation. Archives of Pathology & Laboratory Medicine (2024).6. Robert, M.E., et al. High Interobserver Variability Among Pathologists Using Combined Positive Score to Evaluate PD-L1 Expression in Gastric, Gastroesophageal Junction, and Esophageal Adenocarcinoma. Mod Pathol 36, 100154 (2023).7. Wadapurkar, R.M. & Vyas, R. Computational analysis of next generation sequencing data and its applications in clinical oncology. Informatics in Medicine Unlocked 11, 75-82 (2018).8. Prelaj, A., et al. Artificial intelligence for predictive biomarker discovery in immuno-oncology: a systematic review. Ann Oncol 35, 29-65 (2024).9. Ahn, J.S., et al. Artificial Intelligence in Breast Cancer Diagnosis and Personalized Medicine. J Breast Cancer 26, 405-435 (2023).10. Bilal, M., et al. Role of Al and digital pathology for colorectal immuno-oncology. Br J Cancer 128, 3-11 (2023).11. Huss, R., et al. Artificial intelligence and digital biomarker in precision pathology guiding immune therapy selection and precision oncology. Cancer Rep (Hoboken) 6, el796 (2023).12. Jha, A.K., et al. Emerging role of quantitative imaging (radiomics) and artificial intelligence in precision oncology. Explor Target Antitumor Ther 4, 569-582 (2023).13. Klauschen, F., et al. Toward Explainable Artificial Intelligence for Precision Pathology. Annu Rev Pathol 19, 541-570 (2024).14. Lyall, D.M., et al. Artificial intelligence for dementia-Applied models and digital health. Alzheimers Dement 19, 5872-5884 (2023).15. Naik, K.. et al. Current Status and Future Directions: The Application of Artificial Intelligence / Machine Learning for Precision Medicine. Clin Pharmacol Ther 115, 673-686 (2024).16. Vigdorovits, A., et al. Breaking Barriers: ALs Influence on Pathology and Oncology in Resource- Scarce Medical Sy stems. Cancers (Basel) 15(2023).17. Wen, Z., et al. Deep learning in digital pathology for personalized treatment plans of cancer patients. Semin Diagn Pathol 40, 109-119 (2023).18. Cao, Y., et al. CT Image-Based Radiomic Analysis for Detecting PD-L1 Expression Status in Bladder Cancer Patients. Acad Radiol (2024).19. Wu, Y., et al. Using machine learning for mortality prediction and risk stratification in atezolizumab-treated cancer patients: Integrative analysis of eight clinical trials. Cancer Med 12, 3744-3757 (2023).20. Bhattarai, S., et al. Predicting Neoadjuvant Treatment Response in Triple-Negative Breast Cancer Using Machine Learning. Diagnostics (Basel) 14(2023).21. Reitsam, N.G., et al. Artificial Intelligence in Colorectal Cancer: From Patient Screening over Tailoring Treatment Decisions to Identification of Novel Biomarkers. Digestion 105, 331-344 (2024).22. Wagner, S.J., et al. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell 41, 1650-1661 el654 (2023).23. Kather, J.N., et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 25, 1054-1056 (2019).24. Echle, A., et al. Clinical-Grade Detection of Microsatellite Instability in Colorectal Tumors by Deep Learning. Gastroenterology 159, 1406-1416. e!411 (2020).25. Schirris, Y., et al. DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer. Med Image Anal 79, 102464 (2022).26. Shmatko, A., et al. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology'. Nat Cancer 3, 1026-1038 (2022).27. Shamai, G , et al. Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer. Nat Commun 13, 6753 (2022).28. Rakha, E.A., et al. Digital Technology in Diagnostic Breast Pathology and Immunohistochemistry. Pathobiology 89, 334-342 (2022).29. Ligero, M., et al. Weakly Supervised Deep Learning Predicts Immunotherapy Response in Solid Tumors Based on PD-L1 Expression. Cancer Res Commun 4, 92-102 (2024).30. Abdul -Ghafar, J., et al. Validation of a Machine Learning Expert Supporting System, ImmunoGenius, Using Immunohistochemistry Results of 3000 Patients with Lymphoid Neoplasms. Diagnostics (Basel) 13(2023).31. Baxi, V., et al. Association of artificial intelligence-powered and manual quantification of programmed death-ligand 1 (PD-L1) expression with outcomes in patients treated with nivolumab ± ipilimumab. Mod Pathol 35, 1529-1539 (2022).32. Huang, Z., et al. Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images. NPJ Precis Oncol 7, 14 (2023).33. Hua, S., et al. PathoDuet: Foundation models for pathological slide analysis of H&E and IHC stains. Medical Image Analysis 97, 103289 (2024).34. Chang, X., et al. Predicting colorectal cancer microsatellite instability with a selfattention-enabled convolutional neural network. Cell Reports Medicine 4, 100914 (2023).35. Guo, B., et al. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin Transformer. J Pathol Clin Res 9, 223-235 (2023).36. Gustav, M., et al. Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathology, npj Precision Oncology 8, 115 (2024).37. Hildebrand, L.A., Pierce, C.J., Dennis, M., Paracha, M. & Maoz, A. Artificial Intelligence for Histology-Based Detection of Microsatellite Instability' and Prediction of Response to Immunotherapy in Colorectal Cancer. Cancers (Basel) 13(2021).38. Saillard, C., et al. Validation of MSIntuit as an Al-based pre-screening tool for MSI detection from colorectal cancer histology slides. Nature Communications 14, 6695 (2023).39. Whangbo, J., et al. Predicting Mismatch Repair Deficiency Status in Endometrial Cancer through Multi-Resolution Ensemble Learning in Digital Pathology. Journal of Imaging Informatics in Medicine 37, 1674-1682 (2024).40. Boland, C.R. & Goel, A. Microsatellite instability in colorectal cancer. Gastroenterology 138, 2073-2087.e2073 (2010).41. Cortes, J., et al. Pembrolizumab plus chemotherapy versus placebo plus chemotherapy for previously untreated locally recurrent inoperable or metastatic triplenegative breast cancer (KEYNOTE-355): a randomised, placebo-controlled, double-blind, phase 3 clinical trial. Lancet 396, 1817-1828 (2020).42. Wang, X., et al. How can artificial intelligence models assist PD-L1 expression scoring in breast cancer: results of multi-institutional ring studies, npj Breast Cancer 7, 61 (2021).43. Knudsen, B.S., et al. A Pipeline for Evaluation of Machine Learning / Artificial Intelligence Models to Quantify Programmed Death Ligand 1 Immunohistochemistry. Laboratory Investigation 104(2024).44. Yan, F., et al. Artificial intelligence-based assessment of PD-L1 expression in diffuse large B cell lymphoma, npj Precision Oncology 8, 76 (2024).45. Bajwa, J., et al. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J 8, el88-el94 (2021).46. Alowais, S.A., et al. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Medical Education 23, 689 (2023).47. Calderaro, J., et al. Deep learning-based phenotyping reclassifies combined hepatocellular-cholangiocarcinoma. Nat Commun 14, 8290 (2023).48. Khera, R., et al. Al in Medicine — JAMA’s Focus on Clinical Outcomes, Patient- Centered Care, Quality, and Equity. JAMA 330, 818-820 (2023).49. El Nahhas, O.S.M., et al. Regression-based Deep-Learning predicts molecular biomarkers from pathology slides. Nature Communications 15, 1253 (2024).50. Ligero, M., et al. Artificial intelligence-based biomarkers for treatment decisions in oncology. Trends in Cancer (2025).51. Jiang, R., et al. A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas. Nature Machine Intelligence 6, 876-891 (2024).52. Ling, S.P., et al. Role of Immunotherapy in the Treatment of Cancer: A Systematic Review. Cancers (Basel) 14(2022).53. Vorontsov, E., et al. Virchow: A Million-Slide Digital Pathology Foundation Model. arXiv:2309.07778 (2023).54. Zimmermann, E., et al. Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology'. arXiv:2408.00738 (2024).55. Chen, R. J., et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine 30, 850-862 (2024).56. Saillard, C., et al. H-optimus-0. (2024).57. Unger, M. & Kather, J.N. Deep learning in cancer genomics and histopathology. Genome Med 16, 44 (2024).58. Unger, M. & Kather, J.N. A systematic analysis of deep learning in genomics and histopathology for precision oncology. BMC Med Genomics 17, 48 (2024).59. Lipkova, J., et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40, 1095-1110 (2022).60. Chen, R. J., et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865-878. e866 (2022).61. Mobadersany, P., et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A 115, E2970-e2979 (2018).62. Cheerla, A. & Gevaert, O. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35, i446-i454 (2019).63. Chen, R.J., et al. Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis. IEEE Trans Med Imaging 41, 757-770 (2022).64. Boehm, K.M., et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer 3, 723-733 (2022).65. Vanguri, R.S., et al. Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non-small cell lung cancer. Nat Cancer 3, 1151-1164 (2022).66. Foersch, S., et al. Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer. Nature Medicine 29, 430-439 (2023).67. Fernandez, A.I., et al. Multi-Institutional Study of Pathologist Reading of the Programmed Cell Death Ligand-1 Combined Positive Score Immunohistochemistry Assay for Gastric or Gastroesophageal Junction Cancer. Mod Pathol 36, 100128 (2023).68. Kacew, A.J., et al. Artificial Intelligence Can Cut Costs While Maintaining Accuracy in Colorectal Cancer Genotyping. Frontiers in Oncology 11(2021).69. Bankhead, P., et al. QuPath: Open source software for digital pathology image analysis. Scientific Reports 7, 16878 (2017).70. Li, Chuyi, et al. Yolov6 v3. 0: A full-scale reloading. arXiv preprint arXiv:2301.05586 (2023).71. Wang, X., et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med Image Anal 81, 102559 (2022).72. Liu, Z., et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, in 2021 IEEE / CVF International Conference on Computer Vision (ICCV) 9992- 10002 (2021).73. Kim, Y.J., et al. PAIP 2019: Liver cancer segmentation challenge. Med Image Anal 67, 101854 (2021).74. Use, M., et al. Attention-based Deep Multiple Instance Learning, in Proceedings of the 35th International Conference on Machine Learning, Vol. 80 (eds. Jennifer, D. & Andreas, K.) 2127—2136 (PMLR, Proceedings of Machine Learning Research, 2018).75. Shao, Z., et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems 34, 2136-2147 (2021).76. Abraham, J.P., et al. Clinical Validation of a Machine-leaming-denved Signature Predictive of Outcomes from First-line Oxaliplatin-based Chemotherapy in Advanced Colorectal Cancer. Clinical Cancer Research 27, 1174-1183 (2021).

Claims

1. WHAT IS CLAIMED IS:

1. A method comprising performing, by a computer system: accessing one or more first images of a biological sample from a subject, wherein each of the one or more first images comprises an immunohistochemistry (IHC) stain for a respective biomarker; accessing a second image of the biological sample comprising a hematoxylin and eosin (H&E) stain; segmenting tissue regions in the one or more first images and the second image; partitioning the tissue regions in the one or more first images and the second image, thereby generating a set of tiles; extracting features from the set of tiles using a feature extractor trained on at least some unlabeled H&E and / or at least some IHC images; and generating, by a machine-learning model, an output classification indicating a first phenotype of the subject based on the features extracted from the set of tiles, wherein the machine-learning model includes one or more classifiers and an aggregation model that provides an aggregated output for the set of tiles.

2. The method of claim 1, wherein generating the output classification includes: aggregating, using the aggregation model, the features extracted from the set of tiles to generate an aggregated embedding, wherein the aggregated output includes the aggregated embedding; and generating, by the one or more classifiers, the output classification indicating the first phenotype based on the aggregated output.

3. The method of claim 2, wherein the features extracted from the set of tiles are concatenated and input into the aggregation model.

4. The method of claim 3, further comprising: prior to concatenation, performing a dimensional reduction projector on the features extracted from the set of tiles.

5. The method of claim 4, wherein different dimensional reduction projectors are applied to the features extracted from each of the one or more first images and the second image.

6. The method of claim 2, wherein aggregating the features to generate the aggregated embedding includes: separately aggregating the features extracted from each image of the one or more first images and the second image to obtain respective aggregated embeddings; and concatenating the respective aggregated embeddings to obtain the aggregated output.

7. The method of claim 2, further comprising: generating a concatenation image using the one or more first images and the second image, wherein the set of tiles are partitioned from the concatenation image.

8. The method of claim 7, wherein generating a concatenation of the one or more first images and the second image comprises one or more of: spatial concatenation along axes, channel-wise concatenation of feature maps, independent processing followed by feature fusion, tile-level concatenation during embedding, concatenation after positional encoding, or integration via dual-branch transformer modules.

9. The method of claim 1, wherein generating the output classification includes: for each tile of the set of tiles, generating a tile classification indicating the first phenotype based on the features extracted from the tile, thereby generating a set of tile classifications; and aggregating, using the aggregation model, the set of tile classifications to obtain the output classification.

10. The method of any preceding claim, wherein the aggregation model comprises one or more transformer layers.

11. The method of any preceding claim, wherein the feature extractor is pretrained in a self-supervised manner on unlabeled H&E and IHC images.

12. A method comprising performing, by a computer system:accessing one or more first images of a biological sample from a subject, wherein each of the one or more first images comprises an immunohistochemistry (IHC) stain for a respective biomarker; accessing a second image of the biological sample comprising a hematoxylin and eosin (H&E) stain; segmenting tissue regions in the one or more first images and the second image; partitioning the tissue regions in the one or more first images and the second image, thereby generating a set of tiles; extracting features from the set of tiles using a feature extractor pretrained in a self-supervised manner on unlabeled H&E and IHC images; and aggregating the extracted features using an aggregation model to generate an aggregated embedding, the aggregation model comprising one or more transformer layers; and generating, by a classifier, an output classification indicating a first phenotype of the subject based on the aggregated embedding.

13. The method of any preceding claim, further comprising: identifying a set of images for subsequent processing based on the output classification of the machine-learning model, wherein the set of images includes the one or more first images and the second image; and outputing an indication of the set of images for the subsequent processing.

14. The method of any preceding claim, wherein the one or more first images comprises a plurality of images, each member of the plurality of images comprising a respective IHC stain for a different biomarker.

15. The method of any preceding claim, wherein the biological sample is from a tumor.

16. The method of claim 15, wherein the tumor is a primary tumor or a metastatic tumor.

17. The method of claim 15 or 16, wherein the tumor is a tumor of any tissue, organ, or cell type including but not limited to: the myeloid, breast, bile ducts, colon, rectum, female genital tract, stomach, esophagus, gastrointestinal stromal cells, smallintestine, brain, mouth, sinuses, nose, throat, blood, liver, nervous system, lung, lymph, male genital tract, pleura, skin, plasma cells, neuroendocrine cells, B-cells, T-cells, ovary, pancreas, pituitary gland, spinal cord, prostate, peritoneum, large intestine, soft tissue, connective tissue, fat tissue, thymus, thyroid, or eye.

18. The method of claim 16, wherein the primary tumor is a tumor of the bladder, breast, colon, rectum, endometrium, uterus, ovary, female genital tract, kidney, blood, liver, lung, skin, lymph, pancreas, prostate, or thyroid.

19. The method of any preceding claim, wherein the biological sample comprises a formalin-fixed paraffin-embedded (FFPE) tissue sample, fixed tissue, a core needle biopsy, a fine needle aspirate, fresh frozen (FF) tissue, formalin sample, tissue comprised in a solution that preserves nucleic acid or protein molecules, a fresh sample, or any combination thereof.

20. The method of any preceding claim, wherein the respective biomarker is a first biomarker comprising a cancer biomarker, an actionable biomarker, or a combination thereof.

21. The method of claim 20, wherein the cancer biomarker comprises one or more of ABL, ABL1, ACVR1, AIP, AKT1, AKT2, AKT3, ALK, AMER1, APC, AR, ARAF, ARHGAP26, ARHGAP35, ARID1A, ARID2, AR-V7, ASXL1, ATM, ATR, ATRX, AXIN1, AXIN2, AXL, B2M, BAP1, BARD1, BCL2, BCL9, BCOR, BCR, BLM, BMPR1A, BRAF, BRCA1, BRCA2, BRD3, BRD4, BRIP1, BTK, CALR, CARD11, CASP8, CBFB, CCND1, CCND2, CCND3, CD79B, CDC73, CDH1, CDK12, CDK4, CDK6, CDKN1B, CDKN2A, CHEK1, CHEK2, CIC, CREBBP, CSF1R, CTCF, CTNNA1, CTNNB1, CXCR4, CYLD, CYP17A1, DDR2, DICER1, DNMT3A, EGFR, EGFR vIII, EGLN1, ELF3, EP300, EPHA2, ERBB2, ERBB3, ERBB4, ERCC2, ERG, ESRI, ETV1, ETV4, ETV5, ETV6, EWSR1, EXO1, EZH2, FANCA, FANCB, FANCC, FANCD2, FANCE, FANCF, FANCG, FANCI, FANCL, FANCM, FAS, FAT1, FBXW7, FGFR1, FGFR2, FGFR3, FGFR4, FGR, FH, FLCN, FLT1, FLT3, FLT4, FOLR1, FOXA1, FOXL2, FUBP1, FYN, GALNT12, GATA3, GLI2, GNA11, GNA13, GNAQ, GNAS, H3F3A, H3F3B, HDAC1, HIST1H3B, HIST1H3C, HNF1A, HOXB13, HRAS, IDH1, IDH2, INSR, IRF4, JAK1, JAK2, JAK3, KDM5C, KDM6A, KDR, KEAP1, Ki-67, KIF1B, KIT, KLF4, KMT2A, KMT2C, KMT2D, KRAS, LCK, LYN, LZTR1, MAML2, MAP2K1, MAP2K2, MAP2K4, MAP3K1, MAPK1,MAPK3, MAST1, MAST2, MAX, MED12, MEF2B, MEN1, MET, MET Exon 14 Skipping, MGA, MITF, MLH1, MLH3, MPL, MRE11, MSH2, MSH3, MSH6, MSMB, MST1R, MTOR, MUSK, MUTYH, MYB, MYC, MYCN, MYD88, NBN, NF1, NF2, NFE2L2, NFKBIA, NOTCH1, NOTCH2, NPM1, NRAS, NRG1, NSD1, NTHL1, NTRK1, NTRK2, NTRK3, NUMBL, NUTM1, PALB2, PARP1, PBRM1, PDGFRA, PDGFRB, PHOX2B, PIK3CA, PIK3CB, PIK3R1, PIK3R2, PIM1, PKN1, PMS1, PMS2, POLDI, POLD2, POLD3, POLD4, POLE, POLQ, POTI, PPARG, PPP2R1A, PPP2R2A, PRDM1, PRKACA, PRKAR1A, PRKCA, PRKCB, PRKDC, PTCHI, PTEN, PTPN11, RABL3, RAC1, RAD50, RAD51B, RAD51C, RAD51D, RAD54L, RAFI, RASA1, RBI, RELA, RET, RHOA, RNF43, ROS1, RPA1, RPA2, RPA3, RPA4, RSPO2, RSPO3, RUNX1, SDHA, SDHAF2, SDHB, SDHC, SDHD, SETD2, SF3B1, SMAD2, SMAD4, SMARCA4, SMARCB1, SMARCE1, SMO, SOCS1, SPEN, SPOP, SRC, SSBP1, STAG2, STAT3, STK11, SUFU, SUZ12, TCF7L2, TERT, TET2, TFE3, TFEB, THADA, TMEM127, TMPRSS2, TNFAIP3, TNFRSF14, TP53, TRAF7, TSC1, TSC2, U2AF1, VHL, WRN, WT1, XPO1, XRCC1, XRCC2, and YES 1.

22. The method of claim 20, wherein the cancer biomarker comprises one or more of ALK, AR, CLDN18, ER, FGFR2b, FOLR1, Her2 / Neu, MAGE-A4, MET, MLH1, MSH2, MSH6, PMS2, pl 6, PD-L1, PR, PTEN.

23. The method of any preceding claim, wherein the IHC stain comprises use of at least one primary antibody or aptamer, secondary antibody or aptamer, and a reporter molecule.

24. The method of claim 23, wherein the reporter molecule comprises an enzyme or a dye, optionally wherein the dye comprises a fluorescent dye.

25. The method of any preceding claim, wherein the machine-learning model comprises one or more of: vision transformer (ViT), Swin Transformer, convolutional neural network (CNN), hybrid CNN-transformer model, graph neural network, attentionbased MIL model, or foundation models pretrained on histopathology images.

26. The method of any preceding claim, wherein the feature extractor comprises one or more of Swin Transformer, CNN, hybrid, CNN-transformer, selfsupervised or contrastive learning-based model, transformer pretrained on histopathologyIQdata, or foundation models such as Virchow, UNI, CTransPath, CONCH, TITAN, THREADS, TANGLE, CHIEF, or PRISM.

27. The method of any preceding claim, further comprising generating, by the machine-learning model, at least one additional output classification indicating at least one additional phenotype.

28. The method of any preceding claim, wherein the first phenotype comprises a state of a second biomarker.

29. The method of claim 28, wherein the second biomarker comprises one or more genomic signatures, optionally wherein the one or more genomic signatures comprise at least one of mismatch repair deficiency (MMRd), microsatellite instability (MSI), tumor mutational burden (TMB), loss of heterozygosity (gLOH), homologous recombination deficiency (HRD), and / or human leukocyte antigen (HLA) genotyping.

30. The method of claim 28 or 29, wherein the respective biomarker comprises one or more of MLH1, MSH2, MSH6, PMS2 and the second biomarker comprises MMRd and / or MSI, optionally wherein the biological sample comprises colorectal cancer cells.

31. The method of claim 28 or 29, wherein the respective biomarker and the second biomarker each comprise programmed death ligand 1 (PD-L1), optionally wherein the biological sample comprises breast cancer cells.

32. The method of claim 28 or 29, wherein the respective biomarker comprises one or more of ER, PR, Ki67 and HER2, the second biomarker comprises a breast cancer subtype, the biological sample comprises breast cancer cells, and wherein the breast cancer subtype is determined based on combinations of expression levels or positivity of said biomarkers, optionally in accordance with established subtypes including but not limited to Luminal A, Luminal B, HER2-enriched, and Tripal-negative.

33. The method of any preceding claim, further comprising determining a diagnosis, prognosis, and / or theranosis for a medical condition, based on the output classification.

34. The method of any preceding claim, further comprising determining whether a treatment for a medical condition is of likely benefit, lack of benefit, or indeterminate benefit in treating the subject, based on the output classification.

35. The method of claim 33 or 34, further comprising administering the treatment to the subject based on the determining.

36. The method of claim 34 or 35, wherein the treatment comprises an immunotherapy.

37. The method of claim 36, wherein the immunotherapy comprises an immune checkpoint therapy.

38. The method of claim 37, wherein the immune checkpoint therapy comprises at least one of anti-PD-1 therapy, anti-PD-Ll therapy, anti-CTLA-4 therapy, ipilimumab, nivolumab, pembrolizumab, atezolizumab, avelumab, durvalumab, cemiplimab, and any combination thereof.

39. The method of any one of claims 34-38, further comprising performing the method over a time course to track a progression of the medical condition.

40. The method of any one of claims 34-39, wherein the medical condition comprises a cancer.

41. The method of claim 40, wherein the cancer comprises an acute lymphoblastic leukemia; acute myeloid leukemia; adrenocortical carcinoma; AIDS-related cancer; AIDS-related lymphoma; anal cancer; appendix cancer; astrocytomas; atypical teratoid / rhabdoid tumor; basal cell carcinoma; bladder cancer; brain stem glioma; brain tumor, brain stem glioma, central nervous method atypical teratoid / rhabdoid tumor, central nervous method embryonal tumors, astrocytomas, craniopharyngioma, ependymoblastoma, ependymoma, medulloblastoma, medulloepithelioma, pineal parenchymal tumors of intermediate differentiation, supratentorial primitive neuroectodermal tumors and pineoblastoma; breast cancer; bronchial tumors; Burkitt lymphoma; cancer of unknown primary site (CUP); carcinoid tumor; carcinoma of unknown primary site; central nervous method atypical teratoid / rhabdoid tumor; central nervous method embryonal tumors; cervical cancer; childhood cancers; chordoma; chronic lymphocytic leukemia; chronic myelogenousleukemia; chronic myeloproliferative disorders; colon cancer; colorectal cancer; craniopharyngioma; cutaneous T-cell lymphoma; endocnne pancreas islet cell tumors; endometrial cancer; ependymoblastoma; ependymoma; esophageal cancer; esthesioneuroblastoma; Ewing sarcoma; extracranial germ cell tumor; extragonadal germ cell tumor; extrahepatic bile duct cancer; gallbladder cancer; gastric (stomach) cancer; gastrointestinal carcinoid tumor; gastrointestinal stromal cell tumor; gastrointestinal stromal tumor (GIST); gestational trophoblastic tumor; glioma; hairy cell leukemia; head and neck cancer; heart cancer; Hodgkin lymphoma; hypopharyngeal cancer; intraocular melanoma; islet cell tumors; Kaposi sarcoma; kidney cancer; Langerhans cell histiocytosis; laryngeal cancer; lip cancer; liver cancer; malignant fibrous histiocytoma bone cancer; medulloblastoma; medulloepithehoma; melanoma; Merkel cell carcinoma; Merkel cell skin carcinoma; mesothelioma; metastatic squamous neck cancer with occult primary; mouth cancer; multiple endocrine neoplasia syndromes; multiple myeloma; multiple myeloma / plasma cell neoplasm; mycosis fungoides; myelodysplastic syndromes; myeloproliferative neoplasms; nasal cavity cancer; nasopharyngeal cancer; neuroblastoma; Non-Hodgkin lymphoma; nonmelanoma skin cancer; non-small cell lung cancer; oral cancer; oral cavity cancer; oropharyngeal cancer; osteosarcoma; other brain and spinal cord tumors; ovarian cancer; ovarian epithelial cancer; ovarian germ cell tumor; ovarian low malignant potential tumor; pancreatic cancer; papillomatosis; paranasal sinus cancer; parathyroid cancer; pelvic cancer; penile cancer; pharyngeal cancer; pineal parenchymal tumors of intermediate differentiation; pineoblastoma; pituitary tumor; plasma cell neoplasm / multiple myeloma; pleuropulmonary blastoma; primary central nervous method (CNS) lymphoma; primary hepatocellular liver cancer; prostate cancer; rectal cancer; renal cancer; renal cell (kidney) cancer; renal cell cancer; respiratory tract cancer; retinoblastoma; rhabdomyosarcoma; salivary gland cancer; Sezary syndrome; small cell lung cancer; small intestine cancer; soft tissue sarcoma; squamous cell carcinoma; squamous neck cancer; stomach (gastric) cancer; supratentorial primitive neuroectodermal tumors; T-cell lymphoma; testicular cancer; throat cancer; thymic carcinoma; thymoma; thyroid cancer; transitional cell cancer; transitional cell cancer of the renal pelvis and ureter; trophoblastic tumor; ureter cancer; urethral cancer; uterine cancer; uterine sarcoma; vaginal cancer; vulvar cancer; Waldenstrom macroglobulinemia; or Wilm’s tumor.

42. The method of claim 40, wherein the cancer comprises an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectaladenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma.

43. The method of any preceding claim, further comprising providing a report comprising the first phenotype and / or the at least one additional phenotype, and / or information derived from the same, optionally wherein the information comprises a presence or absence, diagnosis, prognosis, and / or theranosis of a cancer in the subject.

44. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that, when executed, cause a computer system to perform the method of any one of the preceding claims.

45. A system comprising: the computer product of claim 43; and one or more processors configured to execute instructions stored on the computer readable medium.

46. A system comprising means for performing any of the above methods.

47. A system comprising one or more processors configured to perform any of the above methods.

48. A system comprising modules that respectively perform the steps of any of the above methods.

Images