Machine learning based predictive biomarker discovery and patient stratification using standard care data

Abstract

The present disclosure relates generally to biomarker discovery and patient stratification, and more particularly to machine learning techniques for discovering related biomarkers using data collected as part of standard care (SoC), which may be used to identify related patient populations with therapeutic agents of known mechanisms of action (MoA). An exemplary method for predicting patient molecular analyte activity includes: training a first module of a machine learning model based on a plurality of medical images of a first queue; a second module to train the machine learning model based on one or more molecular analyte data sets obtained from a second queue; receiving a medical image from the patient; and predicting the activity of the molecular analyte from the medical image of the patient using the trained first and second modules of the machine learning model.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 445,980, filed February 15, 2023, and U.S. Provisional Application No. 63 / 618,258, filed January 5, 2024, the entire contents of which are incorporated herein by reference for all purposes.

[0003] Invention Field

[0004] This disclosure generally relates to biomarker discovery and patient stratification, and more specifically to machine learning techniques for discovering relevant biomarkers using data collected as part of standard of care (SoC), which can be used to identify relevant patient populations for therapeutics with known mechanisms of action (MoA). Background Technology

[0005] Predictive biomarkers are biomarkers used to identify individuals who are more likely to experience favorable or adverse effects when exposed to medical products or environmental factors than similar individuals without the biomarker. Clinical programs using predictive biomarkers for patient selection are typically significantly more likely to be successful. Historically, predictive biomarkers have been most commonly used in oncology (relative to other therapeutic areas) due to early recognition of disease heterogeneity and the ability to stratify patients using increasingly abundant data collected as part of SoC (System of Societies). Predictive biomarkers in oncology are typically based on specific somatic alterations measured via targeted gene panels, broader genetic changes such as tumor mutational burden (TMB) or microsatellite instability (MSI), changes in certain key proteins (e.g., ER or HER2), often measured via IHC, and less common gene expression changes or traits.

[0006] However, the promise of precision oncology has not yet been fully realized. A significant challenge is that identifying new biomarkers for patient response often relies on the results of small clinical trials that may lack sufficient statistical power for robust discovery. This biomarker discovery process also requires relevant assays to be performed as part of clinical trials, often without prior knowledge of which assays might be informative for predictive biomarkers. Furthermore, response characteristics relying on biological measurements not currently collected as part of SoC (System of Clinical Oncology), such as gene expression data, can be difficult to robustly determine (e.g., through CLIA certification processes) and are slow to gain widespread adoption.

[0007] Therefore, there is a need to provide techniques for discovering relevant biomarkers using data collected as part of standard of care (SoC), data that may lack essential biological measurements typically required to support such discoveries. Relevant biomarkers can be used for a variety of downstream tasks, such as patient stratification, clinical trial design, and treatment recommendations.

[0008] Brief Overview

[0009] An exemplary system for predicting the activity of molecular analytes in a patient includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: training a first module of a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; training the machine learning model based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and predicting the activity of the molecular analyte from the patient's medical images using the trained first and second modules of the machine learning model. In this disclosure, any machine learning model may be replaced by modules of a machine learning model, optionally having one or more heads. Each of the machine learning model or modules of a machine learning model may include a backbone and a head, which may include a final layer or set of layers (e.g., a neural network) in the model.

[0010] In some embodiments, the one or more procedures further include instructions for determining whether the patient belongs to one or more subgroups based on the predicted activity of the molecular analyte.

[0011] In some embodiments, the one or more programs further include instructions for training a third module of the machine learning model based on a third cohort, the third cohort including multiple medical images and associated clinical outcomes, wherein the third module of the machine learning model is configured to predict treatment and / or clinical outcomes.

[0012] In some embodiments, the one or more programs further include instructions for: using the third machine learning model to determine a measure of the significance or prognostic value of the molecular analyte in order to dynamically select a subset of the molecular analytes for subsequent use.

[0013] In some embodiments, the second module and / or the third module of the machine learning model are trained using transfer learning.

[0014] In some embodiments, the one or more molecular analyte datasets include: gene expression data; copy number amplification (CNA) data; amplification signature data; chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

[0015] In some embodiments, the one or more molecular analyte datasets include: gene expression values ​​containing transcript abundance; copy number amplification (CNA) data; amplification feature data; chromosome accessibility scores containing ATAC-seq peaks; abundance of one or more histone modifications containing ChIPseq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

[0016] In some embodiments, the one or more molecular analyte datasets may include two molecular analyte datasets.

[0017] In some embodiments, medical images from the patient are obtained from a fourth queue, which includes multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images.

[0018] In some embodiments, the one or more procedures further include instructions for determining that each patient in the fourth cohort belongs to one or more subgroups.

[0019] In some embodiments, the first queue includes multiple medical images from multiple patients.

[0020] In some embodiments, the plurality of medical images include: one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof.

[0021] In some embodiments, the plurality of medical images are unlabeled, and the first module is trained using unsupervised learning.

[0022] In some embodiments, the first queue and the second queue are the same queue.

[0023] In some embodiments, the second queue includes data on multiple medical images and one or more related molecular analytes.

[0024] In some embodiments, the third cohort includes multiple medical images and related clinical outcomes.

[0025] In some embodiments, the first, second, or third cohort may further include one or more clinical covariates.

[0026] In some embodiments, the one or more clinical covariates include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

[0027] In some embodiments, the one or more programs further include instructions for removing specific biases in the data from the first, second, and third queues.

[0028] In some embodiments, the one or more programs further include instructions for: receiving a medical image of a new patient; obtaining an embedding by providing the medical image of the new patient to the first module; and mapping the embedding based on domain adaptation.

[0029] In some embodiments, the molecular analyte is a first molecular analyte, and the one or more programs further include instructions for training a fourth module of the machine learning model based on the second module using transfer learning, wherein the fourth module is configured to predict a second molecular analyte associated with the first molecular analyte.

[0030] In some embodiments, the one or more programs further include instructions for calculating continuous scores.

[0031] In some embodiments, the second module for training the machine learning model includes: in a first stage, training a general module based on training data from the one or more molecular analyte datasets obtained from the second cohort; and in a second stage, fine-tuning the general module based on a subset of the training data to obtain the second module.

[0032] In some embodiments, the subset of training data corresponds to patient attributes.

[0033] In some embodiments, the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

[0034] In some embodiments, the patient has the patient attributes.

[0035] In some embodiments, a first module of the machine learning model is trained to generate tile-level embeddings based on multiple tiles of the medical image, and the tile-level embeddings are input into a second module of the machine learning model.

[0036] In some embodiments, at least a subset of the tile-level embeddings is averaged before being input into a second module of the machine learning model.

[0037] In some embodiments, the second module of the machine learning model includes an attention mechanism.

[0038] In some embodiments, the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

[0039] In some embodiments, the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

[0040] An exemplary method for predicting the activity of a molecular analyte in a patient includes: a first module that trains a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module that trains the machine learning model based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analyte from the patient's medical images.

[0041] An exemplary non-transient computer-readable storage medium stores one or more programs for predicting the activity of molecular analytes in a patient. The one or more programs include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform: a first module training a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module training the machine learning model based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analyte from the patient's medical images.

[0042] An exemplary system for predicting the activity of molecular analytes in a patient includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: training a first machine learning model on a plurality of medical images from a first cohort; training a second machine learning model on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from a second cohort; receiving medical images from the patient; and using the trained second machine learning model to predict the activity of the molecular analyte from the patient's medical images.

[0043] In some embodiments, the one or more procedures further include instructions for determining whether the patient belongs to one or more subgroups based on the predicted activity of the molecular analyte.

[0044] In some embodiments, the one or more programs further include instructions for training a third machine learning model based on a third cohort, the third cohort comprising multiple medical images and associated clinical outcomes, wherein the third machine learning model is configured to predict treatment and / or clinical outcomes.

[0045] In some embodiments, the one or more programs further include instructions for: using the third machine learning model to calculate a measure of the significance or prognostic value of the molecular analyte to dynamically select a subset of the molecular analytes for subsequent use.

[0046] In some embodiments, the second machine learning model and / or the third machine learning model are trained using transfer learning.

[0047] In some embodiments, the one or more molecular analyte datasets include: gene expression data; copy number amplification (CNA) data; amplification signature data; chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

[0048] In some embodiments, the one or more molecular analyte datasets include: gene expression values ​​containing transcript abundance; copy number amplification (CNA) data; amplification feature data; chromosome accessibility scores containing ATAC-seq peaks; abundance of one or more histone modifications containing ChIPseq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

[0049] In some embodiments, the one or more molecular analyte datasets may include two molecular analyte datasets.

[0050] In some embodiments, medical images from the patient are obtained from a fourth queue, which includes multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images.

[0051] In some embodiments, the one or more procedures further include instructions for determining that each patient in the fourth cohort belongs to one or more subgroups.

[0052] In some embodiments, the first queue includes multiple medical images from multiple patients.

[0053] In some embodiments, the plurality of medical images include: one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof.

[0054] In some embodiments, the plurality of medical images are unlabeled, and the first machine learning model is trained using unsupervised learning.

[0055] In some embodiments, the first queue and the second queue are the same queue.

[0056] In some embodiments, the second queue includes data on multiple medical images and one or more related molecular analytes.

[0057] In some embodiments, the third cohort includes multiple medical images and related clinical outcomes.

[0058] In some embodiments, the first, second, or third cohort may further include one or more clinical covariates.

[0059] In some embodiments, the one or more clinical covariates include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

[0060] In some embodiments, the one or more programs further include instructions for removing specific biases in the data from the first, second, and third queues.

[0061] In some embodiments, the one or more programs further include instructions for: receiving medical images of a new patient; obtaining an embedding by providing the medical images of the new patient to the first machine learning model; and mapping the embedding based on domain adaptation.

[0062] In some embodiments, the molecular analyte is a first molecular analyte, and the one or more procedures further include instructions for training a fourth machine learning model based on the second machine learning model using transfer learning, wherein the fourth machine learning model is configured to predict a second molecular analyte associated with the first molecular analyte.

[0063] In some embodiments, the one or more programs further include instructions for calculating continuous scores.

[0064] In some embodiments, the second module for training the machine learning model includes: in a first stage, training a general module based on training data from the one or more molecular analyte datasets obtained from the second cohort; and in a second stage, fine-tuning the general module based on a subset of the training data to obtain the second module.

[0065] In some embodiments, the subset of training data corresponds to patient attributes.

[0066] In some embodiments, the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

[0067] In some embodiments, the patient has the patient attributes.

[0068] In some embodiments, a first module of the machine learning model is trained to generate tile-level embeddings based on multiple tiles of the medical image, and the tile-level embeddings are input into a second module of the machine learning model.

[0069] In some embodiments, at least a subset of the tile-level embeddings is averaged before being input into a second module of the machine learning model.

[0070] In some embodiments, the second module of the machine learning model includes an attention mechanism.

[0071] In some embodiments, the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

[0072] In some embodiments, the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

[0073] An exemplary method for predicting the activity of a molecular analyte in a patient includes: training a first machine learning model on a plurality of medical images from a first cohort; training a second machine learning model on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from a second cohort; receiving medical images from the patient; and using the trained second machine learning model to predict the activity of the molecular analyte from the patient's medical images.

[0074] An exemplary non-transient computer-readable storage medium stores one or more programs for predicting the activity of molecular analytes in a patient. The one or more programs include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: train a first machine learning model on a plurality of medical images from a first cohort; train a second machine learning model on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from a second cohort; receive medical images from the patient; and predict the activity of the molecular analyte from the patient's medical images using the trained second machine learning model. An exemplary system for patient stratification includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a first plurality of medical images from a first cohort; determining a plurality of embeddings by providing the first plurality of images to a first trained machine learning model; training a second machine learning model to predict one or more molecular analytes by providing the plurality of embeddings from the first machine learning model and activity data of the one or more molecular analytes from the first cohort to a second machine learning model; predicting inferred activity data of the one or more molecular analytes from the second cohort by providing the trained second machine learning model with a second plurality of medical images from a second cohort; identifying one or more relevant biomarkers based on the inferred activity data of the second cohort and the result data of the second cohort; receiving one or more medical images of a patient; and determining whether the patient belongs to one or more patient subgroups based on the presence of the one or more relevant biomarkers.

[0075] In some embodiments, the first queue is smaller than the second queue.

[0076] In some embodiments, the activity data of one or more molecular analytes in the first cohort and / or the extrapolated activity data of the second cohort include: gene expression data; copy number amplification (CNA) data; amplification signature data; chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

[0077] In some embodiments, the activity data of the one or more molecular analytes in the first cohort and / or the extrapolated activity data of the second cohort include: gene expression values ​​containing transcript abundance; copy number amplification (CNA) data; amplification signature data; chromosome accessibility score containing ATACseq peaks; abundance of one or more histone modifications containing ChIP-seq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

[0078] In some embodiments, the first plurality of images in the first queue and / or the second plurality of images in the second queue include: one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof.

[0079] In some embodiments, data associated with the second queue is collected as part of Standard Care (SoC).

[0080] In some embodiments, the data associated with the second cohort includes Cancer Genome Atlas (TCGA) data.

[0081] In some embodiments, the first trained machine learning model includes an unsupervised model or a self-supervised model.

[0082] In some embodiments, the first trained machine learning model includes a contrastive model.

[0083] In some embodiments, the second machine learning model is a linear model.

[0084] In some embodiments, the extrapolated activity data are correlated with ATAC-seq peaks.

[0085] In some embodiments, identifying the one or more relevant biomarkers includes: using a third machine learning model to determine the association between the inferred activity data of the second cohort and the result data of the second cohort.

[0086] In some embodiments, determining the association includes: training the third machine learning model configured to predict outcomes based on the activity data of the molecular analyte using the inferred activity data and the outcome data of the second cohort; and determining a correlation metric indicating the degree of correlation between the activity data of the molecular analyte and the clinical outcome.

[0087] In some embodiments, the relevant metric includes a p-value associated with the third machine learning model.

[0088] In some embodiments, the one or more biomarkers include machine learning-based biomarkers or image-based biomarkers.

[0089] In some embodiments, determining whether a patient belongs to one or more patient subgroups includes: determining one or more embeddings by providing the one or more images of the patient to the first machine learning model; determining inferential activity data associated with the patient by providing the one or more embeddings to the trained machine learning model; and determining whether the inferential activity data associated with the patient indicates the presence of one or more biomarkers.

[0090] In some embodiments, the one or more programs further include instructions for identifying a treatment for the patient based on the one or more biomarkers and a known mechanism of action (MoA) of the treatment.

[0091] In some embodiments, the outcome data indicates mortality, disease diagnosis, disease progression, disease prognosis, disease risk, or any combination thereof, and wherein patient stratification is based on one or more of mortality, disease diagnosis, disease progression, disease prognosis, and disease risk, or any combination thereof.

[0092] In some embodiments, the one or more programs further include instructions for calculating continuous scores.

[0093] In some embodiments, the second module for training the machine learning model includes: in a first stage, training a general module based on training data from the one or more molecular analyte datasets obtained from the second cohort; and in a second stage, fine-tuning the general module based on a subset of the training data to obtain the second module.

[0094] In some embodiments, the subset of training data corresponds to patient attributes.

[0095] In some embodiments, the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

[0096] In some embodiments, the patient has the patient attributes.

[0097] In some embodiments, a first module of the machine learning model is trained to generate tile-level embeddings based on multiple tiles of the medical image, and the tile-level embeddings are input into a second module of the machine learning model.

[0098] In some embodiments, at least a subset of the tile-level embeddings is averaged before being input into a second module of the machine learning model.

[0099] In some embodiments, the second module of the machine learning model includes an attention mechanism.

[0100] In some embodiments, the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

[0101] In some embodiments, the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

[0102] An exemplary method for patient stratification includes: receiving a first plurality of medical images from a first cohort; determining a plurality of embeddings by providing the first plurality of images to a first trained machine learning model; training a second machine learning model to predict one or more molecular analytes by providing the plurality of embeddings from the first machine learning model and activity data of one or more molecular analytes from the first cohort to a second machine learning model; predicting inferred activity data of the one or more molecular analytes from the second cohort by providing the trained second machine learning model with a second plurality of medical images from a second cohort; identifying one or more relevant biomarkers based on the inferred activity data from the second cohort and outcome data from the second cohort; receiving one or more medical images of a patient; and determining whether the patient belongs to one or more patient subgroups based on the presence of the one or more relevant biomarkers.

[0103] An exemplary non-transient computer-readable storage medium stores one or more programs for predicting the activity of molecular analytes in a patient. The one or more programs include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: receive a first plurality of medical images from a first cohort; determine a plurality of embeddings by providing the first plurality of images to a first trained machine learning model; train a second machine learning model to predict one or more molecular analytes by providing the plurality of embeddings from the first machine learning model and activity data of the one or more molecular analytes from the first cohort to a second machine learning model; predict inferred activity data of the one or more molecular analytes from the second cohort by providing the trained second machine learning model with a second plurality of medical images from a second cohort; identify one or more relevant biomarkers based on the inferred activity data from the second cohort and the result data from the second cohort; receive one or more medical images of a patient; and determine whether the patient belongs to one or more patient subgroups based on the presence of the one or more relevant biomarkers. Attached Figure Description

[0104] Figure 1AAn exemplary platform is shown that, according to some embodiments, utilizes machine learning techniques to bridge the gap between research biological data and real-world biological data.

[0105] Figure 1B An exemplary process is shown, according to some embodiments, of using machine learning techniques to bridge the gap between research biological data and real-world biological data.

[0106] Figure 2A An exemplary process for discovering relevant biomarkers and patient stratification according to some embodiments is shown.

[0107] Figure 2B An exemplary process for discovering relevant biomarkers and patient stratification according to some embodiments is shown.

[0108] Figure 3A The training of an exemplary first machine learning model according to some embodiments is shown.

[0109] Figure 3B The use of a first machine learning model, as exemplarily demonstrated according to some embodiments, is shown.

[0110] Figure 4A The training of an exemplary second machine learning model according to some embodiments is shown.

[0111] Figure 4B The use of an exemplary second machine learning model trained according to some embodiments is shown.

[0112] Figure 5 An exemplary process is shown, according to some embodiments, for identifying one or more relevant biomarkers using a trained first machine learning model and a trained second machine learning model.

[0113] Figure 6 An exemplary process for extrapolating molecular analyte activity measurements from histopathological embeddings is shown according to some embodiments.

[0114] Figure 7A and 7B The chromatin accessibility significantly predicted from histopathological embeddings is shown according to some embodiments.

[0115] Figure 7C Examples of gene-based risk stratification are shown according to some implementation methods.

[0116] Figure 8 This illustrates how ATAC-seq signals extrapolated based on some embodiments can help identify novel outcome-related genes.

[0117] Figure 9 Exemplary verification data according to some embodiments is shown.

[0118] Figure 10 Exemplary processes according to some embodiments are shown.

[0119] Figure 11 Exemplary processes according to some embodiments are shown.

[0120] Figure 12 An exemplary electronic device according to some embodiments is shown.

[0121] Figure 13 The ability of an exemplary system to predict somatic mutations from embeddings is shown according to some embodiments.

[0122] Figure 14A The training and use of a second machine learning model for directly predicting copy number amplification, according to some embodiments, are illustrated.

[0123] Figure 14B The training and use of a second machine learning model for predicting gene expression are illustrated according to some embodiments.

[0124] Figure 14C The training and use of a second machine learning model for predicting gene features associated with copy number amplification, according to some embodiments, are illustrated.

[0125] Figure 15 An exemplary method for predicting the activity of molecular analytes by fine-tuning a general model to a specialized model based on a subset of training data, according to some embodiments, is shown.

[0126] Figure 16 The proportion of target genes with at least a threshold prevalence rate is shown according to some embodiments.

[0127] Figure 17A The distribution of the area under the receiver operating characteristic curve (AUROC) according to some embodiments is shown.

[0128] Figure 17B The mean area under the receiver operating characteristic (AUROC) curve is shown according to some embodiments.

[0129] Figure 18A and 18B The area under the receiver operating characteristic (AUROC) curve for predicting copy number amplification (CNA) by cancer type stratification according to some embodiments is shown.

[0130] Figure 19 The expression differences between patients with and without CNA in 347 targets with available RNA are shown according to some embodiments.

[0131] Figure 20 The pan-cancer observation and prediction expression matrix is ​​shown according to some embodiments.

[0132] Figure 21 The comparison between the observed expression / feature matrix and the matrix based on histopathological prediction, stratified by cancer type, is shown according to some embodiments.

[0133] Figure 22A The distribution of the correlation between observed and predicted expression levels across target patients is presented according to some embodiments.

[0134] Figure 22B The average correlation by cancer type is shown according to some embodiments.

[0135] Figure 23A and 23B The diagram illustrates the stratification of amplification features predicted from digital histopathology by cancer type, based on some embodiments.

[0136] Figure 24A and 24B The AUROC and AUPRC stratification by cancer type based on digital histopathology prediction of elevated target expression is shown according to some embodiments.

[0137] Figure 25 The distribution of characteristic scores in patients with and without amplification is shown according to some embodiments.

[0138] Figure 26 The average characteristic scores of patients with amplification (cases) and without amplification (controls) are shown according to some embodiments.

[0139] Figure 27 The distribution of the correlation between cross-target and pan-cancer amplification features and amplified gene expression is shown according to some embodiments.

[0140] Figure 28 The quadratic correlation between amplification characteristics and amplified gene expression is shown according to some embodiments, pan-cancer.

[0141] Figure 29 The diagram illustrates the number of targets predicted by AUROC exceeding a given threshold for binary classification tasks involving CNA, target representation, and amplified features, according to some embodiments.

[0142] Figure 30 The biomarker counts exceeding various cutoff values ​​according to some embodiments are summarized.

[0143] Figure 31A The performance of models trained to predict overexpression or elevated amplification features in the MET case is shown according to some embodiments, as well as the performance when the model is further specialized for prediction within NSCLC and the performance of models trained for colorectal cancer prediction.

[0144] Figure 31BThe performance of models trained to predict overexpression or elevated amplification features in the TACSTD2 case, according to some embodiments, is shown, as well as the performance when the model is further specialized for prediction within NSCLC and the performance of models trained for pancreatic cancer prediction.

[0145] Figure 32 The covariate-adjusted Kaplan-Meier curves are shown for comparing patients with low and high VEGFR2 characteristic scores according to some embodiments.

[0146] Figure 33A Examples of synthesized superpositions according to some embodiments are shown to locate HER2 expression in breast cancer and MET expression in colorectal cancer.

[0147] Figure 33B An example of synthetic overlay based on amplification feature prediction according to some embodiments is shown.

[0148] Figure 34 A comparison of expression and characterization predictions in breast cancer based on some examples with annotations from expert pathologists is described.

[0149] Figure 35 A comparison of expression and characterization predictions in colorectal cancer based on some examples with annotations by expert pathologists is described.

[0150] Figure 36 Examples of HER3 plus MET co-expression prediction and blinded pathologist annotation based on some implementations are provided.

[0151] Figure 37 Examples of TOP1+TOP2A co-expression prediction and blinded pathologist annotations based on some embodiments are shown.

[0152] Figure 38A and 38B Cross-modal comparisons of the predictive quality of binary digital biomarkers stratified by cancer type according to some embodiments are described.

[0153] Figure 39A and 39B The diagram illustrates how target expression levels can be predicted from digital histopathology and stratified by cancer type, based on some embodiments.

[0154] Figure 40A and 40B The diagram illustrates how elevated amplification features are predicted from digital histopathology and stratified by cancer type according to some embodiments.

[0155] Figure 41 The performance of a binary classification task model stratified by cancer type across biomarkers is shown according to some embodiments.

[0156] Figure 42The target counts exceeding a given threshold are shown for the pan-cancer and hierarchical binary classification task AUPRC, according to some embodiments.

[0157] Figure 43 The gene counts that exceed a given threshold according to some embodiments are shown.

[0158] Figure 44A and 44B Cross-modal comparisons of the predictive quality of continuous digital biomarkers stratified by cancer type according to some embodiments are shown.

[0159] Figure 45 Performance on regression tasks across biomarkers, pan-cancer, and two specific cancer types is shown according to some embodiments.

[0160] Figure 46 The target count for Pearson and Spearman R2 exceeding a given threshold is shown according to some embodiments.

[0161] Figure 47 The diagram illustrates the target count for Pearson and Spearman R2 exceeding a given threshold for hierarchical regression tasks, according to some embodiments.

[0162] Figure 48A and 48B Cross-modal comparisons of the predictive quality of digital biomarkers stratified by cancer type according to some embodiments are shown.

[0163] Figure 49 The prevalence of any targeted amplification stratified by cancer type and any amplification feature is shown according to some embodiments.

[0164] Figure 50 The characteristic distribution by patient amplification status is shown. The distribution shown is the average distribution of up to 351 amplification features according to some embodiments.

[0165] Figure 51 The pan-cancer distribution is illustrated with the correlation between amplification features and amplified gene expression according to some embodiments.

[0166] Figure 52 The hierarchical squared correlation between amplification features and amplified gene expression is shown according to some embodiments.

[0167] Detailed description

[0168] The following description is intended to enable those skilled in the art to make and use various embodiments. The descriptions of specific devices, technologies, and applications are provided by way of example only. Various modifications to the examples described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other examples and applications without departing from the spirit and scope of the various embodiments. Therefore, the various embodiments are not intended to be limited to the examples described and shown herein, but should be accorded the scope consistent with the claims.

[0169] This document discloses exemplary devices, apparatuses, systems, methods, and non-transient storage media for discovering relevant biomarkers and patient stratification using an artificial intelligence (AI) platform. Embodiments of this disclosure can bridge the gap between feature-rich but small-scale study cohorts and large-scale real-world cohorts whose data is collected as part of a System of Clinical Assurance (SoC), allowing the use of SoC data to discover new clinical insights despite data gaps. To this end, the system utilizes shared data modalities between the two cohorts, such as histopathological data (e.g., from H&E biopsies or trichrome staining samples), MRI data, CT scans, X-rays, and continuous monitoring data—data types collected for both cohorts. The system can train an imputation model using data from the study cohort. The imputation model can receive embedded data from subjects (e.g., histopathological embeddings) and predict molecular analyte activity data for subjects. The trained imputation model can be applied to process data from the SoC cohort to obtain imputed molecular analyte activity data for the SoC cohort, thereby revealing new clinical insights, such as identifying relevant biomarkers, performing patient stratification, identifying clinical trial patients, and identifying treatment methods based on known MoA, as discussed herein.

[0170] In some embodiments, the system can train a first machine learning model (or a first module of a machine learning model), such as a self-supervised or unsupervised model, configured to receive input data from a shared modality across cohorts and output embedded data. The embedded data is a numerical, low-dimensional feature of the input data that can support downstream analysis. The system can then train a second machine learning model (or a second module of a machine learning model), configured to receive embedded data from a given patient and output predicted activity data for one or more molecular analytes for that patient. Importantly, the second machine learning model can be trained using data from a study cohort, as molecular analyte activity data is available for the study cohort. Once trained, the second machine learning model can be used to obtain imputed molecular analyte activity data for a larger SoC cohort that has never collected molecular analyte activity data. Therefore, the second machine learning model allows for large-scale imputation of study modalities from SoC modalities and can learn fine-grained phenotypes. In some embodiments, the first and second machine learning models can be implemented as a first module (i.e., an embedding module) and a second module (i.e., an imputation module) of the machine learning model.

[0171] Imputation activity data, combined with raw data collected for the SoC cohort (e.g., longitudinal clinical outcome data), can be used to reveal new clinical insights, such as discovering relevant biomarkers (e.g., machine learning-based biomarkers, image-based biomarkers) to improve clinical development and leveraging human genetics to identify high-confidence therapeutic targets. Relevant biomarkers may include biological processes highly correlated with patient outcomes or treatment response (ideally causally related) and modulated using existing therapeutic interventions whose MoA directly targets that biological process. Furthermore, relevant biomarkers can be accurately and robustly predicted from data measured as part of the SoC using machine learning methods (e.g., first and second machine learning models). An exemplary relevant biomarker might be the aberrant activation of a given gene driving tumor proliferation, where a therapeutic agent inhibiting that gene is present, and where the activation of that gene can be detected from histopathological images (e.g., for machine learning methods). Another exemplary relevant biomarker might be the infiltration (or lack thereof) of a specific type of cell into the tumor microenvironment (TME), where the intervention might be the modulation of specific cell migration signaling proteins.

[0172] In some embodiments, the shared data modality includes histopathological images. Given the near-universal availability of H&E image collection and the richness of information in this data modality, histopathological images allow the system to discover robust, H&E-based predictive biomarkers for patient selection across a range of targeted cancer therapies. The discovered biomarkers may be more precise than broad patient demographics, providing higher effect sizes for the targeted patient population, and more inclusive than patient selection based solely on somatic mutations, as it will also include other processes converging to the same biology (e.g., phenotypic copy number). Exemplary biomarkers disclosed herein include ATAC-based biomarkers that provide significantly higher hazard ratios (HRs) than other risk stratification biomarkers, and also higher than those obtained from patient selection based on copy number alteration (CNA).

[0173] While some embodiments of this disclosure target ATAC peaks (and thus genomic activation) imputed from H&E, the same approach can be more broadly applied to other molecular readouts and other shared data modalities. For example, RNA abundance, extensive proteomics, space biology, or other data modalities can be measured. The system can also impute not only H&E readouts but also IHC and / or genetically enhanced histopathological images (both of which are rapidly becoming SoCs for many cancer types). A key aspect is that the MoA of (at least some) therapeutic interventions can be directly mapped to readouts in these imputed detections, just as the MoA of a repressive driver gene corresponds to an ATAC readout showing that gene is activated. For example, the system can use imputed space biology, RNA, or proteomics to identify patient populations where specific cell types infiltrate the TME and are associated with poor prognosis, and align them with MoA regulating cell transport or MoA of depleted cell types.

[0174] Therefore, embodiments of this disclosure provide oncology discovery driven by multimodal clinical data. Exemplary systems can leverage unsupervised machine learning histological phenotypes (e.g., embeddings) that capture rich, multi-scale tumor microenvironment structures, machine learning techniques (e.g., second machine learning models or imputation models) to imput clinical and genomic outcomes providing an additional layer of information, as well as data-driven assessments of the impact of genetic and genomic variations on clinical outcomes, to discover novel targets and biomarkers. Exemplary systems can utilize the techniques described herein to discover novel targets and biomarkers.

[0175] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that this description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0176] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first graphical representation may be called a second graphical representation, and similarly, a second graphical representation may be called a first graphical representation, without departing from the scope of the various described embodiments. Both the first graphical representation and the second graphical representation are graphical representations, but they are not the same graphical representation.

[0177] The terminology used in the descriptions of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the descriptions of the various embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. Further understanding is that the terms "comprising," "including," "comprise," and / or "comprising," when used in this specification, specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0178] The term "if" may optionally be interpreted as "when" or "at" or "in response to determination" or "in response to detection," depending on the context. Similarly, the phrase "if determination" or "if [the condition or event] is detected" may optionally be interpreted as "when determination" or "in response to determination" or "when [the condition or event] is detected" or "in response to the detection of [the condition or event]," depending on the context.

[0179] Figure 1A An exemplary platform is shown that, according to some embodiments, utilizes machine learning techniques to bridge the gap between research biological data and real-world biological data. Figure 1A Two groups of subjects or patients are depicted: Cohort 102 and Cohort 112. Cohort 102 may be a relatively small cohort whose organization collects rich biological information, which may require specialized equipment and settings, typically for research purposes. In contrast, Cohort 112 may be a larger patient population whose data are collected in a real-world standard care (SoC) setting. For example, Cohort 112 may include data collected from patients as part of receiving medical care and treatment. As described below, the data collected for Cohort 102 and the data collected for Cohort 112 may share modalities but also differ in many ways.

[0180] refer to Figure 1AData collected for cohort 102 and data collected for cohort 112 may share a modality. A shared modality refers to the type of data collected for cohort 102 (e.g., for research purposes) and cohort 112 (e.g., as part of a SoC). For example, a shared modality may include histopathological images, magnetic resonance imaging (MRI) images, computed tomography (CT) scans, continuous monitoring data (e.g., EEG, EKG, continuous glucose monitoring, activity monitoring via accelerometer), etc.

[0181] The data collected for cohort 102 differs from that collected for cohort 112 in many ways. For example, the data collected for cohort 102 (e.g., the research cohort) may include rich high-dimensional molecular content, which may require specialized equipment and setups, such as high-content detection. For example, the data may include gene expression data, copy number amplification values ​​(e.g., from WGS, WGBS, or targeted sequencing), amplification signatures (e.g., RNA-seq), chromatin accessibility data (e.g., ATAC-seq), DNA methylation data (e.g., WGBS, RRBS), histone modification data (e.g., histone ChIPseq), RNA data (e.g., from RNA-seq), protein data, space biology data, whole-genome sequencing (WGS) data (e.g., GWAS), somatic mutation data (e.g., sequence data), germline mutation data (e.g., sequence data), or any combination thereof.

[0182] In some embodiments, the data may specifically include: gene expression values ​​containing transcript abundance, chromosome accessibility scores containing ATAC-seq peaks, abundance of one or more histone modifications containing ChIP-seq values, abundance of one or more mRNA sequences, abundance of one or more proteins, presence of one or more somatic mutations, presence of one or more germline mutations, or any combination thereof.

[0183] However, the data collected for Cohort 102 may be small enough to support robust biomarker discovery. Data collected for Cohort 102 may also lack clinical outcome data entirely. An exemplary dataset collected for Cohort 102, rather than Cohort 112, might be ATAC-seq (Assay for Transposase Accessible Chromatin Using Sequencing) data. ATAC-seq data includes molecular measurements that can provide important insights. ATAC-seq measures chromatin accessibility, i.e., the "activity" of genomic fragments. ATAC-seq data can reveal epigenetic mechanisms of low-expression driver genes, non-coding driver mutations, and / or treatment resistance. ATAC-seq is generally highly sensitive to sample quality and is not used clinically, but is only available in limited-scale research datasets. In other words, ATAC-seq data was not collected for Cohort 112 as part of the SoC. Therefore, ATAC-seq data was collected on a smaller scale and may lack representativeness from a wide range of diseases.

[0184] In contrast, the data collected for cohort 112 is larger in scale and typically has longitudinal observations, as it is collected as part of a System of Clinical Investigations (SoC). Furthermore, the data may include high-density modalities generated in different disease contexts and include phenotypic content that is not typically suitable for R&D purposes. In some embodiments, the data collected for cohort 112 may include imaging data, molecular data, genetic data, and outcome data (e.g., mortality, disease diagnosis, disease progression, disease prognosis, disease risk, or any combination thereof, with patient stratification based on one or more of mortality, disease diagnosis, disease progression, disease prognosis, disease risk, etc.). An exemplary dataset associated with cohort 112 might be data from the Cancer Genome Atlas (TCGA) project. The TCGA project, initiated in 2006, involves over 20,000 tumor and normal samples and 33 cancer types. TCGA data is diverse but collected sporadically, including genetic data, histopathological images and other images, molecular covariates, clinical outcomes, etc.

[0185] Embodiments of this disclosure can bridge the gap between feature-rich but small-scale research cohorts (e.g., Figure 1A In cohort 102) and large-scale real-world patients (e.g., Figure 1AThe system addresses the gap between cohort 112 (collected as part of the SoC) and the study cohort, whose data allows for the discovery of new clinical insights using SoC data despite data gaps. To this end, the system leverages shared data modalities between the two cohorts, such as histopathological data (e.g., from H&E or trichrome stained biopsy samples), MRI data, CT scans, X-rays, and continuous monitoring data—data types collected for both cohorts. First, the system can train a first machine learning model (or a first module of a machine learning model), such as a self-supervised or unsupervised model, configured to receive input data from the shared modality and output embedded data. Embedded data are numerical, low-dimensional feature sets of the input data that can support downstream analysis. Second, the system can train a second machine learning model (or a second module of a machine learning model), configured to receive embedded data for a given patient and output predicted activity data for one or more molecular analytes for that patient. Importantly, the second machine learning model can only be trained using data from the study cohort, as molecular analyte activity data is available for the study cohort. Once trained, the second machine learning model can be used to obtain imputed molecular analyte activity data for a larger SoC cohort that has never had molecular analyte activity data collected. Therefore, the second machine learning model allows for large-scale interpolation of one or more research modalities from SoC modalities and can learn fine-grained phenotypes.

[0186] Imputation activity data, combined with raw data collected for the SoC cohort (e.g., longitudinal clinical outcome data), can be used to reveal new clinical insights, such as discovering relevant biomarkers (e.g., machine learning-based biomarkers, image-based biomarkers) to improve clinical development and leveraging human genetics to identify high-confidence targets, such as... Figure 1B As shown. Relevant biomarkers may include biological processes that are highly correlated (ideally causally related) to patient outcomes or treatment response and can be modulated using existing therapeutic interventions whose MoA directly targets this biology. Furthermore, relevant biomarkers can be accurately and robustly predicted from data measured as part of the SoC using machine learning methods (e.g., first and second machine learning models). An exemplary relevant biomarker might be the aberrant activation of a given gene driving tumor proliferation, where a therapeutic agent inhibiting that gene is present, and the activation of that gene is visible in histopathological images (e.g., for machine learning methods). Another exemplary relevant biomarker might be the infiltration (or lack thereof) of a specific type of cell into the tumor microenvironment (TME), where the intervention might be the modulation of specific cell migration signaling proteins. Figure 13 An exemplary system's ability to predict somatic mutations from embeddings is illustrated. As shown, a machine learning model is trained to predict tumor genotypes from histological embeddings. The predictive accuracy of the machine learning model (potentially a linear model) is comparable to that of a fully supervised model configured to receive and process histological data.

[0187] In some embodiments, the shared data modality includes histopathological images. Given the near-universal availability of H&E image collection and the richness of information in this data modality, histopathological images allow the system to discover robust, H&E-based predictive biomarkers for patient selection across a range of targeted cancer therapies. The discovered biomarkers may be more precise than broad patient demographics, providing higher effect sizes for the targeted patient population, and more inclusive than patient selection based solely on somatic mutations, as it will also incorporate other processes converging to the same biology. Exemplary biomarkers disclosed herein include ATAC-based biomarkers that provide significantly higher hazard ratios (HRs) than other risk stratification biomarkers, and also higher than those obtained from patient selection based on copy number alteration (CNA).

[0188] While some embodiments of this disclosure pertain to interpolating ATAC peaks from H&E (and thus genomic activation), the same approach can be more broadly applied to other shared data modalities. For example, large-scale proteomics or space biology measurements can be performed. The system can also be used not only from H&E... Imputation reads can also be enhanced with IHC and / or genetics (both of which are rapidly becoming SoCs for many cancer types). A key aspect is that the MoA of (at least some) therapeutic interventions can be directly mapped to reads in these imputation tests, just as the MoA of a suppressor of a driver gene corresponds to an ATAC read indicating that gene is activated. For example, the system can use the spatial biology of imputation to identify patient populations where specific cell types infiltrate the TME and are associated with poor prognosis, and align them with MoA that regulates cell transport or MoA of depletion-related cell types.

[0189] Therefore, embodiments of this disclosure provide oncology discovery driven by multimodal clinical data. Exemplary systems can leverage unsupervised or self-supervised machine learning histological phenotypes (e.g., embeddings), machine learning techniques (e.g., second machine learning models or imputation models) to capture rich, multi-scale tumor microenvironment structures to imput clinical and genomic covariates that provide additional layers of information, as well as data-driven assessments of the impact of genetic and genomic variations on clinical outcomes, to discover novel targets and biomarkers.

[0190] In some embodiments, the system may utilize co-embedding. For example, the system may use separate queues collecting both as training sets to align the embeddings of two different (in some embodiments, related) modalities. For example, the system may align the embeddings of the ATAC modality and the RNA modality.

[0191] In some embodiments, the system can identify predictive biomarkers for drugs within a patient cohort defined by demographic factors or known biomarkers (e.g., IHC biomarkers). In some embodiments, the system can identify predictive biomarkers for combination therapy of two or more drugs. In both cases, the drug's MoA needs to be directly aligned with an imputable molecular analyte, and the drug's biomarker will be trained to the corresponding analyte. However, the ability to assess biomarker efficacy within a subset of patients or as part of a combination therapy is achieved through the ability to imput biomarkers in large patient populations. This would allow for an "in-computer" process for selecting clinical trial designs using large real-world datasets.

[0192] In some embodiments, the techniques disclosed herein can be extended beyond cases where biomarkers are inferred entirely from the assumed MoA of a drug. The system can pre-train a model using molecular analytes and then fine-tune the weights using a limited patient cohort (e.g., from phase 1a or phase 2 clinical trials) where clinical outcomes of the drug are actually observed. For example, the system can pre-train a neural network to predict ATAC peaks from histopathological embeddings and then use the embedding layers as input to a machine learning model trained to clinical outcomes. The system can also reduce dimensionality and increase efficacy by focusing on a smaller set of analytes that may be relevant to drug outcomes (e.g., based on prior knowledge).

[0193] Figure 2A -B illustrates an exemplary process 200 for discovering relevant biomarkers and patient stratification according to some embodiments. Process 200 is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process 200 is performed using a client-server system, and the steps of process 200 are distributed between a server and one or more client devices in any way. In other examples, process 200 is performed using only client devices or only multiple client devices. In process 200, some steps may be optionally combined, the order of some steps may be optionally changed, and some steps may be optionally omitted. In some examples, additional steps may be performed in conjunction with process 200. Therefore, the operation as shown (and described in more detail below) is exemplary in nature and should not be considered limiting.

[0194] In block 202, the exemplary system receives a first plurality of medical images from a first queue. The first queue may be a small queue, such as... Figure 1A Queue 102 in the context of this data collection. As described above, the data collected for a small queue may include data of a shared modality, such as imaging data. For example, the first plurality of images may include one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof.

[0195] Each of the multiple medical images in the first cohort can be associated with an activity reading of one or more molecular analytes in the first cohort. In other words, data collected for a small cohort can also include high-content modalities where specific MoA (e.g., activity of a specific gene or process) can be identified from the data. For example, activity data of one or more molecular analytes in the first cohort in block 202 may include gene expression data, copy number amplification values ​​(e.g., from WGS, WGBS, or targeted sequencing), amplification signatures (e.g., from RNA-seq), chromatin accessibility data (e.g., ATAC-seq), DNA methylation data (e.g., WGBS, RRBS), histone modifications (e.g., histone ChIP-seq), RNA data (e.g., from RNA-seq), protein data, space biology data, whole-genome sequencing (WGS) data (e.g., GWAS), somatic mutation data, germline mutation data, or any combination thereof.

[0196] In some embodiments, the data may include: gene expression values ​​containing transcript abundance, copy number amplification values, amplification signatures, chromosome accessibility scores containing ATAC-seq peaks, abundance of one or more histone modifications containing ChIP-seq values, abundance of one or more mRNA sequences, abundance of one or more proteins, presence of one or more somatic mutations, presence of one or more germline mutations, presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

[0197] For example, for patients in the first cohort, activity data may include scalar values ​​(e.g., normalized / log-scaled read counts, p-values, or logarithmic fold changes) or base pair level signals (e.g., regression ATAC-seq signal shape) corresponding to that patient.

[0198] In block 204, the system determines multiple embeddings by providing a first plurality of images to a first trained machine learning model. The first machine learning model is configured to receive image data (e.g., a histopathological image or a portion thereof) and output embeddings. An embedding is a numerical, low-dimensional feature of the input image data. In some embodiments, the first machine learning model includes an unsupervised or self-supervised model. In some embodiments, the first machine learning model includes a contrastive model, such as SimCLR and SwAV. Contrastive learning models can extract embeddings from imaging data that have linear predictive power for biological endpoints or labels that may be assigned to such data (e.g., progression of a disease of interest), as described herein. A suitable contrastive learning model is trained to maximize the similarity between different enhanced embeddings from the same sample image and minimize the similarity between embeddings from different sample images. For example, the model can extract embeddings from images invariant to rotation, flipping, cropping, and color jitter. In some embodiments, the embeddings may be mean-aggregated and / or normalized prior to use for downstream analysis. In some embodiments, normalizing the embeddings includes performing a variance-stabilizing transformation, which may improve their ability to linearly predict biological endpoints or labels. As described herein, normalization can improve the performance of linear prediction models based on embedding fitting. In some embodiments, linear models fitted with standardized embeddings have similar or better predictive capabilities than supervised machine learning models and are more computationally efficient in terms of generation and application, as further described herein. The training and use of the first machine learning model in Figure 3A and 3B Detailed information is provided in the text.

[0199] In block 206, the system trains the second machine learning model to predict one or more molecular analytes by providing it with multiple embeddings from the first machine learning model and activity data of one or more molecular analytes from the first cohort. Specifically, for each patient in the first cohort, the system has one or more embeddings corresponding to the patient's image data (obtained from block 204), as well as the patient's molecular analyte activity data. This data can be used as the training dataset for the second machine learning model. Using this training dataset, the second machine learning model can be trained to receive embedding data for a given patient and predict the molecular analyte activity data for that given patient. The training and use of the second machine learning model are described in detail below. Figure 4A and 4B Detailed information is provided in the text.

[0200] In some embodiments, the second machine learning model is a linear model. For example, the second machine learning model might be a linear model configured to receive one or more embeddings for a given patient and predict molecular analyte activity data associated with ATAC-seq peaks for that given patient. For example, the predicted activity data might include scalar values ​​(e.g., normalized / log-scaled read counts, p-values, or LogFC) or base pair level signals (e.g., regressed ATAC-seq signal shape). It should be understood that the second machine learning model could be other types of models that can be trained using training data.

[0201] In some embodiments, the second machine learning model can be trained using transfer learning. For example, the system can first train the second model to predict per-gene activity using one modality (e.g., RNA-seq), and then fine-tune (i.e., "transfer") the model to predict a related modality (e.g., ATAC-seq). This option is particularly attractive if the cohort with RNA-seq is larger, but if ATAC-seq shows a stronger correlation with the outcome. As another example, transfer learning can be used if both cohorts have ATAC-seq data, but one exhibits batch effects, or one lacks outcome / response data.

[0202] In block 208, the system predicts interpolation activity data for one or more molecular analytes in the second cohort by feeding a second cohort of medical images from the second cohort to a trained second machine learning model. The second cohort may be larger than the first cohort. In some embodiments, the second cohort may be part of a large-scale cohort, such as... Figure 1A The second cohort, 112, is used to collect data such as imaging data, continuous monitoring data, and / or clinical outcome data (e.g., as part of the SoC). In some embodiments, the data collected for the second cohort includes Cancer Genome Atlas (TCGA) data.

[0203] As described above, the data collected for the second cohort may share a modality with the first cohort. For example, a second plurality of medical images in the second cohort may share the same modality as a first plurality of medical images in the first cohort, and may include one or more histopathological images, one or more magnetic resonance imaging (MRI) images, one or more computed tomography (CT) scans, or any combination thereof. Furthermore, the data collected for the second cohort has sufficient power to stratify relevant patient outcomes in a given disease (e.g., with sufficient mortality or response events). Outcome data may indicate mortality, treatment response, disease diagnosis, disease progression, disease prognosis, disease risk, or any combination thereof, with patient stratification based on one or more of mortality, disease diagnosis, disease progression, disease prognosis, and disease risk, or any combination thereof.

[0204] However, as mentioned above, the data collected for the second cohort may not include rich, high-dimensional molecular content, such as high-content detection, which may require specialized equipment and setup. Using the second machine learning model obtained in blocks 202-206, the system can predict the imputation activity data for the second cohort and use this imputation activity data for downstream analysis, as discussed in blocks 210-218.

[0205] The system then uses a second trained machine learning model to determine the imputation activity data for one or more molecular analytes in the second cohort. As described above, the second machine learning model can be configured to receive embedding data for a given patient and predict the molecular analyte activity data for that given patient. Therefore, for each patient in the second cohort, the system can receive the embedding data for patients in the second cohort and predict the imputation activity data for that patient in the second cohort. The generation of the imputation activity data is... Figure 5 Detailed description is provided.

[0206] In block 210, the system identifies one or more relevant biomarkers based on imputation activity data from the second cohort and outcome data from the second cohort. In some embodiments, the system uses a third machine learning model to determine the association between the imputation activity data and clinical outcome data from the second cohort. Specifically, the system can train the third machine learning model using the imputation activity data and clinical outcome data from the second cohort, and the model is configured to predict outcomes based on the activity data of the molecular analyte. For example, for each candidate biomarker (i.e., a specific activity of the molecular analyte, a specific molecular analyte), the system can train a candidate biomarker-specific prediction model, which is configured to receive data associated with the candidate biomarker and output predicted clinical outcomes (e.g., treatment response, time to progression, time to death).

[0207] The candidate biomarker-specific model is then evaluated to determine whether a significant association exists between the candidate biomarker and the clinical outcome. In some embodiments, the system may determine an association measure or correlation measure based on the model that indicates the degree of association or correlation between the candidate biomarker and the clinical outcome. Association measures and correlation measures are used interchangeably in this disclosure.

[0208] For example, the association metric or correlation metric might be a hazard ratio, a hazard proportion, or a p-value. In some embodiments, the hazard ratio can be estimated from a Cox proportional hazards model. More generally, the association between a candidate biomarker and time-to-event outcomes (e.g., overall survival, progression-free survival) can be quantified using a (weighted) log-rank test, a Cox proportional hazards model, an Aalen additive hazard model, or a parametric accelerated time to failure model.

[0209] As another example, the model might be a generalized linear regression model or a time-to-event regression model, and the system can calculate a p-value that associates one or more imputed molecular analytes with one or more clinical outcomes. In some embodiments, the p-value can be obtained using standard Wald, scoring, likelihood ratio, or Monte Carlo tests, and the effect size and standard error can be obtained using classical generalized linear model theory. The p-value indicates the association between the candidate biomarker and the clinical outcome.

[0210] Other association testing procedures can be performed to determine whether a significant association exists between a candidate biomarker and clinical outcomes. Association testing procedures can also be based on extensions of generalized linear models, such as linear mixture models, generalized estimating equations, or nonlinear models (random forests, SVMs, etc.). Further information on obtaining histopathological embeddings and performing association tests can be found in U.S. Provisional Application No. 63233707 entitled "DISCOVERY PLATFORM" and PCT Application No. PCT / US2022 / 075006 entitled "DISCOVERY PLATFORM," the contents of which are incorporated herein by reference for all purposes.

[0211] The system then identifies one or more related biomarkers based on the association. For example, the system can determine whether the p-value corresponding to a candidate biomarker exceeds a predefined threshold to determine if a significant association exists. If the p-value corresponding to a candidate biomarker exceeds the predefined threshold, the system can determine that the candidate biomarker is a related biomarker.

[0212] In some embodiments, association metrics may indicate positive or negative associations. In some embodiments, significant (e.g., statistically significant, exceeding a predefined threshold) positive or significant negative associations can be identified as relevant biomarkers.

[0213] In blocks 212 and 214, the system can perform patient stratification based on biomarkers identified in block 210. For example, features aligned with predicted MoA (e.g., histopathological features) define a "probable responder" patient population. In some embodiments, patient stratification can be based on one or more images of a particular patient. The system can determine whether a patient belongs to one or more patient subgroups by determining whether one or more images of the patient indicate alignment with one or more identified relevant biomarkers. In addition to discretized results (one or more discrete subgroups), the system can also return continuous scores (e.g., PD-L1 expression level, TMB, HER2 quantification by FISH, etc.) to the patient / physician.

[0214] Specifically, in block 212, the system receives one or more medical images of the patient. The system may provide one or more images of the patient to a first trained machine learning model to determine one or more embeddings. The system may then provide one or more embeddings to a second trained machine learning model to determine imputation activity data associated with the patient.

[0215] In block 214, the system determines whether a patient belongs to one or more patient subgroups based on the presence of one or more relevant biomarkers. Specifically, if inferred activity data associated with a patient indicates the presence of a relevant biomarker, the system can determine that the patient likely belongs to a patient subgroup associated with that biomarker.

[0216] The system can identify treatments for patients based on one or more biomarkers and a known mechanism of action (MoA) of the treatment. An exemplary relevant biomarker might be the aberrant activation of a given gene driving tumor proliferation, where a therapeutic agent inhibiting that gene is present, and the activation of that gene is visible in histopathological images (e.g., for machine learning methods). Another exemplary relevant biomarker might be the infiltration (or lack thereof) of a specific type of cell into the tumor microenvironment (TME), where the intervention might be the modulation of a specific cell migration signaling protein. Yet another exemplary biomarker might be an aberrant change in the sequence or structure of a protein product (including missense variants, protein truncation, splicing variants, or fusions of different genes), where this change is visible in histopathological images, and a therapeutic agent selectively targeting the mutant protein rather than the wild-type is present.

[0217] In some embodiments, the system may employ a "compound-first" approach to deploy the techniques described herein to accelerate the pathway to patient impact. The system may first identify a group of targeted therapies in a biopharmaceutical pipeline that have a well-defined MoA (e.g., a well-defined target) and are likely to alter cancer. This group may include cancer drugs, but may also include drugs from other therapeutic areas, such as fibrosis or immunology. The system may then test each target in this group to determine (1) whether the activity of the target can be well extrapolated from the histopathological embedding, and (2) whether the extrapolated activity is significantly correlated with clinical outcomes. If so, the target is identified as a relevant biomarker.

[0218] To determine whether the activity of a target can be well extrapolated from histopathological embeddings, the system can compare the activity data predicted by a second machine learning model with the actual activity data. In some embodiments, if the difference between the predicted and actual activity data does not exceed a predefined threshold, the system can determine that the activity is well extrapolated. This requires a cohort for measuring actual activity. In some embodiments, when the second machine learning model is applied to a new cohort with a limited molecular spectrum available, an additional model can be trained to calibrate the output of the second machine learning model for the new cohort.

[0219] To determine whether the inferred activity is significantly related to clinical outcomes, the system can determine a correlation metric between the activity data and the clinical outcome. In some embodiments, the correlation metric may be as described above. Figure 2A and 2B The calculation. In some embodiments, the association metric includes a hazard ratio, and the system can determine whether the extrapolated activity provides a significant predictive hazard ratio in certain cancers (e.g., significantly correlated with time to progression or time to death), as described above.

[0220] The system can then assess whether the relevant biomarker offers an advantage over (if any) currently used in clinical trials or clinical settings in identifying patients who may benefit from the treatment. For example, it might suggest a new cancer type that was not previously a target of the treatment, allowing for indication expansion. As another example, the hazard ratio might be quite high for a given subset of patients, allowing for earlier use of the treatment in the SoC. As yet another example, the biomarker may have a fairly large patient population, making population expansion possible. As yet another example, currently used biomarkers require expensive or not always performed additional testing, which can be avoided using the proposed system, enabling population expansion.

[0221] Figure 3A The training of an exemplary first machine learning model according to some embodiments is illustrated. In the depicted example, the first machine learning model may be a contrastive learning algorithm, or... Figure 3A One of the encoders in the model. In some embodiments, the first machine learning model may be one or more diffusion models. The first machine learning model can be trained using a training dataset associated with a large cohort of subjects with a large amount of image data. For example, the training dataset may be associated with a large patient cohort that has collected histopathological images. The training dataset does not need to include any other covariates, as it is only used for the purpose of training the first machine learning model to convert image data into embeddings. In some embodiments, the cohort may be neither... Figure 1A Queue 102 in the text is not queue 112.

[0222] Contrastive learning can refer to a machine learning technique used to learn general features of a dataset without labels by teaching the model which data points are similar or different. A contrastive learning model can extract embeddings from imaging data that have linear predictive power for the labels that might be assigned to such data. A suitable contrastive learning model is trained by minimizing a contrastive loss that maximizes the similarity between different augmented embeddings from the same sample image and minimizes the similarity between embeddings from different sample images. For example, the model can extract tile embeddings from tile images that are invariant to rotation, flipping, cropping, and color jitter. Exemplary contrastive learning models include SimCLR and SwAV, but it should be understood that any representation learning algorithm can be used as a first machine learning model.

[0223] refer to Figure 3A During training, the original image X is obtained. Data transformation or augmentation can be applied to the original image X to obtain two augmented images. and For example, the system can randomly apply two separate data augmentation operators (e.g., crop, flip, color dithering, grayscale, blur) to obtain... and .

[0224] Two enhanced images and Each encoder is passed through an encoder to obtain a corresponding vector representation in the latent space. In the depicted example, the two encoders share weights. In some examples, each encoder is implemented as a neural network. For example, the encoder can be implemented using a variant of the Residual Neural Network ("ResNet") architecture. As shown, the outputs of the two encoders... (encoder from) (output vector) and (encoder from) The output vector.

[0225] Two vectors represent and Two projections are obtained by passing through the projection head. and In some examples, the projection head includes a series of nonlinear layers (e.g., Dense-ReLU-Dense layers) to apply nonlinear transformations to the vector representation to obtain the projection. The projection head amplifies invariant features and maximizes the network's ability to recognize different transformations of the same image.

[0226] During training, two different projections of the same image and The similarity between them is maximized. For example, based on and The loss is calculated, and the encoder is updated based on the loss to maximize the similarity between the two latent representations. In some examples, to maximize the consistency (i.e., similarity) between z-projections, the system can define the similarity metric as cosine similarity:

[0227] In some examples, the system trains the network by minimizing the normalized temperature-scaled cross-entropy loss:

[0228] in This represents an adjustable temperature parameter. Therefore, through training, the encoder learns to output a vector representation that preserves invariant features of the input image while minimizing image-specific features (e.g., imaging angle, resolution, artifacts).

[0229] In some embodiments, the embeddings are normalized before further processing and then rescaled by the inverse of the square root of the number of embedding dimensions. Normalization can improve the performance of linear prediction models based on embedding fits, as described herein.

[0230] Figure 3B The use of an exemplary first machine learning model configured to convert image data into an embedding, according to some embodiments, is illustrated. Model 304 may be... Figure 2A The first machine learning model used. In some embodiments, model 304 is an unsupervised model or a self-supervised model. Figure 3B As shown, the machine learning model 304 is configured to receive an input image 302 and provide an output embedding 306. The embedding 306 can be a vector representation of the input image 302 in the latent space. Converting the input image into an embedding can significantly reduce the size and dimensionality of the original data. In an exemplary implementation, the size is 224 pixels. The pixel image can be reduced to a 2,048-dimensional vector. Low-dimensional embeddings can be used for downstream processing, as described below.

[0231] Figure 4A An exemplary training process for a second machine learning model according to some embodiments is shown. The training process is... Figure 2A Examples of blocks 202-206. Figure 4AIn the depicted example, images from a first cohort are provided to a first machine learning model 304, which outputs embeddings. For example, a histopathological image 402 is provided to the first machine learning model 402 to obtain an embedding 406, which is a low-dimensional representation of the histopathological image 402. The system can also access known activity data 408 of patients in the first cohort. In other words, the system can access the corresponding embedding data and corresponding activity data for each subject in the first cohort. This data can be used as a training dataset to train a second machine learning model 404, which is configured to receive embedding data for a given patient and predict the activity data for that patient. In some embodiments, the second machine learning model is a linear model (e.g., one or more penalized linear models). For example, the second machine learning model might be a linear model configured to receive one or more embeddings for a given patient and predict the activity data of molecular analytes associated with ATAC-seq peaks for that given patient. For example, the predicted activity data might include scalar values ​​(e.g., normalized / log-scaled read counts, p-values, or LogFC) or base pair level signals (e.g., regressed ATAC-seq signal shape). In some embodiments, the second machine learning model includes one or more attention-based models (including transformer attention and / or multi-instance attention), one or more diffusion models, or any combination thereof.

[0232] Figure 4B The use of an exemplary second machine learning model trained according to some embodiments is illustrated. The second machine learning model 404 can receive one or more embeddings 452 from a patient and output imputed molecular analyte activity data 456. As described above, the second machine learning model can... Figure 2B It is used in block 210. In some embodiments, the input embedded data is obtained by providing the patient's imaging data to the first machine learning model.

[0233] Figure 5 An exemplary process is illustrated, according to some embodiments, for identifying one or more relevant biomarkers using a trained first machine learning model and a trained second machine learning model. Process 500 may correspond to... Figure 2A -B blocks 208-214. An exemplary system (e.g., one or more electronic devices) can receive multiple medical images and outcome data from a queue (e.g., a SoC queue). Reference Figure 5 The queue includes multiple patients 1-n. For each patient, the system can receive one or more images (e.g., histopathological images, MRI images, CT images) and optional outcome data. For example, the system receives image data 502 and optional outcome data 504 for patient 1, image data 552 and optional outcome data 554 for patient n, and so on.

[0234] Then the system can train a machine learning model on the first (e.g., Figure 3B Model 304 in the first machine learning model is provided with received images to determine multiple embeddings. The first machine learning model is configured to receive image data and output one or more embeddings, which are numerical, low-dimensional feature representations of the input. Figure 5 In the depicted example, the system can provide image data 502 of patient 1 to the first machine learning model to obtain embedding 506, and provide patient 1 image data 502 to the first machine learning model. Image data 552 is used to obtain embedding 556, etc.

[0235] The system can then feed the second trained machine learning model (e.g., Figure 4B Model 404 provides multiple embeddings to determine the imputation activity data for one or more molecular analytes. A second machine learning model is configured to receive one or more embeddings and output imputation data. The imputation activity data for one or more molecular analytes may include gene expression data, copy number amplification values ​​(e.g., from WGS, WGBS, or targeted sequencing), amplification signatures (e.g., from RNA-seq), chromatin accessibility data (e.g., ATAC-seq), DNA methylation data (e.g., WGBS, RRBS), histone modifications (e.g., histone ChIP-seq), RNA data (e.g., from RNA-seq), protein data, space biology data, whole-genome sequencing (WGS) data (e.g., GWAS), somatic mutation data, germline mutation data, or any combination thereof.

[0236] In some embodiments, the data may include: gene expression values ​​containing transcript abundance, chromosome accessibility scores containing ATAC-seq peaks, abundance of one or more histone modifications containing ChIPseq values, abundance of one or more mRNA sequences, abundance of one or more proteins, presence of one or more somatic mutations, presence of one or more germline mutations, or any combination thereof.

[0237] exist Figure 5 In the example depicted, the system can provide the second machine learning model with the embedding 506 of patient 1 to obtain imputation data 508, provide the second machine learning model with the embedding 566 of patient n to obtain imputation data 568, and so on.

[0238] In some embodiments, the second machine learning model is trained using a training dataset associated with a relatively small queue, such as Figure 1A Queue 102 in the database. This queue may be smaller than the queue containing patients. A queue of n. As described above, the data collected for a small queue can include data of a shared modality (e.g., imaging data) as well as high-content modalities, where specific MoAs (e.g., the activity of a specific gene or process) can be identified from the data. To generate a training dataset to train a second machine learning model, imaging data (e.g., histopathological images) is fed to a trained first machine learning model to obtain the corresponding embeddings. Thus, for each subject in the queue, the subject's embedding and the corresponding activity data (e.g., gene activation) of one or more molecular analytes are available. Therefore, the second machine learning model can be trained to receive the embedding data and predict the activity data of one or more molecular analytes for the subject, as referenced above. Figure 4A As stated above.

[0239] The system can then identify one or more relevant biomarkers based on the queue's result data and the queue's imputation activity data.530 Figure 5 In the depicted example, the queue outcome data includes outcome data 504 for patient 1, ... and outcome data 554 for patient n. Imputation activity data includes patient... The imputation data 508, and the patient The interpolated data is 559.

[0240] In some embodiments, to identify biomarkers, the system calculates an association measure (e.g., a p-value) that indicates the degree of association between imputation activity data of a molecular analyte (candidate biomarker) and outcome data. In some embodiments, the association measure quantifies the association between the candidate biomarker and the clinical outcome. By assessing the association between candidate biomarkers and clinical outcomes, the system identifies one or more biomarkers that are significantly associated with the clinical outcome. For example, the association measure or correlation measure may be a hazard ratio, a hazard proportion, or a p-value. In some embodiments, the hazard ratio may be estimated from a Cox proportional hazards model. More generally, the association between a candidate biomarker and a time-to-event outcome (e.g., overall survival, progression-free survival) can be quantified using a (weighted) log-rank test, a Cox proportional hazards model, an Aalen additive hazard model, or a parametric accelerated time to failure model.

[0241] In some embodiments, the system performs association tests by generating a candidate biomarker-specific predictive model for each candidate biomarker, configured to receive data associated with the candidate biomarker and output a predicted clinical outcome. The model is then evaluated to determine whether a significant association exists between the candidate biomarker and the clinical outcome. For example, the model might be a linear regression model, and the system can calculate a p-value associated with the model and determine whether the p-value exceeds a predefined threshold to determine if a significant association exists. In some embodiments, the p-value is obtained using standard Wald, scoring, likelihood ratio, or Monte Carlo test procedures, and the effect size and standard error are obtained using classical linear model theory.

[0242] Other association tests can be performed to determine whether there is a significant association between candidate biomarkers and clinical outcomes. Association tests can also be based on extensions of linear models, such as linear mixed models or generalized linear regression, or nonlinear models (random forests, SVMs, etc.).

[0243] Figure 6 An exemplary procedure for embedding interpolated molecular measurements from histopathology is illustrated according to some embodiments. A first cohort comprises 400 patients, and a dataset is collected for this cohort to measure the ATAC-seq profiles of 400 TCGA samples, broadly covering 23 cancer types. Due to the relatively small number of patients and cancers in the dataset, the dataset may be insufficient for many insights into any cancer type.

[0244] On the other hand, the second cohort (e.g., the SoC cohort) includes 11,000 TCGA patients, but no ATAC-seq spectra were collected for the second cohort. According to embodiments of this disclosure, the imputation model is trained using the dataset from the first cohort (i.e., Figure 2A and 2B (The second machine learning model described in the text). The trained imputation model was used to predict the ATAC-seq signal of the second cohort from the histological embeddings of the second cohort, improving the power by nearly two orders of magnitude.

[0245] Figure 7A and 7B This illustrates chromatin accessibility significantly predicted from histopathological embeddings according to some embodiments. Trained machine learning models (e.g., Figure 2A The second machine learning model in the model is configured to receive histopathological embeddings from one or more patients and predict measurements of activation in one or more genomic regions of the patient. Figure 7A This shows a comparison between actual ATAC-seq data 702 and predicted ATAC-seq data 704 from the same 396 patients across 23 cancer types. Figure 7A and 7BAs shown, high-precision predictions were obtained from approximately 5000 genomic regions. Specifically, approximately 5000 ATAC peaks were observed. Spearman The peaks on the retained test set are imputed. These 5000 peaks showing strong correlations were prioritized and focused on in subsequent analyses.

[0246] Figure 7C Exemplary gene-based risk stratifications according to some embodiments are shown. As illustrated, many of these show a strong association with survival (HR). - The associations are much stronger than those with somatic alterations of the same gene, most of which did not even reach statistical significance. The target patient population is also quite large, such as... Figure 7C Two known cancer targets are shown, including one - AKT3 - which is currently being developed in several biopharmaceutical pipelines for multiple indications.

[0247] Figure 8 This illustrates how ATAC-seq signals imputed according to some embodiments can help identify novel outcome-related genes. As shown, ATAC-seq peaks provide a layer of interpretability to the histopathological embedding—pointing to specific genes strongly associated with patient outcomes. For breast cancer and triple-negative breast cancer, the system can obtain some of the strongest known drivers as positive controls, as well as some novel but highly plausible genes, some of which have hazard ratios close to 2.

[0248] Figure 9 Exemplary verification data according to some embodiments is shown. Figure 9 The study demonstrated significant enrichment of ATAC-associated TNBC genes and supports a causal role of gene activation in tumor proliferation and patient outcomes. Confidence in the target can be increased by leveraging strong genetic evidence from additional cohorts with high-content clinical data, and further enhanced by validation experiments using in vitro and / or model systems.

[0249] While some embodiments disclosed herein involve training multiple machine learning models, it should be understood that similar techniques can be used to train a single machine learning model comprising multiple modules. For example, a first machine learning model, a second machine learning model, and a third machine learning model could be implemented as a first module (e.g., an embedding module), a second module (e.g., a molecular analyte prediction head), and a third module (e.g., an outcome prediction head, such as a survival head), as referenced below. Figure 10The head may include a collection of one or more layers trained on one or more specific prediction tasks. The head may be part of the original model or added to it afterwards. In some embodiments, the head may take an embedding as input and predict a given endpoint / molecular analyte, but it may also take other predictions as input (e.g., a new head may be trained to predict survival from molecular analytes).

[0250] Figure 10 An exemplary process 1000 for predicting the activity of a patient's molecular analyte according to some embodiments is shown. Process 1000 is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process 1000 is performed using a client-server system, and the steps of process 1000 are distributed between a server and one or more client devices in any way. In other examples, process 1000 is performed using only client devices or only multiple client devices. In process 1000, some steps may be optionally combined, the order of some steps may be optionally changed, and some steps may be optionally omitted. In some examples, additional steps may be performed in conjunction with process 1000. Therefore, the operation as shown (and described in more detail below) is exemplary in nature and should not be considered limiting.

[0251] In block 1002, an exemplary system (e.g., one or more electronic devices) trains a machine learning model based on multiple medical images from a first queue. The first module may include components similar to... Figure 3A and 3B The first machine learning model 304 is used to perform the embedded module processing.

[0252] In block 1004, the system trains a second module of a machine learning model based on one or more molecular analyte datasets obtained from the second queue. The second module may have one or more heads. The second module may be similar to... Figure 4A and 4B The second machine learning model in the middle performs the processing in the manner of 404.

[0253] In block 1006, the system receives medical images from the patient. In block 1008, the system uses first and second modules trained with machine learning models to predict the activity of molecular analytes from the patient's medical images. For example, the medical images may be provided to the first module to obtain an embedding, which is then provided to the second module to obtain a prediction of the activity of the molecular analyte.

[0254] In some embodiments, the system further determines whether a patient belongs to one or more subgroups based on the predictive activity of a molecular analyte. For example, if the predictive activity of a molecular analyte indicates the presence of a relevant biomarker, the system can determine that the patient belongs to a subgroup associated with that relevant biomarker.

[0255] In some embodiments, the system trains a third module of the machine learning model based on a third cohort, which includes multiple medical images and associated clinical outcomes. The third module of the machine learning model is configured to predict treatment and / or clinical outcomes. The third module of the machine learning model can be similar to a reference... Figure 2A and 2B The third machine learning model is described in terms of how it performs processing.

[0256] In some embodiments, the system may use a third module to determine a measure of the significance or prognostic value of a molecular analyte for dynamically selecting a subset of molecular analytes for subsequent use (i.e., selecting a molecular analyte for downstream use, such as if it is identified as a relevant biomarker, if the association between the molecular analyte and the outcome is significant). As described herein, a significance value may be a cancer hazard ratio.

[0257] In some embodiments, a second module and / or a third module of the machine learning model are trained using transfer learning. For example, the system may first train a second model / module to predict per-gene activity using one modality (e.g., RNA-seq), and then fine-tune (i.e., "transfer") the model to predict a related modality (e.g., ATAC-seq). This option is particularly attractive if the cohort with RNA-seq is larger, but if ATAC-seq shows a stronger correlation with the outcome. As another example, transfer learning can be used if both cohorts have ATAC-seq data, but one exhibits batch effects, or one lacks outcome / response data.

[0258] In some embodiments, one or more molecular analyte datasets include: gene expression data (e.g., from RNA-seq), copy number amplification values ​​(e.g., from WGS, WGBS, or targeted sequencing), amplification signatures (e.g., RNA-seq), chromatin accessibility data (e.g., ATAC-seq), DNA methylation data (e.g., WGBS, RRBS), histone modifications (e.g., histone ChIP-seq), RNA data (e.g., from RNA-seq), protein data, space biology data, whole-genome sequencing (WGS) data (e.g., GWAS), somatic mutation data, germline mutation data, or any combination thereof.

[0259] In some embodiments, one or more molecular analyte datasets include: gene expression values ​​containing transcript abundance, chromosome accessibility scores containing ATAC-seq peaks, abundance of one or more histone modifications containing ChIP-seq values, abundance of one or more mRNA sequences, abundance of one or more proteins, presence of one or more somatic mutations, presence of one or more germline mutations, or any combination thereof. In some embodiments, one or more molecular analyte datasets include two molecular analyte datasets (e.g., from two or more laboratories).

[0260] In some embodiments, medical images from patients are obtained from a fourth cohort, which includes multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images. In some embodiments, the system may determine that each patient in the fourth cohort belongs to one or more subgroups.

[0261] In some embodiments, the first queue includes multiple medical images from multiple patients. In some embodiments, the multiple medical images include: one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof. In some embodiments, the multiple medical images are unlabeled, and the first module is trained using unsupervised learning.

[0262] In some embodiments, the first queue and the second queue are the same queue. In some embodiments, the second queue (e.g., study queue 102) includes data from multiple medical images and one or more related molecular analytes.

[0263] In some embodiments, the first, second, or third cohorts may also include one or more clinical covariates.

[0264] In some embodiments, the third queue (e.g., SoC queue 112) includes multiple medical images and associated clinical outcomes. One or more clinical covariates may include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history (disease / treatment / billing history, physician notes), or any combination thereof.

[0265] In some embodiments, the system may remove data-specific biases in the first, second, and / or third queues. For example, the system may use adversarial domain adaptation to learn to remove dataset-specific biases present in the second or third queues (i.e., map these embeddings to be indistinguishable from the embeddings of the first queue).

[0266] In some embodiments, during test / inference time (i.e., for new / unseen patients), the system can use domain adaptation to map new embeddings to the same space as the embeddings in queues 1-3. For example, the system can receive medical images of new patients; obtain embeddings by providing the medical images of new patients to the first module; and map the embeddings based on domain adaptation. An example might be training the domain adaptation model on one or more additional training queues, or training on augmented / perturbed examples from previous queues using adversarial loss, where the adaptation model is penalized if the adversarial model is able to distinguish the domain.

[0267] In some embodiments, the system can use transfer learning across related molecular analytes using a new cohort. For example, the system can train a second machine learning module to predict gene-level ATAC-seq signals on a large second cohort. The system can then transfer (i.e., fine-tune) that second module to train a fourth module to predict related molecular analytes (which could also be ATAC-seq, or could be RNA-seq of the same gene) on a new cohort with fewer patients than the second cohort.

[0268] Figure 11 An exemplary process 1100 for predicting the activity of a patient's molecular analyte according to some embodiments is shown. Process 1100 is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process 1100 is performed using a client-server system, and the steps of process 1100 are distributed between a server and one or more client devices in any way. In other examples, process 1100 is performed using only client devices or only multiple client devices. In process 1100, some steps may be optionally combined, the order of some steps may be optionally changed, and some steps may be optionally omitted. In some examples, additional steps may be performed in conjunction with process 1100. Therefore, the operation as shown (and described in more detail below) is exemplary in nature and should not be considered limiting.

[0269] In block 1102, an exemplary system (e.g., one or more electronic devices) trains a first machine learning model on multiple medical images from a first queue. The first machine learning model might be... Figure 3A and 3B The first machine learning model in the series is 304.

[0270] In block 1104, the system trains a second machine learning model on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from the second queue. The second machine learning model may be... Figure 4A and 4B The second machine learning model in the model is 404.

[0271] In block 1106, the system receives medical images from the patient. In block 1108, the system uses a second trained machine learning model to predict the activity of a molecular analyte from the patient's medical images. For example, the medical images may be fed to a first model to obtain an embedding, which is then fed to a second model to obtain a prediction of the molecular analyte's activity.

[0272] In some embodiments, the system further determines whether a patient belongs to one or more subgroups based on the predictive activity of a molecular analyte. For example, if the predictive activity of a molecular analyte indicates the presence of a relevant biomarker, the system can determine that the patient belongs to a subgroup associated with that relevant biomarker.

[0273] In some embodiments, the system trains a third machine learning model based on a third cohort, which includes multiple medical images and associated clinical outcomes. The third machine learning model is configured to predict treatment and / or clinical outcomes. The third machine learning model can be similar to a reference model. Figure 2A and 2B The third machine learning model is described in terms of how it performs processing.

[0274] In some embodiments, the system may use a third model to determine a measure of the significance or prognostic value of a molecular analyte; and based on this measure, determine whether the molecular analyte is significantly associated with treatment and / or clinical outcomes, enabling the molecular analyte to be used for subsequent patient stratification. As described herein, a significance value may be a cancer hazard ratio.

[0275] In some embodiments, the second machine learning model and / or the third machine learning model are trained using transfer learning.

[0276] In some embodiments, one or more molecular analyte datasets include: gene expression data, copy number amplification values ​​(e.g., from WGS, WGBS, or targeted sequencing), amplification signatures (e.g., RNA-seq), chromatin accessibility data (e.g., ATAC-seq), DNA methylation data (e.g., WGBS, RRBS), histone modifications (e.g., histone ChIP-seq), RNA data (e.g., from RNA-seq), protein data, space biology data, whole-genome sequencing (WGS) data (e.g., GWAS), somatic mutation data, germline mutation data, or any combination thereof.

[0277] In some embodiments, one or more molecular analyte datasets include: gene expression values ​​containing transcript abundance, chromosome accessibility scores containing ATAC-seq peaks, abundance of one or more histone modifications containing ChIP-seq values, abundance of one or more mRNA sequences, abundance of one or more proteins, presence of one or more somatic mutations, presence of one or more germline mutations, or any combination thereof. In some embodiments, one or more molecular analyte datasets include two molecular analyte datasets.

[0278] In some embodiments, medical images from patients are obtained from a fourth cohort, which includes multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images. In some embodiments, the system may determine that each patient in the fourth cohort belongs to one or more subgroups.

[0279] In some embodiments, the first cohort includes multiple medical images from multiple patients. In some embodiments, the multiple medical images include: one or more histopathological images; one or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; or any combination thereof. In some embodiments, the multiple medical images are unlabeled, and the first model is trained using unsupervised learning.

[0280] In some embodiments, the first queue and the second queue are the same queue. In some embodiments, the second queue (e.g., study queue 112) includes data from multiple medical images and one or more related molecular analytes.

[0281] In some embodiments, the first, second, or third cohorts may also include one or more clinical covariates.

[0282] In some embodiments, the third queue (e.g., SoC queue 111) includes multiple medical images and associated clinical outcomes. One or more clinical covariates may include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

[0283] In some embodiments, the system may remove data-specific biases in the first, second, and / or third queues. For example, the system may use adversarial domain adaptation to learn to remove dataset-specific biases present in the second or third queues (i.e., map these embeddings to be indistinguishable from the embeddings of the first queue).

[0284] In some embodiments, during test / inference time (i.e., for new / unseen patients), the system can use domain adaptation to map new embeddings to the same space as the embeddings in queues 1-3. For example, the system can receive medical images of new patients; obtain embeddings by providing the medical images of new patients to the first model; and map the embeddings based on domain adaptation.

[0285] In some embodiments, the system can use transfer learning across related molecular analytes using a new cohort. For example, the system can train a second machine learning model to predict gene-level ATAC-seq signals on a large second cohort. The system can then transfer (i.e., fine-tune) that second model to train a fourth model to predict related molecular analytes (which could also be ATAC-seq, or could be RNA-seq of the same gene) on a new cohort with fewer patients than the second cohort.

[0286] The above operations can optionally be performed by Figure 12 The component implementation is described in the text. For those skilled in the art, how to implement it based on... Figure 12 The implementation of other processes by the components described in the text is clear.

[0287] Figure 12 An example of a computing device according to one embodiment is shown. Device 1200 may be a host computer connected to a network. Device 1200 may be a client computer or a server. Figure 12 As shown, device 1200 can be any suitable type of microprocessor-based device, such as a personal computer, workstation, server, or handheld computing device (portable electronic device), such as a telephone or tablet. The device may include, for example, one or more of processor 1210, input device 1220, output device 1230, memory 1240, and communication device 1260. Input device 1220 and output device 1230 can generally correspond to those described above and can be connectable or integrated with a computer.

[0288] Input device 1220 can be any suitable device that provides input, such as a touchscreen, keyboard or keypad, mouse, or voice recognition device. Output device 1230 can be any suitable device that provides output, such as a touchscreen, haptic device, or speaker.

[0289] Memory 1240 can be any suitable device providing storage, such as electrical, magnetic, or optical memory, including RAM, cache, hard disk drive, or removable storage disk. Communication device 1260 can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or device. Computer components can be connected in any suitable manner, such as via a physical bus or wirelessly.

[0290] Software 1250 may be stored in memory 1240 and executed by processor 1210, and may include, for example, programming that embodies the functions of this disclosure (e.g., as embodied in the device described above).

[0291] Software 1250 may also be stored and / or transmitted in any non-transient computer-readable storage medium for use by or connected to the aforementioned instruction execution system, apparatus, or device, which may retrieve and execute instructions related to the software. In the context of this disclosure, the computer-readable storage medium may be any medium, such as memory 1240, which may contain or store programming for use by or connected to the instruction execution system, apparatus, or device.

[0292] The software 1250 can also be propagated in any transmission medium for use by or connection to the aforementioned instruction execution system, apparatus, or device, which can retrieve and execute instructions related to the software. In the context of this disclosure, the transmission medium can be any medium capable of conveying, propagating, or transmitting programming for use by or connection to the instruction execution system, apparatus, or device. Transmission readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation media.

[0293] Device 1200 can be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can include any suitable network links that enable the transmission and reception of network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

[0294] Device 1200 can implement any operating system suitable for running on a network. Software 1250 can be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying the functions of this disclosure can be deployed in different configurations, such as in a client / server arrangement or via a web browser as a web-based application or web service.

[0295] The following is a reference example. Figures 14A-14C The exemplary research section provides an overview of exemplary embodiments for configuring predictions of gene-specific copy number amplification (CNA), gene-specific expression, and gene-specific amplification features. Although Figures 14A-14CThis description pertains to the imputation of specific types of molecular analyte activity, but it should be understood that the model can be configured to predict other factors, such as protein levels or the activity of any other molecular analyte described herein. For example, proteins are direct therapeutic targets for many drugs, including those evaluated in the Exemplary Studies section below. Therefore, the multi-task approach described below is also beneficial in this context. Furthermore, the versatility of the framework described herein will allow for multi-task learning of multiple biomarker types within the same model, enabling information transfer between CNAs, RNA, proteins, etc.

[0296] Cancer is a highly heterogeneous disease, and despite significant advances in discovering and developing “precision” management approaches, patient responses to targeted therapies can remain highly variable, and the reasons for this remain unclear. The growth in targeted therapy development has accelerated the use of predictive biomarkers to identify patients more likely to respond to drugs. Indeed, studies have shown that oncology trials using biomarkers have a fairly high success rate, increasing the likelihood of drug approval by nearly 5 times across all indications, and by 12 times, 8 times, and 7 times in breast cancer, melanoma, and non-small cell lung cancer (NSCLC), respectively.

[0297] Current predictive biomarkers typically utilize one of several assay types: immunohistochemistry (IHC) on biopsy slides; genetic analysis, including karyotype analysis, fluorescence in situ hybridization, and DNA sequencing; or transcriptomic levels of small gene sets, measured via polymerase chain reaction (PCR) or (rarely) based on extensive RNA sequencing. The development and deployment of these methods face significant challenges. First, these methods rely on specialized assays that are not universally available in cancer centers, especially in resource-constrained settings. Second, some of these techniques, such as IHC, require manual evaluation by trained individuals, which can increase variability and reduce the reproducibility of results. Third, they incur additional costs and, more importantly, additional time, potentially delaying diagnosis and treatment initiation. Furthermore, while sequencing-based assays utilize widely adopted and fairly standardized techniques, biomarkers using targeted staining or probes (including IHC and PCR) often require the development and extensive testing of specialized assays and reagents.

[0298] Current development paradigms and available technologies tend to select biomarkers at an early stage, when available data are often based on poorly representative preclinical models and / or insufficiently effective Phase 1 studies, neither of which captures the heterogeneity of the human patient population. This leads to a preference for simplistic biomarkers, primarily driven by human understanding of the disease's mechanisms, typically genetic aberrations measured by sequencing or transcript / protein expression measured by chemistry. Consequently, the labeled populations tend to be either overly restrictive—reducing the pool of patients who will benefit—or overly broad—allowing a subset of patients to receive drugs with limited efficacy while still bearing the risk of toxicity and delaying potentially more effective treatments.

[0299] Existing research primarily utilizes per-task supervised learning frameworks, where a single deep learning model is trained to predict specific, defined biomarkers for clinical use in a given type of cancer, directly from H&E images. For example, in the work of S. Arslan, D. Mehrotra, J. Schmidt, et al., deep learning can predict multiple sets of biomarkers from routine pathology images: a systematic, large-scale study. bioRxiv, 2022, trained over 13,000 different models, one for each cancer, biomarker, and fold. This approach limits the available training data to individuals within a single cancer for which known biomarkers have been measured.

[0300] The descriptions in this article (for example, refer to Figure 1-13 above and the references below) Figures 14A-14C (And the exemplary research section) is a method for developing and deploying a class of predictive biomarkers that leverages deep learning to simultaneously predict a set of molecular factors associated with treatment selection and response on data acquired in SoC (such as H&E sample images). These images are routinely collected and processed for almost all patients with solid tumors (worldwide) and are increasingly being digitized. These images can be information-rich and enable automated disease detection, prognostic prediction, cancer grading, histological and molecular subtype classification, and customized treatment planning. Therefore, the system and method described in this paper provide a novel multi-cancer, multi-biomarker prediction framework that leverages commonalities in cancer mechanisms across cancer types and different genes and mutations, which are reflected in SoC data (such as H&E sample images). Both slices and molecular readouts are reflected. The predictive performance of the model described in this paper is significantly improved by shifting from predicting single molecular readouts to multi-task prediction of all defined target genes and then to transcriptome-wide prediction. Extensive comparison with the results of Arslan et al. is challenging due to the very limited overlap between their work and ours in terms of biomarkers. However, for CDK4, the only shared RNA biomarker, they reported an AUROC of approximately 0.72 (average across 3 cancer types), while the model described in this paper achieved a pan-cancer AUROC of 0.84; 0.75 in stratified analysis, averaging across all cancer types; and 0.77 when filtered to 4 relevant cancer types (specifically breast cancer, colorectal cancer, lung cancer, and pancreatic cancer).

[0301] In some embodiments, one or more machine learning models described herein can be trained for multi-biomarker prediction. Embeddings generated from tissue images, such as hematoxylin and eosin (H&E) stained biopsy samples, can be used to train discovery machine learning models to simultaneously predict multiple biomarkers using multi-task learning methods. In some embodiments, a pan-solid tumor H&E base / discovery model is trained to learn general characterizations of tissue H&E images. The base embeddings generated by one or more base / discovery models can then be used as input to downstream machine learning models, such as the second machine learning model described above (and more details below). By using multi-task learning methods to simultaneously predict multiple biomarkers, the first set of downstream models allows for exploratory analysis and broad discovery. To enhance interpretability, these models can make predictions based on tile-level rather than slice-level characterizations, where tiles constitute small elements much larger than the entire slice image. Tile-level predictions can generate and overlay annotation maps highlighting slice regions driving model predictions. Although trained on batch rather than spatially resolved molecular data, the models are able to learn spatial variations of tumor cell molecular markers, which are relevant to blinded pathologists reviewing regions identified as cancerous.

[0302] The initial broad imputation set allows for an assumption-free investigation into which biomarkers are associated with which patient populations and identifies biomarkers that distinguish patient subgroups. Once a smaller set of biomarkers specific to the patient population of interest is selected from the discovery panel, a specialized model can be trained, starting from the same basic characterization, which may outperform the discovery model in predicting key biomarkers in the target subgroup. This two-step process of broad imputation followed by specialization to a more focused subset enables both the discovery of novel biomarkers and the optimization of diagnostic performance.

[0303] Therefore, the techniques described in this paper can optimize patient populations for targeted therapy, going beyond the use of genetic alterations or IHC, but without going to the other extreme of overly broad labeling that covers an excessively heterogeneous patient population. Furthermore, although the imputation models are trained on batch reads, they are able to overlay tile-level predictions of spatial variations on input histological images, providing a lens of interpretability and enabling clinicians to assess, for example, whether pairs or sets of biomarkers are spatially co-localized within the tumor. In summary, the results described in detail below support the use of... The feasibility and ongoing exploration of highly scalable molecular prediction as a flexible and generalizable approach for the development and deployment of predictive biomarkers for cancer targeted therapy.

[0304] To demonstrate the value of this approach, studies were conducted focusing on biomarkers associated with drug efficacy based on mechanisms of action (MOAs) that differentially identify and kill cancer cells through the abundance of specific protein targets: antibodies (monospecific and multispecific), antibody-drug conjugates (ADCs), and T-cell binders. Three exemplary biomarkers were evaluated: copy number amplification (CNA), RNA transcript level / gene expression level, and RNA-derived amplification features that capture the transcriptomic effects of target CNAs, as described in more detail in the Exemplary Studies section below. Among a vast and diverse range of cancer types and biomarkers, the techniques described herein provide highly accurate patient-specific predictions of both sequential and binary versions of molecular readouts. RNA-derived features, also referred to herein as amplification features, have proven to be reliable proxies for CNAs. See below for further details. Figures 14A-14C Exemplary descriptions of models for interpolating such biomarkers are provided, the performance of which is described in the Exemplary Studies section below.

[0305] Antibody-drug conjugates (ADCs) are a class of targeted cancer therapies designed to deliver cytotoxic (cell-killing) drugs directly to cancer cells while minimizing damage to normal healthy cells. ADCs deliver chemotherapy via linkers to monoclonal antibodies that bind to specific targets expressed on cancer cells. Upon binding to the target (oncoprotein or receptor), the ADC releases the cytotoxic drug into the cancer cells. As described herein, multiple target genes can be identified based on existing ADCs. For example, a drug database can be queried to identify drugs tagged with therapeutic categories as antibody-drug conjugates (ADCs), T-cell binders, or antibodies (including monospecific and multispecific antibodies), and the overall list of drugs can then be filtered to identify drugs with specified targets. These targets can be imputed (e.g., by training first and second machine learning models), and based on the imputed values, one or more biomarkers can be identified (e.g., by a third machine learning model described herein). Biomarkers can be used to evaluate new patients and identify / manage treatment plans. For example, biomarker values ​​can be determined for new patients, and if the biomarker values ​​meet one or more criteria (e.g., exceeding a threshold, falling below a threshold, falling within a range, or falling outside a range), the corresponding ADC can be prescribed accordingly.

[0306] In some embodiments, the second machine learning model described herein can be trained to interpolate copy number amplifications of a set of genes (e.g., a set of target genes). Copy number amplification (CNA) can be directly interpolated (as referenced). Figure 14A The second machine learning model is trained using image-based embeddings (e.g., embeddings based on histopathological images, embeddings based on MRI images, etc.) labeled with observed CNA tags (e.g., binary values ​​indicating whether a CNA is detected as zero or one) to impute the patient's target matrix (e.g., having binary values ​​indicating whether a CNA is detected as zero or one and / or values ​​corresponding to a 0 to 1 probability fraction of amplification in the corresponding gene for a given patient). CNAs can also be imputed indirectly, such as... Figure 14B As shown, a target matrix for a patient is imputed by using an embedding training model paired with the target gene expression value, the value of which corresponds to the imputed expression (e.g., expression level) of each target gene within the patient. Patients with a CNA in a given target gene typically have relatively higher expression of that target than patients without a CNA in a given target gene, thus the expression matrix provides an approximation of the CNA for each gene. Figure 14C The following details an additional method for indirectly imputing CNAs using gene-specific amplification signatures, where the amplification signatures are determined based on the weighted gene expression levels of each differentially expressed gene. Model performance for each of the above models was evaluated on a set of target genes, and a summary of the results is provided in the appendix. References Figures 14A-14CThe imputation model described can be trained and validated using data from the Common Research Data Library (TCGA), which includes genetic, molecular, and histological data from over 20,000 primary tumors across 33 cancer types. Additional molecular and histological data are derived from commercially available multicenter cancer research resources (referred to as Cohort A in this paper).

[0307] Target genes can be identified based on available data related to targeted gene therapy. In some examples described in this paper, commercial drug databases are queried to identify drugs whose therapeutic categories are labeled as antibody-drug conjugates (ADCs), T-cell binders, or antibodies (including monospecific and multispecific antibodies). For ADCs and T-cell binders, drugs in any stage of development are retained, while for antibodies (a larger category), drugs whose development has been discontinued are excluded. The overall list of drugs is filtered to those with the specified targets. Each remaining drug is mapped to an HGNC gene symbol, and the union of all gene symbols is taken, resulting in 352 unique targets.

[0308] exist Figure 14A In any of the -C models, the second machine learning model can be a module of a machine learning model. Any of these models can be trained for regression, classification, or other tasks. Any of these models can be trained in a single-step process (e.g., direct imputation of output) or a two-step process (e.g., broad imputation followed by specialization to a more focused subset), as described herein. For example, the pan-cancer, pan-biomarker approach described herein may not maximize the learning of features that capture tumor- or biomarker-specific changes, but may instead focus on learning features that generalize across tasks. This can be addressed by refining (e.g., fine-tuning) the base model (e.g., models 1404a, 1404b, 1404c) to a specific prediction task. This task can be a single cancer, a single biomarker, or a combination of both. Similar processes can be applied to fine-tuning one or more other models described herein (e.g., references...). Figure 2A and 2B The third machine learning model described, for example, is fed into a treatment response dataset of drugs with relevant mechanisms of action (MOA), to better predict patient responses. This fine-tuning may even be possible from the perspective of extensive pre-training of the model. This is feasible in small cohorts available in phase II clinical trials. Exemplary results of such fine-tuned models are provided below in the Exemplary Studies section.

[0309] Figure 14A An exemplary process for training and using the second machine learning model 1404a direct imputation copy number amplification (CNA) is described. Figure 14AIn the depicted example, training data 1402a includes tile embeddings, which can be generated by using a first training machine learning model (e.g., the first model 304 described above) to generate low-dimensional embeddings of histopathological images (e.g., from a first cohort). The tile embeddings can be paired with observed CNA values ​​from subjects. CNA values ​​can be determined by observation to identify whether a given gene is amplified. For example, in some examples, two methods are used to identify genes with focal amplification based on whole-exome data from tumor specimens: GISTIC2 (v2.0.22), which estimates the copy number relative to a matched normal sample, and Sequenza (v3), which estimates the absolute copy number. In some examples, a gene is designated as focally amplified if it receives a GISTIC2 score of 2, or if it shows a copy number greater than twice the ploidy value based on Sequenza. Tile embeddings can be obtained, for example, by feeding the first training machine learning model (e.g., the first model 304) histopathological images from the first cohort to obtain a low-dimensional representation of each histopathological image.

[0310] Training data 1402a can be used to train a second machine learning model 1404a to directly predict a subject's CNA label / value (e.g., a binary value of zero or one indicating whether a CNA is detected and / or a probability value between zero and one indicating the likelihood that the corresponding gene is amplified) given new input embedding data. In some embodiments, the training data may include binary amplification labels (e.g., 1 for amplified genes, 0 for non-amplified genes), and the training model may generate predicted probabilities (e.g., between zero and one) indicating whether a gene is likely to be amplified. That is, in some embodiments, the model may be a regression model configured to output continuous values ​​indicating probabilities. In some embodiments, the model may be a classification model configured to output binary results (e.g., amplified or not amplified).

[0311] In some examples, the second machine learning model 1404a is configured to make individual tile-level predictions, which are then averaged. Training model 1404a to make predictions at the tile-level rather than slice-level characterization can enhance interpretability. For example, tile-level predictions can generate and overlay annotation maps that highlight slice regions driving the model's predictions. In some examples, the second machine learning model 1404a is trained on batch rather than spatially resolved molecular data, enabling it to learn spatial variations in tumor cell molecular markers that are relevant to blinded pathologists reviewing regions identified as cancerous, as illustrated in the exemplary studies section below.

[0312] Alternatively, the second machine learning model 1404a may be trained to interpolate molecular analyte activity based on the average of one or more tile features, and / or the second machine learning model 1404a may be equipped with an attention mechanism (e.g., a cross-tile spatial attention mechanism) allowing the model to make patient-level predictions while paying attention to spatially adjacent tiles. However, in some examples, configuring model 1404a to perform individual tile-level predictions may outperform a model configured with an attention mechanism and / or a model trained to interpolate molecular analyte activity based on the average of one or more tile features. The improved performance resulting from configuring the model to perform individual tile-level predictions may be due to the signal attenuation and loss of resolution that occurs before making predictions.

[0313] Once training is complete, the second trained model 1404a can provide input tile embeddings 1452a from the patient and output interpolated molecular analyte activity data 1456a. In some embodiments, the input embedding data is obtained by providing the patient's imaging data to the first machine learning model. Figure 14A In the depicted example, the interpolated molecular analyte activity data 1456a includes gene-specific copy number amplification values. The output can be a patient matrix, whose values ​​correspond to a 0- to 1 probability score of amplification for each target gene in a given patient and / or a binary tag indicating whether each target gene has been amplified, as described above.

[0314] In some examples, a second training phase can be used to fine-tune the second machine learning model 1404a for a specific task. In some examples, the second machine learning model 1404a can be fine-tuned based on a subset of the training data to predict imputed CNA molecular analyte activity data 1456a. The subset of training data can correspond to patient attributes, which may include a specific patient cohort, disease, biomarker, or a combination thereof. For example, the second machine learning model 1404a can be fine-tuned to predict imputed CNAs for a specific patient / subject cohort, a specific disease (e.g., cancer type), a specific biomarker, or a combination thereof. Such fine-tuning can provide improved performance for a specific prediction task (e.g., as shown in the MET case study below). Due to the extensive pre-training of the model, such fine-tuning may even be possible from… This is feasible in small cohorts available in phase II clinical trials. Furthermore, a similar process can be applied to fine-tuning the model to treatment response datasets for drugs with relevant MOAs, steering the model toward better predicting patient responses.

[0315] As mentioned above, the second machine learning model can be a reference. Figure 2A and 2B The second machine learning model described, where the activity of the interpolated analyte determined by model 1404a can be... Figure 2BBlock 210 is used to identify biomarkers. Therefore, the biomarkers identified in block 210 may include interpolated gene-specific CNA values. Furthermore, as referenced above... Figure 2A and 2B As stated above, various clinical outcome prediction methods can be used based on imputation activity data. Clinical outcomes can be predicted based on imputation CNA values ​​using any of the methods described in reference blocks 210-214 above (e.g., a third machine learning model can be trained to determine the association between imputation activity data and clinical outcome data).

[0316] For example, the system can identify one or more relevant biomarkers based on imputation activity data output by a second machine learning model 1404a and patient / subject outcome data associated with image data obtained from input embedding 1452a. In some embodiments, the system uses a third machine learning model to determine the association between imputation activity data and clinical outcome data from a second cohort. Specifically, the system can train a third machine learning model using imputation activity data and clinical outcome data from the second cohort, the model being configured to predict outcomes based on the activity data of molecular analytes. For example, for each candidate biomarker (e.g., CNA value), the system can train a candidate biomarker-specific prediction model, the model being configured to receive data associated with the candidate biomarker and output predicted clinical outcomes (e.g., treatment response, time to progression, time to death).

[0317] The candidate biomarker-specific model is then evaluated to determine whether a significant association exists between the candidate biomarker and the clinical outcome. In some embodiments, the system may determine an association measure or correlation measure based on the model that indicates the degree of association or correlation between the candidate biomarker and the clinical outcome. Association measures and correlation measures are used interchangeably in this disclosure.

[0318] For example, the association metric or correlation metric might be a hazard ratio, a hazard proportion, or a p-value. In some embodiments, the hazard ratio can be estimated from a Cox proportional hazards model. More generally, the association between a candidate biomarker and time-to-event outcomes (e.g., overall survival, progression-free survival) can be quantified using a (weighted) log-rank test, a Cox proportional hazards model, an Aalen additive hazard model, or a parametric accelerated failure time model.

[0319] As another example, the model might be a generalized linear regression model or a time-to-event regression model, and the system can calculate a p-value that associates one or more imputed molecular analytes with one or more clinical outcomes. In some embodiments, the p-value can be obtained using standard Wald, scoring, likelihood ratio, or Monte Carlo tests, and the effect size and standard error can be obtained using classical generalized linear model theory. The p-value indicates the association between the candidate biomarker and the clinical outcome.

[0320] Other association testing procedures can be performed to determine whether a significant association exists between a candidate biomarker and clinical outcomes. Association testing procedures can also be based on extensions of generalized linear models, such as linear mixture models, generalized estimating equations, or nonlinear models (random forests, SVMs, etc.). Further information on obtaining histopathological embeddings and performing association tests can be found in U.S. Provisional Application No. 63233707 entitled "DISCOVERY PLATFORM" and PCT Application No. PCT / US2022 / 075006 entitled "DISCOVERY PLATFORM," the contents of which are incorporated herein by reference for all purposes.

[0321] The system then identifies one or more relevant biomarkers based on the association. For example, the system can determine whether the p-value corresponding to a candidate biomarker exceeds a predefined threshold to determine if a significant association exists. If the p-value corresponding to a candidate biomarker exceeds the predefined threshold, the system can determine that the candidate biomarker is a relevant biomarker.

[0322] In some embodiments, association metrics may indicate positive or negative associations. In some embodiments, significant (e.g., statistically significant, exceeding a predefined threshold) positive or significant negative associations can be identified as relevant biomarkers.

[0323] The system can also perform patient stratification based on identified biomarkers. For example, features aligned with predicted MoA (e.g., histopathological features) define a "probable responder" patient population. In some embodiments, patient stratification can be based on one or more images of a specific patient. The system can determine whether a patient belongs to one or more patient subgroups by determining whether one or more images of the patient indicate alignment with one or more identified relevant biomarkers. In addition to discretized results (one or more discrete subgroups), the system can also return continuous scores (e.g., PD-L1 expression levels, TMB, HER2 quantification by FISH, etc.) to the patient / physician.

[0324] Specifically, the system can receive one or more medical images of a patient. The system can feed one or more images of the patient to a first trained machine learning model to determine one or more embeddings. The system can then feed one or more embeddings to a second trained machine learning model to determine patient-related imputation activity data. The system can determine whether the patient belongs to one or more patient subgroups based on the presence of one or more relevant biomarkers. Specifically, if patient-related imputation activity data indicates the presence of a relevant biomarker, the system can determine that the patient likely belongs to the biomarker-related patient subgroup. The system can identify treatments for the patient based on one or more biomarkers and a known mechanism of action (MoA) of the treatment.

[0325] Figure 14B An exemplary process is described for training and using a second machine learning model 1404b to interpolate gene-specific gene expression data (e.g., to approximate the CNA value for each target gene). As mentioned above, differential gene expression can provide a relatively strong approximation for copy number amplification because patients with the corresponding target gene CNA typically have higher expression of that target gene. Figure 14B In the depicted example, training data 1402b including tile embeddings can be obtained by generating low-dimensional embeddings of histopathological images (e.g., from a first cohort) using a first machine learning model and pairing them with observed gene-specific gene expression data. Gene-specific gene expression levels can be determined through differential expression analysis between copy number amplified and copy number-normal (e.g., diploid) subjects.

[0326] Data used for differential expression analysis can be preprocessed according to various preprocessing procedures. For example, in some examples described herein (e.g., those described in more detail below with reference to the Exemplary Studies section), enhanced TCGA STAR+RSEM gene counts for 11,155 samples were obtained from the GDC portal. These counts were generated by the Genome Data Sharing (GDC) standard pipeline and aligned with GRCh38, and provided to the system. In some examples, corresponding STAR+RSEM gene count matrices were prepared in cohort A for at least a subset of samples (e.g., 2,733 samples in at least one example). In some examples, the TPM matrix was concatenated, filtered to genes with nontrivial expression (requiring TPM > 1 in at least one subject), and subsetd to protein-coding genes from Gencode V43, resulting in a unique set of genes (e.g., 19,421 unique genes in at least one example).

[0327] In some examples, the resulting TPM matrix is ​​log2 transformed and then quantile-normalized using the limma voom function in R. Subsequently, in some examples, the removeBatchEffects function of edgeR is applied to regress cohort effects (TCGA vs. cohort A). In some examples, the log2 (TPM) matrix obtained through principal component analysis and lmfit evaluation of potential batch drivers is examined to determine if there are possible batch effects from sequencing instruments and sequencing centers. No significant batch effects were identified in the examples described in this paper. In some examples, this joint expression matrix is ​​used as input for downstream analyses.

[0328] In some examples, the limma voom package in R is used for differential expression analysis between copy number amplification and copy number-normal (e.g., diploid) patients. In some examples, for each amplification, a limma model is fitted ( CNA status (presence or absence of CNA status + cohort associated with cancer type) was used to identify differentially expressed genes with a false discovery rate (FDR) corrected p-value (i.e., q-value) < 0.01 and an absolute log2 fold change > 0.3. The gene expression value of each target gene was paired with a corresponding tile embedding generated using a first training machine learning model and used to train a second machine learning model to imputate gene-specific gene expression data.

[0329] In some examples, the second machine learning model 1404b is configured to make individual tile-level predictions, which are then averaged. Training model 1404b to make predictions at the tile-level rather than slice-level characterization can enhance interpretability. For example, tile-level predictions can generate and overlay annotation maps that highlight slice regions driving the model's predictions. In some examples, the second machine learning model 1404b is trained on batch rather than spatially resolved molecular data, learning spatial variations in tumor cell molecular markers that are relevant to blinded pathologists reviewing regions identified as cancerous, as shown in the exemplary studies section below.

[0330] Alternatively, the second machine learning model 1404b can be trained to interpolate molecular analyte activity based on the average of one or more tile features, and / or the second machine learning model 1404b can be equipped with an attention mechanism (e.g., a cross-tile spatial attention mechanism) allowing the model to make patient-level predictions while paying attention to spatially adjacent tiles. However, in some examples, configuring model 1404b to perform individual tile-level predictions may outperform a model configured with an attention mechanism and / or a model trained to interpolate molecular analyte activity based on the average of one or more tile features. The improved performance resulting from configuring the model to perform individual tile-level predictions may be due to the signal attenuation and loss of resolution that occurs before prediction is made.

[0331] The second trained model 1404b can receive input tile embeddings 1452b from the patient and generate output interpolated molecular analyte activity data 1456b. In some embodiments, the input embedding data is obtained by providing the patient's imaging data to the first machine learning model. Figure 14B In the depicted example, the imputation molecular analyte activity data 1456a includes gene-specific gene expression values. The output can be a patient-target matrix, with values ​​corresponding to the imputation expression of each target gene within the patient. As described above, the second machine learning model 1404b can be a reference... Figure 2A and 2B The second machine learning model described, where the activity of the interpolated analyte determined by model 1404b can be... Figure 2BBlock 210 is used to identify biomarkers. Therefore, the biomarkers identified in block 210 may include gene-specific gene expression values.

[0332] In some examples, a second training phase can be used to fine-tune the second machine learning model 1404b for a specific task. In some examples, the second machine learning model 1404b can be fine-tuned based on a subset of training data to predict imputation molecular analyte activity data 1456b, which includes gene-specific gene expression. The subset of training data can correspond to patient attributes, which may include a specific patient cohort, disease, biomarker, or a combination thereof. In some examples, the second machine learning model 1404a can be fine-tuned in a second training phase to predict imputation molecular analyte activity data 1456b, which includes a specific patient / subject cohort, a specific disease (e.g., cancer type), a specific biomarker, or a combination thereof. This fine-tuning of the model can provide improved performance for a specific prediction task (e.g., as shown in the MET case study below). Due to the extensive pre-training of the model, this fine-tuning may even be possible from… It is feasible in small cohorts available in phase II clinical trials.

[0333] Furthermore, a similar process can be applied to fine-tuning the model to a treatment response dataset for drugs with relevant MOAs, steering the model toward better predicting patient responses. As described in reference block 210 above, various clinical outcome prediction methods can be used based on imputed activity data. Reference blocks 210-214 above and reference... Figure 14A Any method described is based on imputation expression values ​​to predict clinical outcomes (e.g., a third machine learning model can be trained to determine the association between imputation activity data and clinical outcome data).

[0334] Figure 14C An exemplary process for training and using the second machine learning model 1404c to interpolate gene-specific amplification features is illustrated. Figure 14C In the depicted example, training data 1402c includes tile embeddings, which can be generated by using a first trained machine learning model (e.g., first model 304) to generate low-dimensional embeddings of histopathological images (e.g., from a first cohort). The tile embeddings can be paired with amplification features determined based on weighted expression levels of differentially expressed genes. For example, each amplification feature can be constructed by taking the dot product of the differentially expressed genes and a set of weights. More specifically, for amplification... Suppose there is There are differentially expressed genes. Let... Indicates the subject Zhonggen The expression level of the subjects. Compared to the first The characteristics of an amplification can be determined using the following methods:

[0335] Weight It may be an amplification. Zhonggen of The sign of a multiple change is determined by absolute value. Value scaling. This approach may assign more weight to genes with more evidence of differential expression. Corrected features. Indicates the subjects With amplification Consistent expression profile, even if the subject did not show a copy number analysis The amplification of amplified features allows for the direct derivation of characteristic biomarkers from the imputed RNA profile, enabling the substitution of a difficult-to-estimate (and potentially limited) biomarker with a more robustly estimable one; other features that combine RNA measurements in different ways can be similarly defined and evaluated. Amplified features can be computed from the imputed expression levels. However, developing machine learning models to directly impute amplified features (trained using labels derived from observed expression levels) may yield better performance, such as... Figure 14C Model 1404c.

[0336] Tile embeddings can be obtained by feeding histopathological images from a first cohort to a first training machine learning model to obtain a low-dimensional representation of each histopathological image and pairing it with corresponding features. Training data 1402c can be used to train a second machine learning model 1404c to predict patient / subject feature values ​​given new input embedding data.

[0337] In some examples, the second machine learning model 1404c is configured to make individual tile-level predictions, which are then averaged. Training model 1404c to make predictions at the tile-level rather than slice-level characterization can enhance interpretability. For example, tile-level predictions can generate and overlay annotation maps that highlight slice regions driving the model's predictions. In some examples, the second machine learning model 1404b is trained on batch rather than spatially resolved molecular data. The second machine learning model 1404b learns spatial variations in tumor cell molecular markers that are relevant to blinded pathologists reviewing regions identified as cancerous, as shown in the exemplary studies section below.

[0338] Alternatively, the second machine learning model 1404c can be trained to interpolate molecular analyte activity based on the average of one or more tile features, and / or the second machine learning model 1404c can be equipped with an attention mechanism (e.g., a cross-tile spatial attention mechanism) allowing the model to make patient-level predictions while paying attention to spatially adjacent tiles. However, in some examples, configuring model 1404c to perform individual tile-level predictions may outperform a model configured with an attention mechanism and / or a model trained to interpolate molecular analyte activity based on the average of one or more tile features. The improved performance resulting from configuring the model to perform individual tile-level predictions may be due to the signal attenuation and loss of resolution that occurs before making predictions.

[0339] Once training is complete, the second trained model 1404c can provide input tile embeddings 1452c from the patient and output interpolated molecular analyte activity data 1456c. In some embodiments, the input embedding data is obtained by providing the patient's imaging data to the first machine learning model. Figure 14C In the depicted example, the interpolated molecular analyte activity data 1456c includes gene-specific amplification feature values. The output can be a patient matrix, with values ​​corresponding to the amplification features of each target gene for each patient. As described above, the second machine learning model can be a reference... Figure 2A and 2B The second machine learning model described, where the activity of the interpolated analyte determined by model 1404c can be... Figure 2B Block 210 is used to identify biomarkers. Therefore, the biomarkers identified in block 210 may include gene-specific amplification features.

[0340] In some examples, a second training phase can be used to fine-tune the second machine learning model 1404c for a specific task. In some examples, the second machine learning model 1404a can be fine-tuned during the second training phase, predicting interpolated molecular analyte activity data 1456c based on a subset of the training data, including gene-specific amplification features. The subset of training data can correspond to patient attributes, which may include a specific patient cohort, disease, biomarker, or a combination thereof. For example, the second machine learning model can be fine-tuned for a specific patient / subject cohort, a specific disease (e.g., cancer type), a specific biomarker, or a combination thereof. Such fine-tuning can provide improved performance for a specific prediction task (e.g., as shown in the MET case study below). Due to the extensive pre-training of the model, this fine-tuning may even be possible from… This is feasible in small cohorts available in phase II clinical trials. Furthermore, a similar process can be applied to fine-tuning biomarker prediction models to treatment response datasets for drugs with relevant MOAs, steering the models toward better predicting patient responses.

[0341] As described in reference block 210 above, various clinical outcome prediction methods can be used based on imputation activity data. Reference blocks 210-214 above and reference... Figure 14A and 14B Any method described is based on interpolation amplification features to predict clinical outcomes (e.g., a third machine learning model can be trained to determine the association between interpolation activity data and clinical outcome data).

[0342] Used to generate any of the above Figures 14A-14C The embedded tile images referenced in the model described herein can be obtained from stained whole-section images. For example, a set of hematoxylin and eosin (H&E) stained whole-section images (WSI) (e.g., 30,032 images in some examples described herein), corresponding to a unique set of patients (e.g., 11,428 unique patients in some examples described herein), can be downloaded from data sources such as GDC. The foreground containing tissue can be extracted from the images, and low-frequency hypercellular artifacts, such as tissue folds, out-of-focus areas, and pen marks, can be removed, for example using WSI spectral threshold artifact removal (WSI-STAR) as described in U.S. Application No. 63 / 548,141, entitled "SYSTEMS AND METHODS FOR ARTIFACT DETECTION AND REMOVAL FROM IMAGE DATA," the entire contents of which are incorporated herein by reference for all purposes. To account for differences in staining protocols between research centers, color channels can be normalized, for example using the Macenko method. Each slice can be... At a resolution of at least one pixel per point (MPP), the area is divided into non-overlapping 256×256 tiles, which can be filtered to at least [amount missing]. Foreground tiles. In some examples, this produced 180 million individual tiles. In some examples, WSIs of 1,000 patients in cohort A were processed in the same way, producing 8 million tiles.

[0343] Embeddings can be generated from the aforementioned image tiles using embedding models as described in this paper. For example, the self-supervised distillation unlabeled (DINO) algorithm can be used to embed tiles randomly selected from TCGA. Training a Visual Transformer (ViT) type model on 1MPP tiles. Given a set of unlabeled images, DINO trains a student network (e.g., a ViT) to match the output of a teacher network. This task becomes more challenging because the student and teacher networks receive different "views" of the input images. Training can be monitored by periodically evaluating the utility of the embeddings extracted from the teacher network for several downstream tasks, including cancer subtype classification and overall survival prediction, within an independently validated tile set (e.g., 100,000 tiles in some examples described in this paper). The tile-level embeddings generated by the final model can be used as input for downstream modeling tasks, such as references. Figures 14A-14C Those mentioned above.

[0344] In some examples, the models described in this paper (e.g., 1404b, 1404c) achieve fairly high performance in predicting RNA, and even higher performance in predicting amplified traits, relative to directly predicting CNA (e.g., using model 1404a), as shown in the Exemplary Experimental Studies section below. This higher performance likely stems from several sources. First, in some examples, quantitative traits provide enhanced statistical power relative to binary or ordinal traits (such as CNA) because continuous data captures more fine-grained phenotypic variation and provides meaningful information across the entire ensemble of individuals, significantly increasing the effective sample size. Second, multiple studies have shown that copy number amplification is just one mechanism for achieving clinically relevant activation of a gene or pathway, and other mechanisms may converge to the same pathway, leading to the same phenotypic consequences. Using alternative genome-wide features to capture a broader range of these “CNA phenotypic copy” mechanisms avoids creating artificially and biologically meaningless distinctions in the training set that could confound ML models. Furthermore, there are indications that patients lacking a specific target mutation but having a transcriptional pattern consistent with that mutation may benefit from the same class of treatments as patients carrying true amplification.

[0345] Figure 15 An exemplary method 1500 is described, which uses a model trained from a general model to a specialized model by fine-tuning a subset of training data to predict the activity of a molecular analyte. In block 1502, a first machine learning model can be trained on multiple medical images from a first cohort. The first machine learning model may be an embedding model that includes any features of the embedding model described herein. The first machine learning model can be trained to generate tile-level embeddings based on multiple tiles of the medical images. The tile-level embeddings can be fed into a second machine learning model described below. In some examples, at least a subset of the tile-level embeddings is averaged before being fed into the second module of the machine learning model.

[0346] In block 1504, during the first training phase, a second machine learning model can be trained as a general model based on training data including embeddings obtained from the first machine learning model and training data from one or more molecular analyte datasets obtained from the second cohort. The second machine learning model can be trained during the first training phase to predict the activity of imputed molecular analytes, as described with reference to the second machine learning model. The second machine learning model may include any features described herein.

[0347] In block 1506, during the second training phase, the second machine learning model can be trained into a specialized model by fine-tuning the general module based on a subset of the training data. The subset of training data can correspond to patient attributes, which can be a patient / subject cohort (e.g., any set of individuals), a disease (e.g., lung cancer, breast cancer, colorectal cancer), a biomarker, or any combination thereof. Case studies demonstrating the improved performance of the specialized model created by fine-tuning the general model are provided under the following subheadings: Use Case: MET Case Study, Use Case: TACSTD2 Case Study, and Use Case: Cabozantinib Case Study.

[0348] In block 1508, medical images from a patient can be received. The patient may include patient attributes (e.g., a specific type of cancer) corresponding to a subset of the training data. In block 1510, first and second machine learning models can predict the activity of a molecular analyte from the patient's medical images. The predicted activity of the molecular analyte can be used to identify biomarkers, predict patient responses to treatment interventions, etc., as described in this disclosure.

[0349] In block 1510, an annotation map of predicted molecular analyte activity can be generated. This annotation map can be overlaid on medical images. The map can include visualizations that distinguish between normal and tumor tissues, for example, such as... Figure 33A-37 As shown.

[0350] Exemplary experimental studies: basic model and specialized model

[0351] An exemplary study utilized data from the Cancer Genome Atlas (TCGA), a public research resource comprising genetic, molecular, and histological data from 11,000 patients and over 20,000 primary tumors, covering 33 cancer types. Molecular and histological data from an additional 2.6,000 patients were obtained from a commercially available multicenter cancer research resource (Cohort A). Target genes were identified as described above. Commercial pharmaceutical databases were queried to identify drugs tagged with therapeutic categories as antibody-drug conjugates (ADCs), T-cell binders, or antibodies (including monospecific and multispecific antibodies). For ADCs and T-cell binders, drugs in any stage of development were retained, while for antibodies (a larger category), drugs whose development had been discontinued were excluded. The overall list of drugs was filtered to those with the specified targets. Each remaining drug was mapped to an HGNC gene symbol, and the union of all gene symbols was taken, resulting in 352 unique targets.

[0352] Three neural network models were trained using 8-fold cross-validation to predict gene expression, copy number amplification, and gene signatures from 768-dimensional H&E tile embeddings. Training was performed in PyTorch (2.1.0). Two main architecture classes were used. The first is a 4-layer sequence network consisting of alternating linear layers with ReLU and dropout layers, as shown below. Training and evaluation data were input to the models in batches of 2000.

[0353] net = torch.nn.Sequential(

[0354] torch.nn.Linear(768, 512), torch.nn.ReLU(), torch.nn.Dropout(0.6), torch.nn.Linear(512, 256), torch.nn.ReLU(), torch.nn.Dropout(0.6), torch.nn.Linear(256, 256), torch.nn.ReLU(), torch.nn.Dropout(0.6), torch.nn.Linear(256, N_GENES)) The second type of model extends the first type by incorporating cross-tile attention, based on the transMIL architecture, using a batch size of 1 and gradient accumulation of 400 batches. It is optimized using Adam, with a learning rate from... Initially, after two consecutive epochs of validation showing no improvement in loss, the loss exhibited exponential decay (gamma). The early stopping threshold, which indicates that the loss has not improved after 3 or 4 consecutive epochs (depending on the model), is used to indicate that training is complete.

[0355] For regression tasks (predicting target expression or amplification features), the objective is Huber loss with delta=1.0. For classification tasks (predicting copy number amplification, elevated target expression, or elevated amplification features), the objective is binary cross-entropy loss, with minority (positive) classes inversely weighted by class prevalence. When classifying elevated target expression or amplification features, a positive class is defined as more than [missing value]. Patients at the percentile (p95). Label smoothing was applied during training: p0-p50 were assigned label 0; p50-p90 were assigned label 0.1; p90-p95 were assigned label 0.9; and p95-p100 were assigned label 1.0. Label smoothing could not be performed in the case of CNA labels because they are binary in nature.

[0356] Based on promising results from the expression and feature classification base model, a specialized model was trained to predict MET expression and features within the NSLC and COAD (colon adenocarcinoma) cohorts. Training was performed on tile-level H&E embeddings using the following architecture: net torch.nn.Sequential( torch.nn.Linear(768, 32), torch.nn.Tanh(), torch.nn.Linear(32,1)) The model's input is restricted to H&E data from the cohorts of interest (NSLC or COAD), maintaining the same subject segmentation as the base model to avoid contamination, but pruning training and evaluation subjects from other cohorts. The specialized model is trained using the Adam optimizer with binary cross-entropy loss and weight decay. The learning rate is 0.001, and early stopping is enabled after three consecutive epochs of evaluation showing no reduction in loss. Binarization and label smoothing are performed as described in the basic model above.

[0357] Model performance was evaluated using an 8-fold cross-validation procedure, where the model was trained at a 7-fold cross-validation and evaluated at a retention cross-validation. For regression tasks, evaluation metrics included Pearson and Spearman correlations. For classification tasks, evaluation metrics included the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC). While the model emits tile-level predictions, tiles are clustered within patients, and labels are patient-level. Performance metrics were aggregated from the tile level to the patient level by averaging. For pan-cancer analyses, performance was evaluated across all patients, while for stratified analyses, performance was first evaluated within each cancer type and then averaged across cancer types. Stratified analyses were limited to cancer types with at least 100 available patients to ensure that performance metrics could be estimated reasonably accurately. Due to the low prevalence of certain CNAs (e.g., <1%), for stratified CNA analyses, only CNAs carrying the target in at least 3 patients of a given cancer type were included.

[0358] Curated OS tags for patients in the Cancer Genome Atlas (TCGA) were obtained from J Liu, T Lichtenberg, KA Hoadley, et al. An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell, 173(2):400-416, 2018. Within cohort A (2.6K patients obtained from commercially available multicenter cancer research resources), treatment-specific OS was defined as the time from treatment initiation to patient death. Patients were censored at last follow-up in cases where no death was reported. Analysis was conducted in cohort A, where more detailed clinical data were available. The hazard ratios for quantifying the association between OS and predictive biomarkers were estimated using a Cox proportional hazards model, adjusted for age at diagnosis, age at disease stage, pre-treatment stage, sex, cancer type, metastatic status, number of prior unique treatments, and time from diagnosis to treatment of interest. Patients were divided into two groups ("high" and "low") based on their amplification characteristics, but without regard to their survival status. The significance of these inter-group differences in survival was assessed using HR from a Cox model. The adjusted Kaplan-Meier curve was calculated using the direct standardization method.

[0359] Results: Biomarker prediction

[0360] CNA Prediction

[0361] Copy number amplification (CNA) was invoked for each of the 352 target genes (hereinafter referred to as "target sites"). In the total cohort ( Within this range, the median amplification prevalence was [missing information]. (scope: arrive See also Figure 16 A multi-task, binary outcome model was developed to simultaneously predict the CNA status of all 352 targets. The input to these predictive models was 256×10⁻⁶ digital whole-slice histopathological images (WSI). Per-pixel tile embedding. Patient-level predictions are obtained by averaging all tiles within the patient's WSI. Here and throughout the process, an 8-fold cross-validation (CV) procedure is used to generate patient predictions, ensuring that the model generating the patient predictions does not see that patient's data during training.

[0362] Figure 17A and 17B A cross-modal comparison of the predictive quality of binary digital biomarkers stratified by cancer type is presented. More specifically, Figure 17A and 17B The AUROC distribution across targets is presented separately, along with the mean AUROC for each of the 26 cancer types in at least 100 patients. Performance was evaluated at the patient level in the retained evaluation set and averaged over 8 cross-validation compromises. Due to the low prevalence of certain CNAs, metrics for targets carrying amplification were reported only when evaluating performance within cancer types, only those carried by at least 3 patients. Means and distributions are shown across up to 352 target genes. For CNAs, the task was to predict whether a patient carried amplification. For target expression (RNA) and amplification signature (SIG), the task was to predict whether a patient's expression / signature level exceeded the 95th percentile. Error bars are... Confidence intervals. The average AUROC across target points is summarized in Table 1. Figure 18A and 18B The heatmaps in the dataset present the area under the receiver operating characteristic (AUROC) curve for each target's predicted CNA, stratified by cancer type. Performance was evaluated at the patient level in the retained evaluation set and averaged over eight cross-validation tradeoffs. Indicators were presented only if at least three patients within a given cancer type carried the mutation in the target gene. A pan-cancer analysis evaluated performance across all available patients.

[0363] Stratified analyses assessed performance separately for each cancer type, then averaged across cancer types. The difference between these methods is that stratified analyses evaluate the model's ability to distinguish CNA risk within cancer types, while pan-cancer analyses examine the model's ability to distinguish risk both within and between cancer types. To illustrate performance within cancer types, performance within two specific cohorts of interest (breast cancer and colorectal cancer) is also presented.

[0364] Table 1: Overall performance of digital histopathological biomarker prediction. Performance was assessed at the patient level in the retained assessment set and averaged over 8 cross-validation trials. Three biomarker types were used: copy number amplification (CNA), target expression level (RNA), and amplification feature score (SIG). For binary classification, the area under the receiver operating characteristic (AUROC) curve is shown. For regression, Spearman correlation between observed and predicted values ​​is shown.

[0365]

[0366] Expression prediction

[0367] Previous studies have linked copy number amplification with differential gene expression across cancer types. Figure 19 The study showed differential expression between patients with and without CNA at 347 targets with available RNA. Of these, 207 (59.7%) showed significant differential expression, with the vast majority (197 / 207; 95.2%) exhibiting higher mean expression in patients with amplification. It is hypothesized that by providing continuous supervisory signals, modeling RNA will be able to train more accurate biomarker prediction models.

[0368] Therefore, a multi-task, continuous-outcome model was developed to simultaneously predict the expression levels of 347 out of 352 targets with available RNA based on histopathological tile embedding. Figure 20 A comparison was made between observed and predicted expression matrices for pan-cancer studies. Predictions were generated through cross-validation, ensuring that patients were not used to train the model that generated the predictions. The matrix on the left shows the observed gene expression or feature matrix. The matrix on the right shows the predictions based on digital pathology. Colors within the matrices describe the magnitude of expression levels or amplified features. Color bars on the left of each figure annotate the cancer type.

[0369] Figure 21 The figure presents a subset of matrices for breast and colorectal cancer, illustrating a comparison of observed expression / feature matrices with histopathology-based prediction matrices stratified by cancer type. Predictions are generated through cross-validation, ensuring that patients are not used to train the model that generates its predictions. The matrices on the left show the observed gene expression or feature matrices. The middle matrix shows the best-performing prediction model. The matrices on the right show a spatially aware model incorporating tile-level transformer attention. Color bars on the left of each figure annotate the cancer type. Figure 22A The distribution of the correlation between observed and predicted expression levels across target patients is presented. Figure 22B The average correlation by cancer type is illustrated. Specifically, patient-level predictions are first generated using 8-fold CV, and then, for each target, the correlation between observed and predicted expression levels across patients is calculated. At least... Indicators were calculated separately for each cancer type in each patient. The distribution revealed up to 352 target genes. For RNA, the task was to predict standardized... Expression levels. For SIG, the task is to predict minimum-maximal normalized amplification features. For pan-cancer, the average cross-validation Spearman correlation is... Stratified by cancer type, the average Spearman correlation is: (Table 1). As expected, the correlations were higher in the pan-cancer analysis, where the model benefited from learning to distinguish differences within and between cancer types. Figure 23A and 23B The correlations of all targets broken down by cancer type are presented. Specifically, Figure 23A and 23B Displays amplification features predicted from digital histopathology, stratified by cancer type. Performance was assessed at the patient level in the retained evaluation set and averaged over 8 cross-validation compromises.

[0370] To compare with a binary CNA prediction task, the expression of each target was bisected at its 95th percentile (p95), and a multi-task binary outcome model was developed to predict whether a patient's expression exceeded p95, indicating high target expression. Figure 24A and 24B Results for all targets stratified by cancer type are presented. Specifically, Figure 24A and 24B AUROC and AUPRC, stratified by cancer type, are displayed based on digital histopathology predictions of elevated target expression. Elevated expression was defined as a patient's expression level exceeding the 95th percentile for a given target. Performance was evaluated at the patient level in the retained evaluation set and averaged across eight cross-validation trials. Figure 17A and 17B The distribution and mean values ​​for each cancer type are presented. As expected, elevated target expression is generally more predictable than CNA status. As shown in Table 1, pan-cancer AUROC from Increase to Layered AUROC from Increase to .

[0371] Amplification characteristics

[0372] As mentioned above (see) Figure 19There is only moderate concordance between CNAs and differential expression. Therefore, a broader transcriptional signature was developed to capture expression changes beyond the target gene itself, which is expected to provide a better predictor of target CNAs. For each of the 352 targets, all genes differentially expressed between patients with and without amplification were identified, and RNA-based amplification signatures were constructed using the differentially expressed genes (for one gene, no differentially expressed gene was identified). Amplification signatures are linear combinations of expression levels, weighted by the magnitude of the differential expression evidence. Signatures were min-max normalized to unit intervals for ease of comparison. Figure 25 The distribution of feature scores is presented in patients with and without amplification. The average distribution of up to 351 amplification features is shown. Patients with amplification have a higher average feature score compared to patients without amplification. ( Figure 26 ). Figure 26 The mean characteristic scores are shown for patients with amplification (cases) and those without amplification (controls). The mean was calculated across up to 351 amplification features. For all 351 amplification features, at least nominally significant evidence of a difference in scores between patients with and without CNA was obtained by Wilcoxon rank-sum test (median P-value: [missing value]). ). Figure 27 The distribution of correlations between amplification signatures and amplified gene expression is described across targets and pan-cancer. For each of the 352 targets, correlations were calculated across 14K subjects. The distribution of the 352 correlations is shown. Overall, the correlations are low, with a median R² of only [value missing]. ( Figure 28 ). Figure 28 The quadratic correlation between amplification signatures and amplified gene expression is shown, pan-cancer. The correlation is calculated for each of the 352 targets in pan-cancer.

[0373] Based on the work on predicting target expression, a multi-task, continuous outcome model, similar to the expression model, was developed to predict 351 amplification features from histopathological tile embeddings. Figure 22A and 22B The distribution of correlations across targets, patient observations, and predicted amplification features is presented, along with the average correlation by cancer type. As shown in Table 1 above, the predictions of amplification features are, on average, more accurate than those of target expression. For pan-cancer, the average cross-validation Spearman correlation is... Stratified by cancer type, the average Spearman correlation is: (Table 1). Figure 23A and 23B The correlations of all targets broken down by cancer type are presented.

[0374] A binary prediction task was also created, where the goal was to predict whether a patient had elevated amplification features by bifurcerating each amplification feature at p95. Cross-target distribution and mean AUROC stratified by cancer type were shown. Figure 17A and 17B In the pan-cancer category, the average AUROC is [value missing]. When stratified by cancer type, the average AUROC was 77.9% (Table 1). Figure 29 The number of targets predicted by AUROC exceeding a given threshold is presented for a binary classification task involving CNA, target expression, and amplification features. Performance is evaluated at the patient level in the retained assessment set and averaged across 8 cross-validation compromises. For the pan-cancer assessment (left), AUROC is calculated across all patients. For the stratified assessment (right), AUROC is calculated separately within each cancer type and then averaged across cancer types. For copy number amplification, the task is to predict whether a patient carries amplification. For target expression and amplification features, the task is to predict whether a patient's expression / feature level exceeds the 95th percentile. Figure 30 Counts exceeding various cutoff values ​​were summarized. For example, CNAs of 142 targets, elevated expression of 339 targets, and elevated features of 335 targets could be predicted with an AUROC exceeding 0.75, pan-cancer. The corresponding counts for stratified analysis were 25, 97, and 201, respectively, for CNA, expression, and features. Note that achieving an AUROC of 0.75 or higher in stratified analysis is a fairly high standard, as it requires the model to distinguish risk within each cancer type and to do so effectively across many cancer types.

[0375] use cases

[0376] The model's utility was evaluated across multiple applications. Because the research design was based on a set of treatment-related targets, a target-driven perspective was adopted when exploring use cases.

[0377] Use Case: MET Case Study

[0378] The first target evaluated was MET, a target for which multiple therapeutic approaches are available and under development. Copy number alterations in MET are associated with worse overall survival across tumor types, particularly in non-small cell lung cancer (NSCLC). Standard evaluation for patient eligibility for MET-targeted therapy utilizes IHC-based biomarkers; indeed, ADCtelisotuzumab vedotin has received FDA Breakthrough Therapy Designation for patients with high MET overexpression levels. However, MET IHC has been shown to have poor concordance with MET CNA. Furthermore, it is known that mechanisms driving MET overexpression other than CNA also generally lead to worse outcomes and are often more common than amplification events. For example, in NSCLC, MET overexpression... It was found in cases, while amplification occurred only in approximately This occurs within the context of MET. Therefore, there is ample opportunity to develop better biomarkers to identify patients eligible for MET-targeted therapy.

[0379] The prevalence of MET amplification in the total cohort being evaluated was First, the performance of the core (non-specific) model was investigated, as mentioned above, to predict MET overexpression and elevated MET amplification characteristics. Figure 31A As shown, for pan-cancer, the prediction of MET overexpression achieved an AUROC of 0.91, and the prediction of elevated amplification features achieved an AUROC of 0.84. Figure 31A The images show the performance of the model trained to predict overexpression or elevated amplification features, derived from the embeddings of the base model (left). The center shows the performance when the model is further specialized for prediction within NSCLC, while the right shows the performance of the model trained for colorectal cancer prediction, in the case of MET. Within NSCLC, the pan-cancer model achieved AUROCs of only 0.69 and 0.78, respectively, for predicting overexpression and elevated amplification features.

[0380] The reasoning suggests that specialized models (e.g., by training the predictive component of a model on a specific cohort) can improve model performance within that cohort. Indeed, when the model was specifically trained on NSCLC patients, it achieved an AUROC of 0.58 for predicting MET amplification, 0.79 for overexpression, and 0.84 for predicting elevated amplification features. The model was also able to predict quantitative MET expression with a correlation of 0.38. In a work conducted concurrently with this work, K Ingale, SH Hong, JSK Bell, et al. Prediction of met overexpression in non-small cell lungadenocarcinomas from hematoxylin and eosin images. arXiv, 2023. preprint, similarly predicted MET overexpression in NSCLC. Ingale et al were able to train on a larger cohort of NSCLC patients—605 MET+ patients compared to the 38 in the study described here—but used typical supervised training methods and a single-task model. Their method achieved an AUROC of 0.74 (compared to 0.79 for the overexpression prediction AUROC achieved by the model described in this paper), but in an artificially balanced test cohort with an equal number of cases and controls. In this case, the method described in this paper provided an AUROC of 0.87. Further tests were conducted to determine whether the biomarkers described in this paper had the potential to increase the patient ensemble predicting increased MET activity. In fact, MET CNA identified only 38 NSCLC patients, MET RNA overexpression identified 72 patients, and MET amplification signature identified 88 patients.

[0381] The pan-cancer approach described in this paper can also be used to identify new opportunities for biomarker deployment. In particular, the analysis revealed strong performance in predicting MET in colorectal cancer. Although MET amplification is rare in colorectal cancer, previous work has noted that MET overexpression is more common and is a prognostic factor for worse survival outcomes. Therefore, a specialized model was similarly trained for colorectal cancer. Figure 31A As shown, the model predicted an AUROC of 0.81 for MET overexpression and an AUROC of 0.85 for increased amplification features.

[0382] Use Case: TACSTD2 Case Study

[0383] Antibody-drug conjugates targeting the TACSTD2-encoded protein (called trophoblast antigen 2 (TROP2)) are being actively developed by several companies. To date, Trodelvy (sacituzumab govitecan) has been approved for urothelial carcinoma and breast cancer (HR+HER2- and TNBC) and is being actively developed for other indications such as NSCLC. Similarly, Dato-DXd (datopotamab deruxtecan) is being actively pursued in breast cancer and NSCLC. As an oncology target, TROP2 has attracted attention due to its expression in many solid tumors and its limited expression in normal tissues. Furthermore, meta-analyses of multiple studies have shown that overexpression of TACSTD2 is associated with poorer overall survival and reduced disease-free survival.

[0384] Using the pan-cancer approach described in this paper, we found that elevated TACSTD2 expression predicted a 0.85 AUROC in pan-cancer, a 0.63 AUROC in BRCA, and a 0.75 AUROC in NSCLC, as expected. However, the pan-cancer baseline model also suggests predictive ability in several other cancer types, including pancreatic cancer (AUROC: 0.79), gastric cancer (AUROC: 0.89), and thyroid cancer (AUROC: 0.73). Previous work has shown that TROP2 overexpression occurs in these cancer types. Others have recently reported preclinical evidence of tumor shrinkage using another TROP2-targeting ADC in a mouse model of pancreatic cancer xenograft, consistent with the findings described in this paper.

[0385] The potential of the TACSTD2 biomarker was further investigated by developing specific, cohort-specific overexpression and characterization models in NSCLC and pancreatic cancer. The performance of the resulting models showed… Figure 31B middle. Figure 31B The left side shows the model performance trained to predict overexpression or amplified features, derived from the embeddings of the base model. The center shows the performance when the model is further specialized for prediction within NSCLC. The right side shows the model performance trained for pancreatic cancer prediction, in the case of TACSTD2. In both cases, specialization improves feature prediction performance at the cost of some loss in expression prediction performance. In NSCLC, the AUROC for feature prediction increases from 0.75 to 0.82, and in pancreatic cancer from 0.89 to 0.90. Meanwhile, for overexpression prediction, the AUROC for NSCLC decreases from 0.79 to 0.72, and in pancreatic cancer from 0.79 to 0.78. In this scenario, a pan-cancer model can be retained for predicting overexpression while a specialized model is deployed for amplified feature prediction.

[0386] Use Case: Cabozantinib Case Study

[0387] A key clinical application of the method described in this article is the ability to stratify patients into responders and non-responders using biomarkers. Unfortunately, the availability of clinical outcomes in the cohort described in this article is limited, particularly for targeted therapies, which are often relatively new in clinical practice. To increase the set of testable hypotheses, the evaluation is extended beyond biologics to consider any targeted therapy against the selected target, where a sufficient number of patients ( This was used to appropriately power the analysis. This resulted in 38 (indication, target) pairs. The association between imputation features and overall survival (OS) was tested after adjusting for age at diagnosis, age at disease stage, pre-treatment stage, sex, cancer type, metastatic status, number of prior unique treatments, and time from diagnosis to treatment of interest.

[0388] Analysis revealed a significant association between VEGFR2 (KDR) amplification characteristics and overall survival (OS) in patients treated with cabozantinib (hazard ratio [HR]: CI, 0.032 to 0.237; Bonferroni adjusted The covariate-adjusted Kaplan-Meier curves comparing patients with low and high VEGFR2 characteristic scores are shown in... Figure 32 Patients were assigned to two groups ("high" and "low") based on their VEGFR2 (KDR) amplification characteristics, but not based on their survival. Reported hazard ratios (HRs) and p-values ​​comparing patients in the low vs. high VEGFR2 characteristic groups were estimated using a Cox model, adjusted for clinical covariates. KM curves were adjusted for covariates using a direct standardization method. Importantly, no outcome data (for cabozantinib or any other drug) were used to guide the design of the amplification characteristics. For comparison, the HR for the MET characteristic in the same patient group was 1.39 (95% CI, 0.573 to 3.351). Analysis of measured VEGFR2 expression levels also showed an association with improved overall survival (OS) in patients treated with cabozantinib (HR: 0.727), although the evidence was inconclusive. This demonstrates the enhancing ability of the imputation feature. Notably, the VEGFR2 feature (measured or imputed) was not associated with a broader improvement in overall survival (OS) in 372 patients in cohort A of RCC, suggesting that the clinical benefit was specific to cabozantinib.

[0389] Cabozantinib is a broad-spectrum tyrosine kinase inhibitor (TKI) with activity against MET, RET, AXL, VEGFR2, FLT3, and c-KIT

[61] , and has been approved for the treatment of renal cell carcinoma (RCC), medullary thyroid carcinoma, and hepatocellular carcinoma. In 31 patients treated with cabozantinib, the majority ( The patient was diagnosed with renal cell carcinoma. VEGF-A is a known prognostic biomarker in metastatic RCC, and high levels of VEGF-A are associated with poorer overall survival (OS) and progression-free survival in patients treated with sunitinib (another TKI). Previous studies have shown that angiogenic microvessel density and mast cell density biomarkers are associated with improved outcomes in metastatic clear cell RCC; however, these do not appear to predict the efficacy of cabozantinib relative to everolimus (an mTOR inhibitor). Despite this finding, given the known relationship between RCC biology and VEGF signaling and the proposed MOA of cabozantinib, the highly significant association between VEGFR2 signatures and OS in patients treated with cabozantinib may be of interest for future biomarker development and could provide suggestive evidence for VEGF as a mechanism by which cabozantinib achieves efficacy in RCC.

[0390] Use Case: Exploring Spatial Heterogeneity

[0391] A key property of the models described in this paper is their ability to generate biomarker predictions at the resolution of a single tile. This provides the capability to generate spatial gene expression predictions on WSI. Specifically, the model can make predictions for each biomarker and each tile, allowing for the creation of synthetic annotations on top of the WSI, where biomarker predictions are overlaid on each tile within the slice. This capability is useful in several ways. First, it “opens the black box” by providing human experts with the ability to explore the processes that produce the results. Second, it creates views of the spatial distribution of multiple biomarkers, providing considerable insight into tumor structure and intratumoral heterogeneity. In fact, since these imputations are derived directly from H&E, this capability supports a form of “label-free” staining across a very large set of molecular readouts.

[0392] Figure 33A Examples of these synthetic stackings are shown, localizing HER2 expression in breast cancer and MET expression in colorectal cancer. Figure 33B A similar overlay of amplification feature predictions was shown. To provide a baseline for these predictions, expert pathologists were asked to annotate random WSI samples from patients with and without amplification in a blinded manner, unaware of all model predictions. The fact that increased expression was consistent with regions annotated as cancerous aligns with clinical knowledge and indicates that the model has learned to distinguish between tumor and normal tissue. Importantly, the model learned this distinction when trained only on batch rather than spatially resolved expression data. Figure 34 and 35 An alternative view of these results is provided, in which target expression and amplification feature predictions are juxtaposed. Figure 34 This paper describes a comparison between expression and feature predictions in breast cancer and annotations by expert pathologists. Pathologists again blinded the predictions, even though the expression / feature models provide tile-level predictions; they are trained only on batch rather than spatially resolved information. Figure 35A similar comparison between expression and characterization prediction in colorectal cancer and expert pathologist annotations is described.

[0393] Since molecular tags are all synthesized, multiple tags can be exported for the same image. Figure 36 and Figure 37 Examples of HER3+MET and TOP1+TOP2A co-expression predictions are provided, along with blinded pathologist annotations. The spatial expression prediction differences of these target pairs highlight that the model is learning more nuanced expression information than just whether a tile falls within a cancerous region of a slice.

[0394] Machine learning modeling insights

[0395] Machine learning modeling insights: cross-cohort portability

[0396] An important aspect of machine learning models is their ability to generalize beyond the training distribution. This generalization is crucial in evaluating the robustness of a method—that is, avoiding overfitting to the specificity of a single dataset. It is also useful from a clinical deployment perspective, increasing confidence that the model will perform well when applied in new clinical settings.

[0397] Table 2 presents the AUROC of an experiment in which a multi-task binary outcome model was trained to predict elevated expression and amplification features using only TCGA data, and then evaluated only on patients in cohort A. Results show pan-cancer outcomes as well as within breast and colorectal cancer. Stratified results are not presented because the cancer set available only in cohort A differs from that available in TCGA+ cohort A, making the results incomparable to those presented elsewhere. While significant predictive power was preserved, performance degradation is always expected when applying the model to new datasets. Surprisingly, for breast and colorectal cancer, elevated feature prediction improved across datasets, potentially indicating that these cohorts are more heterogeneous in TCGA than in cohort A.

[0398] Table 2: Portability of binary expression and amplification predictions across datasets. Average AUROCs across targets are reported. Intra-dataset results were obtained through cross-validation, using data from TCGA and cohort A for training and testing. Cross-dataset results were obtained by training only on TCGA and then evaluating only on cohort A.

[0399]

[0400] Machine Learning Modeling Insights: Machine Learning Architecture

[0401] Three different model architectures were evaluated in the exemplary study described in this paper, as summarized in Table 3. Panels (a) and (b) of Table 3 explore different ways of combining different tile information. Panel (a) considers whether the model should receive individual embeddings of each tile in the patient's WSI as input, or an average embedding across tiles. Maintaining individual embeddings for each tile performs better, which essentially provides the model with more (albeit relevant) training examples. Panel (b) considers whether to generate predictions individually for each tile, or to incorporate an attention mechanism that allows the model to make patient-level predictions while paying attention to spatially adjacent tiles. While spatial attention did not benefit the model overall, predictions for certain targets did benefit; for example, in TLR9, the Spearman correlation between observed and predicted expression increased from 20% to 48%. Panel (c) considers the value of training across multiple biomarker tasks. Specifically, the following approaches were compared: (1) training a single model to predict each target, (2) training a single model to predict all targets simultaneously, or (3) training a single model to predict whole transcriptome expression and then a subset of targets of interest. The last strategy performed best, with both multi-task strategies significantly outperforming the single (single-task) strategy of the model architecture.

[0402] Table 3

[0403] (a) Average number of tiles

[0404] (b) Attention

[0405] (c) Multitasking

[0406] Other data from the exemplary study

[0407] Figure 38A and 38B A cross-modal comparison of the predictive quality of binary digital biomarkers stratified by cancer type is described. Performance was evaluated at the patient level in the retained assessment set and averaged over eight cross-validation tradeoffs. Indicators were calculated separately for each cancer type in the patients. The distribution showed up to 352 target genes. For CNA, the task was to predict whether the patient carried amplification. For target expression (RNA) and amplification signature (SIG), the task was to predict whether the patient's expression / signature level exceeded the 95th percentile.

[0408] Figure 39A and 39B This study demonstrates the prediction of target expression levels from digital histopathology, stratified by cancer type. Performance was evaluated at the patient level in the retained assessment set and averaged across 8 cross-validation compromises.

[0409] Figure 40A and 40B This study describes the prediction of elevated amplification features from digital histopathology, stratified by cancer type. A patient was defined as having elevated amplification features if their score exceeded the 95th percentile for a given target. Performance was evaluated at the patient level in the retained assessment set and averaged across eight cross-validation compromises.

[0410] Figure 41 The performance of the binary classification task model across biomarkers and stratified by cancer type is illustrated. Performance metrics are the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic (AUROC) curve. Performance was evaluated at the patient level within the retained evaluation set. For copy number amplification (CNA), the task was to predict whether a patient carried the amplification. For target expression (RNA) and amplification signature, the task was to predict whether a patient's expression / signature level exceeded the 95th percentile. The distribution shows 352 target genes.

[0411] Figure 42 This describes the target counts exceeding a given threshold for both pan-cancer and stratified binary classification tasks using AUPRC. Performance was evaluated at the patient level within the retained assessment set and averaged across 8 cross-validation tradeoffs. For pan-cancer assessment, AUPRC was calculated across all patients. For stratified assessment, AUPRC was calculated separately within each cancer type and then averaged across cancer types. For copy number amplification, the task was to predict whether a patient carried amplification. For target expression and amplification features, the task was to predict whether a patient's expression / feature level exceeded the threshold. .

[0412] Figure 43 This describes the number of genes for which AUROC exceeds a given threshold. Performance was evaluated at the patient level within the retained assessment set and averaged across 8 cross-validation tradeoffs. For pan-cancer assessment, AUROC was calculated across all patients. For stratified assessment, AUROC was calculated separately within each cancer type and then averaged across cancer types.

[0413] Figure 44A and 44B This study illustrates a cross-modal comparison of the predictive quality of continuous digital biomarkers stratified by cancer type. Performance was assessed at the patient level in the retained assessment set and averaged over eight cross-validation compromises. Indicators were calculated separately for each cancer type in each patient. The distribution revealed up to 352 target genes. For RNA, the task was to predict standardized... Expression level. For SIG, the task is to predict min-max normalized amplification features.

[0414] Figure 45Performance on regression tasks across biomarkers, pan-cancer, and two specific cancer types is illustrated. The task is to predict continuous expression levels or amplified feature scores. Performance was evaluated at the patient level in the retained evaluation set and averaged over eight cross-validation tradeoffs. Distributions show up to 352 target genes.

[0415] Figure 46 This describes the number of targets for which Pearson and Spearman R2 exceed a given threshold for the pan-cancer regression task. Performance was evaluated at the patient level in the retained evaluation set and averaged over 8 cross-validation compromises. Pearson and Spearman are calculated across the pan-cancer domain.

[0416] Figure 47 This describes the target counts for which Pearson and Spearman R2 exceeded a given threshold for the hierarchical regression task. Performance was evaluated at the patient level in the retained evaluation set and averaged over 8 cross-validation compromises. Pearson and Spearman R2 exceeded the target counts for at least [number missing]. The results were calculated separately for each cancer type in each patient, and then averaged across cancer types.

[0417] Figure 48A and 48B This study illustrates a cross-modal comparison of the predictive quality of digital biomarkers stratified by cancer type. Performance was evaluated at the patient level in the retained assessment set and averaged over eight cross-validation tradeoffs. (The last sentence appears to be incomplete and requires further context.) Indicators were calculated separately for each cancer type in each patient, and then averaged across cancer types. The distribution revealed up to 352 target genes. Figure 48A This displays predictions of continuous target expression (RNA) and amplification characteristic levels. Figure 48B This indicates a binary copy number amplification (CNA) status or a prediction of being in the upper 5 percentile (RNA and trait). RNA expression is... Measured per million transcripts. Amplification characteristics are based on differentially expressed genes in patients with and without amplification.

[0418] Figure 49 This describes the prevalence of any target amplification and any elevated amplification trait stratified by cancer type. Prevalence is calculated at the patient level for up to 352 target genes. A patient is considered to have an elevated amplification trait if their score exceeds the 95th percentile for a given target.

[0419] Figure 50 This illustrates the characteristic distribution of amplification features according to the patient's amplification status. The average distribution of up to 351 amplification features is shown. Figure 51 This illustrates the pan-cancer distribution of the correlation between amplification features and amplified gene expression. For each of the 352 targets, the correlation was calculated across 14K subjects. The distribution of the 352 correlations is shown.

[0420] Figure 52 This illustrates the stratified squared correlation between amplification characteristics and amplified gene expression. Correlation was first calculated within each cancer type, then averaged across all cancer types. The summary statistics show up to 352 correlations.

[0421] The method described in this paper derives a complete molecular profile from routinely collected histopathological images, defining a "semi-synthetic" cohort in which imputed molecular data inferred from real H&E supplement other measured covariates, including patient demographics, medical history, treatment, and clinical outcomes. Given the richness of the cohort including H&E and these other covariates, a very large semi-synthetic cohort can be generated, providing high statistical power for extensive exploratory analyses. Specifically, diverse multimodal biomarkers can be explored and even constructed, evaluated as described herein to determine which are good predictors, and associations with clinically relevant covariates such as CNA, survival, or treatment response can be identified.

[0422] In addition to identifying biomarkers for a given target within a selected tumor type, the methods described herein are also capable of identifying potential novel therapeutic opportunities. Specifically, the pan-cancer results described herein demonstrate the ability to accurately imput expression levels from a variety of cancers originating from very different tissues. These predictions can help highlight cancers where the cancer target is significantly expressed, reaching levels that may be therapeutically relevant (compared to other cancers where the MOA is deployed). This may suggest new opportunities to expand the set of indications for a given targeted therapy. While these insights may be obtained from molecular data collected across tumor types, such data are not collected regularly as part of standard care, making it difficult to detect these opportunities, particularly for rare cancers and / or smaller patient subgroups. Where clinical outcomes of treatment response are available, associations between features (e.g., amplification features) and these outcomes can be assessed. As shown in the preliminary analysis of cabozantinib responses described in the Exemplary Studies section below, these associations may help understand which aspects of the drug MOA drive efficacy, thus suggesting potential pathways to generate improved chemicals. Preliminary results from the cabozantinib case study demonstrate the potential of machine learning-defined features to predict superior clinical outcomes for a particular targeted therapy without training on any response or outcome data for that drug.

[0423] In summary, this approach enables the identification of patients who may benefit from targeted therapy using universally collected H&E images. This capability can be deployed in multiple ways. For example, it can be used as a rapid triage step to suggest a set of treatment interventions that may be relevant to a patient; this step can then be followed by the deployment of other, more standard biomarker assays, such as gene sequencing or IHC, to verify that the patient is indeed eligible for medication, given the currently approved label. This ML-based H&E biomarker can be rapidly deployed across geographic regions, requiring no specialized equipment or reagents other than H&E staining and scanning, making it widely accessible. As another example, the techniques described herein may allow for the direct use of individual patient-derived H&E biomarkers to identify and prescribe treatment interventions. Unlike most other biomarkers, which typically focus on one or two molecular measurements, the H&E biomarkers described herein rely on the complete context of whole-slice images, providing a broad, detailed, multi-scale phenotype. Therefore, they may detect more diffuse slice-level evidence, better capturing coherent patient populations that may have similar outcomes in treatment. This analysis may help identify patients who are unlikely to benefit, enabling clinicians to recommend different treatment options. Furthermore, it may identify new patients. In fact, the patient pool at the higher end (95th percentile) of RNA (expression) and amplification biomarkers is much larger than the patient pool defined directly by CNA; therefore, expressing and amplifying biomarkers can help expand the patient population that may benefit from the drug. It is noteworthy that this approach is generalizable to a wide range of targeted therapies.

[0424] For purposes of explanation, the foregoing description has been made with reference to specific examples or aspects. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. For the purpose of clarity and concise description, features are described as part of the same or individual variations; however, it should be understood that the scope of the disclosure includes variations having all or part of the described features. In view of the foregoing teachings, many modifications and variations are possible. These variations were chosen and described in order to best explain the principles of the technology and its practical application. Others skilled in the art will therefore be able to best utilize the technology and various variations, as well as various modifications suitable for the particular application considered.

[0425] Although the disclosures and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. These changes and modifications should be understood to be included within the scope of the disclosures and examples as defined by the claims. Finally, the full disclosures of the patents and publications mentioned in this application are hereby incorporated by reference.

Claims

1. A system for predicting the activity of molecular analytes in patients, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: a first module for training a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module for training the machine learning model based on a dataset of one or more molecular analytes obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analytes from the medical images of the patient.

2. The system of claim 1, wherein the one or more programs further include instructions for determining whether the patient belongs to one or more subgroups based on the predicted activity of the molecular analyte.

3. The system according to any one of the preceding claims, wherein the one or more programs further include instructions for: training a third module of the machine learning model based on a third cohort, the third cohort comprising multiple medical images and associated clinical outcomes, wherein the third module of the machine learning model is configured to predict treatment and / or clinical outcomes.

4. The system of claim 3, wherein the one or more programs further include instructions for: using the third machine learning model to determine a measure of significance or prognostic value of the molecular analyte to dynamically select a subset of the molecular analytes for subsequent use.

5. The system according to any one of the preceding claims, wherein the second module and / or the third module of the machine learning model are trained using transfer learning.

6. The system according to any one of the preceding claims, wherein the one or more molecular analyte datasets comprise: Gene expression data; Copy number amplification (CNA) data; Amplify feature data; Chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole-genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

7. The system of claim 6, wherein the one or more molecular analyte datasets comprise: Gene expression values ​​including transcript abundance; copy number amplification values; amplification characteristic values; Chromosome accessibility score including ATAC-seq peaks; abundance of one or more histone modifications including ChIP-seq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

8. The system according to any one of the preceding claims, wherein the one or more molecular analyte datasets comprise two molecular analyte datasets.

9. The system according to any one of the preceding claims, wherein medical images from the patient are obtained from a fourth queue, the fourth queue comprising multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images.

10. The system of claim 9, wherein the one or more procedures further include instructions for determining that each patient in the fourth cohort belongs to one or more subgroups.

11. The system according to any one of the preceding claims, wherein the first queue comprises multiple medical images from multiple patients.

12. The system of claim 11, wherein the plurality of medical images comprises: One or more histopathological images; One or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; Or any combination thereof.

13. The system of claim 11 or claim 12, wherein the plurality of medical images are unlabeled, and the first module is trained using unsupervised learning.

14. The system according to any one of the preceding claims, wherein the first queue and the second queue are the same queue.

15. The system according to any one of the preceding claims, wherein the second queue comprises data of a plurality of medical images and one or more related molecular analytes.

16. The system of claim 3, wherein the third queue comprises multiple medical images and associated clinical outcomes.

17. The system of claim 3, wherein the first, second, or third cohort further comprises one or more clinical covariates.

18. The system of claim 17, wherein the one or more clinical covariates include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

19. The system of claim 3, wherein the one or more programs further include instructions for removing specific deviations in the data in the first, second and third queues.

20. The system according to any one of the preceding claims, wherein the one or more programs further include instructions for: receiving a medical image of a new patient; obtaining an embedding by providing the medical image of the new patient to the first module; and mapping the embedding based on domain adaptation.

21. The system according to any one of the preceding claims, wherein the molecular analyte is a first molecular analyte, and the one or more procedures further include instructions for: training the machine learning model based on the second module using transfer learning, wherein the fourth module is configured to predict a second molecular analyte associated with the first molecular analyte.

22. The system according to any one of the preceding claims, wherein the one or more programs further include instructions for calculating a continuous score.

23. The system of claim 1, wherein the second module for training the machine learning model comprises: In the first phase, a general module is trained based on training data from the one or more molecular analyte datasets obtained from the second queue; In the second phase, the general module is fine-tuned based on a subset of the training data to obtain the second module.

24. The system of claim 23, wherein the subset of the training data corresponds to patient attributes.

25. The system of claim 24, wherein the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

26. The system of claim 24, wherein the patient has the patient attributes.

27. The system of claim 1, wherein the first module of the machine learning model is trained to generate tile-level embeddings based on a plurality of tiles of the medical image, and wherein the tile-level embeddings are input into the second module of the machine learning model.

28. The system of claim 27, wherein at least a subset of the tile-level embeddings is averaged before being input into a second module of the machine learning model.

29. The system of claim 1, wherein the second module of the machine learning model includes an attention mechanism.

30. The system of claim 1, wherein the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

31. The system of claim 30, wherein the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

32. A method for predicting the activity of a patient's molecular analyte, comprising: A first module for training a machine learning model based on multiple medical images from a first queue, wherein the first module includes an embedding module; A second module of the machine learning model is trained based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analytes from the patient's medical images.

33. A non-transient computer-readable storage medium storing one or more programs for predicting the activity of molecular analytes in a patient, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform: a first module training a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module training the machine learning model based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analyte from the medical images of the patient.

34. A system for predicting the activity of molecular analytes in patients, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for training a first machine learning model on a plurality of medical images from a first queue; A second machine learning model is trained on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from the second cohort; medical images from the patient are received; and the activity of the molecular analytes is predicted from the patient's medical images using the trained second machine learning model.

35. The system of claim 34, wherein the one or more procedures further include instructions for determining whether the patient belongs to one or more subgroups based on the predicted activity of the molecular analyte.

36. The system according to any one of claims 34-35, wherein the one or more programs further include instructions for training a third machine learning model based on a third cohort, the third cohort comprising multiple medical images and associated clinical outcomes, wherein the third machine learning model is configured to predict treatment and / or clinical outcomes.

37. The system according to any one of claims 34-36, wherein the one or more procedures further include instructions for: using the third machine learning model to determine a measure of significance or prognostic value of the molecular analyte to dynamically select a subset of the molecular analytes for subsequent use.

38. The system according to any one of claims 34-37, wherein the second machine learning model and / or the third machine learning model are trained using transfer learning.

39. The system according to any one of claims 34-38, wherein the one or more molecular analyte datasets comprise: Gene expression data; Copy number amplification (CNA) data; Amplify feature data; Chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole-genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

40. The system of claim 39, wherein the one or more molecular analyte datasets comprise: Gene expression values ​​including transcript abundance; copy number amplification values; amplification characteristic values; Chromosome accessibility score including ATAC-seq peaks; abundance of one or more histone modifications including ChIP-seq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

41. The system according to any one of claims 34-40, wherein the one or more molecular analyte datasets comprise two molecular analyte datasets.

42. The system according to any one of claims 34-41, wherein the medical images from the patient are obtained from a fourth queue, the fourth queue comprising multiple medical images from multiple patients and optionally one or more related molecular analyte datasets for each of the multiple medical images.

43. The system of claim 42, wherein the one or more procedures further include instructions for determining that each patient in the fourth cohort belongs to one or more subgroups.

44. The system according to any one of claims 34-43, wherein the first queue comprises multiple medical images from multiple patients.

45. The system of claim 44, wherein the plurality of medical images comprises: One or more histopathological images; One or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; Or any combination thereof.

46. ​​The system of claim 44 or claim 45, wherein the plurality of medical images are unlabeled, and the first machine learning model is trained using unsupervised learning.

47. The system according to any one of claims 34-46, wherein the first queue and the second queue are the same queue.

48. The system according to any one of claims 34-47, wherein the second queue comprises data of multiple medical images and one or more related molecular analytes.

49. The system according to any one of claims 34-48, wherein the third queue comprises multiple medical images and associated clinical outcomes.

50. The system according to any one of claims 34-49, wherein the first, second, or third cohort further comprises one or more clinical covariates.

51. The system of claim 50, wherein the one or more clinical covariates include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

52. The system according to any one of claims 34-51, wherein the one or more programs further include instructions for removing specific deviations in the data in the first, second and third queues.

53. The system according to any one of claims 34-52, wherein the one or more procedures further include instructions for: receiving medical images of a new patient; obtaining an embedding by providing the medical images of the new patient to the first machine learning model; and mapping the embedding based on domain adaptation.

54. The system according to any one of claims 34-53, wherein the molecular analyte is a first molecular analyte, and the one or more procedures further include instructions for: training a fourth machine learning model based on the second machine learning model using transfer learning, wherein the fourth machine learning model is configured to predict a second molecular analyte associated with the first molecular analyte.

55. The system according to any one of claims 34-54, wherein the one or more programs further include instructions for calculating a continuous score.

56. The system of claim 55, wherein training the second machine learning model comprises: In the first phase, a general model is trained based on training data from the one or more molecular analyte datasets obtained from the second queue; In the second stage, the general model is fine-tuned based on a subset of the training data to obtain the second machine learning model.

57. The system of claim 56, wherein the subset of training data corresponds to patient attributes.

58. The system of claim 57, wherein the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

59. The system of claim 57, wherein the patient has the patient attributes.

60. The system of claim 34, wherein the first machine learning model is trained to generate tile-level embeddings based on a plurality of tiles of the medical image, and wherein the tile-level embeddings are input into the second machine learning model.

61. The system of claim 60, wherein at least a subset of the tile-level embeddings is averaged before being input into the second machine learning model.

62. The system of claim 34, wherein the second machine learning model includes an attention mechanism.

63. The system of claim 34, wherein the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

64. The system of claim 63, wherein the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

65. A method for predicting the activity of a patient's molecular analyte, comprising: The first machine learning model was trained on multiple medical images from the first cohort; A second machine learning model is trained on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from the second cohort; medical images from the patient are received; and the activity of the molecular analytes is predicted from the patient's medical images using the trained second machine learning model.

66. A non-transient computer-readable storage medium storing one or more programs for predicting the activity of a molecular analyte in a patient, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: train a first machine learning model on a plurality of medical images from a first cohort; train a second machine learning model on embeddings obtained from the first machine learning model and one or more molecular analyte datasets obtained from a second cohort; receive medical images from the patient; and predict the activity of the molecular analyte from the patient's medical images using the trained second machine learning model.

67. A system for patient stratification, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a first plurality of medical images from a first queue; determining a plurality of embeddings by providing the first plurality of images to a first trained machine learning model; and training a second machine learning model to predict one or more molecular analytes by providing the plurality of embeddings from the first machine learning model and activity data of the one or more molecular analytes from the first queue to a second machine learning model. The calculated activity data of one or more molecular analytes in the second cohort are predicted by providing the trained second machine learning model with a second cohort of medical images. Identify one or more relevant biomarkers based on the inferred activity data and the result data of the second cohort; Receive one or more medical images from the patient; The presence of one or more relevant biomarkers determines whether the patient belongs to one or more patient subgroups.

68. The system of claim 67, wherein the first queue is smaller than the second queue.

69. The system according to any one of claims 67-68, wherein the activity data of the one or more molecular analytes in the first cohort and / or the extrapolated activity data of the second cohort comprises: Gene expression data; Copy number amplification (CNA) data; Amplify feature data; Chromatin accessibility data; DNA methylation data; histone modification data; RNA data; protein data; space biology data; whole-genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

70. The system of claim 69, wherein the activity data of the one or more molecular analytes in the first cohort and / or the extrapolated activity data of the second cohort comprises: Gene expression values ​​that include transcript abundance; Copy number amplification value; Amplified eigenvalues; Chromosome accessibility score including ATAC-seq peaks; abundance of one or more histone modifications including ChIP-seq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

71. The system according to any one of claims 67-70, wherein the first plurality of images in the first queue and / or the second plurality of images in the second queue comprise: One or more histopathological images; One or more magnetic resonance imaging (MRI) images; One or more computed tomography (CT) scans; Or any combination thereof.

72. The system according to any one of claims 67-71, wherein the data associated with the second queue is collected as part of Standard Care (SoC).

73. The system according to any one of claims 67-72, wherein the data associated with the second cohort includes Cancer Genome Atlas (TCGA) data.

74. The system according to any one of claims 67-73, wherein the first trained machine learning model comprises an unsupervised model or a self-supervised model.

75. The system of claim 74, wherein the first trained machine learning model includes a contrastive model.

76. The system according to any one of claims 67-75, wherein the second machine learning model is a linear model.

77. The system according to any one of claims 67-76, wherein the extrapolated activity data is correlated with ATAC-seq peaks.

78. The system according to any one of claims 67-77, wherein identifying the one or more relevant biomarkers comprises: A third machine learning model is used to determine the correlation between the inferred activity data of the second cohort and the result data of the second cohort.

79. The system of claim 78, wherein determining the association comprises: The third machine learning model, configured to predict the activity data of molecular analytes, is trained using the inferred activity data and the result data from the second cohort. Determine a correlation measure indicating the degree of correlation between the activity data of the molecular analyte and clinical outcomes.

80. The system of claim 79, wherein the relevant metric includes: The p-value associated with the third machine learning model.

81. The system according to any one of claims 67-80, wherein the one or more biomarkers include machine learning-based biomarkers or image-based biomarkers.

82. The system according to any one of claims 67-81, wherein determining whether the patient belongs to one or more patient subgroups comprises: One or more embeddings are determined by providing the first machine learning model with one or more images of the patient; Inferential activity data related to the patient is determined by providing the trained machine learning model with one or more embeddings. And determine whether the extrapolated activity data associated with the patient indicates the presence of one or more biomarkers.

83. The system according to any one of claims 67-82, wherein the one or more procedures further include instructions for identifying a treatment for the patient based on the one or more biomarkers and a known mechanism of action (MoA) of the treatment.

84. The system according to any one of claims 67-83, wherein the outcome data indicates mortality, disease diagnosis, disease progression, disease prognosis, disease risk, or any combination thereof, and wherein patient stratification is based on one or more of mortality, disease diagnosis, disease progression, disease prognosis, disease risk, or any combination thereof.

85. The system according to any one of claims 67-84, wherein the one or more programs further include instructions for calculating a continuous score.

86. The system of claim 85, wherein training the second machine learning model comprises: In the first phase, a general model is trained based on training data from the one or more molecular analyte datasets obtained from the second queue; In the second stage, the general model is fine-tuned based on a subset of the training data to obtain the second machine learning model.

87. The system of claim 86, wherein the subset of the training data corresponds to patient attributes.

88. The system of claim 87, wherein the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

89. The system of claim 87, wherein the patient has the patient attributes.

90. The system of claim 67, wherein the first machine learning model is trained to generate tile-level embeddings based on a plurality of tiles of the medical image, and wherein the tile-level embeddings are input into the second machine learning model.

91. The system of claim 90, wherein at least a subset of the tile-level embeddings is averaged before being input into the second machine learning model.

92. The system of claim 67, wherein the second machine learning model includes an attention mechanism.

93. The system of claim 67, wherein the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

94. The system of claim 93, wherein the annotated diagram includes visualizations that distinguish between normal tissue and tumor tissue.

95. A method for patient stratification, comprising: Receive the first multiple medical images from the first queue; Multiple embeddings are determined by providing the first plurality of images to a first trained machine learning model; The second machine learning model is trained to predict one or more molecular analytes by providing the second machine learning model with the plurality of embeddings from the first machine learning model and the activity data of the one or more molecular analytes from the first queue; The calculated activity data of one or more molecular analytes in the second cohort are predicted by providing the trained second machine learning model with a second cohort of medical images. Identify one or more relevant biomarkers based on the inferred activity data and the result data of the second cohort; Receive one or more medical images from the patient; The presence of one or more relevant biomarkers determines whether the patient belongs to one or more patient subgroups.

96. A non-transient computer-readable storage medium storing one or more programs for predicting the activity of molecular analytes in a patient, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: receive a first plurality of medical images from a first queue; determine a plurality of embeddings by providing the first plurality of images to a first trained machine learning model; and train a second machine learning model to predict one or more molecular analytes by providing the plurality of embeddings from the first machine learning model and activity data of the one or more molecular analytes from the first queue to a second machine learning model; The calculated activity data of one or more molecular analytes in the second cohort are predicted by providing the trained second machine learning model with a second cohort of medical images. Identify one or more relevant biomarkers based on the inferred activity data and the result data of the second cohort; Receive one or more medical images from the patient; The presence of one or more relevant biomarkers determines whether the patient belongs to one or more patient subgroups.

97. A system for predicting the activity of molecular analytes in patients, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: a first module for training a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module for training the machine learning model based on a dataset of one or more molecular analytes obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analytes from the medical images of the patient.

98. The system of claim 97, wherein the predicted activity of the molecular analyte includes amplification characteristic data and / or a chromosome accessibility score containing ATAC-seq peaks.

99. The system of claim 98, wherein the amplification feature data is generated based on a plurality of differentially expressed genes and a plurality of weights relative to the amplification.

100. The system of claim 97, wherein the one or more programs further include instructions for: training a third module of the machine learning model based on a third cohort comprising multiple medical images and associated clinical outcomes, wherein the third module of the machine learning model is configured to predict treatment and / or clinical outcomes.

101. The system of claim 100, wherein the one or more programs further include instructions for: using the third machine learning model to determine a measure of significance or prognostic value of the molecular analyte to dynamically select a subset of the molecular analytes for subsequent use.

102. The system of claim 101, wherein the patient is a first patient, and the method further comprises: Using the first and second modules trained by the machine learning model, the activity of at least one of the molecular analytes in the subset is predicted from the medical images of the second patient, and based on the predicted activity, antibody-drug conjugate (ADC) therapy is identified for the second patient.

103. The system of claim 97, wherein the second module of the machine learning model and / or the third module of the machine learning model are trained using transfer learning.

104. The system of claim 97, wherein the one or more molecular analyte datasets comprise: Gene expression data; Copy number amplification (CNA) data; chromatin accessibility data; DNA methylation data; histone modifications; RNA data; protein data; space biology data; whole-genome sequencing (WGS) data; somatic mutation data; germline mutation data; or any combination thereof.

105. The system of claim 104, wherein the one or more molecular analyte datasets comprise: Gene expression values ​​that include transcript abundance; Copy number amplification value; amplification characteristic value; Abundance of one or more histone modifications including ChIP-seq values; abundance of one or more mRNA sequences; abundance of one or more proteins; presence of one or more somatic mutations; presence of one or more germline mutations; presence or absence of one or more specific DNA methylation markers in one or more specific genomic regions, or any combination thereof.

106. The system of claim 97, wherein medical images from the patient are obtained from a fourth queue, the fourth queue comprising multiple medical images from multiple patients and optionally one or more relevant molecular analyte datasets for each of the multiple medical images.

107. The system of claim 106, wherein the one or more procedures further include instructions for determining that each patient in the fourth cohort belongs to one or more subgroups.

108. The system of claim 97, wherein the first queue comprises multiple medical images from multiple patients.

109. The system of claim 108, wherein the plurality of medical images comprises: One or more histopathological images; One or more magnetic resonance imaging (MRI) images; one or more computed tomography (CT) scans; Or any combination thereof.

110. The system of claim 108, wherein the plurality of medical images are unlabeled, and the first module is trained using unsupervised learning.

111. The system of claim 97, wherein the first queue and the second queue are the same queue.

112. The system of claim 100, wherein the third queue comprises multiple medical images and associated clinical outcomes.

113. The system of claim 100, wherein the first, second, or third cohort further comprises one or more clinical covariates.

114. The system of claim 113, wherein the one or more clinical covariates include patient sex, patient age, height, weight, patient diagnosis, patient histological data, patient radiological data, patient medical history, or any combination thereof.

115. The system of claim 100, wherein the one or more programs further include instructions for removing specific deviations in the data in the first, second and third queues.

116. The system of claim 97, wherein the one or more programs further include instructions for: receiving a medical image of a new patient; obtaining an embedding by providing the medical image of the new patient to the first module; and mapping the embedding based on domain adaptation.

117. The system of claim 97, wherein the molecular analyte is a first molecular analyte, and the one or more procedures further include instructions for: training a fourth module of the machine learning model based on the second module using transfer learning, wherein the fourth module is configured to predict a second molecular analyte associated with the first molecular analyte.

118. The system of claim 97, wherein the second module for training the machine learning model comprises: In the first phase, a general module is trained based on training data from the one or more molecular analyte datasets obtained from the second queue; In the second phase, the general module is fine-tuned based on a subset of the training data to obtain the second module.

119. The system of claim 118, wherein the subset of the training data corresponds to patient attributes.

120. The system of claim 118, wherein the patient attributes include patient cohorts, diseases, biomarkers, or any combination thereof.

121. The system of claim 118, wherein the patient has the patient attributes.

122. The system of claim 97, wherein the first module of the machine learning model is trained to generate tile-level embeddings based on a plurality of tiles of the medical image, and wherein the tile-level embeddings are input into the second module of the machine learning model.

123. The system of claim 97, wherein the one or more programs further include instructions for: generating an annotation map of the predicted activity of the molecular analyte; and overlaying the annotation map onto the medical image.

124. A method for predicting the activity of a patient's molecular analyte, comprising: A first module for training a machine learning model based on multiple medical images from a first queue, wherein the first module includes an embedding module; A second module of the machine learning model is trained based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analytes from the patient's medical images.

125. A non-transient computer-readable storage medium storing one or more programs for predicting the activity of molecular analytes in a patient, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform: a first module training a machine learning model based on a plurality of medical images from a first cohort, wherein the first module includes an embedding module; a second module training the machine learning model based on one or more molecular analyte datasets obtained from a second cohort, wherein the second module includes one or more heads; receiving medical images from the patient; and using the trained first and second modules of the machine learning model to predict the activity of the molecular analyte from the medical images of the patient.

126. A biomarker generated by: training one or more machine learning models configured to predict values ​​of a plurality of candidate biomarkers based on embeddings representing medical images; and selecting the biomarker from the plurality of candidate biomarkers.

Images