Clinical predictor based on multiple machine learning models

Abstract

The method includes receiving data corresponding to a plurality of data categories of a patient; selecting a first machine learning model and a second machine learning model from a plurality of trained machine learning models and based on the plurality of data categories, the first machine learning model being trained with first data of a first subset of the plurality of data categories and having first weights indicative of a value of a first performance metric, and the second machine learning model being trained with second data of a second subset of the plurality of data categories and having second weights indicative of a value of a second performance metric; generating first and second prediction results using the first and second models, respectively; and generating a composite prediction result based on the first prediction result, the second prediction result, the first weights, and the second weights.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

[0001] This patent application claims the benefit of priority to U.S. patent application Ser. No. 16 / 871,933, filed May 11, 2020, which is incorporated herein by reference for all purposes. [Background technology]

[0002]

[0002] Predictive machine learning models trained with real-world clinical data offer great potential for providing patients and their clinicians with patient-specific information regarding diagnosis, prognosis, or optimal treatment course. Machine learning models can be trained, for example, to predict a patient's chances of survival as a function of time from diagnosis (e.g., worsening cancer). Survival predictions can also be classified based on the treatment the patient will receive. Clinical decisions, such as which treatment to give the patient, are then made in part by the clinician based on the prediction (e.g., the clinician will often prefer the treatment that is predicted to have the longest survival). Prognostic information can also be provided to patients, for example, to improve the patient's ability to plan for their future, which can improve the patient's quality of life.

[0003]

[0003] Machine learning models can be trained to make predictions about patients based on patient data that includes multiple data categories. The absence of certain data categories in patient data can limit the applicability of machine learning models in making predictions. Machine learning models that require predictor data from one set of data categories may not be able to make predictions using actual patient data that includes only a subset of the required categories. For example, a model that makes predictions based on hematology and laboratory molecular test results may not be able to make predictions about patients who have hematology data but no molecular test results. Summary of the Invention

[0004]

[0004] Disclosed herein is a technique for making clinical predictions based on a meta-model including multiple machine learning models. Clinical predictions can include, for example, predicting a patient's survival rate at a time point from disease diagnosis. Each machine learning model can be trained using a different set of data categories including, for example, biographical data, medical history data, laboratory test results, biomarkers, etc. Each machine learning model can also be associated with a weight that represents a performance metric, such as, for example, the area under the curve (AUC) of the receiver operating characteristic (ROC) curve of the machine learning model.

[0005]

[0005] To make a clinical prediction for a particular patient, multiple machine learning models can be identified based on the data categories present in the patient's data. Each of the identified machine learning models can be trained using clinical data of other patients associated with a particular data category subset present in the patient data. The patient data subset associated with the data category subset can be extracted and input to the associated machine learning model to generate a prediction result. The prediction results from the machine learning models can be combined based on the weights of the machine learning models to generate a composite prediction result for the patient.

[0006]

[0006] These and other embodiments of the invention are described in detail below. For example, other embodiments relate to systems, devices, and computer-readable media associated with the methods described herein.

[0007]

[0007] The nature and advantages of embodiments of the present invention may be better understood with reference to the accompanying detailed description and drawings.

[0008]

[0008] The detailed description will now be given with reference to the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1A] 1A and 1B show examples of clinical predictions that can be made using the techniques described in this disclosure. [Figure 1B] 1A and 1B show examples of clinical predictions that can be made using the techniques described in this disclosure. [Figure 2A] 2A, 2B, 2C, and 2D show examples of using machine learning models to make clinical predictions, according to certain features of the present disclosure. [Figure 2B] 2A, 2B, 2C, and 2D show examples of using machine learning models to make clinical predictions, according to certain features of the present disclosure. [Figure 2C] 2A, 2B, 2C, and 2D show examples of using machine learning models to make clinical predictions, according to certain features of the present disclosure. [Figure 2D] 2A, 2B, 2C, and 2D show examples of using machine learning models to make clinical predictions, according to certain features of the present disclosure. [Figure 3A] 3A, 3B, 3C, and 3D show examples of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 3B] 3A, 3B, 3C, and 3D show examples of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 3C] 3A, 3B, 3C, and 3D show examples of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 3D] 3A, 3B, 3C, and 3D show examples of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 4A] 4A, 4B, 4C, 4D, and 4E show examples of experimental results of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 4B] 4A, 4B, 4C, 4D, and 4E show examples of experimental results of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 4C] 4A, 4B, 4C, 4D, and 4E show examples of experimental results of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 4D] 4A, 4B, 4C, 4D, and 4E show examples of experimental results of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Figure 4E] 4A, 4B, 4C, 4D, and 4E show examples of experimental results of using a meta-model to make clinical predictions, according to certain features of the present disclosure. [Diagram 5] FIG. 5 illustrates a method for making clinical predictions in accordance with certain features of the present disclosure. [Figure 6] FIG. 6 illustrates an example of a computer system that can be used to implement the techniques disclosed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0015] According to an embodiment of the present disclosure, a machine learning model can be trained to make clinical predictions about a patient based on patient data including a set of data categories. The clinical prediction can include, for example, predicting a patient's survival rate as a function of time since diagnosis (e.g., worsening cancer). Survival rate predictions can also be made for different types of treatments. The machine learning prediction model of the present disclosure can be a very powerful tool for making clinical predictions and assisting clinicians in making clinical decisions based on the prediction results because the model can uncover hidden relationships between the predicted clinical parameters (e.g., survival rate) and different data categories of the patient data that may not have been considered by the clinician. The machine learning model can be trained with data from a very large population of patients, thereby learning different relationships between the clinical parameters and different data categories, and making predictions about the patient based on the patient data and the relationships that the model has learned through training.

[0011]

[0016] Although machine learning predictive models can be very powerful tools in terms of making clinical predictions and assisting clinicians in making clinical decisions, challenges are limited in deploying machine learning-based predictive models in real clinical settings. One set of challenges is related to the inter-patient variability in data availability. In most real-world datasets, many patients are missing records of findings for many data categories. On the other hand, machine learning models are typically trained using data from a large data category set to uncover more hidden links between the predicted parameters and the large data category set. If a machine learning model is trained to make predictions based on a data category set, but the patient data is missing one or more categories from that set, the machine learning model may not be able to make predictions based on the patient data.

[0012]

[0017] The data categories included in the data for a patient group may vary among different patients for various reasons, thus resulting in high data heterogeneity. For example, some hospitals may have a laboratory test menu that includes more than 1000 unique orderable tests. Depending on the patient situation (e.g., clinical environment, clinical conditions and diagnoses, clinical severity, comorbidities, treatments received, age, sex, economic status, etc.), most patients may have received at most a few of these possible tests, and they may have received different ones of these tests. In another example, different patients may lack various non-laboratory clinical data categories, including other clinical studies, elements of patient history, biopsy data, etc. For patients who have received similar diagnostic tests or physical exam maneuvers, they may have been performed at different times or at various intervals, and the problem of data heterogeneity becomes especially important when considering time series data. In all these cases, the absence of various data categories in the patient data may prevent the machine learning model from making predictions for a large number of patients, which limits the usefulness of the machine learning model.

[0013]

[0018] One approach to overcome the challenges presented to machine learning models by data heterogeneity is through an imputation process, where a patient's clinical data set can be preprocessed to generate additional clinical data elements for the missing data categories, such that the patient data, including the original and imputed clinical data elements, can contain the complete set of data categories as input to the machine learning model. The imputation process can estimate the distribution or value of each data element for the missing data categories based on the original patient data. However, imputation processes can have significant limitations. Specifically, most imputation algorithms assume that the data are "missing at random" (MAR), but this MAR assumption is typically not valid for most clinical data. This is because the diagnostic studies that generate the clinical data are selected and ordered according to the clinical environment rather than randomly, and therefore the presence (or absence) of certain data categories in the patient data is also not random. Furthermore, imputation can introduce additional uncertainty and imprecision into the predictions and can obscure some of the hidden relationships between the predicted parameters and the data categories.

[0014]

[0019] In order to address some of the above problems, a technique is disclosed for making clinical predictions using a composed learning model (hereinafter referred to as a "meta-model") including multiple machine learning models. Each of the multiple machine learning models of the meta-model is trained individually with data from a data category set, and the various machine learning models are trained with data from various data category sets, although there may be overlapping data categories among the various sets. When the machine learning model is trained to predict patient survival, the training of the machine learning model can be based, for example, on classifying patients into groups such that the similarity in survival statistics of patients within the groups is maximized and the difference in survival statistics of patients between the groups is maximized. Each of the multiple machine learning models can be associated with a performance metric, which can be obtained from a validation process of the trained machine learning model. When the machine learning model is trained to predict patient survival, the performance metric can measure the relationship between the true positive rates and false positive rates of the model output, which can be represented in the form of a receiver operating characteristic (ROC) curve. In some examples, the performance metric may be based on the area under the curve (AUC) of the model's ROC curve.

[0015]

[0020] Each of the plurality of machine learning models can be used to make a clinical prediction for the patient. The patient data can include a plurality of data categories that can be mapped to different sets of data categories of the plurality of machine learning models. Based on the mapping, a meta-model can be selected that includes at least a first machine learning model and a second machine learning model, each of the machine learning models trained with data of different data category subsets of the patient data. For example, the first machine learning model is trained with data of a first subset of the plurality of data categories, and the second machine learning model is trained with data of a second subset of the plurality of data categories. The first subset of patient data corresponding to the first subset of the plurality of data categories can be input to the first machine learning model to generate a first prediction output, and the second subset of the plurality of data categories can be input to the second machine learning model to generate a second prediction output. The meta-model can then generate a composite prediction output based on the first prediction output, the first performance metric, the second prediction output, and the second performance metric. A clinical prediction can then be made based on the composite prediction output.

[0016]

[0021] In some examples, the multiple data categories may include various personal data such as a patient's age and gender, each of which may correspond to a data category. The multiple data categories may also include a patient's history, such as a patient's medical history (e.g., what medical treatments the patient has received), a patient's habits (does the patient smoke), etc. The multiple data categories may also include various laboratory test result categories of the patient, such as white blood cell count, hemoglobin count, platelet count, hematocrit count, red blood cell count, creatinine count, lymphocyte count, and measurements of protein, bilirubin, calcium, sodium, potassium, glucose, etc., each of which may correspond to a data category. Additionally, the multiple data categories can include measurements of different biomarkers for different cancer types, such as estrogen receptor (ER), progesterone receptor (PR), human epidermal growth factor receptor 2 (HER2), epidermal growth factor receptor (EGFR or HER1) for breast cancer, ALK (anaplastic lymphoma kinase) for lung cancer, KRAS gene for lung and colorectal cancer, BRAF gene for colorectal cancer, etc. Additionally, each biomarker measurement can correspond to a data category. Each machine learning model of the multiple machine learning models can be trained with different subsets of the multiple data categories (e.g., different subsets of personal information, different subsets of laboratory test results, different subsets of biomarker measurements, etc.) to make predictions.

[0017]

[0022] In some examples, multiple machine learning models can be trained to determine patient survival rates as a function of time. Survival rates can predict the likelihood that a patient will be alive at a given time (e.g., 500 days, 1000 days, 1500 days, etc.) after the patient is diagnosed with a health condition (e.g., advanced cancer). Survival rates over time can be used to generate patient-specific Kaplan-Meier plots for the patient. Machine learning models can be implemented using a variety of techniques, such as random forest models, regression models, etc.

[0018]

[0023] Specifically, a Random Forest model can include multiple decision trees, each associated with a different set of data categories. Each decision tree can include multiple terminal nodes, each associated with a cumulative hazard function (CHF) value, representing a particular combination of data value ranges for the set of data categories. The cumulative hazard function can be interpreted as the probability of a patient's survival (or death) at a particular time from diagnosis, assuming that the patient survives at least until that particular time. Each decision tree in the Random Forest model can classify a patient into one of the terminal nodes based on the value of the patient's corresponding data category and can output a cumulative hazard function value for the patient. The multiple values ​​from the multiple decision trees in the Random Forest model can be averaged as the cumulative hazard function output for the patient, and the predicted survival rate for the patient can be calculated based on the cumulative hazard function output.

[0019]

[0024] On the other hand, a regression model, such as a proportional hazards regression model, can include a set of parameters that, in combination with the patient's data values, can be used to calculate a cumulative hazard function for the patient. Multiple random forest and / or regression models, each associated with a particular time since diagnosis, can be provided to output various values ​​of the cumulative hazard function, which can be used to generate a predicted survival rate for the patient as a function of time.

[0020]

[0025] The training process can generate (1) each decision tree in the random forest model, (2) a data category assigned to each decision tree, (3) a classification criterion at each parent node of the decision tree, and (4) a cumulative hazard function value at each terminal node. The training process can use patient data and survival statistics for a set of data categories for a large population of patients. Specifically, the training process can begin with randomly assigning a data category subset to a root node of a decision tree, and different data category subsets can be assigned to the root nodes of different decision trees. Furthermore, in a bagging process, each decision tree can obtain a set of random patient data samples for the assigned data category subset. The root node (and each parent node thereafter) can then split into child nodes in a recursive node splitting process.

[0021]

[0026] In the node splitting process, a node representing a set of patients can be split into two child nodes, each representing a subset of patients based on a threshold on the data category subset, where the data category subset and threshold are selected to maximize the difference in the number of surviving patients / deaths between the two child nodes. This process can then be repeated at the child nodes to create additional child nodes, for example, until a threshold minimum number of deaths / non-surviving patients is reached at a particular child node, which then becomes a terminal node. Patients can then be classified by the decision tree into the terminal nodes according to their data category subset values. The cumulative hazard function output at each terminal node can be expressed as the percentage of surviving patients classified into that terminal node.

[0022]

[0027] Furthermore, proportional hazards regression models can also be trained to make predictions. The regression model can include coefficients that relate various patient data categories to the probability of survival (e.g., cumulative hazard function) at a given time after diagnosis. Based on the regression model and the patient's values ​​for the various data categories as inputs, the probability that the patient is alive at a given time can also be calculated as an output of the regression model. The parameters of the regression model can also be determined in a training process that includes fitting the regression model across the patient data and patient survival statistics of various data categories for a large number of patients. For example, based on the survival statistics, the percentage of surviving patients of that group at various times (the group size is adjusted for time to account for death) can be determined to represent a target survival rate. The parameters of the regression model can then be determined based on the patient's values ​​for the various data categories at various times, and the regression model is adapted to output a target survival rate.

[0023]

[0028] As mentioned above, each of the machine learning models can be associated with a performance metric that can be used to create a composite output of the meta-model. The performance metric can include a confidence that a particular machine learning model will generate a correct prediction. The performance metric of the machine learning models can be determined using various techniques, such as determining the area under the curve of the receiver operating characteristic (ROC) curve of the machine learning models. The ROC curve can provide a plot between the corresponding true positive rate and false positive rate of the machine learning models. A value of 0.5 for the area under ROC means that the true positive rate and the false positive rate are equal on average and can indicate that the machine learning model is indistinguishable and undesirable, while a large area under ROC of more than 0.5 can indicate that the predictions are more reliable. Weights can be assigned to the machine learning models based on the area of ​​the ROC of the machine learning models, with larger weights being assigned to machine learning models with larger ROC areas and vice versa. A composite predicted output can be created based on an average of the weighted predicted outputs of the machine learning models, with each predicted output being adjusted by the respective weight of the machine learning models.

[0024]

[0029] In the case where a machine learning model is trained to output a patient's survival rate, a true positive occurs when the machine learning model outputs that the patient's survival probability at a given time exceeds a probability threshold, and the patient actually survives at that given time. A false positive occurs when the patient's survival probability exceeds the probability threshold at that given time, but the patient does not survive. A validation process can be performed in which data from a group of patients is input into the machine learning model and the survival rate of each patient is calculated. To determine a performance metric, a true positive rate, which represents the percentage of patients who have a survival rate above the probability threshold and are alive, and a corresponding false positive rate, which represents the percentage of patients who have a survival rate above the probability threshold but are not alive, can be obtained for a range of probability thresholds. Various pairs of true positive rate and false positive rate for various probability thresholds can then be determined to generate an ROC plot, and the area under ROC for the machine learning model can be determined. In various implementations, true negative rates and / or false negative rates can be used in addition to or instead of true positive rates and / or false positive rates.

[0025]

[0030] The training of each machine model and the determination of the machine model's performance metrics can be based on different patient data subsets of the same patient group in a cross-validation process. Specifically, from the patient data from the patient group, patients having data that includes the data category required by the machine learning model can be identified. The identified patients can be divided into two subsets. One subset (e.g., patient data of 80% of the patients) can be used to train the machine model, and the other subset (e.g., patient data of 20% of the patients) can be input to the trained machine model as part of the cross-validation process to generate the ROC plot and calculate the area under ROC. In some examples, the training and cross-validation can be repeated, with each iteration using a different patient data subset (e.g., a different 80% and 20% portion of the patients), and the weights assigned to the machine learning models can be determined by averaging the ROC areas obtained from the multiple iterations. The training process can be repeated with a second machine learning model, beginning with identifying patients having data that includes the data category required by the second machine learning model. More iterations of training and cross-validation can improve the confidence level of a machine learning model in producing correct predictions.

[0026]

[0031] In the disclosed embodiments, rather than using one machine learning model trained with a large set of fixed data categories to make clinical predictions for a patient, a metamodel can be provided that includes multiple machine learning models, each trained with a small but different set of data categories, to make clinical predictions for a patient. The machine learning models of the metamodel can be selected based on data categories of the patient data that match the data categories of the machine learning model, such that there are no missing data categories in the patient data for the selected machine learning model. This can reduce or eliminate the need to impute data elements from the patient data for the machine learning model and the uncertainty in the predictions caused by the imputed data. Also, the reduced number of data categories input to the machine learning model can reduce the number of possible permutations of the inputs to the machine learning model, which can reduce the time to train the machine learning model (or at least make the training more manageable). Furthermore, the prediction outputs of the selected machine learning models are combined to produce a composite prediction output, which can reflect all data categories of the patient data, and the combination reflects the confidence level of each machine learning model. All of this can improve the confidence level of the clinical predictions produced by the meta-model, and furthermore, when the clinical predictions are used to predict a patient's survival in response to treatment, making the predictions more accurate and allowing them to be made earlier in the patient journey, and allowing corrective actions such as consideration of additional treatment options and possible clinical trials to be taken earlier, all of which increases the patient's chances of recovery and survival.

[0027] I. Examples of clinical predictions and applications

[0032] 1A and 1B show examples of clinical predictions that can be provided by examples of the present disclosure. FIG. 1A shows a mechanism for predicting the survival rate of a patient with respect to the time from when a diagnosis of cancer is made, and FIG. 1B shows an example of the application of survival rate prediction. Referring to FIG. 1A, chart 100 shows an example of a Kaplan-Meier (KM) plot, which provides a representative of the statistics of survival among patients with a certain type of cancer (e.g., lung cancer). The patients can receive a particular treatment. The KM plot shows the change in the survival rate of a group of patients with respect to the measured time from when the patients were diagnosed with cancer. In the case where the patients receive treatment, the KM plot also shows the survival rate of the patients according to the treatment. As time progresses, some patients may die and the survival rate decreases. Some other patients are removed (dropped) from the plot due to other events that are not related to the event being studied, but those unrelated events are represented by a tick in the KM plot. The length of each horizontal line represents the survival time for that interval, and all survival estimates for a given point represent the cumulative probability of survival for that time.

[0028]

[0033] In FIG. 1A, chart 100 includes two KM plots of survival rates for different patient cohorts A and B. Cohorts A and B can include cohorts of patients with different characteristics, undergoing different treatments, etc. In FIG. 1A, the median survival rate (50% of patients are estimated to survive) is about 11 months for cohort A, but about 6.5 months for cohort B. Additionally, the percentage of patients surviving is shown by the horizontal line on the plot, which can also provide a prediction of the survival rate of patients within the cohorts over time. For example, at 4 months from diagnosis, the survival rate of patients in cohort A is about 70% (0.7) and the survival rate of patients in cohort B is about 30% (0.3).

[0029]

[0034] Population-based survival statistics are available across a wide range of cancer types and patients, but patient-specific information can be difficult to discern. Specifically, while the 5-year survival rate for patients with stage IV colon cancer overall is relatively low (e.g., 14% according to the National Cancer Institute's (NCI) Surveillance, Epidemiology, and End Results (SEER) statistics), some individual patients have survival times that are significantly better than the median survival time. Thus, the clinical question for oncologists when looking at individual patients is not about population survival times, but rather, what is the prognosis for individual patient survival?

[0030]

[0035] 1B shows an example application of survival prediction for a patient. As shown in FIG. 1B, data 102 representing a survival prediction for a patient, such as the KM plot of FIG. 1A, can be input into a clinical decision support tool 104. The data 102 can include a survival prediction for the patient without treatment and / or a survival prediction for the patient with various treatments.

[0031]

[0036] Based on the data 102, the clinical decision support tool 104 can generate various information to assist the clinician in caring for / treating the patient. For example, to facilitate the care of the patient, the clinical decision support tool 104 can generate information 106 that allows the clinician to better assess the prognosis of the patient, such as the patient's predicted survival rate compared to other patients, the patient's life expectancy, etc. The information 106 can facilitate discussions between the clinician and the patient regarding the evaluation of the patient's prognosis and treatment options, and planning for the patient's life events. As an example given here, if the clinical decision support tool 104 predicts that the patient has a relatively long remaining life expectancy (e.g., 5 years), the patient may decide to undergo aggressive treatment that is physically demanding and has significant side effects. However, if the clinical decision support tool 104 indicates that the patient has a relatively short remaining life expectancy (e.g., less than 1 year), the patient may decide to forgo treatment or undergo alternative treatment, and may plan for care and life events for the patient's remaining life expectancy.

[0032]

[0037] Additionally, in cases where the data 102 includes predictions of patient survival for various treatments, the clinical decision support tool 104 can also output information 108 to facilitate selecting an optimal treatment for the patient. For example, the clinical decision support tool 104 can display various KM plots of the patient for various treatments based on the data 102. The clinician can then select, for example, the treatment that maximizes the patient's survival rate at a given time, maximizes the patient's expected remaining life span, etc.

[0033] II. Clinical prediction using machine learning models

[0038] A machine learning predictive model can be trained to make clinical predictions about patients based on patient data that includes a set of data categories. Machine learning predictive models can be very powerful tools for making clinical predictions because the models can uncover hidden relationships between the clinical parameters to be predicted (e.g., survival rates) and various data categories of the patient data that may not have been considered by clinicians. A machine learning model can be trained with data from a large population of patients, so that it learns various relationships between the clinical parameters and various data categories, and makes predictions about patients based on the patient data and the relationships that the model has learned through training.

[0034]

[0039] FIG. 2A illustrates an example of the use of a machine learning model to make a prediction of a patient's survival rate at a given time after being diagnosed with cancer. As shown in FIG. 2A, the machine learning prediction model 200 can receive as input various types of data for a patient, including, for example, personal data 202, medical history data 204, laboratory test results 206, biomarkers 208, etc., and generate a prediction result 210 of the patient's survival rate based on the input. The survival rate can provide the likelihood that the patient will be alive at a given time (e.g., 500 days, 1000 days, 1500 days, etc.) after the patient is diagnosed with a health condition (e.g., worsening cancer). The survival rate over time can be used to generate a patient-specific Kaplan-Meier (KM) plot for the patient.

[0035]

[0040] 2B illustrates a table 220 providing example categories of personal data 202, medical history data 204, laboratory test results 206, and biomarkers 208. By way of example, the personal data 202 may include various categories of information such as age, gender, race, etc. By way of example, the history data 204 may include various categories of information such as diagnosis results (e.g., stage of cancer, site of colorectal cancer (CRC), etc.), patient habits (e.g., does the patient smoke), patient treatment history (e.g., what treatments the patient has received), etc.

[0036]

[0041] By way of example, the laboratory test results 206 may include various laboratory test result categories for the patient, such as white blood cell count, hemoglobin count, platelet count, hematocrit count, red blood cell count, creatinine count, lymphocyte count, and measurements of protein, bilirubin, calcium, sodium, potassium, alkaline phosphatase, carbon dioxide, monocytes, chloride, lactate dehydrogenase, glucose, etc.

[0037]

[0042] By way of example, the biomarker data 208 can include measurements of various biomarkers for various cancer types, such as estrogen receptor (ER), progesterone receptor (PR), human epidermal growth factor receptor 2 (HER2), epidermal growth factor receptor (EGFR or HER1) for breast cancer, ALK (anaplastic lymphoma kinase) for lung cancer, KRAS gene for lung and colon cancer, BRAF gene for colon cancer, etc. It should be understood that other categories of clinical data not shown in FIG. 2B, such as biopsy image feature data, etc., can also be input into the machine learning prediction model 200 to make clinical predictions.

[0038] A. Random Forest

[0043] The machine learning predictive model 200 can be implemented using various techniques such as random forest trees, regression models, etc. FIG. 2C illustrates an example of a random forest model 230 that can be used to implement the machine learning predictive model 200. As illustrated in FIG. 2C, the random forest model 230 can include multiple decision trees, including, for example, decision trees 232 and 234. Each decision tree can include multiple nodes, including a root node (e.g., root node 232a of decision tree 232, root node 234a of decision tree 234, etc.) and child nodes (e.g., child nodes 232b, 232c, 232d, and 232e of decision tree 232, child nodes 234b and 234c of decision tree 234, etc.). Each parent node that has child nodes (e.g., nodes 232a, 232b, and 234a) can be associated with a predefined classification criterion to classify a patient into one of its child nodes. Child nodes that do not have child nodes are terminal nodes, including nodes 232d and 232e (in decision tree 232) and nodes 234b and 234c (in decision tree 234), each of which is assigned a cumulative hazard function (CHF) value.

[0039]

[0044] The value of CHF can represent the risk of death (or survival rate) at a particular time. CHF represents the sum of hazard functions h(t), which represent the risk of immediate death at a particular time t, and thus CHF itself is also a function of time. Each decision tree can classify the patient into one of the terminal nodes and output a CHF value, be it CHF value 236 from decision tree 232, CHF value 238 from decision tree 234, etc. The CHF values ​​for the patient from each of the decision trees are averaged to produce an overall CHF value 240 for the patient (CHF combine) can be generated to represent the survival rate of a patient at a particular time t. A random forest model 230, as shown in FIG. 2C, can be trained to predict CHF values ​​for a particular time, and multiple random forest models 230 can be used to predict overall CHF values ​​240 for a patient for various times.

[0040]

[0045] The survival probability m(t) of a patient at time t is calculated by the combined cumulative hazard function CHF combine It can be calculated based on the exponential function (exp). m(t) = exp(-CHF combine (t)) (Equation 1)

[0041]

[0046] Each decision tree can be assigned to process a different data category subset of the patient data to generate a CHF value. For example, as shown in FIG. 2C, the patient data 242 is divided into a data category set {S0, S1, S2, S3, S4, . . . , S n Decision tree 232 can be assigned and trained to handle data categories S0 and S1, decision tree 234 can be assigned and trained to handle data category S2, and other decision trees can be assigned to handle other data category subsets.

[0042]

[0047] A parent node in the decision tree may then compare a subset of the patient data 242 corresponding to one or more of the assigned data categories to one or more thresholds to classify the patient into one of the child nodes of the parent node. Specifically, with reference to the decision tree 232, the root node 232a may classify the patient into child node 232b if the patient data in data category S0 exceeds threshold x0, and into terminal node 232c if not. The child node 232b may further classify the patient into either terminal node 232d or 232e based on the patient data in data category S1. Depending on which terminal node the patient is classified into based on the data categories S0 and S1, the decision tree 232 may output a CHF value of CHF0, CHF1, or CHF2. Additionally, the decision tree 234 may also output a CHF value of CHF3 or CHF4 depending on which terminal node the patient was classified into based on the data category S2.

[0043]

[0048] Each decision tree of the random forest model can be generated in a training process over the patient data and survival statistics of a set of data categories of a large population of patients. Furthermore, the training process can determine the data category subsets assigned to each decision tree, the classification criteria at each parent node of the decision tree, and the value of the cumulative hazard function at each terminal node. Specifically, the training process can start by randomly assigning the subset of data categories to the root node of the decision tree, and different subsets of data categories can be assigned to the root nodes of different decision trees. The process of generating trees can be iterative until a target number of decision trees is reached, which may be determined by the hyperparameters of the training process. Furthermore, in the bagging process, the root nodes of the decision trees can be assigned to a random sample set of patient data of the assigned data category subset to perform training.

[0044]

[0049] As part of the training process, the root node (and each subsequent parent node) may be split into child nodes in a recursive node splitting process based on the random sample set of patient data assigned to the root node. In the node splitting process, a node representing a set of patients may be classified into two child nodes, each representing a patient subset, based on a threshold on the data category subset, where the data category subset and threshold are selected to maximize the difference in the number of surviving patients / deaths between the two child nodes, such that the patients represented in the two child nodes have survival statistics as similar as possible. For example, with reference to decision tree 232, during the training process, it may be determined that by splitting the random patient data samples assigned to decision tree 232 into two groups based on data category S0 and threshold x0, the difference between the number of surviving patients in the two groups may be maximized compared to other classification criteria (e.g., based on data category S1, setting a different threshold on S0, etc.).

[0045]

[0050] The process can then be repeated at the child nodes to generate further child nodes, for example, until a threshold minimum number of dead / non-surviving patients is reached at a particular child node, which becomes a terminal node. For example, among the patients classified into terminal nodes 232c, 232d, and 232e, the number of dead / non-surviving patients reaches a threshold minimum number, so the root splitting operation stops at those nodes. The output of the cumulative hazard function at each of those terminal nodes can be represented by the percentage of surviving patients classified into that terminal node at a given time. For example, for the 20 patients classified into terminal node 232c, a hazard function can be calculated based on the number of surviving and non-surviving patients at various times until a particular time (e.g., 4 days) is reached. A CHF value at a particular time can then be generated based on summing the hazard functions for that particular time. The training can be iterative based on the survival statistics of patients at different times to generate different decision trees and different thresholds for classifying patients into different child nodes, thereby predicting different CHF values ​​as a function of time.

[0046] B. Mathematical Model

[0051] In addition to the random forest model 230, a regression model can also be used to predict patient survival as a function of time. FIG. 2D shows an example of a regression model 250. As shown in FIG. 2D, the regression model 250 can be a Cox proportional hazards model for predicting patient survival. The survival function m(t) can be expressed as: m(t)=m0(t)×(b0S0+b1S1+···+b n S n ) (Formula 2)

[0047]

[0052] In Equation 2, m0(t) can represent the base hazard, which can be calculated empirically using actual patient survival data and with censoring, as a function of time t. For example, if 20 patients are known to be alive on day 9, and one of the 20 dies, then the base hazard on day 9 is 0.05 (1 / 20). Patients who are lost to follow-up or who are still alive in the study are censored and therefore removed from both the numerator and denominator at all time points after the time of censoring. For example, if a patient was alive on day 1400, when the study ended, he or she would be removed from the numerator and denominator in the hazard calculation for all days after day 1400. The survival rate of patients can be calculated by dividing the base hazard by the S0, S1, and S2 hazards, respectively. n The survival function M(t) can be determined based on multiplying a set of variables representing data categories by a linear function such as KM plot. Each variable can be adjusted by a corresponding coefficient in the linear function. For example, variables of data category S0 can be adjusted by a corresponding coefficient b0, variables of data category S1 can be adjusted by a corresponding coefficient b1, and so on. Each coefficient reflects the influence of a data category on the risk of death. The survival function M(t) can show the change (increase) in the risk of death over time, as shown in the KM plot.

[0048]

[0053] The coefficients of the regression model 250 can also be determined in a training process that involves fitting a mathematical model across patient data and survival statistics for various data categories for a large population of patients based on regression. For example, based on survival times of patients (represented in FIG. 2D as P0, P1, P2, etc.), a KM plot of the percentage of surviving patients in a group (where the size of the group is adjusted for time to account for death) at various times can be determined to represent a target survival rate at various times. The target survival rate can be represented by a base hazard m0(t) calculated from a patient group that includes patients P0, P1, P2, etc. Coefficients can then be determined based on the patient values ​​for the various data categories at various times to fit the output of the regression model 250 to the target survival rate.

[0049] III. Clinical prediction using a single machine learning model

[0054] Although machine learning predictive models can be very powerful tools when it comes to making clinical predictions and assisting clinicians in making clinical decisions, challenges are limitations in deploying machine learning-based predictive models into real-world clinical environments. One set of challenges is related to the inter-patient variability in data availability. In most real-world datasets, many patients lack records of findings for many data categories. On the other hand, machine learning models are typically trained using data from a large data category set to uncover more hidden links between the predicted parameters and the large data category set. For example, the random forest model 230 and the regression model 250 can be trained based on multiple data categories from the data categories listed in table 220. If the machine learning model is trained to make predictions based on a data category set, but the patient data is missing one or more categories from that set, the machine learning model may not be able to make predictions based on the patient data.

[0050]

[0055] The data categories included in the data of a patient group may vary among different patients for various reasons, thus resulting in high data heterogeneity. For example, some hospitals may have a laboratory test menu with more than 1000 unique orderable tests. Depending on the patient's circumstances (e.g., clinical environment, age, sex, economic status, etc.), most patients may have received at most a few of those possible tests, and different circumstances of the patients may have received different ones of those tests. In another example, different patients may be missing various non-laboratory clinical data categories, including other clinical studies, elements of patient history, biopsy data, etc. Even for patients who have undergone similar diagnostic tests or physical examination maneuvers, they may have been performed at different times or at different intervals, and the problem of data heterogeneity becomes especially important when considering time series data. In all these cases, the absence of various data categories in the patient data may prevent the machine learning model from making predictions for a large number of patients, which limits the usefulness of the machine learning model.

[0051]

[0056] One approach to overcome the challenges presented to machine learning models by data heterogeneity is through an imputation process, where a patient's clinical data set can be preprocessed to produce additional clinical data elements for the missing data categories, such that the patient data, including the original and imputed clinical data elements, can contain the complete set of data categories as input to the machine learning model. The imputation process can estimate the distribution or value of each data element for the missing data categories based on the original patient data. However, imputation processes can have significant limitations. Specifically, most imputation algorithms assume that the data are "missing at random" (MAR), but the MAR assumption is typically not valid for most clinical data because the diagnostic studies that generate the clinical data are selected and ordered according to the clinical environment rather than randomly, and therefore the presence (or absence) of certain data categories in the patient data is also not random. Furthermore, imputation can introduce additional uncertainty and imprecision into the predictions and can obscure some of the hidden relationships between the predicted parameters and the data categories.

[0052] IV. Clinical Prediction Using Metamodels of Machine Learning Models

[0057] Machine learning models such as random forest decision tree models (as shown in Figure 2C) and regression models (as shown in Figure 2D) can be trained with a large number of data categories representing different characteristics of patients to improve the accuracy of predictions. However, patient data typically do not have all of those data categories. Imputing for the missing data categories allows machine learning models to be used to make predictions about patients, but such constructions introduce errors and uncertainties, as discussed above.

[0053]

[0058] To address some of the above problems, techniques are disclosed herein for making clinical predictions using a composite learning model (hereinafter referred to as a "meta-model") that includes multiple machine learning models. Each of the multiple machine learning models in the meta-model is trained individually with data from a set of data categories and assigned a weight indicative of the performance of the machine learning model. To make a prediction for a patient, a machine learning model that has been trained with the data categories present in the patient data can be selected to generate a prediction result. The prediction results can be combined based on the weights of the selected machine learning models to generate a composite prediction result for the patient.

[0054] A. Metamodel

[0059] 3A, 3B, and 3C illustrate an example of a clinical prediction system 300 that addresses at least some of the problems discussed above. As shown in FIG. 3A, the clinical prediction system 300 includes a machine learning model database 302, a selection module 304, and a predictor module 306. In some examples, the various models of the clinical prediction system 300, including the selection module 304 and the predictor module 306, can be implemented as software instructions executable by a hardware processor to perform the following functions:

[0055]

[0060] The selection module 304 can accept as input patient data 308. Based on data categories included in the data 308, the selection module 304 can select multiple machine learning models from the machine learning model database 302 to form a metamodel and provide the metamodel to the predictor module 306. The predictor module 306 can then process the data 308 with the machine learning models included in the metamodel to generate predictive outputs for the patient, such as survival rates 310a and 310b, and can combine the predictive outputs from the machine learning models to generate a composite predictive output, such as a composite patient survival rate 312. Clinical predictions (e.g., patient survival rate as a function of time and / or treatment, expected patient life expectancy, etc.) can then be made based on the composite predictive output.

[0056]

[0061] Specifically, the machine learning model database 302 can store multiple machine learning models, including models 304a, 304b, 304c, etc. Each machine learning model can include, for example, a random forest model (e.g., random forest model 230 of FIG. 2C), a mathematical model (e.g., regression model 250 of FIG. 2D), or other types of machine learning models to make predictions (e.g., survival rates). The random forest model and the mathematical model can include a set of sub-models, each of which is associated with a respective time, thereby generating different survival rates for different times. Each machine learning model in the machine learning model database 302 is trained individually with data from a data category set to make predictions, and different machine learning models are trained with data from different data category sets, although there may be overlapping data categories among the different sets. For example, the machine learning model 304a is trained with data category sets S0, S1, S2, and S3. Further, the machine learning model 304b is trained with data category sets S2, S4, and S5. Additionally, machine learning model 304c is trained using data category sets S0, S8, and S9. Machine learning model database 302 may also include other machine learning models trained using the data category subsets used to train machine learning models 304a-304c. For example, machine learning model database 302 may include machine learning models trained using data categories S4 and S5.

[0057]

[0062] Additionally, each of the multiple machine learning models may be associated with a performance metric. The performance metric may include a confidence level that a particular machine learning model will generate a correct prediction. For example, the machine learning model 304a may have a performance metric value M A and the machine learning model 304b can be associated with a performance metric value M B and the machine learning model 304c can be associated with a performance metric value M CAs will be explained later, the performance metric values ​​can be obtained from a validation process after the machine learning model is trained.

[0058]

[0063] To make a prediction about the patient based on the data 308, the selection module 308 can search the machine learning model database 302 for machine learning models that accept a subset (or all) of the data categories present in the data 308. In one example, the selection module 308 can obtain the required data categories of each machine learning model in the database (e.g., data category sets S0, S1, S2, and S3 for machine learning model 304a, data category sets S2, S4, and S5 for machine learning model 304b, etc.), compare those data categories with the data categories of the data 308, and determine whether the required data categories of the machine learning model represent a subset (or all) of the data categories in the data 308. If so, the selection module 308 can select that machine learning model to form the meta-model. The selection module 308 can then provide the selected machine learning model to the predictor module 306. In the example of FIG. 3A, the data 308 includes data categories S0, S1, S2, S3, S4, and S5. Based on the presence of these data categories in the data 308, the selection module 304 may select as part of the meta-model a machine learning model 304a trained with data of data categories S0, S1, S2, and S3 included in the data 308, and a machine learning model 304b trained with data categories S2, S4, and S5 included in the data 308, and may select other machine learning models (one or more) trained with any subset of the data categories S0, S1, S2, S3, S4, and S5. On the other hand, the selection module 304 does not select the machine learning model 304c because it is trained with data of data categories S0, S8, and S9, and the data categories S8 and S9 are not present in the data 308. The selection module 304 then selects the machine learning models 304a and 304b and their associated metrics M A and M. B can be fetched into the predictor module 306.

[0059]

[0064] The predictor module 306 can make predictions using the machine learning models 304a and 304b fetched by the selection module 304. The predictor module 306 can input a first subset of the data 308 corresponding to data categories S0, S1, S2, and S3 into the machine learning model 304a to generate a survival rate 310a, and the predictor module 306 can input a second subset of the data 308 corresponding to data categories S2, S4, and S5 into the machine learning model 304b to generate a survival rate 310b.

[0060]

[0065] The predictor module 306 further includes a synthesis module 320, which synthesizes the performance metric value M A and M. B , and combines the survival rates 310a and 310b to generate a composite survival rate 312. In one example, the combination module 320 can associate a weight for each survival rate based on a performance metric of the machine learning model generating the survival rate, and then generate the composite survival rate 312 based on the weighted survival rate average, where each survival rate can be adjusted with the associated weight. The weights can be generated to reflect a confidence level in the prediction of the survival rate by the machine learning model, as reflected by the performance metric. For example, a higher weight can be assigned to survival rates from machine learning models with a higher confidence level, and a lower weight can be assigned to survival rates from machine learning models with a lower confidence level. The composite survival rate 312 can then be provided to a clinical decision support tool, such as the clinical decision support tool 104, to generate various information to support clinical decisions, as described above.

[0061] B. Performance Metrics

[0066] As described above, each of the multiple machine learning models in the machine learning model database 302 can be associated with a performance metric that can be used to generate a composite output of the metamodel. The performance metric can indicate a confidence level that a particular machine learning model will generate correct predictions. The performance metric of a machine learning model can be determined using various techniques, such as determining the area under the curve (AUC) of the receiver operating characteristic (ROC) curve of the machine learning model.

[0062]

[0067] FIG. 3B shows an example of a ROC curve 330 for a machine learning prediction model. As shown in FIG. 3B, the ROC curve 330 provides a plot between the corresponding true positive rate and false positive rate of the machine learning model when making a prediction. The true positive rate is the rate at which the machine learning model correctly predicts that an event will occur, and the false positive rate is the rate at which the machine learning model predicts that an event will occur but the event does not occur. A value of 0.5 for the area under the curve (AUC) of the ROC, represented by the dashed line, means that the true positive rate and the false positive rate are equal, meaning that the machine learning model cannot discriminate and is undesirable, whereas a large AUC above 0.5, represented by the solid line, may indicate high confidence in the prediction. Weights can be assigned to the machine learning models based on the AUC of the ROC of the machine learning models, with larger weights assigned to machine learning models with larger AUC of the ROC and vice versa.

[0063]

[0068] When the machine learning model is trained to output the survival rate of the patient, the definition of true positive and false positive can be based on various definitions.In one example, the true positive case can be defined as the machine learning model outputs that the survival probability of the patient at a given time exceeds a probability threshold, and the patient is actually alive at the given time, while the false positive case can be defined as the patient's survival probability at a given time exceeds a probability threshold, but the patient is not alive.In another example, the true negative case can be defined as the machine learning model outputs that the survival probability of the patient at a given time falls below a probability threshold, and the patient is not actually alive at the given time, while the false negative case can be defined as the patient's survival probability at a given time falls below a probability threshold, but the patient is actually alive at the given time.

[0064]

[0069] FIG. 3C shows the weights (e.g., M A , M B Illustrates a validation process 340 for generating a prediction probability for a patient at a given time after diagnosis, etc. In this example, a machine learning model is trained to predict patient survival at a given time after diagnosis.

[0065]

[0070] In step 342, data for a group of patients with known survival statistics at a given time can be input into a machine learning model to predict the survival rate of each patient in the group. The data corresponds to categories for which the machine learning model is trained.

[0066]

[0071] In step 344, a probability threshold is set. The probability threshold can be selected from one of a set of probability thresholds used to test the machine learning model. For example, the probability threshold set can include a set of discrete values ​​including 0.2, 0.4, 0.6, and 0.8, and the probability threshold can be selected from this set. Fine or coarse increments can be used. Also, the difference between the values ​​need not be uniform.

[0067]

[0072] In step 346, each patient in the group can be labeled as either alive or non-alive based on comparing the patient's predicted survival rate to a probability threshold. Specifically, if the predicted survival rate is higher than the probability threshold, the patient can be labeled as a alive patient, whereas if the predicted survival rate is lower than the probability threshold, the patient can be labeled as a non-alive / deceased patient.

[0068]

[0073] For each patient labeled as alive, a determination is made based on survival statistics as to whether the patient is actually alive at a given time, at step 348. If the patient is labeled as alive and is actually alive at a given time, the patient may be counted as a true positive case, at step 350. On the other hand, if the patient is labeled as alive but is not actually alive at a given time, the patient may be counted as a false positive case, at step 352.

[0069]

[0074] In step 354, a true positive rate and a false positive rate can be generated for that probability threshold. The true positive rate can be generated by determining the percentage of true positive cases out of the number of patients represented in the data, while the false positive rate can be generated by determining the percentage of false positive cases out of the number of patients represented in the data.

[0070]

[0075] Following step 354, steps 344-352 can then be repeated for different probability thresholds selected from the probability threshold set. In this way, pairs of false positive and true positive rates for various probability thresholds can be obtained, and the corresponding pairs of false positive and true positive rates can be plotted to generate a ROC for the machine learning model.

[0071]

[0076] A validation process 340 may be performed for each machine learning model in the machine learning model database 302 to determine the area under the curve (AUC) of the ROC for each model. The weights for each model may be determined based on the AUC of its individual ROC according to the formula: Weight = (AUC-0.5) n (Formula 3)

[0072]

[0077] In Equation 3, the weight of the model can be based on the difference between the AUC and 0.5. As mentioned above, 0.5 means that the true positive rate and the false positive rate are the same, which means that the output of the machine learning model is completely random and does not discriminate, and the weight of such a model should be close to zero. On the other hand, an AUC greater than 0.5 means that the model can discriminate, and a large AUC reflects a high confidence level, and therefore the weight can be increased with the AUC. The difference between the AUC and 0.5 can be raised to the power of n, where n is greater than or less than 1, to emphasize (or de-emphasize) the influence of AUC on the weight. The value of n can be determined, for example, to maximize the AUC of the metamodel. For example, as part of the validation process, the AUC of the ROC of the output of the composite survival rate of the metamodel, which is based on the output of the weighted survival rate average of the machine learning models included in the metamodel, can be determined for various values ​​of n. Then, when the metamodel is used to calculate the composite survival rate of a new patient, the value of n that maximizes the AUC of the ROC of the metamodel can be determined and included in the calculation of the weight.

[0073]

[0078] In addition to the AUC, the weights of the machine learning model can be determined based on other performance metrics such as Harrell's C-index (also known as the concordance index). To determine the Harrell's C-index for a machine learning model, the model can be used to calculate the survival rate of a group of patients whose survival times are known. Concordant and discordant pairs of patients can be identified. A concordant pair of patients is a pair of patients where the first patient is predicted to have a higher survival rate than the second patient and the first patient has a longer survival time than the second patient. A discordant pair of patients is a pair of patients where the first patient is predicted to have a higher survival rate than the second patient, but the first patient has a shorter survival time than the second patient. The Harrell's C-index for a machine learning model can be determined as follows: C-index = (number of matched pairs) / (number of matched pairs + number of mismatched pairs) (Equation 4)

[0074]

[0079] A machine learning model with a larger C-index can be assigned a larger weight, and vice versa.

[0075]

[0080] In the machine learning model of the present disclosure, the concordance between two pairs may be partially a function of time, and the definition of concordance may be adapted to accommodate this time dependency. In one example, a patient pair at a given time point where one patient is known to be alive and the other deceased may be defined as "concordant" if the living patient has a high predicted probability of survival at that time point, and "mismatched" if the deceased patient has a high predicted probability of survival at that time point. In some examples, concordance / mismatch within a patient group may be defined using a bootstrap method, where patient pairs are randomly sampled at various time points and matching and mismatching bootstrap replicates are counted, where replicates representing combinations of patient pairs and time points where both patients are known to be alive or dead, or where the survival status of one or both patients is unknown (e.g., those who would be removed from the KM plot by censoring) are excluded from the count.

[0076] C. Training and validating machine learning models

[0081] In some examples, the training of each machine model in the machine learning model database 302 and the determination of the performance metrics for each machine model may be based on different subsets of patient data for the same patient group in a cross-validation process. Figure 3D illustrates an example of a training process 360 for the machine learning models in the machine learning model database 302, including machine learning models 304a, 304b, etc.

[0077]

[0082] As shown in FIG. 3D, the training process 360 can receive training data 362 as input. The training data 362 can include patient data of a large population of patients. The data categories included in the patient data can vary among various patients. Patient data of a group of patients having data categories required for the machine learning model can be identified for each machine learning model. For example, to perform training for the machine learning model 304a, patient data 364a of a first group of patients having data categories S0, S1, S2, S3 required by the machine learning model 304a can be identified. Furthermore, to perform training for the machine learning model 304b, patient data 364b of a second group of patients having data categories S2, S4, S5 required by the machine learning model 304b can be identified.

[0078]

[0083] The groups of patients identified for each training model and their patient data can then be divided into a first subset and a second subset. The first subset can be about 80% of the patient data (of the group's patients), and the second subset can be about 20%. The patient data of the first subset of the patient group can be used to train the machine learning model, while the patient data of the second subset of the patient group can be used to generate a performance metric (e.g., AUC of ROC) of the trained machine learning model. For example, as shown in FIG. 3D, the first subset of the patient data 364a, the patient data 366a, can be used to perform the training process 368 of the machine learning model 304a. As described above, if the machine learning model classifies patients into one of the groups (e.g., a random forest decision tree), the training can be based, for example, on maximizing the similarity in the survival statistics of the patients within the group and maximizing the difference in the survival statistics of the patients between different groups. If the machine learning model includes a regression model, training can be based on fitting parameters of the regression model to patient survival statistics.

[0079]

[0084] Additionally, the second subset of patient data 364a, patient data 366b, can be input to the trained machine learning model 304a in a validation process 370 to determine a performance metric (e.g., AUC) as described in FIG. 3C. Next, in process 372, the weights M A (eg, based on Equation 2 above) can be determined.

[0080]

[0085] Further, the first subset of the patient data 364b, the patient data 376a, can be used to train the machine learning model 304b in a training process 378, and the second subset of the patient data 364b, the patient data 376b, can be input to the trained machine learning model 304b in a validation process 380 to determine a performance metric (e.g., AUC) of the machine learning model 304b, followed by validation of the weights M of the machine learning model 304b in a validation process 382. B (e.g., based on Equation 2 above). The model parameters of the machine learning models generated in steps 368 and 378 (e.g., thresholds x0, s1, s2, x5, etc. of FIG. 2C, regression parameters b0, b1, bn, etc. of FIG. 2D) and weights associated with the machine learning models generated in steps 370 and 380 can then be stored in the machine learning models database 302 of FIG. 3A and retrieved to construct the meta-model described above.

[0081]

[0086] In some examples, the training and cross-validation of the machine learning model can be repeated for multiple iterations. For each iteration, different subsets of patient data (e.g., different 80% and 20% portions of patients) can be selected to perform the above-mentioned training and validation operations, and the weights of the machine learning model can be determined by averaging the ROC ranges obtained from multiple iterations. More iterations of training and cross-validation can improve the confidence level of the machine learning model in generating correct predictions.

[0082] V. Experimental Results A. Experimental Method Overview

[0087] The meta-model techniques described above are tested using clinical patient data from Flatiron Health clinical DataMarts for patients with three cancer types: metastatic colorectal cancer, metastatic breast cancer, and advanced lung cancer. A set of individual prediction models is developed, each of which predicts patient survival based on a different set of data categories for metastatic colorectal cancer, metastatic breast cancer, and advanced lung cancer. In this example, eight individual prediction models are developed for metastatic colorectal cancer, seven individual prediction models are developed for metastatic breast cancer, and five individual prediction models are developed for advanced lung cancer. Each individual model is trained and tested using clinical patient data for patients with all the data categories required for that model. The individual prediction models for each cancer are then combined and retrained into a meta-model for each cancer type. The meta-model for each cancer type and its component individual prediction models are evaluated based on the performance metric AUC.

[0083] B. Patient Cohorts and Data Categories

[0088] Three patient cohorts, one each for metastatic colorectal cancer, metastatic breast cancer, and advanced lung cancer, can be defined from three Flatiron DataMarts: i) Metastatic CRC (colorectal cancer), ii) Advanced NSSLC (non-small cell lung cancer), and iii) Metastatic Breast Cancer. For each patient in the patient cohort, patient survival outcome data is extracted. Patient survival outcome data can be either the time between the date of death and the date of advanced diagnosis, or, if the date of death is not recorded, the time between the last recorded visit and the date of advanced diagnosis.

[0084]

[0089] Additionally, various data categories for each patient are extracted that may be potential predictors for the survival of other patients with the same data category. Figure 4A illustrates the data categories input that are selected and entered into each individual predictive model for each cancer type. As discussed with respect to Figure 2A, the data categories may include laboratory test results, personal data, medical history data, biomarkers, etc.

[0085]

[0090] For laboratory test results (labeled "Lab" in Figure 4A), the available results for selected quantitative laboratory tests for each patient in each cohort are extracted as samples collected around the time of advanced diagnosis (+ / - 90 days). The maximum and minimum test results for the patients are extracted and converted to median absolute deviation (absolute value of result minus population median) for use in the model. The use of median absolute deviation in the model can reflect the idea that high or low values ​​for many tests may be diagnostically informative. Additionally, personal data such as age and sex, and diagnostic history data such as cancer stage are also extracted.

[0086]

[0091] Various biomarker data are also extracted for various cancer types. For example, for the CRC cohort, the gene expression status of KRAS and BRAF is extracted for patients in the cohort. For the breast cancer cohort, the ER, PR, and HER2 status of patients is extracted. Additionally, for the lung cancer cohort, the mutation status of ALK and EGFR is extracted. For molecular biomarkers (ER, PR, HER2, KRAS, BRAF, EGFR, ALK), only results for each marker for each patient that are closer in time to the date of advanced diagnosis (e.g., based on specimen collection date) are considered. Molecular biomarker results are then classified as "positive," "negative," or "other," with "positive" indicating that the corresponding marker expression level was above a threshold or that a mutation was present. A classification of "negative" indicates that the marker was absent or that no mutation was observed, and "other" indicates equivocal results, tests that could not be performed, or other non-conclusive results.

[0087] C. Predictor set

[0092] Few, if any, patients have data for all possible predictors (laboratory tests, molecular biomarkers, and clinical / demographic variables) to predict survival as shown in Figure 4A. In this experiment, individual prediction models are developed for each cancer type, where each individual prediction model is built and trained using a subset of possible predictors. The individual prediction models are then combined into a meta-model. As shown in Figure 4A, seven individual prediction models were developed for colorectal cancer with distant metastasis (CRC in Figure 4A). A -CRC G ) were developed, and five individual prediction models (labeled BC in Figure 4A) were developed for breast cancer with distant metastasis. A -BC E ) were developed, and eight individual prediction models for advanced lung cancer (labeled LC A -LC HA set of individual predictive models (labeled as “1”) have been developed. Each individual predictive model takes as input a set of predictors represented by a set of data categories and makes a prediction. For each predictive model, a data category labeled as “1” indicates that the data category is used to train the predictive model, and a data category labeled as “0” indicates that the data category is not used to train the predictive model. For example, a predictive model for advanced lung cancer, LC A was trained based on the gender and race data of patients. In Figure 4A, the gender and race data categories are labeled as “1”. Furthermore, the predictive model for advanced lung cancer, LC B was trained based on age, histology, and smoking status; in Figure 4A, these data categories are labeled "1".

[0088]

[0093] Depending on the data categories represented by the predictor set (e.g., the number of data categories, the distribution of the data categories among the patient cohort, etc.), each individual predictive model can be trained using clinical data of different populations of the patient cohort with the corresponding cancer type that includes the data categories. Since different portions of the patient cohorts may overlap, the same patient may be represented in multiple portions of the patient cohort, and the clinical data of that patient is used to train multiple individual predictive models for the cancer types corresponding to the patient cohort.

[0089]

[0094] Figure 4B illustrates the portion of the patient cohort for each cancer type with the data categories required for each individual prediction model. Ais trained based on 100% clinical data of the corresponding patient cohort. This can be due to the fact that, for example, only two data categories (gender and race) are provided as inputs to the model, and these two data categories are common and present in the clinical data of all patients. Furthermore, for the same reason, an individual prediction model for CRC with distant metastasis was developed. B and CRC C , individual prediction model BC A -BC D , and an individual prediction model for advanced lung cancer LC A -LC C is trained on a relatively large collection (94%-100%) of clinical data from the corresponding patient cohort.

[0090]

[0095] In contrast, the individual prediction model CRC G is trained on the clinical data of only 15% of the corresponding patient cohort. This may be due, for example, to the fact that a large number of data categories are input to the model, and some of them (e.g., biomarkers) are present in the clinical data of only a small portion of the patient cohort. Furthermore, for the same reason, the individual prediction model CRC for CRC with distant metastasis is trained on the clinical data of only 15% of the corresponding patient cohort. D and CRC G , individual prediction model BC E , and an individual prediction model for advanced lung cancer LC D -LC H are trained on a relatively small subset (2%-63%) of the clinical data of the corresponding patient cohort.

[0091] D. Individual Predictive Models and Metadata

[0096] The identified portions of the clinical data having the data categories required for each individual predictive model, as shown in FIG. 4B, are then used to build and train individual predictive models. In the first part of the experiment, each individual predictive model includes a random forest model as described in FIG. 2C. In the second part of the experiment, each individual predictive model includes a mathematical model as described in FIG. 2D. Each portion of the clinical data is randomly split into a training and testing partition, respectively, in a ratio of approximately 75:25. The training partition is used for model development, and the testing partition is kept separate for testing the meta-model, as described below.

[0092]

[0097] The training partition is further divided into a first clinical data subset for training an individual predictive model and a second clinical data subset for validating the performance of the trained model, as described in FIG. 3D. For example, 80% of the training partition can be used for training, and 20% of the training partition can be used for validation. Five different partitions can be performed to create five different first and second subsets of clinical data. The validation partition is used in a cross-validation operation to calculate the AUC for the trained model with respect to survival at 500 days, 1000 days, and 1500 days after diagnosis, as shown in FIG. 3C. The training and validation operations are performed five times with different training and validation partitions to obtain five AUC values. The weights of each individual predictive model are then determined based on the average of the five AUC values ​​obtained from the training and validation operations.

[0093] E. Experimental Results

[0098] The meta-model for each cancer type (colon cancer with distant metastasis, breast cancer with distant metastasis, and advanced lung cancer) is then tested using the test plots. Two sets of tests are performed. The first test involves using the meta-model to make a prediction of patient-specific survival for randomly selected patients, and the second test involves determining the AUC of the meta-model.

[0094]

[0099] In the first test, a random group of patients is selected from the test partition of each patient cohort. For each patient, the individual predictive models for which the patient has the data categories required by the individual predictive models are identified and combined with their associated weights to form a meta-model, as described in Figure 3A. The meta-model is then used to determine survival rates at various time points from the time of diagnosis to generate a KM plot for the patient.

[0095]

[0100] FIG. 4C illustrates the KM plot of the metamodel for 12 patients (patient AL) from a patient cohort with CRC with distant metastasis (represented by solid lines) and also illustrates the KM plot of the patients represented in the training partition of the patient cohort (represented by dotted lines). The KM plot in FIG. 4C illustrates that the survival rate of a particular patient can vary significantly from the survival rate of the cohort. As shown in FIG. 4C, a patient can have a different set of applicable individual prediction models. For example, patient A has a set of applicable prediction models CRC A , CRC B , and CRC C (In Fig. 4C, CRC A -CRC C ) and patient H has the applicable prediction model CRC A , CRC B , CRC C , CRC D , CRC E , CRC F , and CRC G (In Fig. 4C, CRC A -CRC G Furthermore, with the exception of patient A, the KM plots for the other patients do not match the KM plots for the training plots of the patient cohort. This further highlights the need to provide patient-specific predictions of survival rather than relying on population-based survival statistics.

[0096]

[0101] In the second test, the AUC of the meta-model is determined and compared to the AUC of the individual predictive models. Specifically, referring back to FIG. 3C, for each patient in the training partition, the individual predictive models for which the patient has the data categories required by the individual predictive models are determined, and the identified individual predictive models are combined to obtain the meta-model. The survival rate or survival rate is determined for the patient using the meta-model and compared to a probability threshold to predict whether the patient will be alive at a particular time after diagnosis (e.g., 500 days, 1000 days, and 1500 days). The patient's actual survival information is compared to the survival prediction to count the patient as a true positive or false positive. The analysis is performed for all patients in the training partition and for various probability thresholds to generate the AUC for the meta-model approach.

[0097]

[0102] Figure 4E illustrates the AUC of each individual prediction model and the meta-model for the cohort of patients with CRC with distant metastasis in the second test. The meta-models represented in the AUC plot can include different sets of prediction models based on the data categories present in the patient data used to evaluate each prediction model. In Figure 4E, the AUC is calculated for a) prediction models using only the subset of patient data with the required data categories, b) meta-models including all of the prediction models where the patient data subset has the data categories required for the prediction models, and c) prediction models using all patient data and, for patients without the required data categories, imputed data for the missing data categories.

[0098]

[0103] As shown in Figure 4E, the meta-model approach provides improved AUC compared to each individual predictive model. Substantial improvements in AUC are seen in cases where the individual predictive models only receive a small set of data categories as input (e.g., predictive model CRC using only gender and race). AFor example, for survival prediction at 500 and 1000 days, the AUC was A The AUC improved from 0.61 when using only the CRC model to 0.74 when using the meta-model approach. Furthermore, for the prediction of survival at 1500 days, the AUC was significantly higher than that of the CRC model. A The improvement is from 0.57 when using only the model to 0.74 when using the metamodel approach.

[0099]

[0104] Predictive models that take multiple data categories as input (e.g., predictive model CRC G For example, for survival prediction at 500 days, the AUC was 1.2 times higher for the prediction model CRC. G The AUC improved from 0.81 when using only the CRC model to 0.84 when using the meta-model approach. Furthermore, for the prediction of survival at 1000 days, the AUC was significantly higher than that of the CRC model. G The AUC improved from 0.78 when using only the CRC model to 0.79 when using the meta-model approach. Furthermore, for the prediction of survival at 1500 days, the AUC was significantly higher than that of the CRC model. G The improvement is from 0.73 when using only the model to 0.75 when using the metamodel approach.

[0100]

[0105] Moreover, the AUC of the meta-model is also higher than that of a single model using imputed data for all predictive models, which may be due to the fact that, as mentioned above, the imputed data are generated based on the “missing at random” assumption, which is not a valid assumption in clinical data.

[0101] VI. Method

[0106] 5 illustrates a method 500 for making a clinical prediction for a patient. The clinical prediction may include, for example, the patient's survival rate at a particular time after diagnosis with a disease (e.g., a particular type of cancer). Method 500 may be performed by various components of clinical prediction system 300.

[0102]

[0107] In step 502, the clinical prediction system 300 (e.g., the selection module 304) receives data 308 corresponding to a plurality of data categories for a patient. The plurality of data categories may include, for example, personal data, medical history data, laboratory test results, biomarkers, etc., as shown in FIG. 4A.

[0103]

[0108] In step 504, the clinical prediction system (e.g., selection module 304) selects a first machine learning model and a second machine learning model from the plurality of trained machine learning models and based on the plurality of data categories, the first machine learning model being trained with first data of a first subset of the plurality of data categories and having a first performance metric, and the second machine learning model being trained with second data of a second subset of the plurality of data categories and having a second performance metric. These machine learning models may be selected from machine learning model database 302 of clinical prediction system 300.

[0104]

[0109] Specifically, the machine learning model database 302 of the clinical prediction system 300 can store multiple machine learning models, including models 304a, 304b, 304c, etc. Each machine learning model can include, for example, a random forest model (e.g., random forest model 230 of FIG. 2C), a regression model (e.g., regression model 250 of FIG. 2D), or other types of machine learning models to make a prediction (e.g., survival rate). Each machine learning model in the machine learning model database 302 is trained separately with data from a set of data categories to make a prediction, and different machine learning models are trained with data from different sets of data categories, although there may be overlapping data categories among the different sets. For example, referring to FIG. 3A , if the data 308 includes data categories S0, S1, S2, S3, S4, and S5, the selection module 304 may identify a first machine learning model 304a trained using the data categories S0, S1, S2, and S3, and a second machine learning model 304b trained using the data categories S0, S4, and S5.

[0105]

[0110] Further, each of the multiple machine learning models can be associated with a performance metric, such as AUC, which can be expressed in the form of a weight. The selection module 304 can perform a search of the machine learning models in the machine learning model database 302 to identify machine learning models for which the data received in step 502 includes the data categories required for the machine learning model, and provide the identified machine learning models and their associated performance metrics (e.g., weights) to the prediction module 306.

[0106]

[0111] Referring again to FIG. 3A, the selection module 304 can search the machine learning model database 302 for machine learning models that accept a subset (or all) of the data categories present in the data 308. In one example, the selection module 308 can obtain the required data categories of each machine learning model in the database (e.g., data categories S0, S1, S2, and S3 for machine learning model 304a, data categories S2, S4, and S5 for machine learning model 304b, etc.), compare those data categories with the data categories of the data 308, and determine whether the required data categories of the machine learning model represent a subset (or all) of the data categories in the data 308. If they do, the selection module 304 can select that machine learning model to generate a meta-model. The selection module 304 can then provide the selected machine learning model to the prediction module 306.

[0107]

[0112] In step 506, a first subset of the data corresponding to a first subset of the categories is input to a first machine learning model to generate a first prediction result. Referring to FIG. 3A, the prediction module 306 can input a first subset of the data 308 including data categories S0, S1, S2, and S3 to the first machine learning model 304a. The first prediction result can include, for example, a first survival rate predicted for the patient by the first machine learning model 304a.

[0108]

[0113] At step 508, a second subset of the data corresponding to a second subset of the plurality of categories is input to the second machine learning model to generate a second prediction result. Referring to FIG. 3A, the prediction module 306 can input a second subset of the data 308 including data categories S0, S4, and S5 to the second machine learning model 304b. The second prediction result can include, for example, a second survival rate predicted for the patient by the second machine learning model 304b.

[0109]

[0114] In step 510, the prediction module 306 generates a composite prediction result based on the first prediction result, the second prediction result, the first weight indicative of the first performance metric, and the second weight indicative of the second performance metric. For example, the prediction module 306 can generate a composite survival rate for the patient based on a weighted average of the first survival rate and the second survival rate based on the first weight and the second weight. The first weight can reflect a first AUC of the first machine learning model 304a, and the second weight can reflect a second AUC of the second machine learning model 304b.

[0110]

[0115] In step 512, a clinical prediction for the patient can be made based on the composite prediction result. For example, the composite survival rate can then be provided to a clinical decision support tool, such as the clinical decision support tool 104, to generate various information to support clinical decisions, as described above with respect to FIG. 1B. For example, to facilitate treatment for the patient, the clinical decision support tool 104 can generate information to allow clinicians to better access the patient's prognosis, such as a comparison of the patient's predicted survival rate with other patients and the patient's life expectancy. The information can facilitate discussions between clinicians and patients regarding the evaluation of the patient's prognosis and treatment options and planning for the patient's life events. As an example, if the clinical decision support tool 104 predicts that the patient has a relatively long life expectancy (e.g., 5 years), the patient may decide to undergo aggressive treatment that is physically demanding and has significant side effects. However, if the clinical decision support tool 104 indicates that the patient has a relatively short remaining life expectancy (e.g., less than one year), the patient may decide to forgo treatment or receive alternative treatment and may also plan for care and life events for the patient's remaining life.

[0111]

[0116] Additionally, the clinical decision support tool 104 can also output information 108 to facilitate selecting an optimal treatment for a patient. For example, the clinical decision support tool 104 can display various KM plots for a patient for various treatments based on the patient's data. The clinician can then select, for example, the treatment that maximizes the patient's survival rate at a given time, the treatment that maximizes the patient's expected remaining life span, etc.

[0112]

[0117] In all of these examples, clinical predictions are made based on predicted survival for the patient and not for the patient cohort, so clinical predictions made for the patient tend to be more accurate, or at least more relevant, to the patient. As shown in Figure 4D, patient-specific survival predictions can be significantly different from the survival of the patient cohort. The disclosed techniques can improve the accuracy and validity of clinical predictions and avoid the problem of missing data categories when machine learning models require many data categories that the patient does not have.

[0113] VII. Computer Systems

[0118] Any of the computer systems described herein may employ any suitable number of subsystems. An example of such a subsystem is shown in computer system 10 in FIG. 6. In some embodiments, the computer system may include one computer device, and the subsystems may be components of the computer device. In other embodiments, the computer system may include multiple computer devices, each of which may be a subsystem and have internal components. The computer system may include desktop and laptop computers, tablets, mobile phones, and other mobile devices. In some embodiments, the disclosed techniques may be implemented using cloud infrastructure (e.g., Amazon Web Services), graphical processing units (GPUs), and the like.

[0114]

[0119] The subsystems shown in FIG. 6 may be interconnected via a system bus 75. Additional subsystems are shown, such as a printer 74, a keyboard 78, storage device(s) 79, a monitor 76 coupled to a display adapter 82, and others. Input / output (I / O) devices coupled to the peripherals and I / O controller 71 may be connected to the computer system by any number of means known in the art, such as input / output (I / O) ports 77 (e.g., USB, FireWire, etc.). For example, the I / O ports 77 or external interface 81 (e.g., Ethernet, Wi-Fi, etc.) may be used to connect the computer system 10 to a wide area network, such as the Internet, a mouse input device, or a scanner. The interconnection via the system bus 75 allows the central processor 73 to communicate with each subsystem and control the execution of instructions from the system memory 72 or storage device(s) 79 (e.g., fixed disks such as hard drives, optical disks, etc.) and the exchange of information between the subsystems. The system memory 72 and / or the storage device(s) 79 may embody a computer readable medium. Another subsystem is a data collection device 85 such as a camera, microphone, accelerometer, etc. Any of the data described herein may be output from one component to another or to a user.

[0115]

[0120] A computer system may include multiple identical computers or subsystems connected together, for example, by external interfaces 81 or internal interfaces. In some embodiments, the computer systems, subsystems, or devices may communicate over a network. In such cases, one computer may be considered a client and another computer may be considered a server, each of which may be part of the same computer system. Each of the clients and servers may include multiple systems, subsystems, or components.

[0116]

[0121] Features of the embodiments may be implemented in the form of control logic using hardware (e.g., application specific integrated circuits or field programmable gate arrays) and / or using computer software in conjunction with a generally programmable processor in a modular or integrated fashion, including a single-core processor, a multi-core processor on the same integrated chip, or multiple processing units on a single circuit board or networked together. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will know and appreciate other means and / or methods for implementing embodiments of the present invention using hardware and combinations of hardware and software.

[0117]

[0122] Any of the software components and functions described in this application may be implemented as software code using any suitable computer language, such as Java, C, C++, C#, Objective-C, Swift, or scripting languages, such as Perl or Python, using traditional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer-readable medium for storage and transmission. Suitable non-transitory computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media, such as hard drives and floppy disks, optical media, such as compact disks (CDs) and digital versatile disks (DVDs), flash memory, and the like. The computer-readable medium may be any combination of such storage and transmission devices.

[0118]

[0123] Also, such a program may be encoded and transmitted using a carrier signal adapted for transmission over wired, optical, and / or wireless networks following various protocols, including the Internet. Thus, a computer-readable medium may be created using a data signal encoded with such a program. A computer-readable medium encoded with program code may be packaged with a computing device or provided separately from other devices (e.g., via download over the Internet). Any such computer-readable medium may reside in one computing product (e.g., a hard drive, a CD, or an entire computing system) or may reside in various computing products within a system or network. A computing system may include a monitor, printer, or other suitable display for providing any of the results described herein to a user.

[0119]

[0124] Any of the methods described herein may be performed in whole or in part using a computer system including one or more processors configurable to perform the steps. Thus, embodiments may be directed to a computer system configured to perform any of the method steps described herein, possibly with various components performing each step or each group of steps. Although steps are shown as numbered steps, steps of the methods herein may be performed simultaneously or in different orders. Furthermore, portions of these steps may be used with portions of other steps from other methods. Also, portions or entire steps may be optional. Furthermore, any of the steps of any of the methods may be performed using modules, units, circuits, or other means for performing those steps.

[0120]

[0125] The specific details of the particular embodiments may be combined in any suitable manner without departing from the spirit and scope of the embodiments of the invention, however other embodiments of the invention may be directed to particular embodiments relating to each of the individual features or particular combinations of those individual features.

[0121]

[0126] The above description of example embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and many modifications and variations are possible in light of the above teachings.

[0122]

[0127] The words "a," "an," and "the" are intended to mean "one or more," unless specifically stated otherwise. The use of "or" is intended to mean "inclusive or," and does not mean "exclusive or," unless specifically stated otherwise. A reference to a "first" component does not necessarily require that a second component is also present. Furthermore, a reference to a "first" or "second" component does not limit the referenced component to a particular location, unless specifically stated otherwise.

[0123]

[0128] All patents, patent applications, publications, and statements mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art.

Claims

1. 1. A computer-implemented method for making clinical predictions, comprising: receiving patient medical data corresponding to a plurality of data categories regarding biological or behavioral characteristics or treatment history of a particular patient; In response to receiving the patient medical data, selecting a first machine learning model and a second machine learning model from a plurality of trained machine learning models based on the plurality of data categories about biological or behavioral characteristics or treatment history of the particular patient; the first machine learning model is trained with first clinical data for a first subset of the plurality of data categories and has a value for a first clinical performance metric; the second machine learning model is trained with second clinical data for a second subset of the plurality of data categories and has a value for a second clinical performance metric; the second subset of the plurality of data categories being distinct from the first subset of the plurality of data categories regarding biological or behavioral characteristics or medical history; inputting a first subset of the patient medical data corresponding to the first subset of the plurality of data categories of biological or behavioral characteristics or treatment history into the first machine learning model to generate a first clinical prediction outcome; inputting a second subset of the patient medical data corresponding to the second subset of the plurality of data categories of biological or behavioral characteristics or treatment history into the second machine learning model to generate a second clinical prediction outcome; generating a composite clinical prediction outcome based on the first clinical prediction outcome, the second clinical prediction outcome, a first weight indicative of a value of the first clinical performance metric, and a second weight indicative of a value of the second clinical performance metric, a value of the second clinical performance metric that is different from a value of the first clinical performance metric; making a clinical prediction for the particular patient based on the composite clinical prediction result; outputting said clinical prediction, said clinical prediction identifying a diagnosis or prognosis for a medical condition of said particular patient, or a treatment to be administered to said particular patient; The method includes:

2. 2. The method of claim 1, wherein the first subset of the plurality of data categories and the second subset of the plurality of data categories include at least one common data category.

3. 3. The method of claim 1 or 2, wherein the data categories include at least one of personal data of the particular patient, results of one or more laboratory tests of the particular patient, biopsy image data of the particular patient, molecular biomarkers of the particular patient, cancer site of the particular patient, and cancer stage of the particular patient.

4. The method according to any one of claims 1 to 3, The plurality of machine learning models includes a third machine learning model trained with a third clinical data of a data category that is not part of the plurality of data categories, the data category being: wherein the third machine learning model is not selected for the particular patient based on the data category not present within the plurality of data categories for the particular patient.

5. 5. The method of claim 1, wherein the clinical prediction comprises predicting the survival rate of the particular patient at a given time from when the particular patient was diagnosed with advanced cancer.

6. 6. The method of claim 5, further comprising determining the treatment for the particular patient based on the predicted survival rate.

7. 7. The method according to claim 5 or 6, the plurality of data categories includes a category associated with a treatment received by the particular patient; wherein the clinical prediction comprises predicting the survival rate of the particular patient at the given time in response to the treatment. method.

8. 8. The method of claim 5, wherein the first and second machine learning models include at least one of a random forest model and a hazard function.

9. 9. The method of claim 8, the first machine learning model comprises a random forest model; the random forest model includes a plurality of decision trees; each decision tree configured to process a subset of the first subset of the patient medical data to generate a cumulative hazard function (CHF) value; The method of claim 1, wherein the survival rate of the particular patient at the given time is determined based on an average of the CHF values ​​output by the plurality of decision trees.

10. 10. The method according to claim 8 or 9, 11. The method of claim 10, wherein the first machine learning model includes a hazard function that outputs a probability of non-survival for the particular patient at the given time, the hazard function being a function of values ​​of the first subset of the plurality of data categories.

11. The method according to any one of claims 5 to 10, The method, wherein the value of the first clinical performance metric and the value of the second clinical performance metric relate to rates of correct and incorrect survival predictions by the first and second machine learning models for a group of patients at the given time.

12. 12. The method according to claim 10 or 11, the value of the first clinical performance metric and the value of the second clinical performance metric are based on an area under the curve (AUC) of a receiver operating characteristic (ROC) curve of each of the first machine learning model and the second machine learning model; the first weight is based on an area measure of a first AUC of a first ROC curve of the first machine learning model; The method of claim 1, wherein the second weights are based on an area measure of a second AUC of a second ROC curve of the second machine learning model.

13. 13. The method of claim 12, wherein the composite clinical prediction outcome represents a linear combination of the first clinical prediction outcome adjusted by the first weight and the second clinical prediction outcome adjusted by the second weight.

14. The method according to any one of claims 1 to 13, the first machine learning model is trained using a first subset of the first clinical data; a value of the first clinical performance metric is determined based on an output of the trained first machine learning model upon processing of a second subset of the first clinical data; the second machine learning model is trained using a third subset of the second clinical data; wherein the value of the second clinical performance metric is determined based on an output of the trained second machine learning model upon processing of a fourth subset of the second clinical data.

15. 15. The method of claim 14, the first machine learning model is trained using a first distinct subset of the first clinical data; a value of the first clinical performance metric is determined based on an output of the trained first machine learning model upon processing of a second distinct subset of the first clinical data; the second machine learning model is trained using a third, different subset of the second clinical data; The method of claim 1, wherein the value of the second clinical performance metric is determined based on an output of the second trained machine learning model upon processing of a fourth distinct subset of the second clinical data.

16. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 15.

Images