Systems and methods for quantitative clinical trial database modeling to inform project decisions

Abstract

Embodiments of the present disclosure include systems, methods, and non- transitory computer-readable storage media for generating a combined standardized clinical trial database providing a graphical user interface for analysis using the combined standardized clinical trial database. The disclosure of some embodiments includes processes of standardizing and unifying clinical trial data obtained from a variety of database systems that store similar clinical trial data using different datatypes and formats.

Description

SYSTEMS AND METHODS FOR QUANTITATIVE CLINICAL TRIAL DATABASE MODELING TO INFORM PROJECT DECISIONSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to European Patent Application No. 23306518.4 filed September 14, 2023, and to European Patent Application No. 24305220.6 filed February 9, 2024, the entire contents of both of which are incorporated herein by reference.TECHNICAL FIELD

[0002] Some embodiments disclosed herein relate to the field of medical informatics, and more particularly to systems and methods using medical informatics primarily to plan, conduct and analyze clinical trials and their results.BACKGROUND

[0003] Clinical trial knowledge that is stored in or accessed by various conventional clinical trial database systems can be disparate in its structure and management. For example, clinical trial knowledge may include information about the outcome of clinical trials including safety and efficacy data that is based on results published in different journal publications, information about drugs evaluated during the clinical trials as well as results about the drugs evaluated in the clinical trials. Clinical trial knowledge that is stored in or accessed by various conventional clinical trial database systems can also include information about the comparison of different clinical trials based on the safety and efficacy data. However, the clinical trial knowledge stored across multiple clinical trial database systems can often be unstructured, or structured, according to a specific format associated with an individual clinical trial database system of the multiple clinical trial database systems. As a result, data and information representing the clinical trial knowledge can be structured differently across the different clinical trial database systems.SUMMARY

[0004] Embodiments of the present disclosure comprises a method comprising: extracting or accessing, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generating, by the at least one processor, aplurality of structured datasets including a plurality of data types based on the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; applying, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generating, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing; and providing a graphical user interface to receive input from a user regarding an analysis to be performed using the plurality of structured datasets.

[0005] The method can include a method, wherein the plurality of structured datasets also includes a regulatory event dataset. The method can include a method, wherein the at least a portion of the extracted information including unstructured data includes recent information not yet available as structured clinical trial information from the one or more databases. The method can include a method, further comprising: extracting or accessing, by at least one processor, second information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the one or more databases; and one or both of: generating, by the at least one processor, second data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the second information; and applying, by the at least one processor, natural language processing to at least a portion of the second information including unstructured text, and generating, by the at least one processor, second data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

[0006] The method according to some embodiments may further comprise: periodically extracting or accessing, by at least one processor, additional information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the one or more databases; and one or both of: generating, by the at least one processor, additional data corresponding to at least one of the plurality of data types to be added to at least one of theplurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the additional information; and applying, by the at least one processor, natural language processing to at least a portion of the additional information including unstructured text, and generating, by the at least one processor, additional data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

[0007] The method according to some embodiments may further comprise: executing, by the at least one processor, a machine learning module to determine a probability of success of a clinical drug trial advancing to a next clinical trial stage based on the plurality of structured datasets. The method may further comprise: executing, by the at least one processor, a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets. The method may further comprise: executing, by the processor, a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets for each of a plurality of regulatory routes. The method may further comprise: identifying, by the at least one processor, one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the historical likelihood of approval of for at least some of the plurality of clinical drug trials.

[0008] The method, in some embodiments may include a method, wherein the one or more scientific factors include a mechanism of action associated with each of the one or more clinical drug trials, the one or more design factors include endpoints associated with each of the one or more clinical trials, and the one or more regulatory factors includes a designation of the one or more clinical drug trials as a breakthrough. The method may further comprise assigning a category of trial status to each clinical trial in the plurality of structured datasets, wherein the categories of trial status include one or more of: completed with final results, terminated or ongoing with no final results, and no results. The method may further comprise: simulating, by the at least one processor, one or more phase three clinical trial outcomes based on one or more proof of concept study outcomes. The method may include a method, wherein the GUI includes: a first field for receiving a first input including information associated with a disease, specifying variables or variable combinations of interest, or both;and a second field for identifying an output to be generated. The method may include a method, wherein the first input includes or identifies a disease-specific database or a subset of a disease-specific database, the disease-specific database is based at least in part on standardized clinical trials associated with the disease. The method may include a method, wherein the identified output to be generated includes one or more of a description of one or more first clinical trials associated with the disease; characteristics or outcomes associated with the one or more first clinical trials; or prediction of an outcome of the one or more first clinical trials for a drug used to treat the disease.

[0009] The method, in some embodiments, may include a method wherein the trial arm dataset includes data types for associating trial outcomes with treatment information. The method may further comprise generating a predictive model or in-silico trial simulations for a prediction of a trial outcome. The method may include a method, wherein the extracted or accessed trial outcome information includes endpoints associated with each of the clinical trials. The method may further comprise generating a predictive model or in-silico trial simulations for a prediction of the endpoints, wherein the predictive model or in-silico trial simulations is based at least in part on a machine learning model.

[0010] Embodiments of the present disclosure comprise a system comprising: storage configured to store a unified clinical trial database; a memory storing one or more instructions; and at least one processor operationally coupled to the storage, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: extract or access, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generate, by the at least one processor, a plurality of structured datasets including a plurality of data types based on at least a portion of the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; apply, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing;store the plurality of structured datasets in the storage, where the unified clinical trial database includes the plurality of structured datasets; and provide a graphical user interface to receive input from a user regarding an analysis to be performed using the unified clinical trial database.

[0011] Embodiments of the system may comprise a system, wherein the plurality of structured datasets also includes a regulatory event dataset. The system may further be one, wherein the at least a portion of the extracted information including unstructured data includes recent information not yet available as structured clinical trial information from the unified clinical trial database. The system may further be a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: extract or access, by at least one processor, second information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the unified clinical trial database; and one or both of: generate, by the at least one processor, second data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the second information; and apply, by the at least one processor, natural language processing to at least a portion of the second information including unstructured text, and generating, by the at least one processor, second data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

[0012] The system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: periodically extract or access, by at least one processor, additional information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the clinical trial databases and one or both of: generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the additional information; andapply, by the at least one processor, natural language processing to at least a portion of the additional information including unstructured text, and generate, by the at least one processor, additional data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

[0013] The system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: execute a machine learning module to determine a probability of success of a clinical drug trial advancing to a next clinical trial stage based on the plurality of structured datasets. The system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: execute a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets. The system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: execute a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets for each of a plurality of regulatory routes.

[0014] The system may further be a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: identify, by the at least one processor, one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the historical likelihood of approval of for at least some of the plurality of clinical drug trials. The system may further be a system, wherein the one or more scientific factors include a mechanism of action associated with each of the one or more clinical drug trials, the one or more design factors include endpoints associated with each of the one or more clinical trials, and the one or more regulatory factors includes a designation of the one or more clinical drug trials as a breakthrough. The system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: assign a category of trial status to each clinical trial in the plurality of structured datasets, wherein the categories of trial statusinclude one or more of: completed with final results, terminated or ongoing with no final results, and no results.

[0015] Embodiments of the system may further comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: simulate, by the at least one processor, one or more phase three clinical trial outcomes based on one or more proof of concept study outcomes. The system may further be a system, wherein the GUI includes: a first field for receiving a first input including information associated with a disease, specifying variables or variable combinations of interest, or both; and a second field for identifying an output to be generated. The system may further be a system, wherein the first input includes or identifies a disease-specific database or a subset of a disease-specific database. The system may further be a system, wherein the identified output to be generated includes one or more of: a description of one or more first clinical trials associated with the disease; characteristics or outcomes associated with the one or more first clinical trials; or prediction of an outcome of the one or more first clinical trials for a drug used to treat the disease. The system may further be a system, wherein the trial arm dataset includes data types associating trial outcomes with treatment information. The system may further be a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: generate a predictive model or in-silico trial simulations for a prediction of a trial outcome.

[0016] Embodiments of the system may further be a system, wherein the extracted or accessed trial outcome information includes endpoints associated with each of the clinical trials. The system may comprise a system, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: generate a predictive model or in-silico trial simulations for a prediction of progression free survival (PFS) based on overall response rate (ORR), also referred to herein as objective response rate ORR, using a tree-based regression machine learning model or associated in-silico trial simulations. The system may further comprise a system, wherein the information further includes publication, conference proceedings, journal articles, regulatory information associated with one or more clinical trials.

[0017] Embodiments of the present disclosure comprise a non-transitory computer-readable medium comprising instructions that when executed by a processing device causes the processing device to: extract or access, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generate, by the at least one processor, a plurality of structured datasets including a plurality of data types based on at least a portion of the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; apply, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing; store the plurality of structured datasets in the storage, where the unified clinical trial database includes the plurality of structured datasets; and provide a graphical user interface to receive input from a user regarding an analysis to be performed using the unified clinical trial database.

[0018] Embodiments of the non-transitory computer-readable medium comprise media, wherein the plurality of structured datasets also includes a regulatory event dataset. The non- transitory computer-readable medium also comprise media, wherein the at least a portion of the extracted information including unstructured data includes recent information not yet available as structured clinical trial information from the unified clinical trial database. The non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: extract or access, by at least one processor, second information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the unified clinical trial database; and one or both of: generate, by the at least one processor, second data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the second information; and apply, by the at least one processor, natural language processing to at least a portion of the second information including unstructured text, and generating, by the at least one processor, second data or revised data corresponding to at least one of the pluralityof data types for at least some of the plurality of structured datasets based on the natural language processing.

[0019] Embodiments of the non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: periodically extract or access, by at least one processor, additional information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the clinical trial databases and one or both of: generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the additional information; and apply, by the at least one processor, natural language processing to at least a portion of the additional information including unstructured text, and generate, by the at least one processor, additional data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing. The non- transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: execute a machine learning module to determine a probability of success of a clinical drug trial advancing to a next clinical trial stage based on the plurality of structured datasets.

[0020] Embodiments of the non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: execute a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets. The non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: execute a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets for each of a plurality of regulatory routes. The non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: identify, by the at least one processor, one or more scientificfactors, design factors, regulatory factors, or operational factors that contributed to the historical likelihood of approval of for at least some of the plurality of clinical drug trials.

[0021] Embodiments of the non-transitory computer-readable medium may comprise media, wherein the one or more scientific factors include a mechanism of action associated with each of the one or more clinical drug trials, the one or more design factors include endpoints associated with each of the one or more clinical trials, and the one or more regulatory factors includes a designation of the one or more clinical drug trials as a breakthrough. The non- transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: assign a category of trial status to each clinical trial in the plurality of structured datasets, wherein the categories of trial status include one or more of: completed with final results, terminated or ongoing with no final results, and no results. The non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: simulate, by the at least one processor, one or more phase three clinical trial outcomes based on one or more proof of concept study outcomes.

[0022] The non-transitory computer-readable medium may comprise media, wherein the GUI includes: a first field for receiving a first input including information associated with a disease, specifying variables or variable combinations of interest, or both; and a second field for identifying an output to be generated. The non-transitory computer-readable medium may comprise media, wherein the first input includes or identifies a disease-specific database or a subset of a disease-specific database. The non-transitory computer-readable medium may comprise media, wherein the identified output to be generated includes one or more of: a description of one or more first clinical trials associated with the disease; characteristics or outcomes associated with the one or more first clinical trials; or prediction of an outcome of the one or more first clinical trials for a drug used to treat the disease. The non-transitory computer-readable medium may comprise media, wherein the trial arm dataset includes data types associating trial outcomes with treatment information. Embodiments of the non- transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: generate a predictive model or in- silico trial simulations for a prediction of a trial outcome.

[0023] Embodiments of the non-transitory computer-readable medium may comprise media, wherein the extracted or accessed trial outcome information includes overall response rate (ORR) data and progression free survival (PFS) data. The non-transitory computer-readable medium may further comprise instructions that when executed by the processing device causes the processing device to: generate a predictive model or in-silico trial simulations for a prediction of PFS based on ORR using a tree-based regression machine learning model or associated in-silico trial simulations. The non-transitory computer-readable medium may comprise media, wherein the information further includes publication, conference proceedings, journal articles, regulatory information associated with one or more clinical trials.

[0024] Exemplary embodiments may include methods for another tree-based regression model for predicting a median progression free survival (mPFS) of a test subject or a median overall survival (mOS) of the test subject in response to drug treatment using overall response rate (ORR) data. In particular the tree-based regression model is a weighted regression that uses the trial arm sample size as a weight. In some embodiments, the treebased regression model relies on some of all of treatment class data, treatment line data, biomarker status information, as well as a current cancer stage for a subject.

[0025] Some embodiments provide a method of determining a predictive model of a median progression free survival (mPFS) or median objective survival (mOS) for a late phase clinical trial based on overall response rate (ORR) in an early phase clinical trial. The method includes extracting or accessing clinical trial information including ORR information, mPFS information, mOS information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information for a plurality of clinical trials obtained from one or more database. The method also includes defining variable classes including at least some of or all of variables based on ORR information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information. The method further includes determining a tree-based regression machine learning model for mPFS based on ORR or for mOS based on ORR, the determination including: performing forward segmentation of a predictor space based on the variable classes into a number of subspaces until there is nomPFS based on ORR heterogeneity or until there is no mOS based on ORR heterogeneity; and determining a regression model for each subspace.

[0026] In some embodiments, one or more of the treatment classes include an anti-PDl and / or anti-PLDl therapy (PD1 / PDL1 therapy); and the biomarker status information includes PD1 and / or PDL1 status information.

[0027] In some embodiments, performing forward segmentation of a predictor space into a number of subspaces until there is no mPFS based on ORR heterogeneity or until there is no mOS based on ORR heterogeneity includes: determining a slope for each variable class and grouping the variable classes with homogenous slope to form grouped variable classes; and for each grouped variable class, splitting the grouped variable class where there is significant heterogenous slope within the variable classes in the grouped variable class.

[0028] In some embodiments, determining a slope for each variable class and grouping the variable classes with homogenous slope to form grouped variable classes includes: determining a first variable class having a largest sample size as compared with sample sizes of the other variable classes which are one or more second variable classes; determining for each variable class, a slope for the variable class by comparing: a first regression of the mPFS or mOS against a product of the ORR and a value of the class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the class variable. Determining a slope for each variable class and grouping the variable classes with homogenous slope to form grouped variable classes also includes for each of the one or more second variable classes: determining a slope difference between a slope of the second variable class and a slope of the first variable class; and grouping variable classes where a slope difference for the variable classes have a P value over a specified grouping threshold or a second variable class sample size under a specified sample size threshold. In some embodiments, the steps above are performed for each remaining ungrouped second variable class.

[0029] In some embodiments, splitting the grouped variable class where there is significant heterogenous slope within the variable classes in the grouped variable class for each grouped variable class includes: for each variable having a number of classes after grouping of greater than one, comparing: a first regression of the mPFS or mOS against a product of the ORRand a value of the grouped class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the grouped class variable; and selecting a grouped variable class to split where an ANOVA P value is less than a specified splitting threshold and the ANOVA P value is the most significant for that variable as compared with all other variables.

[0030] In some embodiments, the method further includes, for each split grouped variable class: determining a first variable class of the split grouped variable class having a largest sample size as compared with sample sizes of the other variable classes in the split grouped variable class, which are one or more second variable classes; and determining for each variable class of the split grouped variable class, a slope for the variable class by comparing: a first regression of the mPFS or mOS against a product of the ORR and a value of the class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the class variable; and for each of the one or more second variable classes: determining a slope difference between a slope of the second variable class and a slope of the first variable class; and grouping variable classes where a slope difference for the variable classes have a P value over the specified grouping P threshold or a second variable class sample size under the specified sample size threshold. \

[0031] In some embodiments, determining a regression model for each subspace comprises: for each variable, comparing: a first regression of the mPFS information or mOS information against the ORR information added to the value of the class variable, and a second regression of the mPFS information or mOS information against the ORR; and selecting the variable whose P value is less than a specified P threshold.

[0032] In some embodiments, the predictive model of the mPFS on the ORR is a logarithm of the mPFS regressed on a logarithm of the ORR for at least some of the subspaces; or the predictive model of the mOS on the ORR is a logarithm of the mOS regressed on a logarithm of the ORR for at least some of the subspaces.

[0033] In some embodiments, the predictive model for at least some of the subspaces is further based on one or more of indication information and disease stage information.

[0034] In some embodiments, treatment class information comprises chemotherapy, a combination of chemotherapy and PD1 / PDL1, or a combination of two or more of chemotherapy, PD1 / PDL1, or a targeted therapy.

[0035] In some embodiments, the indication information comprises one or both of non-small cell lung cancer data and melanoma data.

[0036] In some embodiments, the treatment line information comprises one or both offirst line (IL) treatment and first line therapy plus (1L+) treatment.

[0037] In some embodiments, the methods described herein are performed by a computing system including at least one processor or one or more processors.

[0038] In some embodiments, the extracted or accessed clinical trial information is obtained from a quantitative clinical trial database. In some embodiments, the quantitative clinical trial database is generated a method including: extracting or accessing, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generating, by the at least one processor, a plurality of structured datasets including a plurality of data types based on the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; applying, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; and generating, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG. 1 is a flowchart of a process that can be executed in accordance with some embodiments of the present disclosure.

[0040] FIG. 2 schematically depicts a logical information flow between various exemplary databases, modules, and a computing environment in accordance with some embodiments of the present disclosure.

[0041] FIG. 3 is a chart including different datasets that can be incorporated into a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0042] FIG. 4 is a logical diagram depicting a process for generating a clinical trial summary dataset of a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0043] FIG. 5 is a logical diagram depicting a process for generating a clinical trial arm dataset of a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0044] FIG. 6 is a logical diagram depicting a process for generating a clinical trial baseline dataset of a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0045] FIG. 7 is a logical diagram depicting a process for generating a clinical trial outcome dataset of a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0046] FIG. 8 is a logical diagram depicting a process for generating a clinical trial outcome comparison dataset of a unified clinical trial database in accordance with some embodiments of the present disclosure.

[0047] FIG. 9 is a logical diagram depicting a process of applying natural language processing to unstructured text stored in a database to extract clinical trial outcome and clinical trial outcome comparison data in accordance with some embodiments of the present disclosure.

[0048] FIG. 10 schematically depicts an example computer system that can be used to execute some embodiments of the present disclosure.

[0049] FIG. 11 is a logical diagram depicting a process of accessing clinical trial data on a host device from a client device using an application programming interface in accordance with some embodiments of the present disclosure.

[0050] FIG. 12 is an example of the status of clinical trials organized according to different tiers in accordance with some embodiments of the present disclosure.

[0051] FIG. 13 is a logical diagram depicting a process of extracting clinical trial data from one or more external databases for a unified clinical trial database and using the extracted clinical trial data as input to trial simulator in accordance with some embodiments of the present disclosure.

[0052] FIG. 14 is a clinical trial simulator graphical user interface depicting a regression of progression free survival on overall response rate in accordance with some embodiments of the disclosure.

[0053] FIG. 15 is a clinical trial simulator graphical user interface depicting a probability that drug treatments used in clinical trials will advance from one development stage to the next development stage or advancing to approval for a total number of progression free survival events and a probability that drug treatments used in clinical trials will advance from one development stage to the next development stage or advancing to approval for a given size of an event in accordance with some embodiments of the disclosure.

[0054] FIG. 16 is a flow diagram illustrating a process of a tree based regression of progression free survival associated with patients in a clinical trial based on an overall response rate of the patients in the clinical trial to drug treatment for a particular disease including graphs showing results of the analysis in accordance with some embodiments of the disclosure.

[0055] FIG. 17 is a table including a predictor’s distribution of included trial arms in an Example in accordance with some embodiments of the disclosure.

[0056] FIG. 18 is a tree-based machine learning model of overall response rate (ORR) regressed on median progression free survival (mPFS) in the Example in accordance with some embodiments of the disclosure.

[0057] FIGS. 19A-19E include graphs of an ORR regressed on a mPFS with heterogeneity within a tree-based machine learning modelin the Example in accordance with some embodiments of the disclosure.

[0058] FIG. 19A is a graph of ORR regressed on a mPFS with heterogeneity over treatment class (all trial arms) in the Example.

[0059] FIG. 19B is a graph of ORR regressed on a mPFS with heterogeneity over line (treatment class group 1) within the tree-based machine learning model in the Example.

[0060] FIG. 19C is a graph of ORR regressed on a mPFS with heterogeneity over line (treatment class group 2) within the tree-based machine learning model in the Example.

[0061] FIG. 19D is a graph of ORR regressed on a mPFS with heterogeneity over indication (treatment class group 1 1L / 1L+) within the tree-based machine learning model in the Example.

[0062] FIG. 19E is a graph of ORR regressed on a mPFS with heterogeneity over indication (treatment class group 2 Remaining line) within the tree-based machine learning model in the Example.

[0063] FIG. 20 is a tree-based machine learning model of ORR regressed on median objective survival (mOS) in the Example in accordance with some embodiments of the disclosure.

[0064] FIGS. 21A-21C include graphs of an ORR regressed on a mOS with heterogeneity within tree-based machine learning model in the Example in accordance with some embodiments of the disclosure.

[0065] FIG. 21A is a graph of an ORR regressed on a mOS with heterogeneity over treatment class for all trial arms within the tree-based machine learning model in the Example.

[0066] FIG. 21B is a graph of an ORR regressed on a mOS with heterogeneity over indication for treatment class group 2 within the tree-based machine learning model in the Example. FIG. 21C is a graph of an ORR regressed on a mOS with heterogeneity over line for treatment class group 2 NSCLC / Melanoma within the tree-based machine learning model in the Example.

[0067] FIG. 22 includes mean square error (MSE) comparison of different regression models of ORR regressed against mPFS and ORR regressed against mOS in the Example.

[0068] FIGS. 23A-23D include probability of survival estimations for a phase II and phase III study using progression free survival (PFS) as an endpoint and objective survival (OS) asan endpoint in the Example where FIG. 23A is a graph of probability of success for scenario 1, FIG. 23B is a graph of probability of success for scenario 2, FIG. 23C is a graph of probability of success for scenario 3, and FIG. 23D is a graph of probability of success for scenario 4.DETAILED DESCRIPTION

[0069] Some embodiments of the present disclosure include computer-implemented methods, systems, and non-transitory computer-readable storage media providing standardized, quantified clinical trial data associated with one or more drugs that are used in the treatment of diseases in various stages of approval and that is received or extracted from a plurality of database systems. The disclosure of some embodiments describes processes of standardizing and unifying clinical trial data obtained from a variety of database systems that store similar clinical trial data using different datatypes and formats. Some of the clinical trial data stored in certain conventional clinical trial database systems can be unstructured text that includes relevant outcome results that are similar to structured clinical trial data stored in other database systems, but because of the unstructured nature of the text, any meaningful information associated with the unstructured text cannot easily be standardized in accordance with clinical trial data stored in other databases. Some embodiments disclosed herein, provide a process that obtains clinical trial data associated with certain measured results or outcome quantities, also known as variables, from multiple different database systems, generates standardized and unified data based on the obtained clinical trial data, and stores the standardized and unified clinical trial data in disease (indication) specific standardized databases.

[0070] Some embodiments disclosed herein provide methods of generating a unified clinical trial database, and a system including or employing a unified clinical trial database. Some embodiments disclosed herein provide methods employing a unified clinical trial database.

[0071] Some embodiments disclosed herein provide users (e.g., researchers, sponsors of clinical trials, physicians, investors, etc.) with a probability or likelihood that a particular drug used in clinical trials will advance through various stages of a clinical trial, as well as determine what factors, if any, significantly contribute to the likelihood that the particular drug will advance through certain stages of the clinical trials to approval.

[0072] Some conventional individual database systems can provide some predictive analytics that generate a probability or likelihood of success regarding whether certain drugs will advance to each stage of a clinical trial. However, simply summing or combining these probabilities or likelihoods produced from multiple different individual conventional database systems merely provides a sum or combination of the individual probabilities or likelihoods with which a drug will advance through the different stages of the clinical trials and ultimately to approval. Furthermore, methods used to compute the probabilities and / or likelihoods is likely to be different for different individual conventional database systems given that the measures associated with the outcomes of the clinical trials corresponding to the drugs are different across different databases storing clinical trial data. As a result, any combination of the individual probabilities or likelihoods may not provide a more accurate representation of the probability or likelihood with which a drug will advance through the various stages of the clinical trials and to approval, than the individual probabilities or likelihoods.

[0073] Some described methods, systems, and non -transitory computer-readable media described herein accurately provide a probability and likelihood of advancement of a drug through various stages of a clinical trial and ultimately to approval based on a normalized, or standardized and structured set of clinical trial data. In addition to this, some described methods, systems, and non-transitory computer-readable media identify factors that significantly contribute to the probability and likelihood with which the drug associated with the clinical trials will advance through the stages and to approval. In some embodiments, the described methods, systems, and non-transitory computer-readable media can identify different lines of therapy that contribute to a probability and likelihood with which the drug associated with the clinical trials will advance through the stages and to approval.

[0074] In some embodiments, safety and efficacy information stored across the clinical trial database systems may be used to determine or model a likelihood that a drug will advance through the different stages of clinical trials and ultimately to approval. The safety and efficacy information may also be used to determine certain factors that contribute to the likelihood that the drug will advance through the different stages of the clinical trials and to approval in some embodiments. This information is associated with each individual clinical trial, which provides a likelihood that the performance drugs will advance through the different stages of the clinical trials and potentially approval for each of the clinical trials.

[0075] To this end standardizing the safety and efficacy data obtained from different clinical trial databases improves the ability to determine or model a likelihood that performance drugs will advance through the different stages of the clinical trials and ultimately to approval and determine certain factors that contribute the advancement through the different stages of the clinical trials and ultimately to approval.

[0076] In some embodiments, described methods, systems, and non-transitory computer- readable media apply natural language processing (NLP) to unstructured text associated with the clinical trials obtained from database systems to identify relevant outcome events associated with the outcome of the clinical trials corresponding to drugs administered during the clinical trials. In some embodiments, methods, systems, and non-transitory computer- readable media detect sections of unstructured text such as clinical trial design information, as well as results associated with clinical trial. For example, in some embodiments methods, systems, and non-transitory computer-readable media apply the NLP to the unstructured text to detect information associated with a description of how the study will be conducted, including comparison group design and strategies for masking and allocating participants. In response to applying the NLP to the unstructured text, in some embodiments, the methods, systems, and non-transitory computer-readable media automatically annotate sentences that are associated with the clinical trial design information and the corresponding results associated with the outcomes of the clinical trials, apply labels to the sentences and phrases in the unstructured text, and apply a pattern matching technique to identify and extract data associated with the outcomes of clinical trials such as treatment arms, biomarker types, efficacy estimates, confidence interval, and P values. In some embodiments, the extracted data is used by the methods, systems, and non-transitory computer-readable media to determine the probability of success or likelihood that the drugs associated with the clinical trials will progress throughout the various stages of approval and also any factors that contribute significantly to the probability of success or likelihood.

[0077] In some embodiments, the described methods, systems, and non-transitory computer- readable media provide a graphical user interface (GUI) that includes an application programming interface (API) that enables the user to access and interact with the structured clinical trial data from one or more indication-specific unified clinical trial databases. In some embodiments, the GUI also provides the user with the ability to locate clinical trial data based on certain areas of research and project specific needs. For example, the GUI canhave an interface that is best designed for oncology research, and another interface that is best designed for another area of research. In some embodiments, the GUI receives input from a user, and, in response, the system simulates an outcome of a clinical trial based on clinical trial data in one or more unified clinical trial databases.

[0078] FIG. 1 depicts an example method 100 that can be executed in accordance with some embodiments of the present disclosure. Method 100 provides a user with access to an analysis of structured clinical trial data from one or more databases (e.g., a plurality of clinical trial databases) via a graphical user interface. One or more steps of the method can be executed by one or more computing devices or computing systems (e.g., the computing device 205 in FIG. 2 or the computing system 1000 of FIG. 10). At 101, the method extracts or accesses clinical trial information associated with one or more clinical trials from one or more databases (e.g., a plurality of clinical trial databases). For example, an account hosted on the computing device 205 can request an access token by submitting a client id and password to an identity service and can receive an access token from the identity service confirming the identity of the account. The computing device can then use the issued access token to extract or access the information associated with the clinical trials stored or hosted on the clinical trial databases 201 and 202. Further description of obtaining access to one or more clinical trial databases appears below in connection with FIG. 11.

[0079] The clinical trial information obtained can include drug study design information, biomarker information, drug treatment information, and efficacy information associated with one or more clinical trials from one or more databases (e.g., clinical trial databases 201 and 202). The drug study design information can include, but is not limited to, descriptions of how the study has been, is being, or will be conducted, including one or more of comparison group design and strategies for masking and allocating participants. The biomarker information can include, but is not limited to, measurable biological parameters that can be evaluated throughout a clinical trial as an indicator of normal biological, pathogenic, and / or pharmacologic responses to certain drugs. Efficacy information can include, but is not limited to, efficacy endpoints. For some disease areas, e.g., cancer, the efficacy endpoints may include overall response rate (ORR), progression free survival (PFS), and overall survival (OS) in some embodiments.

[0080] After extracting or accessing the relevant information about the clinical trials from clinical trial databases 201 and 202, the method can then generate datasets associated withsummaries of the clinical trials, arms of the clinical trials, and at least some or all of baselines corresponding to the clinical trials, outcome of the clinical trials, and outcome comparisons between the clinical trials. For example, the computing device 205 can generate a plurality of structured datasets including a trial summary dataset 203a, a trial arm dataset 203b (FIG. 2), and at least some or all of a baseline dataset 203c (FIG. 2), an outcome dataset 203d (FIG. 2), and an outcome comparison dataset 203 e (FIG. 2) in accordance with some embodiments. Some clinical trials may not have baseline information available for generation of a baseline dataset 203c. Some clinical trials may not have outcome information available yet for generation of an outcome dataset 203d. Some clinical trials may not have outcome comparison information available yet for generation of an outcome comparison dataset 203e. Where regulatory information is available, the computing device 205 can also generate a regulatory information 203f dataset (FIG. 2) in accordance with some embodiments.

[0081] The trial summary 203a dataset can include study level information that has been extracted from clinical trial databases 201 and / or 202. In some embodiments this information can include raw datasets associated with the study itself. For example, in some embodiments the raw datasets can include one or more of a new clinical trial identifier (NCT ID) associated with a drug of the clinical trials, an industry associated with the development of the drug, a number of arms associated with the clinical trials, a start year of the clinical trial, and a completion year of the clinical trial where the clinical trial is completed. In some embodiments, the raw datasets can also include information associated with a primary completion year, which is the date on which the final participant is examined or received an intervention for the purposes of final collection of data for a primary outcome, whether the clinical study concluded according to the pre-specified protocol or was terminated. In some embodiments, the raw datasets can also include the name of a sponsor and / or the type of sponsor (e.g., academic, industry, government agency). In some embodiments, the trial summary dataset 203 can also include information associated with the drug administered throughout the clinical trial or associated with the clinical trial.

[0082] The trial arm dataset 203b can include treatment information associated with the clinical trial. The baseline dataset 203 c can include any or all of raw datasets from the clinical trial databases 201 and / or 202 that include the results of a group of participants in the clinicaltrials, baseline measurements associated with the participants in the groups of the clinical trials, and baseline counts associated with the clinical trials.

[0083] The outcome dataset 203d can include any or all of measurements associated with the outcome of the clinical trial, counts associated with the outcome of the clinical trial, information about the population of the groups participating in the clinical trials, as well as drug information associated with the outcome of the clinical trial. This information can include any or all of drug event information, drug phase, and drug approval information.

[0084] The outcome comparison dataset 203e can include analysis data comparing the outcomes across the clinical trials.

[0085] In some embodiments, the plurality of structured data sets also includes a safety dataset. The outcome comparison dataset can include between treatment comparison results on either efficacy or safety endpoints within the study. As the pair-wise comparisons can be conducted between different treatments, on different clinical endpoints and for different analysis populations, relevant comparisons need to be standardized for potential pooled analyses. Clinical safety measurements include but are not limited to adverse events, safety laboratory values, which can be pooled together for analysis.

[0086] The method 100 then progresses to step 102 and determines whether the extracted or accessed clinical trial information includes unstructured text. If the method determines that the extracted or accessed clinical trial information does include unstructured text (Yes), the method 100 progresses to step 104 and applies natural language processing to the unstructured text.

[0087] At 104, the method 100 applies a natural language processing module, such as natural language processing module 204, to the unstructured text associated with the clinical trial information stored or hosted on the clinical trial databases 201 and 202. The output of the natural language processing module 204 can include structured data that is based on unstructured text that is extracted or accessed from the clinical trial databases 201 and 202 in accordance with some embodiments. The method 100 progresses to step 105, where structured data is generated corresponding to at least one of a plurality of data types for at least some of a plurality of structured data sets based on the output of the natural language processing to the unstructured text. Further description of an example of a natural language processing module is included below with respect to FIG. 9.

[0088] Structured data is generated corresponding to at least one of the plurality of data types, the method 100 provides a user with a graphical user interface (GUI) that receives input form the user regarding an analysis to be performed using the plurality of structured data sets (step 110).

[0089] Returning to step 102, if the method 100 determines that the extracted or accessed clinical trial information does not include unstructured text (No), the method 100 generates data corresponding to at least one of the plurality of data types for at least some of a plurality of structured datasets based on extracted or obtained clinical trial information at step 103.

[0090] In some embodiments, at 106 the method 100 generates one or more variables based on at least some of the data types for some of the structured datasets. For example, the method can combine information from the clinical trials stored in the clinical trial databases 201 and 202 associated with a trial summary, and then generate a variable that can filter through outcomes of all of the clinical trials in order to determine which outcomes are associated with clinical trial data stored in the clinical trial database 201 or the clinical trial database 202. In some embodiments, the generated variables based on the data types are stored in a unified clinical trial database.

[0091] At 110 a graphical user interface (GUI) 205 is provided to a user for an analysis to be performed or that has been performed on one or more of the structured datasets 203. In some embodiments, the GUI can receive information or criteria from a user regarding data to be copied or accessed from the different structured datasets 205. As an example, GUI can receive a request from the user via the GUI 205 to determine a probability of success that a drug will progress through phase three of the clinical trials. In some embodiments, the GUI may receive from the user, information identifying the drug, and / or information regarding a set of parameters to be included in the analysis.

[0092] FIG. 2 is a schematic depicting a logical information flow between exemplary databases, a module, and a computing device to generate and provide a unified clinical trial database and perform analysis on datasets stored in the database (e.g., to simulate the outcome of a clinical trial) in accordance with some embodiments of the present disclosure. Flow 200 illustrates the relationship between the clinical trial data stored in the clinical trial database 201 and in the clinical trial database 202 and the one or more datasets (trial summary dataset 203a, trial arm dataset 203b, baseline dataset 203c, outcome dataset 203d, outcome comparison dataset 203e, and regulatory information dataset 203f) that aregenerated from the clinical trial information obtained from the clinical trial database 201 and the clinical trial database 202 and that are stored in the unified clinical trial database.

[0093] Examples of clinical trial databases include, but are not limited to, the Aggregate Content of Clinical Trials (AACT) database, which is a publicly available relationship database that contains all information (e.g., protocol and result data elements) about every study registered in clinicaltrials.gov, and Informa Pharma Intelligence Trialtrove (Trialtrove) and Biomedtracker, which is a database that provides clinical trial data including sets, clinical research information, and data from FDA clinical trials. In some embodiments, one or more additional clinical trial databases may be employed. In some embodiments, clinical trial databases other than AACT or Trialtrove may be employed. In some embodiments, information regarding clinical trials may also be obtained from one or more publicly accessible sources that are not clinical trial databases (e.g., news articles). In some embodiments, information regarding clinical trials may also be obtained from one or more sources that are internal to an organization (e.g., internal reports, internal databases, etc.)

[0094] The one or more datasets can be stored in one or more unified standardized clinical trial database(s) 203g, which may be stored on a server 203 or servers in accordance with some embodiments. The clinical trial databases 201 and 202 include clinical trial data that can be used in the creation of in any of the one or more structured datasets or manipulated in response to the application of the NLP module 204 to the clinical trial data stored in the clinical trial databases 201 and 202. For example, some of the clinical trial data stored in either of the clinical trial databases 201 or 202 can be unstructured text data that the NLP module 204 can create meaningful structured data out of using the unstructured text data that can be then added to one or more of the datasets. In some embodiments, the structured data generated by the NLP module 204 can be data that is related to a variable that already exists in one or more of the datasets. For example, the NLP module 204 can generate structured data associated with the outcome dataset 203d and / or the outcome comparison dataset 203f based on unstructured data stored in one of the clinical trial databases 201 or 202. The unstructured data can be data associated with an event type and date of the data of one of the clinical trials, and the NLP module 204 can determine based on the event type and date an outcome or outcome comparison in accordance with some embodiments. The outcome and the outcome comparison associated with the event type and the date when the event type took place can be additional data that is added to the outcome dataset 203d and the outcomecomparison dataset 203f in accordance with some embodiments. An example of the natural language processing in accordance with some embodiments is described in more detail with respect to FIG. 9 below.

[0095] The user can retrieve, interact with, and / or analyze the data in any of the datasets using the GUI 205a on the computing device 205. In some embodiments, the user can create a dataset that is based on data that is requested by a user, that may include additional data that is not included in any one of the individual datasets discussed herein or may not include the data that is included in any one of the individual datasets discussed herein. The user can enter a command via the GUI 205a to generate an analysis of the one or more datasets. In some embodiments, the analysis includes determining a probability with that a drug will successfully complete all stages of the clinical trials.

[0096] FIG. 3 is a chart including different datasets that can be included in the unified clinical trial database 203g for use by a user via the GUI 205a of a computing device 205 (see FIG. 2). In some embodiments, the datasets obtained from the unified clinical trial database are used to simulate the outcome of a clinical trial. The trial summary dataset 303 describes general study attributes. The attributes can include a clinical trial identifier (e.g., NCT ID), a name of the trial or study (e.g., Trial title), a phase that the trial or study is in (e.g., stage), a number of arms included in the trial or study, the planned or actual enrollment information provided in a protocol section of the trial or study record (e.g., timing variables), a design of the trial or study, and subpopulation information that can be curated from patient segment information in the trial or study record. The subpopulation attribute can be a variable that is generated based on a combination of information associated with the patient.

[0097] The server 203 creates the trial summary dataset 303 by first unifying or standardizing, certain clinical trial data from one or more clinical trial databases (e.g., the clinical trial database 201 and the clinical trial database 202) as explained below in reference to FIG. 4 in accordance with some embodiments.

[0098] The trial summary 303 can also include primary investigational drug data that is being used in the clinical trials in addition to data about other drugs that are being used in the clinical trials that are used in combination with the primary investigational drug or are being used as comparators to the primary investigational drug in accordance with some embodiments, which are also referred to as other involved drugs herein. Class data and subclass data are coded into variables for both the primary investigational drug (e.g.,drugclass_primary, drugsubclass_primary) and other involved drug(s) (e.g., either combination component(s) or comparator(s)) (e.g., drugclass other, drugsubclass other) in accordance with some embodiments The server can generate one or more of the variables included in the trial summary dataset 303 as shown in FIG. 3 from information obtained from the clinical trial databases in accordance with some embodiments. Each of these variables in the trial summary dataset are discussed in detail below in the description of an example embodiment in FIG. 4.

[0099] The trial arm dataset 306 can be generated based on the AACT data stored in the clinical trial database 201 in accordance with some embodiments. The trial arm dataset 306 can be used to curate all treatment arms and treatment subpopulations related data in accordance with some embodiments. It contains treatment arms per study focusing on one or more drug components.

[0100] The baseline dataset 304 can also include clinical trial patient segment variables such as a stage of the illness or disease, type of the illness or disease, for example a type of cancer, a line of therapy, or a gene expression. The baseline dataset 304 can also include variables that are associated with the treatment provided in each arm of the clinical trial. In some embodiments, the variables can include data about a combination of drugs that are used to treat the illness or disease, a classification of the combination of the drugs that are used to treat the illness or disease, or a sub-classification of the combination of the drugs that are used to treat the illness or disease. The variables employed in the combined standardized clinical trial database for the structured datasets are specified by datatypes. Accordingly, information related to the treatment in each arm of the clinical trial can be used to curate the treatment arms.

[0101] In some embodiments, additional subpopulation and dosage data for each arm of the study can be curated and added as supplemental data that further enriches the data set. In some embodiments, the coding of drugs can be based on the world health organization’s drug dictionary (WHODD) standard coding. In some embodiments, coding can be generated for the drugs to better classify the drugs if an accurate coding for a drug is not included in the WHODD. The server can generate one or more of the variables included in the trial arm dataset 306 as shown in FIG. 3 from information obtained from the clinical trial databases in accordance with some embodiments. Each of these variables in the trial arm dataset 306are discussed in detail below with respect to the description of an example embodiment in FIG. 5

[0102] The baseline dataset 304 can be created based at least in part on information obtained from a clinical trial database (e.g., the AACT data). The baseline dataset 304 can include demographic data and baseline measures and data per clinical trial arm or comparison group for the entire population of participants in the clinical trials in some embodiments. Some of the information included in the baseline data can include a description of each baseline or demographic characteristic measured in the clinical trials. For example, baseline data can include Age, Sex / Gender, Race, Ethnicity (if collected under the protocol), and any other measurement(s) that are assessed at baseline and used in the analysis of the primary outcome measurement(s). The server can generate one or more of the variables included in the baseline dataset 304 as shown in FIG. 3 from information obtained from the clinical trial databases in accordance with some embodiments. Each of these variables in the baseline are discussed in detail below in the description of an example embodiment in FIG. 6.

[0103] In order to unify baseline variables reported across different clinical trials, baseline category (bl eat), baseline sub category (bl subcat), baseline sub category level (bl subcat level), and baseline subcategory sub level (bl subcat sublevel) variables can be derived from information obtained from the clinical trial data in the clinical trial databases 201 and 202 in order to code any baseline variables using a hierarchical structure. In some embodiments, the baseline data may be obtained from only a single clinical trial database (e.g., from only AACT).

[0104] The server 230 creates the outcome dataset 305 by first unifying or standardizing certain data associated with the outcomes of the clinical trials that are reported for each arm of the clinical trials that are obtained from the clinical trial database 201 and the clinical trial database 202 as explained below in reference to FIG. 7. Some data associated with the outcomes of the clinical trials can include efficacy data such as an overall response rate (ORR), progression free survival (PFS), or overall survival (OS) endpoint variables of patients observed during the clinical trials in accordance with some embodiments.

[0105] The ORR, PFS, or OS variables are exemplary efficacy endpoints that are used for oncology related studies. For other disease areas, variables corresponding to different efficacy endpoints may be employed. The relevant endpoints can be extracted, and can be based on the information obtained from one or more clinical trial databases (e.g., AACTclinical trial data or Biomedtracker clinical trial data). Other data that can be included in the outcome 305 dataset can include a dictionary that is used to define the outcomes of the clinical trials. The outcome 305 dataset can also include trial arm patient segment information, which can be based at least in part on the information from the clinical trial databases (e.g., AACT clinical trial data and Trialtrove clinical trial data) regarding arm variables in accordance with some embodiments. In some embodiments, the variables can include data about drugs or a combination of drugs that are used to treat the illness or disease, a classification of the combination of the drugs that are used to treat the illness or disease, or a sub-classification of the combination of the drugs that are used to treat the illness or disease. The server can generate one or more of the variables included in the outcome dataset 305 as shown in FIG. 3 from information obtained from the clinical trial databases in accordance with some embodiments. Each of these variables in the outcome dataset are discussed in detail below in the description of an example embodiment with respect to FIG. 7.

[0106] The server 230 creates the outcome comparison dataset 307 by first unifying or standardizing certain data associated with the outcome comparison results between treatment arms of the clinical trials in information obtained from the clinical trial database 201 and the clinical trial database 202 as explained below in reference to FIG. 8. Some of the data can include variables associated with one or more of a parameter value type (e.g., treatment difference, treatment ratio, odds ratio etc.), p values associated with the clinical trials, confidence intervals associated with the clinical trials, and statistical testing methods used in the clinical trials. The server can generate one or more of the variables included in the outcome comparison dataset 307 as shown in FIG. 3 from information obtained from the clinical trial databases in accordance with some embodiments. Each of these variables in the outcome comparison dataset 307 are discussed in detail below in the description of an example embodiment in FIG. 8.

[0107] As noted above, the server 203 creates the trial summary dataset 303 by first unifying or standardizing clinical trial information obtained from one or more clinical trial databases (e.g., both the clinical trial database 201 and the clinical trial database 202) that is related to the drugs used during the clinical trials. The server 203 can begin the process of standardizing the clinical trial information that is used to summarize the study by extracting data associated with certain variables stored in the clinical trial databases 201 and 202 that capture particular attributes of the study. In the example embodiment depicted in FIG. 4, theserver 203 standardizes, or unifies, information related to variables associated with AACT data 301, Trialtrove data 302a, and BMT data 302b to generate a trial summary dataset (Trial_Summary 402).

[0108] In some embodiments, variables in the trial summary dataset can be a default set of variables based on information that can be accessed or extracted from the clinical trial databases 201 and 202. These variables can be presented to the user via the GUI. In some embodiments, the GUI can receive an input from the user that instructs the server 203 to extract data from the clinical trial databases 201 and 202 associated with other variables that are not included in the list of variables associated with AACT 301 data, Trialtrove 302a data, and BMT 302b data. In other embodiments, the GUI can receive an input from the user that instructs the server 203 to remove the variables from the trial summary that are associated with AACT 301 data, Trialtrove 302a data, and BMT 302b data.

[0109] FIGS. 4 to 8 are described with respect to data from the AACT, Trialtrove, and Biomedtracker (BMT) databases for illustrative purposes. One of ordinary skill in the art in view of the present disclosure will appreciate that embodiments employing other clinical trial databases with other formats of data or types of data also fall within the scope of the present invention.

[0110] FIG. 4 is a logical diagram 400 depicting a process for generating the clinical trial summary 303 dataset in accordance with some embodiments of the present disclosure. The process can begin with the server 203 connecting to the clinical trial databases 201 and 202 that extracts raw summary data associated with the AACT clinical trial data 301. The data associated with the AACT clinical trial data 301 can include variables such as the NCT ID associated with the clinical trials, the number of arms (num of arms) associated with the clinical trials, year the clinical trials start (start year), the year the clinical trials are completed (completion_year), the primary year that the clinical trials are completed (primary completion year), region of a country, state, or province in where the clinical trials are being held (region), the name of the investigator(s) who are conducting the clinical trials (investigator), the title of the clinical trials (trial title), the overall status of the clinical trials (overall status).

[0111] The data associated with the AACT clinical trial data 301 can also include a name of the sponsor(s) (sponsor), and a type of sponsor (sponsor type). The type of sponsor could be an industry sponsor, academic sponsor, or government sponsor. The AACT clinical trialdata 301 can further include the time at which enrolment has been completed (enr comp time). The data associated with the AACT clinical trial data 301 can also include an estimated total number of participants to be enrolled (target number) or the actual total number of participants that are enrolled in the clinical study (enrl num).

[0112] The AACT clinical trial data 301 can also include a result type variable that can be used to filter through a series of resulting event types associated with the clinical trials. The event types can include data associated with the outcome of the clinical trials, data associated with a reported event of the clinical trials, and data associated with a participant flow of the clinical trials. The data associated with reported event can include summary information about reported adverse events (any untoward or unfavourable medical occurrence to participants, including abnormal physical exams, laboratory findings, symptoms, or diseases), including serious adverse events, other adverse events, and mortality. The data associated with the participant flow of the clinical trials can include recruitment data that is relevant to the recruitment process & pre-assignment details (i.e., significant events in the study that occur after participant enrolment, but prior to assignment of participants). The data associated with the participant flow includes data about participant flow that applies to all milestones of the clinical trials.

[0113] The server 203 can set the result type variable to outcome so that that the outcome data associated with the clinical trials is extracted from the clinical trial databases 201 and 202 associated with the NCT ID.

[0114] The server 203 can also extract raw data associated with the Pharma Intelligence clinical trial data from the clinical trial databases 201 or 202. The raw data can include Trialtrove data 302a as well as BMT data 302b. The raw data can include variables associated with an identifier (ID) associated with the clinical trials (trialld), a title associated with the clinical trials (trialTitle), and a status of the clinical trials (trialStatus). The raw data can also include a start date of the clinical trials (trial StartDate), a design associated with the clinical trials (trial StudyDesign), and a date of last modification of the clinical trial (tri alLastModifi edDate) .

[0115] The raw data can further include data associated with the outcome of the clinical trials (trialOutcomeDetails), data associated with the results of the clinical trials (trialResults), and any notes about the clinical trials (trialNotes).

[0116] The raw data can also include a primary endpoint for the clinical trials (trialPrimaryEndpoint), as well as any other end points associated with the clinical trials (trialOtherEndpoint). In addition, the clinical trials data can include a list of the sponsors of the clinical trials (trial Sponsors), and certain therapeutic areas (trialTherapeuticAreas) (e.g., oncology). In some embodiments, the clinical trials data can include a first set of primary drugs that were tested during the clinical trials (trialPrimaryDrugsTested). In some embodiments, the clinical trials data can also include one or more of a second set of primary drugs that were tested during the clinical trials (bmtPrimaryDrugsTested), a first set of alternative drugs, different from the first set of primary drugs, that were tested during the clinical trials (trial OtherDrugsTested), and a second set of alternative drugs, different from the second set of primary drugs that were tested during the clinical trials (bmtOtherDrugsTested).

[0117] The Pharma Intelligence clinical trial data can also include a unique identifier that is used to identify a specific drug (Drug ID) included in the Pharma Intelligence clinical trial data, as well as any brand name (brandName) affiliated with the DrugID, and indicationGroup (drug indication + pivotal study (Yes / No) e.g.: Breastcancer_Pivotal:Yes).

[0118] The server 203 can generate disease subtype data (diseaseSubtype) (e.g., data associated with melanoma, non-small cell lung cancers such as squamous cell carcinoma and adenocarcinoma, etc.) based at least in part on the data associated with the design associated with the clinical trials (TrialStudyDesign) and the patient segments data (PatientSegments) associated with the clinical trials. A line of therapy for treating the diseases corresponding to the different disease subtypes identified in the disease subtype data. In addition to determining the disease subtype data and the data associated with the design associated with the clinical trials, the server 203 can also generate data associated with a gene expression (geneExpression) associated with the diseases corresponding to the different disease subtypes identified in the disease subtype data.

[0119] After the server 203 has extracted the AACT clinical trial data and the Pharma Intelligence clinical trial data from the clinical trial databases 201 and 202, the server 203 can apply a text mining keyword matching algorithm to extract data associated with the cancer subtypes, gene expression, stages, line of therapy variables based on the patientSegment variable. For example, if a column of the patientSegment includes information related to different subtypes of a disease, for example, "Adenocarcinoma","Squamous Cell”, “Large Cell", the different subtypes of the diseases, the server 203 can group the different subtypes of the disease in a single value, which can be expressed as Adenocarcinoma | Squamous Cell| Large Cell. Similarly, if a column of the patientSegment includes information related to different stages of a disease, for example, "Stage I" and "Stage II” of a cancer, the server 203 can group the different stages of the disease in a single value which can be expressed as Stage I| Stage II. The server 203 can also apply the text mining keyword matching algorithm gene expression and linear of therapy variables to generate a single value for both gene expression variable and the line of therapy variable. In response to the application of the text mining keyword matching algorithm to the combined AACT clinical trial data and the Pharma Intelligence clinical trial data, the server 203 can generate an indicator that identifies the data source from which the data associated with the cancer subtypes, gene expression, stages, line of therapy variables was extracted. This can be done to keep track of the source(s) of the types of data associated with the patientSegment.

[0120] Because drug information is not unified across the AACT clinical trial data and the Pharma Intelligence clinical trial data, due to the complexity of drug naming and mechanism of action classification used by different entities conducting the clinical trials, a drug dictionary is created as a standard reference to code drug name and classify mechanisms of action in accordance with some embodiments.

[0121] The server 203 generates a series of variables that can be used to uniquely describe a drug that is being administered in the clinical trials, based at least in part on the on the trialPrimaryDrugsTested variable, the trialOtherDrugsTested variable, the bmtprimarydrugstested variable, and the bmtotherdrugtested variable.

[0122] The server 203 can unnest the trialPrimaryDrugsTested and extract the drugld, drugPrimaryName, drugName, and mechanismOfAction , mechanismSynonyms, directMechanism variables, etc. associated with the primary drugs that were administered during the clinical trials corresponding to the Trialtrove clinical trial data. The server 203 can also unnest the trial SecondaryDrugsTested and extract variables associated with the other drugs in the Trialtrove data. The server 203 can unnest the bmtprimarydrugstested variable, and extract a drug ID, brandname and Target variables, etc. associated with the primary drugs that were administered during the clinical trials corresponding to the BMT clinical trial data. The server 203 can also unnest the bmtotherdrugtested variable and extracta drug ID associated with the other drugs that were administered during the clinical trials also corresponding to the BMT clinical trial data.

[0123] The server 203 can merge all 4 unnested datasetd by drugid. The result of the full join of the Trialtrove dataset and the BMT dataset is a complete listing of all of the drugs designed to address the disease(s) (e.g. a variable All .Names), that are included in the first subset of the Pharma Intelligence clinical trial data as well as the second subset of the Pharma Intelligence clinical trial data.

[0124] The Target, directMechanism, mechanism Synonyms variable etc. can be used to help classify the drugs according to a drug dictionary like the World Health Organization (WHO) drug dictionary.

[0125] After the complete list of drugs has been created, and the predetermined list of variables has been selected by the server 203, the list of drugs needs to be coded according to the drug dictionary. The drug dictionary can be stored locally on server 203 in accordance with some embodiments. The drug dictionary includes data associated with the Trialtrove clinical trials data as well as data associated with the BMT clinical trial data. The drug coding is then done as drug class and drug sub class according to either WHODD or specified disease specific rules.

[0126] Each drug can be classified into certain classes and subclasses using the established drug dictionary that have similar or identical names in accordance with some embodiments. The server 203 can extract trial arm data associated with each drug that is being studied in each clinical trial and generate the combination drug used in each trial if more than one drug was used in the arm. With merged drug dictionary, each treatment drug and drug combination are coded and classified into drug combo class and drug combo subclass. Logical diagram 500 in FIG. 5 illustrates the process by which a clinical trial arm dataset is created by the server 203 in order to add a classification and sub-classification to the drugs being used as a treatment in the trial arm.

[0127] The server 203 can begin the process by loading trial arm data from clinical trial databases 201 that includes information about different design groups, interventions, terms, phrases, or names that are synonymous with a particular intervention, and interventions applied to the different design groups in accordance with some embodiments. The server 203 can extract design groups data from the clinical trial databases 201 that is associatedwith the trial arm. The data can include information about a protocol-specified group, subgroup, or cohort of participants in a clinical trial assigned to receive specific intervention(s) or observations according to a protocol that is followed for the administration of certain drugs to participants of the specified group, subgroup, or cohort or participants.

[0128] The server 203 can extract intervention data that is associated with the trial arm from the clinical trial databases 201. The data can include particular medical interventions or exposures, including, but not limited to drugs, medical devices, procedures, vaccines, and other products of interest in the clinical trials, or associated with the trial arm or groups in the trial arm. The server 203 can extract terms, phrases, or names that are synonymous with the interventions applied to the different design groups, from the clinical trial databases 201. Each term, phrase, or name is linked to one of the interventions associated with the clinical trials. For example, the server 203 can extract three different names or phrases that are each related to a particular intervention that is associated with the trial arm.

[0129] The server 203 can extract design group specific interventions data from the clinical trial databases 201 that is associated with particular interventions that are applied to specific groups in the clinical trial in accordance with some embodiments. The design group specific interventions data serves as a cross-reference for groups and the corresponding interventions applied to those groups. For example, if the clinical trials have multiple groups and multiple interventions, the design group specific interventions data can specify which interventions are associated with which groups in accordance with some embodiments.

[0130] The server 203 can also be programmed to extract specific variables associated with the interventions data. For example, the server 203 can identify and extract data associated with the NCT ID of the clinical trial, data that describes the type of intervention that is being applied in the trial arm, which can include a particular procedure along with drugs, a biological intervention, radiation therapy, etc. Additional interventions data that the server 203 can extract includes a name of a drug or drugs used during the clinical trial as well as a description of the drug or drugs.

[0131] The server 203 can also be programmed to extract design group specific variables including the NCT ID, the title of the trial arm, and a description of the trial arm. The description of the trial arm can include information that identifies the role of the intervention that participants receive and / or can include the different types of trial arms including, but notlimited to, experimental arms, active comparator arms, placebo comparator arms, sham comparator arms, and no intervention arms.

[0132] The server 203 then combines the data associated with the information corresponding to the different design groups, interventions, terms, phrases, and / or names that are used to refer to a particular intervention and interventions applied to the different design groups, thereby creating a table that includes all of the extracted variables mentioned above in accordance with some embodiments. The server 203 can combine the data by merging the data associated with the information corresponding to the different design groups, interventions, terms, phrases, or names that are used to refer to a particular intervention, and interventions applied to the different design groups using common key variables that can include either the NCT ID, identifier associated with the intervention, the identifier associated with the intervention, or identifier associated with the design group. The resulting table (drug table) is then merged with the drug dictionary, which results in the drugs that have been administered in the trial arm being assigned a drug classification and subclassification, as explained below.

[0133] The server 203 can select a variable associated with the NCT ID, name associated with the drugs administered in the trial arm, the identifier associated with each design group, the description of the drug, a title associated with trial arm, the description of the trial arm, and the type of group included in the trial arm from the drug table, and can group the data associated with the table resulting from the merger of the drug table and the drug dictionary according to the selected variables in accordance with some embodiments. The type of group included in the trial arm can be either experimental or a control. As seen in FIG. 5, AACT 301 can include the variables NCT ID, the type of group (group type), the associated with each design group (design group ID), name, the title associated with trial arm (arm title), the type of intervention (intervention type), the drug associated with the intervention type (drug intervention), as well as the description of the drug (description) in accordance with some embodiments.

[0134] The drug dictionary 401 can contain a list of all of the different names associated with the drugs (All Names) that are administered in the trial arm. The drug dictionary 401 can also include classification information (drugclass) associated with each drug included in the drug dictionary, sub-classification (drugsubclass) information associated with each drug included in the drug dictionary.

[0135] The server 203 can then tokenize the name variable in the drug table and can merge the drug table and the drug dictionary based on the tokenized name variable and the recast name variable in the drug dictionary in accordance with some embodiments. The drugs in the drug table are then coded into drug class and drug subclass.

[0136] After the server 203 merges the modified drug table and the modified drug dictionary, the data in the table resulting from the merger of the modified drug table and the modified drug dictionary, is grouped together by variables associated with the NCT ID, the name associated with the drugs administered in the trial arm, the identifier associated with each design group, description of the drug, a title associated with trial arm, the description of the trial arm, and the type of different trial arms, from the drug table.

[0137] After the server 203 has grouped the data in the resulting table in accordance with the aforementioned variables, the server 203 can generate a variable associated with the most commonly used drug name for the drugs in the group in accordance with some embodiments. This variable can be expressed as drugname combo as shown in Trial Arm 501 by combining each drug component of the Trial Arm. This drugname combo variable can be coded as drugclass combo and drugsubclass combo as shown in Trial Arm 501. Furthermore, the Trial Arm 501 can include the NCT ID (NCT ID), trial arm title (Ta arm title), and group type (group type).

[0138] Each clinical trial can have corresponding baseline data or outcome data in accordance with some embodiments. The baseline data can include data collected at the beginning of the clinical trial for all participants and for each trial arm. This data can include, but is not limited to, demographics, such as age, sex / gender, race and ethnicity, and studyspecific measures (e.g., systolic blood pressure, prior antidepressant treatment, etc.). The server 203 can extract baseline data associated with the clinical trials in accordance with the logical diagram 600 in FIG. 6.

[0139] The server 203 can begin the process of extracting the baseline data associated with the clinical trials by first determining whether baseline data exists for a clinical trial. For some clinical trials, baseline data may not exist, e.g., because the clinical trial has been abandoned or has not started yet. The server 203 can determine whether any baseline data is present in the clinical trial database 201 by attempting to access baseline data stored in the clinical trial database 201 in accordance with some embodiments.

[0140] Where the server 203 determines that baseline data does exist for the clinical trials, the server 203 can extract baseline data from clinical trials database 201. The baseline data associated with the clinical trials can include results data associated with one or more groups of participants. For example, the results data associated with the one or more groups can include a consolidated or aggregate list of group titles and descriptions that can be used for reporting summary results information in accordance with some embodiments.

[0141] The results data associated with the one or more groups of participants can include the NCT ID, a result type variable that can indicate the result of the clinical trials. For example, the value of the result type variable can be “Baseline”, “Outcome”, “Reported Event”, or “Participant Flow” in accordance with some embodiments. In some embodiments, the server 203 can be programmed to create a filter that will only extract certain result type data, e.g., Baseline result type data, when extracting data associated with the result type variable from the clinical trial database 201. The results data associated with the one or more groups of participants can also include a title associated with the baseline of the trial arm that the one or more groups of participants are included in, as well as a description of the baseline of the trial arm in accordance with some embodiments. The description of the baseline of the trial arm can include data collected at the beginning of the clinical trial for all participants and for each trial arm or comparison group. These data can include demographics, such as age, sex / gender, race and ethnicity, and study-specific measures (e.g., systolic blood pressure, prior antidepressant treatment, etc.).

[0142] As mentioned above, the baseline data can further include a set of baseline measurements which can include the NCT ID, a title associated with the baseline of the trial arm, a description of the baseline of the trial arm, a classification, a set of units, parameter type, a value associated with the parameter type, a number associated with the parameter value, a dispersion type, a dispersion value associated with the particular type of dispersion, a lower bound, or limit, on the dispersion, or an upper bound, or limit, on the dispersion in accordance with some embodiments.

[0143] As noted above, the baseline data can also include a set of baseline count information in accordance with some embodiments. In some embodiments, the baseline may be generated from data obtained from a single clinicial trial database (e.g., AACT). In some embodiments, baseline measurement and baseline count are used to generate the baselinedata. The corresponding baseline data is represented in the variable list AACT 301 in FIG.6

[0144] In some embodiments, the server 203 can merge the variable list AACT 301 with the trial summary 402 in order to create a relationship between the baseline data and disease subtype categorization, as well as the stage of the disease in the participants, the line of therapy being used to treat the participants, and an expression of the gene associated with the disease of the participants in accordance with some embodiments. For example, the disease could be a type of cancer and the stage could include the stage of the type of cancer of all of the participants, the line of therapy, could include the particular order in which different therapies are given to the participants as the diseases progresses through different stages, and the expression of the gene can include the gene or genes that are expressed or not expressed in association with the disease. After the sever 203 merges the variable list AACT 301 with the trial summary 402, the server can further merge the trial arm 501 into the merged variable list AACT 301 and the trial summary 402 using the NCT ID in accordance with some embodiments. The server 203 merges the trial arm 501 with the merged variable list AACT 301 and the trial summary 402 in order to determine if there are any similarities between the title associated with the trial arm 501 and the title associated with the baseline of the trial arm 501. The server 203 can compare the titles by tokenizing the title associated with the trial arm 501 and baseline of the trial arm 501 and then comparing the number of tokenized words in the titles to determine if the two titles are an exact match, closely match, or resemble one another. The sever 203 compares the number of tokenized words included in the baseline arm title to the number of tokenized words included in the trial arm 501 title that are the same and determines which associated with the trial arm 501 has the greatest number of tokenized words that are the same as the tokenized words in the baseline trial arm title in accordance with some embodiments. The server 203 can then add information from the trial arm, with the greatest number of tokenized words in common with the title of the baseline trial arm to the baseline dataset. For example, in some embodiments, patient segmentation information available in summary data is then added to the baseline dataset for future patient subgroup identification.

[0145] The server 203 can also determine whether the title of the trial arm includes phrases such as “all participants” or “total” which indicate whether all participants were included in the trial arm in accordance with some embodiments. A determination that the title includesthese phrases or phrases that are synonymous with these phrases indicates whether the count of the baseline is inclusive of all of the participants or not. If the title does include phrases indicating that all of the participants are included in the trial arm, then the server 203 can create a flag indicating that the trial arm is inclusive of all of the participants. If the title does not include phrases indicating that all of the participants are included in the trial arm, then the server 203 can set the flag to a value that indicates that the trial arm is not inclusive of all of the participants in accordance with some embodiments. The flag can be called a total population flag, and can be represented by (tot_pop_flg) in baseline 602 dataset.

[0146] In order to unify the baseline variable naming across different sources, a hierarchical structure is used to code them uniformly in accordance with some embodiments. Three variables can be exported from the AACT table, which are bl variable, bi description and classification in order to derive bl catl, bl_cat2, bi level and bl class std. This dataset can be named as baseline_code_dictionary_v4. Further, the exported variables baseline catl to bl eat, bl_cat2 to bl subcat, bi level can be renamed bl subcat level, bl class std to bl subcat sublevel .

[0147] After the server 203 has completed the various merges described above, the server 203 can generate the baseline 602 dataset as shown in FIG. 6. The server can receive a request from the computing device 205, via the GUI 205a, for data included in the baseline 602 dataset that is stored on the server 203, and the server 203 can send the requested data to the computing device 205.

[0148] When the server 203 sets the value of the result type variable to “Outcome” the server 203 can be programmed to only extract Outcome result type data when extracting data associated with the result type variable from the clinical trial database 201 consistent with the logical diagram 700 in FIG. 7 in accordance with some embodiments. Outcome data associated with the clinical trials can include results data associated with one or more groups of participants. For example, the results data associated with the one or more groups can include a consolidated or aggregate list of group titles and descriptions that can be used for reporting summary results information in accordance with some embodiments. The summary results information can include a summary of results associated with the clinical trials. The Outcome result type data can include the NCT ID associated with the clinical trials, an identifier (result group id) associated with the results of the group of the one or more participants being observed in the clinical trials, a title (oc arm title) associated withthe trial arm corresponding to the outcome data, and a description (oc arm description) of the trial arm corresponding to the outcome data.

[0149] The outcome data can further include a set of outcome measurements, which can include summary data associated with primary and secondary outcome measures for each group included in the clinical trials in accordance with some embodiments. The summary data can include parameter estimates (e.g., overall response rate, median progression survival, etc. for some embodiments) and measures of dispersion or precision. The outcome data can further include outcome count information, which can include a sample size or count included in an analysis of each outcome for each group in the clinical trials. The count or sample size can be representative of the number of participants, but can also represent other units of measure, such as eyes’ 'lesions', or some other feature associated with bodies of the participants. The outcome data can further include an outcome population, which is the patient population related to the outcome measures. For example, in some embodiments this would include patients with high PD1 expression. The aforementioned outcome data can be extracted by the server 203 from the clinical trial database 201.

[0150] The data associated with the set of outcome measurements can be expressed via a plurality of the following variables. The set of outcome measurements can be characterized by a title (oc title), a description of the outcome of the clinical trials (oc description), and the units used during the clinical trials to capture (e.g. in some embodiments this would include median progression free survival in weeks, months or years). The set of outcome measurements can be further characterized by a parameter type, a value associated with the parameter type, a number associated with the parameter value, a dispersion type, a dispersion value associated with the particular type of dispersion, a lower bound, or limit, on the dispersion, or a upper bound, or limit, on the dispersion in accordance with some embodiments,

[0151] After the server 203 has extracted the different outcome data from the clinical trial database 201, it can merge the different outcome data together using the NCT ID, an identifier associated with the results of the group of the one or more participants being observed in the clinical trials, and / or an identifier associated with the outcome, in accordance with some embodiments.

[0152] The server 203 can extract additional outcome data from clinical trial database 202, which can include information associated with one or more drug efficacy endpoint measurescorresponding to the clinical trials. The event data table, expressed as the variable eventDataTable, can include information about the description of the trial arm or the treatment administered during the trial arm to a group of participants in accordance with some embodiments. This information can be expressed as the variable (oc arm title). The reason why the information can be expressed as the variable oc arm title is that oftentimes the description of the trial arm is included in the title of the trial arm. The event data table can also include information about the number of participants to which that the treatment was administered. This information can be expressed as the variable oc sample size, which is also known as the count of the number of participants. The results data table, expressed as the variable dataTableResults, can include information associated with a description associated with efficacy endpoints for the trial arm. The description can be expressed as the variable oc title.

[0153] The server 203 can combine the data that is extracted from clinical trial databases201 and 202 by simply stacking the data extracted from the clinical trial databases 201 and202 on top of one another. The resulting dataset can be expressed as the variable result sub.

[0154] The eventType variable can include different types of results associated with the clinical trials including Top Line Results that indicate the statistical significant of the trial results regardless of the stage or phase of the trial, Final Results, Published Results, and Updated Results. The Final Results can be results associated with the completion of a clinical trial. The Published Results can include results associated with the completion of a clinical trial that have also been published. The Updated Results can be results that may or may not have been published, but include new results data that was not present at a time prior to the creation of the Updated Results. The server 203 can extract one or more drug efficacy endpoints from any of the aforementioned results. For example, the server 203 can extract overall response rate (ORR), progression free survival (PFS), and / or overall survival (OS) efficacy endpoints, from one or more of the aforementioned results.

[0155] The server 203 can determine which clinical trials include the efficacy endpoints by determining whether the outcome title variable (oc title) in result sub for each clinical trial included in result sub contains any keywords associated with ORR, PFS, or OS as an example. The server 203 can compare the words in each title corresponding to a clinical trial contained in the outcome variable to a key word, e.g the phrase “overall survival” and other relevant key works to identify and unify an outcome. Then the server 203 can determinethat the clinical trial does include, for example overall survival data associated with that clinical trial in some embodiments. The similar algorithms apply to other trial outcomes, e.g. overall response rate, partial response, duration of response, etc.

[0156] After performing the keyword search described above, the server 203 can code the efficacy endpoints across the different clinical trials in accordance with some embodiments. The coding of the efficacy endpoints corresponds to a classification and sub-classification. The classification can be expressed as an outcome class variable (oc class) and an outcome sub-classification variable (oc subclass). The server 203 generates an outcome dictionary that includes both the outcome class variable and the outcome subclass variable.

[0157] The outcome dictionary can be generated by filtering out the data associated with the outcome time variable oc title that is extracted from the clinical trial databases 201 and 202 and filtering out data associated with an outcome classification variable (oc classification) that is extracted from the clinical trial database 201 in accordance with some embodiments. Outcome classes and outcome subclasses are defined by different disease area needs and analysis needs, and can be updated when a new endpoint of interest needs to be added. In addition to comparing the keywords associated with efficacy endpoints listed above, to the words in the outcome title oc title, the server 203 can also compare the same keywords to the words that are included in the outcome description variable oc description. Each of the efficacy endpoints corresponds to a classification in the outcome dictionary and the class of the outcome dictionary can be expressed as oc class. The outcome dictionary also includes a corresponding outcome sub-classification and the efficacy endpoints correspond to the outcome sub-classification as well.

[0158] In addition to the oc title variable, the outcome dictionary can include the outcome description variable oc description, and the outcome classification variable (oc classification). After the outcome dictionary is generated by the server 203, the server 203 can merge the outcome dataset variable result sub with the outcome dictionary on the oc title variable, oc description variable, and the oc classification variable. In FIG. 7 the merge can be illustrated as the joining of the AACT 301 data and the BMT 302b data with outcome variable dic 702 in order to create, or add, the oc class and oc subclass variables to the outcome 701 dataset resulting from the merge and classify outcome variables by the dictionary.

[0159] As shown in FIG. 7, the list of variables included in the AACT 301 data can include any or all of the oc classification variable, oc description variable, oc dispersion type variable, oc dispersion value num variable, oc dispersion value num variable, oc dispersion lower limit, oc dispersion upper limit, time frame, and population (which can also be expressed as oc_population) variables in accordance with some embodiments. The AACT 301 data can further include the NCT ID, sample size, oc units, oc title, oc_param_value_num, oc type, oc arm title, and oc arm desc in accordance with some embodiments. The AACT 301 data is data that is extracted from the clinical trial database 201. The BMT 302b data can include any or all of the eventType, eventDate, eventPhase, drugPhase, liklihoodofApproval, averageApprocal, changetoLikelihoodApproval, numberOffivaluablePatients, and Endpoint description, and treatment description (treatment descr in FIG. 7) variables.

[0160] The numberOfEvaluablePatients corresponds to the sample size, or count, of participants in the clinical trials. Because this value is the same as the count, or sample size, in the AACT 301 data, the server 203 can store the numerical value associated with the sample size, count, and numberOfEvaluablePatients variable in the sample size variable. The Endpoint description variable can be related to the oc title variable because as explained above the outcome title can contain words that are related to one or more efficacy endpoints. Accordingly, the server 203 can store the description provided in the Endpoint description variable in the oc title variable. The treatment descr variable can include a description of the trial arm, which is similar to the description of the trial arm associated with the oc arm desc variable, so the server 203 can store the description stored in the treatment descr variable in the oc arm desc variable.

[0161] After the server 203 merges the outcome dictionary with the result sub variable, the resulting dataset can be merged with trial arm 501 using the NCT ID in order to generate the outcome 701 dataset as shown in FIG. 7 in accordance with some embodiments. After merging the trial arm 501 with resulting data set, the server 203 can tokenize the outcome trial arm title variable (oc arm title) as well as the treatment or trial arm title (arm title), and merge outcome trial arm title and trial arm title by similarity in title name.

[0162] In order to extract genesub and cancersub, three functions (e.g., R functions) can be generated such as match gene, match sign, and match cancer from oc arm title, trial title, oc description, oc arm description, oc classification using a program.

[0163] PopulationSub can be extracted using keywords search algorithm on the variables such as from arm title, arm description, outcome title, title, classification and description (by order) in accordance with some embodiments.

[0164] The server 203 can cause the outcome 701 dataset to be displayed via GUI 205a on a display of computing device 205 in response to the user requesting outcome data associated with one or more related clinical trials.

[0165] FIG. 8 is a logical diagram depicting a process 800 for generating a clinical trial outcome comparison dataset in accordance with some embodiments of the present disclosure. The outcome comparison dataset can include statistical analysis results such as Odds Ratio, Hazard ratio, difference in response rates, risk difference, risk ratio, or another estimation parameter between different treatment comparisons. The server 203 can extract comparison results from the clinical trial database 201 and extract comparison results from the clinical trial database 202, and stack them vertically.

[0166] The server 203 can send a request to the clinical trial database 201 to access and extract data associated with statistical outcome analyses performed on primary and secondary clinical trial outcomes. The data can include results for treatment effect estimates, confidence intervals and other measures of dispersion, and p-values in accordance with some embodiments. This data can be referred to as outcome analysis data and can be expressed in variable form as outcome analysis. The server 203 can also send a request to the clinical trial database 201 to access and extract data that identifies the comparison groups that were involved with each statistical outcome analysis. This data can be referred to as outcome analysis groups data and can be expressed in variable form as outcome analysis group. The server 203 can also send a request to the clinical trial database 201 to access and extract the data associated with a consolidated, or aggregate list of group clinical trial titles and descriptions that are used for reporting summary results information. The server 203 can also send a request to the clinical trial database 202 to access and extract data associated with the outcome of the clinical trial data stored in the clinical trial database 202.

[0167] The outcome analysis groups data can include the NCT ID associated with different clinical trials, an identifier associated with the outcome analysis of the different clinical trials, and an identifier associated with the different groups of participants that are involved in the comparison of outcome results of the different groups of participants.

[0168] The outcome analysis data can include the NCT ID, an identifier associated with the outcome analysis that can be expressed as the variable outcome analysis id, and a type of statistical test that is applied to the clinical trials data in order to analyze the clinical trials data. That statistical test can be any or all of superiority test, non-inferiority test, or equivalence test in accordance with some embodiments. The outcome analysis data can also include a description of the statistical test.

[0169] The outcome analysis data can also include dispersion type variables that measures the dispersion of the clinical trial arm data for a particular group relative to another group in accordance with some embodiments. An example could be the Standard Error of the Mean. The dispersion type variable can be expressed as dispersion type. The outcome analysis data can also include a corresponding value for the dispersion type variable in accordance with some embodiments. The outcome analysis data can also include a modifier, symbol, or sign that indicates whether a p-value associated with the statistical test is less than or greater than a certain value in accordance with some embodiments. The modifier can be expressed as a variable of p value modifier. The outcome analysis data can also include the p-value variable associated with the statistical test, which can be expressed as a p value.

[0170] The outcome analysis data can also include information associated with a confidence interval of the statistical test in accordance with some embodiments. For example, the outcome analysis data can include a variable associated with the number of sides to the confidence interval (one or two), and can be expressed as ci n sided. Additional information associated with the confidence interval can include the level associated with the confidence interval, which can be expressed as a percentage and captured by the variable ci_percent. A lower limit, which can be expressed the variable ci lower limit, as well as an upper limit, which can be expressed the variable ci upper limit, associated with the confidence interval are also included in the outcome analysis data in accordance with some embodiments.

[0171] The outcome analysis data can also include a statistical method that was used to analyze the clinical trial arm data in accordance with some embodiments. For example, the method can be a statistical test used to calculate the p-value. It can be expressed as a variable method, and can include any of the following tests: ANCOVA, ANOVA, Chi-Squared, Chi- Squared, Corrected, Cochran-Mantel-Haenszel, Fisher Exact, Kruskal -Wallis, Log Rank, Mantel Haenszel, McNemar, Mixed Models Analysis, Regression, Cox, Regression, Linear,Regression, Logistic, Sign Test, t-Test, 1-Sided, t-Test, 2-Sided, Wilcoxon (Mann-Whitney), or any other statistical test.

[0172] The outcome analysis data can also include information about the procedure used to estimate the effect of intervention, and can be, for example, a statistical test of hypothesis in accordance with some embodiments. The information about the procedure used to estimate the effect of the intervention can be expressed as a variable estimate description. The outcome analysis data can also include a description of the groups and can be expressed as a variable groups description, e.g., intended treatment population, safety population, or high PD1 expression population in some embodiments.

[0173] The server 203 can also select the variables nct id, oc arm title, oc geneSub, oc geneSign, oc cancerSub, populationSub, drug name combo, drug intervention, drugclass combo, drugsubclass combo to be included in outcome data 70. The server 203 can also extract outcome analysis related data from the clinical trial database 202, NCT ID, Title, result type, group type, arm title, arm description from the clinical trial database 201 in accordance with some embodiments.

[0174] For BMT outcome analysis, the server 203 can consider result sub raw data and filter result sub raw data for variables result_type= “BMT outcome” and group_type= “difference”. In some embodiments, a BMT and AACT comparison dictionary is generated separately, using the outcome dictionary generation approach summarized above. The bmt comparison is joined with the bmt comparison dictionary on outcome title. Then, the bmt comparison dataset is joined with the outcome table on nct id to bring back the trial arm title and population variables. For AACT analysis group.rdata and analysis r data are merged using NCT ID, ctgov group code, result group id, and then merged with result sub on NCT ID to bring the oc arm title. In some embodiments, outcome bmt comparison data is stacked with aact comparison data.

[0175] The server merges outcome comparison dataset with outcome final dataset on oc title and NCT DI to get arm title and arm title2 as a key variable to get drug level information.

[0176] FIGS. 4 to 8 are described above with respect to specific variables names merely for illustrative purposes. One of ordinary skill in the art in view of the present disclosure will appreciate that any other variables names could be employed.

[0177] FIG. 9 is a logical diagram depicting a process of applying natural language processing to unstructured text stored in a database to extract clinical trial outcome and clinical trial outcome comparison data in accordance with some embodiments of the present disclosure. The natural language processing can include one or more computer readable instructions that once executed by the server 203 can cause the server to send a prompt to the clinical trial database or databases 201 and / or 202 requesting clinical trial data associated with outcomes or results corresponding to clinical trials. The requested clinical trial data can include unstructured text explaining the outcomes or results of the clinical trials. When the server 203 executes the computer readable instructions of the natural language processing it can detect sections within the unstructured text that correspond to design and result information associated with the clinical trials. The server 203 can enter credentials, such as a username and password of a user, to a webpage and request raw data or the preconfigured datasets disclosed herein from a database (e.g., Trialtrove 901) hosting the raw data or the preconfigured datasets which is stored in a file 902.

[0178] The natural language processing can extract 906 the requested data from the database 901 by detecting sections within the file corresponding to the requested data. For example, when the server executes instructions associated with the natural language processing, the server can detect certain sections of the raw data such as the results section, or a section associated with the design of the clinical trials. The server can also clean, or standardize the data in association with the format of the data that will be used to generate one or more of the preconfigured datasets discussed herein. The natural language processing instructions can further cause the server 203 to generate a file 910 containing the cleaned text, the sections containing data that is relevant to the requested data, and in addition specific sentences that are associated with the requested data. File 910 can be used by the server 202, when executing the natural language processing instructions, to automatically annotate 907 the sentences with phrases and / or labels that are related to the requested data.

[0179] A configuration function 904, which when executed by the server 203 causes the server 203 to use one or more named entity models 905 as it is executing the instructions associated with the natural language processing, to detect specific data, such as the disease, cell type, dose, strength of dose, endpoint, drug name, gene expression data, etc. in the cleaned text.

[0180] For example, the natural language processing can include instructions that cause the server 203 to annotate the unstructured text with labels that identify certain named entities such as any or all of diseases, cell types, a dose of a drug administered during the clinical trial to the participants, the strength of the dose, an efficacy endpoint, a type of drug used, statistics collected throughout the clinical trial, and a number of patients or count of the outcome or results of the clinical trial in some embodiments. The server 203 can annotate 907 the unstructured text by adding labels to sentences, phrases, or words that identify any of the aforementioned entities, or any other relevant information that can be used to identify data that corresponds to each of the datasets stored in server 203.

[0181] After the server 203 has completed auto annotation 907, the server can then perform an entity pattern matching 908 between the entities identified above and the entities recognized in the unstructured text. For example, the natural language processing can be fed the aforementioned list of entities, and the natural language processing can cause the server 203 to perform an entity association in which the server 203 can recognize certain information such as biomarker levels, treatment types, tumor types, treatment doses, treatment arms or trial arms, and groups all associated with the clinical trials within the unstructured text. After the server 203 has completed the pattern matching, the server 203 can generate a database 909 along with a mapping between the recognized information and the sentences or phrases that have been automatically annotated. The extracted information can then be used by the server 203 to create one or more of the datasets stored on server 203. After the database (e.g., the combined clinical database) has been generated, the server 203 can also output a file containing one or more structured datasets in accordance with data requested by the user.

[0182] In some embodiments, information or data is obtained from the clinical trial databases periodically (e.g., daily, weekly, etc.) and new or changed data is processed and integrated into the combined standardized clinical trial database. In some embodiments, the combined standardized clinical trial database supports and enables fast and comprehensive meta-analysis with update-to-date and unbiased data as compared with conventional systems for analysis of clinical trial data. In some embodiments, the information curation and integration can be adjusted and guided by therapeutic area knowledge and drug development experiences. In some embodiments, the combined standardized clinical trial database can better support robust decisions of clinical development and planning.

[0183] FIG. 10 schematically depicts an exemplary computing device or system 1000 (e.g., the computing device 205 in Fig. 2). The system 1000 may be used to perform the operations described with regard to one or more implementations according to any of the methods of the present disclosure. For example, the system 1000 may be included in any or all of the server components, or other computing device(s), discussed herein. The system 1000 may include one or more processors 1010, one or more memories 1020, one or more storage devices 1030, and one or more input / output (I / O) devices 1040. The components 1010, 1020, 1030, and 1040 may be interconnected using a system bus 1050.

[0184] The processor 1010 may be configured to execute instructions within the system 1000. The processor 1010 may include a single-threaded processor or a multithreaded processor. In some embodiments, the one or more processors 1000 may include one or more graphical processing units. The processor 1010 may be configured to execute or otherwise process instructions stored in one or both of the memory 1020 or the storage device 1030. Execution of the instruction(s) may cause graphical information to be displayed or otherwise presented via a user interface on the VO device 1040. In some embodiments, the one or more processors may include one or more graphical processing units.

[0185] The memory 1020 may store information within the system 1000. In some implementations, the memory 1020 is a computer-readable medium. In some implementations, the memory 1020 may include one or more volatile memory units. In some implementations, the memory 1020 may include one or more non-volatile memory units.

[0186] The storage device 1030 may be configured to provide mass storage for the system 1000. In some implementations, the storage device 1030 is a computer-readable medium. The storage device 1030 may include a floppy disk device, a hard disk device, an optical disk device, a tape device, or other type of storage device. The VO device 1040 may provide VO operations for the system 1000. In some implementations, the VO device 1040 may include a keyboard, a pointing device, or other devices for data input. In some implementations, the VO device 1040 may include output devices such as a display unit for displaying graphical user interfaces or other types of user interfaces.

[0187] FIG. 11 is a logical diagram depicting a process 1100 of accessing clinical trial data on a host device from a client device 1101 using an application programming interface (API) 1104 in accordance with some embodiments of the present disclosure. This process may be employed to obtain information and data from clinical trial databases for generation of thecombined standardized clinical trial database. Process 1100 illustrates the process of exchanging credentials between client 1101 such as for example computing device 205, and API 1104 hosted on the server 203. The client 1101 can request an access token from an identity service 1102 that stores access tokens, by executing instructions in accordance with request access token 1111. The client 1101 can send credentials to the identity service 1102, which include a client id, password, grant type, and scope. The identity service 1102 returns an access token in response to executing instructions returns access token 1112 in response to approving the credentials submitted by the client 1101. Identify service 1102 can be hosted by a third party provider’s server. The client 1101 can use access token in API requests 1113 when attempting to requesting data from API 1104. The API 1104 can return the requested data 1114. Process 1100 is executed by computing device 205 in response to a user entering credentials that are used to request data from the server 203.

[0188] FIG. 12 is a table of the status of clinical trials organized according to different tiers in accordance with some embodiments of the present disclosure. The table 1200 illustrates how different clinical trials are organized in accordance with the status of the clinical trial. The table 1200 includes a plurality of tiers including a first tier (tier 1) for clinical trials that have been completed and include final results such as outcome data that is included in outcome 203d. The table 1200 also includes a second tier (tier 2) for clinical trials that have been terminated or are ongoing with no final results. The table 1200 also includes a third tier (tier 3) for clinical trials in which there are no results at all. And the table 1200 can include a fourth tier (tier 4) that can be a catchall for clinical trials in which the status of the clinical trial is not known. The endpoint associated with each of the tiers can be the efficacy or safety of the drugs being administered throughout the clinical trials.

[0189] Because tier 1 includes final results associated with clinical trials, the data associated with the final results can be used to structure the endpoints of the clinical trials. The combination of different tiers can be used by a user to design future clinical trials based on the data recorded across the different tiers. The table 1200 also includes the percentage of all clinical trials that are categorized into one of the different tiers, as well as the percentage of clinical trials that are sponsored by or conducted by a certain number of pharmaceutical companies.

[0190] FIG. 13 is a logical diagram depicting a process of extracting clinical trial data from one or more external databases for generation of a combined standardized clinical trialdatabase, and using clinical trial data from the combined standardized clinical trial database as input to trial simulator in accordance with some embodiments of the present disclosure. Process 1300 can begin by the server 203 extracting or accessing data stored in clinical trial databases 201 and 202, which are represented logically as external information 1215. The data stored in the external information 1215 is data that is collected by third parties and is relevant to one or more clinical trials. The external information 1215 can be standardized, combined or integrated, and stored in a combined standardized clinical trial database 1211 in the form of one or more of the different datasets addressed above (e.g., trial summary 203a, trial arm 203b, baseline 203c, outcome 203d, outcome comparison 203e, and / or regulatory information 203f). The combined standardized clinical trial database may also be referred to herein as a quantitative knowledge database.

[0191] In some embodiments, internal information 1216 (e.g., nonpublic information or information generated within an organization) can be data associated with one or more of the clinical trials that the user who is requesting access to the combined standardized clinical trial database 1211, has access to, but that is not included in the external information 1215. The server 203 can extract or access the internal information 1216 in response to the computing device 205 issuing a request to a server storing the internal information 1216 to provide access to the server 203. Once the external information 1215 and internal information 1216 are combined and integrated, the combined clinical trial database 1211, a study population specification 1201 is applied to the data stored in the combined clinical trial database and stored in a study specific database 1210. The study specific database and control arm historical databases can be project-specific databases curated from the original combined standardized clinical trial database. For example, a chemotherapy dataset in oncology can be further extracted from the combined standardized clinical trial database and curated to focus on trials with chemotherapy treatments. The established chemotherapy dataset can be used in clinical trial design and decision making with the chemotherapy dataset as the control arm. An example of a study population can be metastatic non-small cell lung cancer patient receiving 1stline therapy including chemotherapy. For example, treatment ORR->PFS prediction model 1202 can be investigated using the data stored in the study specific database 1210, which can be stored on server 203 or another server or locally, in order to generate a meta-analy sis model 1213. As one example, a tree-based meta-analysis model 1213 can be generated in accordance with the process 1600 in FIG. 16. The database1210 stores treatments, treatment outcomes and study design and baseline factors specific to the disease and population of interest, which enable the establishment of the prediction model of interest. One or more control arm assumptions, e.g. chemotherapy in the first line non-small lung cancer metastatic patients, can be applied to the data stored in the study specific database 1210 and the resulting data can be stored in control arm historical study database 1212. For example, a control arm assumption can be that chemotherapy will be used in first line non-small lung cancer metastatic patients. The data stored in the control arm historical study database 1212 and the meta-analysis linear model 1213 can be used to simulate the results and probability of success of ongoing clinical trials or a clinical trial that is designed from scratch by the user. The database 1212 can also be originated from 1211, but with focus on the control arm of interest, i.e. chemotherapy in the first line non-small lung cancer metastatic patients.

[0192] In some embodiments, a customized user interface (e.g., a GUI) can be implemented to access data in the combined, standardized clinical trial database and to conduct analysis or simulation. In some embodiments, the GUI is customized based on the therapeutic area of interest and therapeutic needs. For example, an R-shiny based interface for oncology was implemented and is shown in FIG. 14. In some embodiments, the GUI enables non-modelers to explore the database interactively and test different modelling and simulation scenarios by the interface design. FIG. 14 is a screenshot of a clinical trial simulator graphical user interface depicting a regression of progression free survival on overall response rate in accordance with some embodiments of the disclosure. The various trial simulators can be created by user needs, with this example simulated to predict late phase trial outcomes by entering early phase trial outcomes (i.e. proof of concept study (POC)) using a meta-analysis model established after POC study results are available (Post to POC). Other prediction modeling can be conducted before the POC study (Prior to POC). Additional modeling (sensitivity analysis) can also be conducted at each stage to ensure the analysis robustness based on the relevant data in the database. A trial simulation GUI interface can be implemented with the GUI for accessing the data. In some embodiments the GUI can be designed based on the needs or desires of the user.

[0193] FIG. 15 is a screenshot 1500 of a clinical trial simulator graphical user interface depicting a probability that drug treatments used in clinical trials will advance from one development stage to the next development stage or advancing to approval for a total numberof progression free survival events and a probability that drug treatments used in clinical trials will advance from one development stage to the next development stage or advancing to approval for a given size of an event in accordance with some embodiments of the disclosure. The user can use the simulator to predict the next trial success and optimize the next trial design.

[0194] Some embodiments provide estimates of a probability of approval of a drug based, at least in part, on the structured data sets in the unified clinical trial database. Determination of an estimate for probability of approval of a drug may be conducted using a machine learning approach in some embodiments. For example, analysis may be conducted based on methods described in the publication, Machine Learning with Statistical Imputation for Predicting Drug Approvals, Harvard Data Science Review, Issue 1.1, Oct. 4, 2019, page 1- 42, by Lo et al, which is incorporated by reference herein in its entirety.

[0195] The Lo publication describes predictive models for assessing the probability of approval of drug candidates in two scenarios: after phase two testing, and after phase three testing. The Lo publication describes using statistical imputation to address missing data. For example, k-nearest neighbor (kNN) (e.g., 5NN) imputation. Lo describes using machinelearning techniques for generating predictions, including cross-validation for training and a held out testing set for performance evaluation, and using the standard “area under the receiver operating characteristic curve” (AUC) metric to measure model performance. AUC is the estimated probability that a classifier will rank a positive outcome higher than a negative outcome. The Lo publication described employing a random forest (RF) classifier model for predicting approval.

[0196] Some embodiments include methods and applications for analysis based on the structured data sets in the unified clinical trial database. FIG. 16 schematically depicts an overview of a tree-based regression analysis model to predict progression free survival (PFS) associated with treatments for a particular disease based on an overall response rate (ORR) for patients in a clinical trial based in accordance with some embodiments of the disclosure. An additional explanation of an example tree-based regression analysis is provided below. The PFS can be determined based on in-silico trial simulations.

[0197] A unified clinical trial database for cancers was constructed based on clinical trial information obtained from the AACT, trialtrove, and BMT databases. A graphical userinterface was generated for analysis based on the structured data sets in the unified clinical trial database (see GUI in FIG. 14).

[0198] Analyses were conducted based on the structured data sets to predict progression free survival (PFS) based on the proof of concept (POC) object response rate (ORR) for a PD1 checkpoint response inhibitor. Only post 2010 industry sponsored intervention trials were included. The PD1 related late phase clinical trials enrolling late-stage patients were included. The arms were classified into the following treatment classes: PD1 monotherapy, chemo monotherapy, other target therapy monotherapy, PD1 combined with chemotherapy, PD1 combined with other target therapy, chemotherapy combined with other target therapy, and PD1 combined with chemotherapy and other target therapy. Only trials having an arm with paired ORR / PFS were included. Only primary analysis population (PAS) ORR / PFS were included. The analysis included approximately 200 arms with paired ORR / PFS. Important variables were: ORR; Line: 1L / 1L+ and 2L / 2L+; Stage: III / IV and IV; PAS PDL1 status: Positive and Not specified; Cancer type: Melanoma, NSCLC and Other; and Treatment class: PD1 mono, Chemo mono, Other target therapy mono, PD1 + Chemo, PD1 + Other target therapy, Chemo + Other target therapy, PD 1 + Chemo + Other target therapy. An assumption was made that for each therapy development, the phase II and phase III underlying population / ORR are the same.

[0199] FIG. 16 schematically depicts the tree-based model of PFS based on ORR including graphs of Log(ORR) versus Log(PFS) for the splits in accordance with some embodiments of the disclosure. The first split in the tree-based regression analysis is by line where ORR / PFS slope is most heterogeneous across Line compared to Stage, Cancer class and PAS PDL1 status. The second split in the tree-based regression is by treatment class where ORR / PFS slope is most heterogeneous across treatment class (Chemo related and not) compared to Stage, Cancer class and PAS PDL1 status within “IL / 1L+”. The third split in the treebased regression analysis is by stage where ORR / PFS slope is most heterogeneous across stage compared to Cancer class and PAS PDL1 status within “IL / 1L+ and Not chemo related”. The splitting is stopped if no clinically meaningful and significant ORR / slope heterogeneity is found or if the subgroup same size is less than or equal to 5 after the split. The predictive model for each later after the split is Log(PFS) ~ Log(ORR) + U where U are the variables with significant intercept. The final tree-based regression model can becompared with other regression based models to assess and compare the model performance (e.g. mean square errors (MSE) of the model prediction in a training and testing split setting).

[0200] The features described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus may be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device) for execution by a programmable processor; and method steps may be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features may be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that may be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0201] Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer may also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magnetooptical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, application-specific integrated circuits (ASICs).

[0202] To provide for interaction with a user, the features may be implemented on a computer having a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user may provide input to the computer.

[0203] The features may be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system may be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include, e.g., a local area network (LAN), a wide area network (WAN), and the computers and networks forming the Internet.

[0204] The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the described one. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0205] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems.Example: Predict progression free survival and overall survival using objective response rate for PD1 / PDL1 combination therapy development

[0205] In oncology PD1 / PDL1 combination therapy development for solid tumors, objective response rate (ORR) is commonly used clinical endpoint in early phase study while progression free survival (PFS) and overall survival (OS) are widely used in late phase study. With early phase clinical outcome such as ORR, developing predictive models to median PFS (mPFS) and median OS (mOS) could inform late phase trial study design optimization and probability of success (POS) evaluation. In existing literature, there are ORR / mPFS / mOS association and surrogacy investigations with limited number of included clinical trials. In this example, without establishing surrogacy, mPFS and mOS were predicted based on early efficacy in ORR and optimize late phase trial design for PD1 / PDL1 combination therapy development. In order to include adequate eligible clinical trials, a comprehensivequantitative clinical trial landscape database (QLD), which may also be referred to as a unified clinical trial database, was built up by combining information from different sources such as clinicaltrial.gov, publications, company press releases for relevant indications and therapies. A generalizable algorithm was developed to systematically extract structured data and manual curation for scientific accuracy and completeness. Finally, more than 150 late phase clinical trials were identified for ORR / mPFS (ORR / mOS) predictive model development while existing literature included at most 40. A tree-based machine learning regression model has been derived to borrow strength and account for ORR / mPFS (ORR / mOS) relationship heterogeneity simultaneously to ensure that the predictive model is robust and have explicit structure for clinical interpretation. Over 1000 times cross validation, the predictive mean square error of the proposed model is competitive to random forest and extreme gradient boosting, and much better than commonly used additive or interaction regression model by cross validation. An example application of the proposed ORR / mPFS (ORR / mOS) predictive model on late phase trial POS evaluation for PD1 / PDL1 combination therapy was illustrated.1. INTRODUCTION

[0207] Anti-PDl / anti-PDLl therapy, for example Pembrolizumab (FDA, approved in 2014), Nivolumab (FDA, approved in 2014) and Atezolizumab (FDA, approved in 2016), has been developed to treat various cancer such as NSCLC and Melanoma in last decade. Recently, PD1 / PDL1 combination therapy, for example PD1 / PDL1 combined with other target therapy, became increasingly popular due to better clinical efficacy. For the clinical trial of PD1 / PDL1 combination therapy in solid tumors, ORR is a commonly used clinical endpoint for early phase study while PFS / OS are widely used for late phase study. ORR is an attractive early clinical endpoint because it allows short trial duration, the use of smaller patient cohort and even single-arm trial designs (Clarke J.M., Wang X. and Ready N.E. (2015). Surrogate clinical endpoints to predict overall survival in non-small cell lung cancer trials - are we in a new era? Translational Lung Cancer Research, 4, 804 - 808.). The correlation or surrogacy of ORR with PFS / OS remains a central consideration in determining its validity and thus, there are various literatures studying the association of ORR / mPFS / mOS. For example, Zhu et al. (Zhu Andrew X, Lin Yong, Ferry David Raymond, Widau Ryan C and Saha Abhijoy (2022). Meta-analysis of surrogate endpoints for survival in patients with unresectable hepatocellular carcinoma treated with immunecheckpoint inhibitor-based regimens. American Society of Clinical Oncology.) studied ORR / mPFS / mOS association for immune checkpoint inhibitor, Ye et al. (Ye Jiabu, Ji Xiang, Dennis Phillip A, Abdullah Hesham and Mukhopadhyay Pralay (2020). Relationship between progression-free survival, objective response rate, and overall survival in clinical trials ofPD-l / PD-Ll immune checkpoint blockade: a meta-analysis, Clinical Pharmacology and Therapeutics, 108(6), 1274 - 1288), Nie et al. (Nie Runcong, Chen Foping, Yuan Shuqiang, Luo Yingshan, Chen Shi, Chen Yongming, Chen Xiaojiang, Chen Yingbo, Li Yuanfang and Zhou Zhiwei (2019). Evaluation of objective response, disease control and progression-free survival as surrogate endpoints for overall survival in anti -programmed death-1 and anti-programmed death ligand 1 trials, European Journal of Cancer, 106, 1 - 11) and Ito et al. (Ito Kentaro, Miura Satoru, Sakaguchi Tadashi, Murotani Kenta, Horita Nobuyuki, Akamatsu Hiroaki, Uemura Kohei, Morita Satoshi and Yamamoto Nobuyuki (2019). The impact of high PD-L1 expression on the surrogate endpoints and clinical outcomes of anti-PD-l / PD-Ll antibodies in non-small cell lung cancer. Lung Cancer, 128, 113 - 119) studied ORR / mPFS / mOS relationship for PD1 / PDL1 related therapy, Blumenthal et al. (Blumenthal Gideon M, Karuri Stella W, Zhang Hui, Zhang Lijun, Khozin Sean, Kazandjian Dickran, Tang Shenghui, Sridhara Rajeshwari, Keegan Patricia and Pazdur Richard (2015). Overall response rate, progression-free survival, and overall survival with targeted and standard therapies in advanced non-small-cell lung cancer: US Food and Drug Administration trial-level and patient-level analyses. Journal of Clinical Oncology, 33(9), 1008), Goring et al. (Goring Sarah, Varol Nebibe, Waser Nathalie, Popoff Evan, Lozano- Ortega Greta, Lee Adam, Yuan Yong, Eccles Laura, Tran Phuong and Penrod John R. (2022). Correlations between objective response rate and survival -based endpoints in first-line advanced non-small cell lung Cancer: a systematic review and meta-analysis. Lung Cancer, 170, 122 - 132) and Hua et al. (Hua Tiantian, Gao Yuan, Zhang Ruyang, Wei Yongyue and Chen Feng (2022). Validating ORR and PFS as surrogate endpoints in phase II and III clinical trials for NSCLC patients: difference exists in the strength of surrogacy in various trial settings, BMC cancer, 22(1), 1 - 13) investigated ORR / mPFS / mOS relationship for NSCLC. However, all the existing literature focused on ORR / mPFS / mOS association include conflicting results. Some tumor types or therapies showed surrogacy of ORR to PFS or OS, but some investigations failed to make the linkage. One reason could be that only a limited number of clinical studies were included in the investigation and there is relationshipheterogeneity. Therefore, till now, no predictive model was unambiguously developed for all tumor types or all therapies. In this example, without establishing endpoint surrogacy, an adequate number of clinical trials across various tumor types, treatment class, etc. were integrated and a comprehensive prediction model for PD1 / PDL1 combination therapy development was established. Thus, observed efficacy in early phase study can be leveraged to better predict late phase study success or better plan late phase studies.

[0208] In order to include adequate number of clinical trials to capture ORR / mPFS (ORR / mOS) heterogeneity in predictive model development, the comprehensive quantitative landscape data base (QLD) was derived by including all the PD1 / PDL1, NSCLC and Melanoma related clinical trials from clinicaltrial.gov and Informa. As the inventors aimed to focus on establishing a prediction model for PD1 / PDL1 combination therapy development, all the PD1 / PDL1 related clinical trials were included. All the NSCLC and Melanoma related clinical trials were included to enrich the arms with less prevalent treatment class (Chemotherapy, Other target therapy, Chemotherapy + Other target therapy, which is systematically defined in section 2.2) in PD1 / PDL1 related trials to facilitate the predictive model development. NSCLC and Melanoma were selected as two most prevalent solid tumors with more clinical trials than other tumor types. All the PD1 / PDL1, NSCLC, Melanoma related trials available in clinicaltrial.gov and Informa until July 2022, September 2022, and December 2022, respectively, were included to generate QLD. In detail, the inventors first leveraged clinicaltrial.gov with its relevant clinical studies and sponsors uploaded studies efficacy outcomes. In addition, Informa that provides additional data with publications, conference abstract and presentations, and company press releases etc. were incorporated next.

[0209] Next, all the eligible clinical trials from QLD were selected and a tree-based regression predictive model for PD1 / PDL1 combination therapy development was built. An important idea of the tree-based regression machine learning model is forwardly dividing predictor’s space into a number of simple subspaces until there is no ORR / mPFS (ORR / mOS) heterogeneity. Compared to existing simple predictive models such as linear regression models, the proposed method is more flexible to account for ORR / mPFS (ORR / mOS) heterogeneity. Compared to existing machine learning predictive models such as random forest and extreme gradient boosting models, the proposed method has explicit structure for clinical interpretation. The predictive performance of the proposed method wascompared with a variable selected multiple linear regression model with / without interaction, random forest and extreme gradient boosting. An example application of developed treebased machine learning predictive model on late phase trial POS evaluation is illustrated. Also, the primary application in the example was to predict late phase mPFS / mOS based on early ORR to inform late phase trial POS. Late phase studies with both ORR and mPFS / mOS in the same study were selected.2. METHOD2.1 QLD

[0210] An objective of creating the QLD, which can also be referred to as a unified clinical trial database, was to curate historical clinical studies related information into indication or drug mechanism of action (MO A) specific databases. The unified clinical trial database QLD was used to generate insights to prioritize development plans, optimize and de-risk projects and studies, and finally promote critical exploration and timely decision making. A major focus was on efficacy and safety profiling and linking to model probability of success and influential factor identification. Current input data sources were clinicaltrial.gov and Informa. Specifically, in this example, all the PD1 / PDL1, NSCLC and Melanoma related clinical trials from clinicaltrial.gov and Informa with structured information were included to generate QLD. The QLD contained at least 3 types of information including trial summary, outcome, and outcome comparison.

[0211] Trial summary: This dataset included general study level attributes such as study ID, study name, phase, number of arms, study start / completion date, study design, cancer type, line of therapy, cancer stage, primary investigational drug / drug class / drug subclass and other involved drug / drug class / drug subclass (either combination component or comparator).

[0212] Outcome: This dataset contained reported clinical outcomes for each study arm. For each outcome record, the study ID, study name, arm title, drug / drug combo, outcome title and description, outcome value, number of patients was provided. The corresponding drug / drug combo class / subclass and endpoint class (e.g., ORR, PFS, OS etc.) were curated by algorithm combined with manually.

[0213] Outcome comparison: This dataset contained outcome comparison results between treatment arms. For each outcome comparison record, the study ID, study name, outcome comparison p value, confidence interval, statistical testing method, primary and comparativedrug / drug combo were provided. The corresponding primary and comparative drug / drug combo class / subclass, endpoint class (e.g., ORR, PFS, OS etc.) and comparison class (e.g., difference, odds ratio, hazard ratio, etc.) were curated by algorithm combined with manually.

[0214] In this example, information from trial summary and outcome were used to develop the predictive model.2.2 DATA SELECTION AND PREDICTORS

[0215] The following inclusion and exclusion criteria were applied to filter the eligible trial arms from QLD to develop ORR / mPFS (ORR / mOS) predictive model for PD1 / PDL1 combination therapy development. (1) All the clinical trials started after 2010 were included due to more recent and popular MOA. (2) The unknown or early phase (phase I, phase I / II, phase II) clinical trials were excluded due to potential unmature PFS and OS. (3) The trials including unadvanced cancer patients (cancer stage I / II) were excluded. (4) The trial arm without paired ORR / mPFS (ORR / mOS) were excluded.

[0216] For each included trial arm, only the primary analysis population paired ORR / mPFS and ORR / mOS were included in predictive model development. Primary analysis population clinical outcome was selected because it is the most important metric affecting trial POS and study design. In addition, besides ORR, mPFS and mOS, treatment class, indication, treatment line, cancer stage and PD1 / PDL1 status were extracted for each included trial arm for the primary analysis population. Note all information was either at study level or treatment arm level, and no patient level data was available for use. Treatment class was defined into the following seven categories, including Chemotherapy, PD1 / PDL1 therapy, Other target therapy, PD1 / PDL1 + Chemo combination therapy, PD1 / PDL1 + Other target combination therapy, Other target + Chemo combination therapy, PD1 / PDL1 + Other target + Chemo combination therapy, where Other target therapy indicated all the target therapy other than PD1 / PDL1 therapy. All the target therapy other than PD1 / PDL1 therapy was grouped as Other target therapy because the main objective in this example was for PD1 / PDL1 combination therapy development, and therefore, the number of included arms about each target therapy class other than PD1 / PDL1 were relatively small. Thus, it is realistic to combine them and assume they share the same ORR / mPFS (ORR / mOS) relationship. Indication was grouped into three categories NSCLC, Melanoma and Others (Hua et al. 2022; Nie et al. 2019). Treatment line was grouped into three categories IL, 1L+ and Remaining. Cancer stage was grouped into two categories stage IV only and III / IVmixed. PD1 / PDL1 status was grouped into two categories positive and not specified. Remaining biomarker (other than PD1 / PDL1) status on primary analysis population was not extracted for the analysis. It is because their relevant therapies are not enriched and thus the biomarker positive prevalence in the analysis data set was relatively low. In addition, the ORR / mPFS (ORR / mOS) relationship heterogeneity across biomarker (other than PD1 / PDL1) status is not addressed in this example as the primary goal was for PD1 / PDL1 related therapy development. Therefore, it was assumed that the ORR / mPFS (ORR / mOS) relationship was homogeneous across biomarker (other than PD1 / PDL1) status.2.3 TREE-BASED MACHINE LEARNING REGRESSION MODEL

[0217] In this section, tree-based machine learning regression models were developed to predict ORR to mPFS / mOS. Beside ORR, treatment class, indication, treatment line, PD1 / PDL1 status and cancer stage were also included as predictors to better account for ORR / mPFS (ORR / mOS) homogeneity / heterogeneity. Compared to a traditional linear predictive model such as an additive model, the proposed model accounted for the ORR / mPFS (ORR / mOS) heterogeneous relationship. Compared to modem black box machine learning models such as random forest and extreme gradient boosting, the proposed model has clear structure for clinical interpretation. An important idea of proposed model was forwardly segmenting predictor’s space into a number of simple regions until there is no significant ORR / mPFS (ORR / mOS) relationship heterogeneity. The proposed tree-based machine learning regression model was summarized into following steps.

[0218] Step 1 (Group variable classes with homogeneous slope): Denoteas the classes of each variable VbG 0 with decreasing corresponding sample size | Vb| >■■■, >| .Evaluate slope difference between each of (Vb, ... , Vblbto Vbby comparing \ogmPFS(mOS) \og ORR X Vbto \ogmPFS(mOS) \og ORR + Vb. Combine every Vb(J = with Vbif P value > 0.25 or< 5. Repeat above process for the remaining classes until no classes are left. For each variable Vb, the class after grouping is VGb, ... , VGbb), denoted as class grouping variable VGb.

[0219] Step 2 (Split by grouped variable classes in Step 1 given significant heterogeneous slope): Evaluate slope difference by comparing \ogmPFS(mOS)~ og ORR x VGbto \ogmPFS mOS)~ \og ORR + VGbfor each variable ^satisfying Lb> 1. Select Vb* to splitinto branches by VGb* if (1) ANOVA P value < 0.01 (2) ANOVA P value most significant among all candidate variables.

[0220] Step 3:

[0221] Case 1 : If there is a split in Step 2, repeat Step 1 and 2 on remaining variables 0 \ VGb* for each branch of VGb*.

[0222] Case 2 (Potential heterogeneous intercept adjustment): If no split in Step 2, evaluate intercept effect by comparing log mPFS (mOS)~ log ORR + Vbwith \ogmPFS(mOS) \og ORR for each variable VbG 0 satisfying min{|Vb1|,> 5. Select all Vb> with P value < 0.01. The predictive model is ogmPFS(mOS)~ og ORR + J] Vb' on given branch.

[0223] Specifically, in Steps 1 to 3, all the regression models are weighted regression with each trial arm sample size as weight. In the ORR / mPFS (ORR / mOS) predictive model development, 0 contains treatment class, indication, treatment line, PD1 / PDL1 status and cancer stage.3. RESULTS

[0224] For the predictive model of ORR to mPFS, finally 154 trials (321 trial arms) were included. For the predictive model of ORR to mOS, finally 123 trials (249 trial arms) were included. The number of included late phase clinical trials was much larger than existing literature where at most 40 were included (Ye et al., 2020). This demonstrated the comprehensiveness of QLD and ensured generalizability and reproducibility of the developed predictive model. The distribution summary of each predictor appears in Table 1. For a continuous variable such as ORR, the mean and standard deviation were summarized. For a categorical variable such as treatment class, indication, treatment line, PD1 / PDL1 status and cancer stage, the amount and proportion of each category were provided.Table 1. Predictor’s distribution of included trial armsNumber of trial arms Data set 1 (n=321) Data set 2 (n=249)ORR - Mean (SD) 35.2 (21.4) 31.1 (20.7)Treatment class - n (%)Chemo 85 (26.5%) 77 (30.9%)PD1 / PDL1 58 (18.1%) 51 (20.5%)Other target 79 (24.6%) 49 (19.7%)PD1 / PDL1 + Chemo 32 (10.0%) 25 (10.0%)PD1 / PDL1 + Other target 28 (8.7%) 16 (6.4%)Chemo + Other target 28 (8.7%) 21 (8.4%)PD1 / PDL1 + Chemo + Other target 11 (3.4%) 10 (4.0%)Indication - n (%)NSCLC 176 (54.8%) 133 (53.4%)Melanoma 44 (13.7%) 23 (9.2%)Others 101 (31.5%) 93 (37.3%)Treatment line - n (%)IL 191 (59.5%) 139 (55.8%)1L+ 28 (8.7%) 12 (4.8%)Remaining 102 (31.8%) 98 (39.4%)Primary analysis population PD1 status - n (%)Positive 40 (12.5%) 40 (16.1%)Not specified 281 (87.5%) 209(83.9%)Cancer stage - n (%)III / IV mixed 217 (67.6%) 158 (63.5%)IV only 104 (32.4%) 91 (36.5%)Data set 1 is for analyzing ORR / mPFS and Data set 2 is for analyzing ORR / mOS

[0225] The tree-based machine learning predictive model of ORR / mPFS was summarized in FIG. 18. Specifically, for treatment classes, Chemotherapy, Other target + Chemo combination and PD1 / PDL1 + Other target + Chemo combination were grouped together (treatment class group 1) due to homogeneous ORR / mPFS relationship within these three treatment classes, and similarly PD1 / PDL1 therapy and PD1 / PDL1 + Other target combination were grouped together (treatment class group 2) (FIG. 19A). (Note: C indicates Chemotherapy, P indicates PD1 / PDL1 therapy, T indicates Other target therapy in FIG. 19A) The ORR / mPFS relationships across treatment class group 1, treatment class group 2, Other target therapy, PD1 / PDL1 + Chemo combination were heterogeneous and left as tree branches. This indicated that the same ORR may lead to different mPFS across different treatment class. Moreover, within treatment class group 1 and treatment class group 2,different treatment lines lead to heterogeneous ORR / mPFS relationship (FIGS. 19B and 19C). Furthermore, within the IL / 1L+ of treatment class group 1 and Remaining Line of treatment class group 2, different indications lead to heterogeneous ORR / mPFS relationship (FIGS. 19D and 19E). The tree-based regression approach enabled ORR / mPFS (ORR / mOS) relationship of various treatment classes, tumor types, line of therapy, etc. to borrow strength when homogeneity was identified among these influential factors.

[0226] The tree-based machine learning predictive model of ORR / mOS was summarized in FIG. 20. Specifically, Chemotherapy, Chemo + Other target combination and PD1 / PDL1 + Other target + Chemo combination were grouped together (treatment class group 1) due to homogeneous ORR / mOS relationship, and similarly for grouping Other target therapy, PD 1 / PDL1 + Chemo combination and PD1 / PDL1 + Other target combination (treatment class group 2) (FIG. 21 A). (Note: C indicates Chemotherapy, P indicates PD1 / PDL1 therapy, T indicates Other target therapy in FIG. 21A.) The ORR / mOS relationship across treatment class group 1, treatment class group 2, PD1 / PDL1 therapy were heterogeneous. Moreover, within treatment class group 2, different indication leads to heterogeneous ORR / mOS relationship (FIG. 21B). Within NSCLC & Melanoma of treatment class group 2, different treatment lines lead to heterogeneous ORR / mOS relationship (FIG. 21C).

[0227] Next, the predictive performance of proposed tree-based machine learning regression model was evaluated compared to other commonly used weighted regression model and black box machine learning model including ML mPFS (mOS) ~ ORR, M2: mPFS (mOS) - ORR + Treatment class + Indication + Treatment line + PD1 / PDL1 status + Cancer stage, M3: Variable selection from M2, M4: mPFS (mOS) - ORR x (Treatment class + Indication + Treatment line + PD1 / PDL1 status + Cancer stage), M5: Variable selection from M4, M6: Random forest, M7: Extreme gradient boosting, M8: Our proposed tree-based machine learning regression model. The predictive performance is evaluated by 80% - 20% cross validation. The whole data set is divided into 80% and 20% for 1000 times. In each replicate, 80% of data is used for training the model and 20% is used for testing the predictive mean square error (MSE). Specifically, for random forest and extreme gradient boosting, various hyper-tuning parameters combinations were searched and the one with smallest median predictive mean square error (MSE) over 1000 replicates was reported. In random forest, hyper-tuning parameters number of trees in forest e {500, 1000}, number of maximum nodes in each tree e { 10, 20, 30} and number of minimum observations in each tree terminal nodee { 10, 20, 30} were searched. In extreme gradient boosting, hyper-tuning parameters number of maximum iterations e {25, 50, 100}, maximum depth of each tree e { 1, 2, 3, 4, 5} and number of minimum observations in each tree terminal node e { 10, 20, 30} were searched.

[0228] In FIG. 22, it is clear that the predictive MSE of tree-based machine learning model was competitive to M6 and M7 (black box machine learning model) and much smaller than Ml, M2, M3, M4, M5 (regression model) no matter using ORR predicting mPFS or mOS. This demonstrated that the model and structure developed by tree-based machine learning regression was robust. Specifically, Ml, M2 and M3 did not account for ORR / mPFS (ORR / mOS) heterogeneity, M4 and M5 did not borrow enough strength, M6 and M7 were in short of clearly clinical interpretation. In general, compared to other competitors, the proposed tree-based machine learning regression model was robust, borrowed strength and detected ORR / mPFS (ORR / mOS) heterogeneity simultaneously, as well as had clear structure for clinical interpretation.4. APPLICATION

[0229] In this section, how to apply a developed tree-based machine learning predictive model for late phase trial POS evaluation is illustrated. The following two scenarios were considered. Scenario 1 : 1 : 1 randomized phase III clinical trial with treatment arm PD1 / PDL1 + target therapy (other than PD1 / PDL1) combination versus control arm Keytruda with target population previous untreated PD1 / PDL1 positive advanced / metastatic NSCLC patients. Scenario 2: 1 : 1 randomized phase III clinical trial with treatment arm PD1 / PDL1 + Chemo combination versus control arm Chemo with target population previous untreated advanced / metastatic NSCLC patients. The sample size was N for each arm. For simplicity, the censoring rate was assumed to be 0%. For treatment arm, the phase II study was single arm with sample size n. For scenario 1, the mPFS and mOS for Keytruda were assumed to be 5.40 months (standard deviation 0.48 months) and 16.7 months (standard deviation 1.48 months), respectively (Mok et al., 2019). For scenario 2, the mPFS and mOS for Chemotherapy were assumed to be 5.00 months (standard deviation 0.48 months) and 13.0 months (standard deviation 1.07 months), respectively (Gogishvili et al., 2022). In addition, it was assumed that the survival curve for both treatment arms and control arm were exponentially distributed. The phase III trial POS was evaluated using PFS and OS as clinical endpoint respectively as follows.Step 1: Sample ORR for treatment arm by phase II study.Step 2: Sample 100% patients with replacement to develop ORR / mPFS (ORR / mOS) tree-based machine learning predictive model and predict mPFS (mOS) given sampled ORR in Step 1 for treatment arm.Step 3: Sample mPFS (mOS) for control arm.Step 4: Derive the survival curve for each arm by sampled mPFS (mOS) in Step 2 and 3.Step 5: Simulate N patients for each arm from the survival curve in Step 4 and derive hazard ratio P value by proportional hazard model.Step 6: Repeat Step 1 to 5 for 5000 times to evaluate the proportional of simulated trials with P value < 0.05.

[0230] The POS was summarized in FIGS. 23A-23D over different phase II / III study sample size n and N. For scenario 1 using OS as endpoint, the POS was relatively small when ORR is 40% (FIG. 23 A). This was because predicted mOS mean was 18 months, which was similar to the mOS of competitor Keytruda. It is unlikely to move to phase III study in such situation. For the remaining, as the phase II / III study sample size n and N increased, the POS increased due to smaller variance on mPFS (mOS) prediction and simulated trial hazard ratio estimation. In addition, the POS increasing magnitude decreased as n and N increased. For example, for phase III study sample size N, POS increased more for N changing from 100 to 300 compared to N changing from 300 to 500. For phase II study sample size n, POS increased more for n changing from 10 to 30 compared to n changing from 30 to 50. The POS cannot be significantly improved as the number of samples in phase II / III study was adequate, which may help justify the desirable sample size in phase II / III study design.5. DISCUSSION

[0231] In this example, a comprehensive QLD and tree-based machine learning model to enhance the prediction of mPFS / mOS by ORR for PD1 / PDL1 combination therapy development was established. The QLD integrated information from different sources such as clinicaltrial.gov and Informa to include adequate number of historical clinical trials. A generalizable algorithm was proposed to systematically extract trial level summary and clinical outcome from structured data in clinicaltrial.gov and Informa, complemented withmanual curation for scientific accuracy and completeness. Next, a tree-based machine learning regression model to was developed to borrow strength and account for ORR / mPFS (ORR / mOS) relationship heterogeneity simultaneously with explicit structure for clinical interpretation. The model and structure developed by the proposed method is more robust than popular existing machine learning predictive models (e.g., random forest and extreme gradient boosting) as determined by by cross validation. Through the predictive model, observed ORR in early phase study can be leveraged to better predict late phase study success or better plan late phase studies.

[0232] One potential extension of this example is to incorporate the historical clinical trials from other tumor types beyond NSCLC and Melanoma for ORR / mPFS (ORR / mOS) predictive model development in accordance with some embodiments. This may enrich each target therapy class and enable systematic exploration of the potential ORR / mPFS (ORR / mOS) heterogeneity across each target therapy class and target biomarker status on primary analysis population. Thus, in some embodiments, the predictive model derived from more tumors could support various target therapy related late phase study design, and embodiments are not restricted to use in PD1 / PDL1 combination therapy development. In addition, some embodiments could include other potential predictors beyond treatment class, tumor type, line of therapy, cancer stage, biomarker status of primary analysis population future depending on the availability in data source (e.g., clinicaltrial.gov and Informa).

[0233] A number of implementations of the present disclosure have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

CLAIMSWe claim:

1. A method comprising: extracting or accessing, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generating, by the at least one processor, a plurality of structured datasets including a plurality of data types based on the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; applying, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generating, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing; and providing a graphical user interface to receive input from a user regarding an analysis to be performed using the plurality of structured datasets.

2. The method of claim 1, wherein the plurality of structured datasets also includes a regulatory event dataset.

3. The method of claim 1 or claim 2, wherein the at least a portion of the extracted information including unstructured data includes recent information not yet available as structured clinical trial information from the one or more databases.

4. The method of any one of claims 1 to 3, further comprising: extracting or accessing, by at least one processor, second information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the one or more databases; and one or both of:generating, by the at least one processor, second data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the second information; and applying, by the at least one processor, natural language processing to at least a portion of the second information including unstructured text, and generating, by the at least one processor, second data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

5. The method of any one of claims 1 to 3, further comprising: periodically extracting or accessing, by at least one processor, additional information including one or more of drug study design information, biomarker information, drug treatment information, and efficacy information for at least one clinical trial from at least one of the one or more databases; and one or both of: generating, by the at least one processor, additional data corresponding to at least one of the plurality of data types to be added to at least one of the plurality of datasets or to update at least some of the data in at least one of the plurality of datasets based on the additional information; and applying, by the at least one processor, natural language processing to at least a portion of the additional information including unstructured text, and generating, by the at least one processor, additional data or revised data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing.

6. The method of any one of claims 1 to 5, further comprising: executing, by the at least one processor, a machine learning module to determine a probability of success of a clinical drug trial advancing to a next clinical trial stage based on the plurality of structured datasets.

7. The method of any one of claims 1 to 6, further comprising: executing, by the at least one processor, a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets.

8. The method of any one of claims 1 to 7, further comprising: executing, by the processor, a machine learning module to determine a probability of success of obtaining regulatory approval of a drug in a current clinical trial based on the plurality of structured datasets for each of a plurality of regulatory routes.

9. The method of any one of claims 1 to 8, further comprising: identifying, by the at least one processor, one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the historical likelihood of approval of for at least some of the plurality of clinical drug trials.

10. The method of claim 9, wherein the one or more scientific factors include a mechanism of action associated with each of the one or more clinical drug trials, the one or more design factors include endpoints associated with each of the one or more clinical trials, and the one or more regulatory factors includes a designation of the one or more clinical drug trials as a breakthrough.

11. The method of any one of claims 1 to 10, further comprising assigning a category of trial status to each clinical trial in the plurality of structured datasets, wherein the categories of trial status include one or more of: completed with final results, terminated or ongoing with no final results, and no results.

12. The method of any one of claims 1 to 11, further comprising simulating, by the at least one processor, one or more phase three clinical trial outcomes based on one or more proof of concept study outcomes.

13. The method of any one of claims 1 to 12, wherein the GUI includes: a first field for receiving a first input including information associated with a disease, the first field specifying variables, variable combinations of interest, or both; and a second field for identifying an output to be generated.

14. The method of claim 13, wherein the first input includes or identifies a disease-specific database or a subset of a disease-specific database, the disease-specific database being based at least in part on standardized clinical trials associated with the disease.

15. The method of claim 13 or claim 14, wherein the identified output to be generated includes one or more of: a description of one or more first clinical trials associated with the disease; characteristics or outcomes associated with the one or more first clinical trials; or prediction of an outcome of the one or more first clinical trials for a drug used to treat the disease.

16. The method of any one of claims 1 to 15, wherein the trial arm dataset includes data types for associating trial outcomes with treatment information.

17. The method of any one of claims 1 to 16, further comprising generating a predictive model or in-silico trial simulations for a prediction of a trial outcome.

18. The method of any one of claims 1 to 16, wherein the extracted or accessed trial outcome information includes endpoints associated with each of the clinical trials.

19. The method of claim 18, further comprising generating a predictive model or in-silico trial simulation for a prediction of the endpoints, wherein the predictive model or in- silico trial simulation is based at least in part on a machine learning model.

20. A method of determining a predictive model of a median progression free survival (mPFS) or median objective survival (mOS) for a late phase clinical trial based on objective response rate (ORR) in an early phase clinical trial, the method comprising:extracting or accessing clinical trial information including ORR information, mPFS information, mOS information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information for a plurality of clinical trials obtained from one or more databases; defining variable classes including at least some of or all of variables based on ORR information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information; and determining a tree-based regression machine learning model for mPFS based on ORR or for mOS based on ORR, the determination including: performing forward segmentation of a predictor space based on the variable classes into a number of subspaces until there is no mPFS based on ORR heterogeneity or until there is no mOS based on ORR heterogeneity; and determining a regression model for each subspace.

21. The method of claim 20, wherein one or more of the treatment classes include an anti- PD1 and / or anti-PLDl therapy (PD1 / PDL1 therapy); and wherein the biomarker status information includes PD1 and / or PDL1 status information.

22. The method of claim 20 or 21, wherein performing forward segmentation of a predictor space into a number of subspaces until there is no mPFS based on ORR heterogeneity or until there is no mOS based on ORR heterogeneity comprises: determining a slope for each variable class and grouping the variable classes with homogenous slope to form grouped variable classes; and for each grouped variable class, splitting the grouped variable class where there is significant heterogenous slope within the variable classes in the grouped variable class.

23. The method of claim 22, wherein determining a slope for each variable class and grouping the variable classes with homogenous slope to form grouped variable classes comprises: determining a first variable class having a largest sample size as compared with sample sizes of the other variable classes which are one or more second variable classes; determining for each variable class, a slope for the variable class by comparing: a first regression of the mPFS or mOS against a product of the ORR and a value of the class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the class variable; for each of the one or more second variable classes: determining a slope difference between a slope of the second variable class and a slope of the first variable class; and grouping variable classes where a slope difference for the variable classes have a P value over a specified grouping threshold or a second variable class sample size under a specified sample size threshold.

24. The method of claim 23, further comprising, performing the steps of claim 23 for each remaining ungrouped second variable class.

25. The method of claim 23 or 24, wherein for each grouped variable class, splitting the grouped variable class where there is significant heterogenous slope within the variable classes in the grouped variable class comprises: for each variable having a number of classes after grouping of greater than one, comparing: a first regression of the mPFS or mOS against a product of the ORR and a value of the grouped class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the grouped class variable; andselecting a grouped variable class to split where an ANOVA P value is less than a specified splitting threshold and the ANOVA P value is the most significant for that variable as compared with all other variables.

26. The method of claim 25, further comprising, for each split grouped variable class, performing the steps of determining a first variable class of the split grouped variable class having a largest sample size as compared with sample sizes of the other variable classes in the split grouped variable class, which are one or more second variable classes; determining for each variable class of the split grouped variable class, a slope for the variable class by comparing: a first regression of the mPFS or mOS against a product of the ORR and a value of the class variable, and a second regression of the mPFS information or mOS information against the ORR information added to the value of the class variable; and for each of the one or more second variable classes: determining a slope difference between a slope of the second variable class and a slope of the first variable class; and grouping variable classes where a slope difference for the variable classes have a P value over the specified grouping P threshold or a second variable class sample size under the specified sample size threshold.

27. The method of any one of claims 20 to 26, wherein the extracted or accessed trial outcome information includes endpoints associated with each of the clinical trials.

28. The method of any one of claims 20 to 27, wherein treatment class information comprises chemotherapy, a combination of chemotherapy and PD1 / PDL1, or a combination of two or more of chemotherapy, PD1 / PDL1, or a targeted therapy.

29. A system comprising: storage configured to store a unified clinical trial database; a memory storing one or more instructions; andat least one processor operationally coupled to the storage, wherein the at least one processor is configured to or programmed to read the one or more instructions stored in the memory, thereby causing the at least one processor to: extract or access, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generate, by the at least one processor, a plurality of structured datasets including a plurality of data types based on at least a portion of the extracted information, the plurality of structured datasets including a trial summary dataset, a trial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; apply, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing; store the plurality of structured datasets in the storage, where the unified clinical trial database includes the plurality of structured datasets; and provide a graphical user interface to receive input from a user regarding an analysis to be performed using the unified clinical trial database.

30. A non-transitory computer-readable medium comprising instructions that when executed by a processing device causes the processing device to: extract or access, by at least one processor, information including drug study design information, biomarker information, drug treatment information, and efficacy information for a plurality of clinical trials from one or more databases; generate, by the at least one processor, a plurality of structured datasets including a plurality of data types based on at least a portion of the extracted information, the plurality of structured datasets including a trial summary dataset, atrial arm dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; apply, by the at least one processor, natural language processing to at least a portion of the extracted or accessed information including unstructured text; generate, by the at least one processor, additional data corresponding to at least one of the plurality of data types for at least some of the plurality of structured datasets based on the natural language processing; store the plurality of structured datasets in the storage, where the unified clinical trial database includes the plurality of structured datasets; and provide a graphical user interface to receive input from a user regarding an analysis to be performed using the unified clinical trial database.

Images