System and method for quantitative clinical trial database modeling to notify project decisions

Abstract

Embodiments of this disclosure include systems, methods, and non-temporary computer-readable media for generating combined standardized clinical trial databases, which provide a graphical user interface for analysis using the combined standardized clinical trial databases. Some embodiments of the disclosure include processes for standardizing and integrating clinical trial data obtained from various database systems that store similar clinical trial data using different data types and formats.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to European Patent Application Publication No. 23306518.4, filed on 14 September 2023, and European Patent Application Publication No. 24305220.6, filed on 9 February 2024, both of which are incorporated herein by reference in their entirety.

[0002] Some embodiments disclosed herein relate to the field of medical informatics, and more specifically, to systems and methods that use medical informatics to plan, conduct, and analyze clinical trials and their results. [Background technology]

[0003] Clinical trial knowledge stored in or accessed by various conventional clinical trial database systems can differ in its structure and management. For example, clinical trial knowledge may include information on clinical trial outcomes, including safety and efficacy data based on results published in different journal publications, information on drugs evaluated during clinical trials, and results regarding drugs evaluated in clinical trials. Clinical trial knowledge stored in or accessed by various conventional clinical trial database systems may also include information on comparisons between different clinical trials based on safety and efficacy data. However, clinical trial knowledge stored across multiple clinical trial database systems is often structured or unstructured according to a specific format associated with each individual clinical trial database system among the multiple clinical trial database systems. As a result, the data and information representing clinical trial knowledge may be structured differently across different clinical trial database systems. [Overview of the project] [Means for solving the problem]

[0004] Embodiments of the present disclosure include methods comprising: extracting or accessing information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information relating to a plurality of clinical trials, using at least one processor; generating a plurality of structured datasets, including a plurality of data types, based on the extracted information, wherein the plurality of structured datasets include a trial summary dataset, a trial group dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; applying natural language processing to at least a portion of the extracted or accessed information, including unstructured text, using at least one processor; generating additional data, based on natural language processing, for at least a portion of the plurality of structured datasets, corresponding to at least one of the plurality of data types; and providing a graphical user interface for receiving user input regarding analyses performed using the plurality of structured datasets.

[0005] The method may include a method in which multiple structured datasets also include a regulatory events dataset. The method may include a method in which at least a portion of the extracted information, including unstructured data, includes recent information that is not yet available as structured clinical trial information from one or more databases. The method may further include one or both of the following: extracting or accessing second information, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information, relating to at least one clinical trial, from at least one of one or more databases by at least one processor; generating second data, corresponding to at least one of multiple data types, based on the second information, and adding it to at least one of multiple datasets, or updating at least a portion of the data in at least one of multiple datasets; and applying natural language processing to at least a portion of the second information, including unstructured text, by at least one processor, and generating second data or revised data, corresponding to at least one of multiple data types, for at least a portion of multiple structured datasets, based on the natural language processing by at least one processor.

[0006] Methods according to some embodiments may further include: at least one processor periodically extracting or accessing additional information from at least one of one or more databases, including one or more of drug trial design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial; at least one processor generating additional data corresponding to at least one of a plurality of data types based on the additional information and adding it to at least one of a plurality of datasets, or updating at least a portion of the data in at least one of a plurality of datasets; and at least one processor applying natural language processing to at least a portion of the additional information, including unstructured text, and at least one processor generating additional or revised data corresponding to at least one of a plurality of data types for at least a portion of a plurality of structured datasets based on the natural language processing.

[0007] Methods according to some embodiments may further include, by at least one processor, running a machine learning module to determine the probability of success of a clinical drug trial progressing to the next clinical trial stage based on multiple structured datasets. Methods may further include, by at least one processor, running a machine learning module to determine the probability of success of obtaining regulatory approval for a drug in the current clinical trial based on multiple structured datasets. Methods may further include, by a processor, running a machine learning module to determine the probability of success of obtaining regulatory approval for a drug in the current clinical trial for each of multiple regulatory routes based on multiple structured datasets. Methods may further include, by at least one processor, identifying one or more scientific, design, regulatory, or operational factors that contributed to the likelihood of past approval for at least a portion of multiple clinical drug trials.

[0008] The method may, in some embodiments, include a method in which one or more scientific factors include a mechanism of action related to each of one or more clinical drug trials, one or more design factors include an endpoint related to each of one or more clinical trials, and one or more regulatory factors include the designation of one or more clinical drug trials as a breakthrough. The method may further include assigning a trial status category to each clinical trial in a plurality of structured datasets, the trial status categories including one or more of completed with final results, terminated or continued without final results, and no results. The method may further include simulating one or more phase 3 clinical trial outcomes based on one or more proof-of-concept study outcomes by at least one processor. The method may include a method in which a GUI includes a first field for receiving a first input containing disease-related information, the first field for specifying a variable, a combination of variables of note, or both, and a second field for identifying the output to be generated. The method may include a method in which the first input includes or identifies a disease-specific database or a subset of disease-specific databases, and the disease-specific database is at least partially based on standardized clinical trials related to the disease. The method may include a method in which the identified generated output includes one or more of the following: a description of one or more first clinical trials related to a disease; a characteristic or outcome related to one or more first clinical trials; or a prediction of the outcome of one or more first clinical trials relating to a drug used to treat a disease.

[0009] The method may, in some embodiments, include a method in which the trial group dataset includes data types for relating trial outcomes with therapeutic information. The method may further include generating a predictive model or in silico trial simulation for predicting trial outcomes. The method may include a method in which the extracted or accessed trial outcome information includes endpoints relevant to each of the clinical trials. The method may further include generating a predictive model or in silico trial simulation for predicting endpoints, wherein the predictive model or in silico trial simulation is at least partially based on a machine learning model.

[0010] Embodiments of the present disclosure are systems comprising storage configured to store an integrated clinical trial database, memory for storing one or more instructions, and at least one processor operably coupled to the storage, wherein the at least one processor is configured or programmed to read one or more instructions stored in memory, thereby allowing the at least one processor to extract or access information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information relating to multiple clinical trials, and allowing the at least one processor to generate multiple structured datasets, including multiple data types, based on at least a portion of the extracted information, wherein the multiple structured datasets include a trial summary dataset, a trial group dataset, a baseline dataset, A system comprising generating outcome datasets and outcome comparison datasets, applying natural language processing to at least a portion of the extracted or accessed information, including unstructured text, by at least one processor, generating additional data corresponding to at least one of a plurality of data types for at least a portion of a plurality of structured datasets, by at least one processor, based on the natural language processing, and storing the plurality of structured datasets in storage, wherein the integrated clinical trial database includes storing the plurality of structured datasets and providing a graphical user interface for receiving user input regarding analyses performed using the integrated clinical trial database, by having at least one processor perform these tasks.

[0011] Embodiments of the system may include a system in which multiple structured datasets also include a regulatory events dataset. The system may further include a system in which at least a portion of the extracted information, including unstructured data, includes recent information that is not yet available as structured clinical trial information from an integrated clinical trial database. The system may further involve at least one processor being configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to perform one or both of the following: extract or access second information from at least one of an integrated clinical trial database, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial; generate second data corresponding to at least one of a plurality of data types based on the second information, and add it to at least one of a plurality of datasets, or update at least a portion of the data in at least one of a plurality of datasets; and apply natural language processing to at least a portion of the second information, including unstructured text, and generate second data or revised data corresponding to at least one of a plurality of data types for at least a portion of a plurality of structured datasets based on the natural language processing.

[0012] The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to periodically extract or access additional information from at least one clinical trial database, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial; to generate additional data corresponding to at least one of a plurality of data types based on the additional information, to be added to at least one of a plurality of datasets, or to update at least a portion of the data in at least one of a plurality of datasets; and to apply natural language processing to at least a portion of the additional information, including unstructured text, and to generate additional or revised data corresponding to at least one of a plurality of data types for at least a plurality of structured datasets based on the natural language processing, or both of the above.

[0013] The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to execute a machine learning module to determine the probability of success of a clinical drug trial progressing to the next clinical trial stage, based on a plurality of structured datasets. The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to execute a machine learning module to determine the probability of success of obtaining regulatory approval for a drug in the current clinical trial, based on a plurality of structured datasets. The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to execute a machine learning module to determine the probability of success of obtaining regulatory approval for a drug in the current clinical trial for each of a plurality of regulatory routes, based on a plurality of structured datasets.

[0014] The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to identify one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the likelihood of past approval for at least a portion of a plurality of clinical drug trials. The system may further include a system in which one or more scientific factors include a mechanism of action related to each of one or more clinical drug trials, one or more design factors include an endpoint related to each of one or more clinical trials, and one or more regulatory factors include the designation of one or more clinical drug trials as a breakthrough. The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to assign a trial status category to each clinical trial in a plurality of structured datasets, the trial status categories including one or more of completed with final results, terminated or continued without final results, and no results.

[0015] Embodiments of the system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to simulate one or more Phase 3 clinical trial outcomes based on one or more proof-of-concept study outcomes. The system may further include a GUI which includes a first field for receiving a first input containing disease-related information, a first field for specifying a variable, a combination of variables of interest, or both, and a second field for identifying a generated output. The system may further include a system in which the first input contains or identifies a disease-specific database or a subset of a disease-specific database. The system may further include a system in which the identified generated output contains one or more of the following: a description of one or more first clinical trials related to the disease, features or outcomes related to one or more first clinical trials, or predictions of outcomes of one or more first clinical trials relating to drugs used to treat the disease. The system may further include a system in which a trial group dataset contains a data type for relating trial outcomes to treatment information. The system may further include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to generate a predictive model or in silico test simulation for predicting test outcomes.

[0016] Embodiments of the system may further include a system in which the extracted or accessed trial outcome information includes endpoints relevant to each clinical trial. The system may include a system in which at least one processor is configured or programmed to read one or more instructions stored in memory, thereby causing at least one processor to generate a predictive model or in silico trial simulation for predicting progression-free survival (PFS) based on overall response rate (ORR), also referred herein to as objective response rate (ORR), using a tree-based regression machine learning model or a related in silico trial simulation. The system may further include a system in which the information further includes publications, conference proceedings, academic papers, and regulatory information relevant to one or more clinical trials.

[0017] Embodiments of the present disclosure include a non-temporary computer-readable medium containing instructions, the instructions causing the processing device to: extract or access information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information relating to a plurality of clinical trials, by at least one processor; generate a plurality of structured datasets, each containing a plurality of data types, based on at least a portion of the extracted information, wherein the plurality of structured datasets includes a trial summary dataset, a trial group dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset; apply natural language processing to at least a portion of the extracted or accessed information, each containing unstructured text, by at least one processor; generate additional data, based on natural language processing, for at least a portion of the plurality of structured datasets, corresponding to at least one of a plurality of data types; store the plurality of structured datasets in storage, wherein an integrated clinical trial database contains the plurality of structured datasets; and provide a graphical user interface for receiving user input regarding analyses performed using the integrated clinical trial database.

[0018] Embodiments of non-temporary computer-readable media include media containing multiple structured datasets, including regulatory event datasets. Non-temporary computer-readable media may include media containing at least a portion of extracted information, including unstructured data, that is recent information not yet available as structured clinical trial information from an integrated clinical trial database. Non-temporary computer-readable media may further include instructions which, when executed by a processing device, cause the processing device to perform one or both of the following: extract or access second information from at least one of at least one integrated clinical trial databases, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial; generate second data based on the second information, corresponding to at least one of a plurality of data types, and add it to at least one of a plurality of datasets, or update at least a portion of the data in at least one of a plurality of datasets; and apply natural language processing to at least a portion of the second information, including unstructured text, and generate second data or revised data based on the natural language processing for at least a portion of a plurality of structured datasets, corresponding to at least one of a plurality of data types.

[0019] Embodiments of non-temporary computer-readable media may further include instructions, which, when executed by a processing device, cause the processing device to: periodically extract or access additional information from at least one clinical trial database, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial; generate additional data corresponding to at least one of a plurality of data types based on the additional information, to be added to at least one of a plurality of datasets, or update at least a portion of the data in at least one of a plurality of datasets; and apply natural language processing to at least a portion of the additional information, including unstructured text, and generate additional or revised data corresponding to at least one of a plurality of data types for at least a portion of a plurality of structured datasets based on the natural language processing, or both of the above. Non-temporary computer-readable media may further contain instructions that, when executed by a processing device, cause the processing device to run a machine learning module to determine the probability of success of a clinical drug trial progressing to the next clinical trial phase, based on multiple structured datasets.

[0020] Embodiments of the non-transitory computer-readable medium may further include instructions that, when executed by a processing device, cause the processing device to execute a machine learning module to determine the success probability of obtaining regulatory approval for a drug in an ongoing clinical trial based on a plurality of structured datasets. The non-transitory computer-readable medium may further include instructions that, when executed by a processing device, cause the processing device to execute a machine learning module to determine the success probability of obtaining regulatory approval for a drug in an ongoing clinical trial based on a plurality of structured datasets. The non-transitory computer-readable medium may further include instructions that, when executed by a processing device, cause the processing device to identify, by at least one processor, one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the likelihood of past approval for at least a portion of a plurality of clinical drug trials.

[0021] Embodiments of the non-transitory computer-readable medium may comprise a medium wherein the one or more scientific factors include a mechanism of action associated with each of the one or more clinical drug trials, the one or more design factors include an endpoint associated with each of the one or more clinical trials, and the one or more regulatory factors include designation of the one or more clinical drug trials as a breakthrough. The non-transitory computer-readable medium may further include instructions that, when executed by a processing device, cause the processing device to assign a test status category to each clinical trial in the plurality of structured datasets, wherein the test status category includes one or more of completed with final results, discontinued or ongoing without final results, and no results. The non-transitory computer-readable medium may further include instructions that, when executed by a processing device, cause the processing device to simulate one or more Phase III clinical trial outcomes based on the outcomes of one or more proof-of-concept studies by at least one processor.

[0022] The non-transitory computer-readable medium may comprise a medium wherein the GUI includes a first field for receiving a first input containing disease-related information, the first field specifying a variable, a notable variable combination, or both, and a second field for identifying an output to be generated. The non-transitory computer-readable medium may comprise a medium wherein the first input comprises or identifies a disease-specific database or a subset of a disease-specific database. The non-transitory computer-readable medium may comprise a medium wherein the identified generated output includes one or more of: a description of one or more first clinical trials related to the disease, features or outcomes associated with the one or more first clinical trials, or a prediction of outcomes of the one or more first clinical trials relating to a drug used to treat the disease. The non-transitory computer-readable medium may comprise a medium wherein the trial cohort dataset includes data types for associating trial outcomes with treatment information. Embodiments of the non-transitory computer-readable medium may further comprise instructions that, when executed by a processing device, cause the processing device to generate a predictive model or in silico trial simulation for prediction of trial outcomes.

[0023] Embodiments of the non-transitory computer-readable medium may comprise a medium wherein the extracted or accessed trial outcome information comprises Overall Response Rate (ORR) data and Progression-Free Survival (PFS) data. Embodiments of the non-transitory computer-readable medium may further comprise instructions that, when executed by a processing device, cause the processing device to generate a predictive model or in silico trial simulation for prediction of PFS based on ORR using a tree-based regression machine learning model or associated in silico trial simulation. In the non-transitory computer-readable medium, the information further includes publications, academic conference proceedings, scholarly papers, and regulatory information related to one or more clinical trials.

[0024] Exemplary embodiments may include methods for alternative tree-based regression models that use overall response rate (ORR) data to predict the median progression-free survival (mPFS) of subjects or the median overall survival (mOS) of subjects in response to drug therapy. In particular, the tree-based regression model is a weighted regression that uses the test group sample size as weights. In some embodiments, the tree-based regression model relies on some of the treatment class data, treatment line data, biomarker status information, and current cancer stage of subjects.

[0025] Some embodiments provide a method for determining a predictive model for median progression-free survival (mPFS) or median objective survival (mOS) in a late-phase clinical trial based on the overall response rate (ORR) in an early-phase clinical trial. The method includes extracting or accessing clinical trial information, including ORR information, mPFS information, mOS information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information, for multiple clinical trials obtained from one or more databases. The method also includes defining a variable class, which includes at least some or all of the variables, based on the ORR information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information. The method further includes determining a tree-based regression machine learning model for mPFS based on ORR or for mOS based on ORR, wherein the determination includes performing forward segmentation of the predictor space into several subspaces based on the variable class until there are no more mPFS based on ORR heterogeneity or no more mOS based on ORR heterogeneity, and determining a regression model for each subspace.

[0026] In some embodiments, one or more therapy classes include anti-PD1 and / or anti-PLD1 therapy (PD1 / PDL1 therapy), and the biomarker status information includes PD1 and / or PDL1 status information.

[0027] In some embodiments, forward segmentation of the predictor space into several subspaces until there is no longer an mPFS based on ORR heterogeneity or until there is no longer an mOS based on ORR heterogeneity includes determining the gradient with respect to each variable class, grouping the variable classes by uniform gradient to form grouped variable classes, and for each grouped variable class, splitting the grouped variable class if there is a significant heterogeneous gradient within the variable class.

[0028] In some embodiments, determining the gradient for each variable class and grouping the variable classes by a uniform gradient to form a grouped variable class includes determining a first variable class having the largest sample size compared to the sample sizes of the other variable classes, which are one or more second variable classes, and determining the gradient for each variable class by comparing a first regression of mPFS or mOS on the product of ORR and the class variable value with a second regression of mPFS or mOS information on ORR information added to the class variable value. Determining the gradient for each variable class and grouping the variable classes by a uniform gradient to form a grouped variable class also includes, for each of the one or more second variable classes, determining the gradient difference between the gradient of the second variable class and the gradient of the first variable class, and grouping the variable class if the gradient difference for the variable class has a p-value above a specified grouping threshold or a second variable class sample size below a specified sample size threshold. In some embodiments, the above steps are performed for each of the remaining ungrouped second variable classes.

[0029] In some embodiments, for each grouped variable class, if there is a significant heterogeneous gradient within the grouped variable class, splitting the grouped variable class includes comparing a first regression of mPFS or mOS on the product of ORR and the value of the grouped class variable with a second regression of mPFS or mOS information on ORR information added to the value of the grouped class variable, for each variable having a grouped class number greater than 1, and selecting the grouped variable class for splitting if the ANOVA P value is less than a specified splitting threshold and the ANOVA P value is the most significant for that variable compared to all other variables.

[0030] In some embodiments, the method further includes: determining, for each divided grouped variable class, a first variable class of the divided grouped variable class which is one or more second variable classes having the largest sample size compared to the sample sizes of other variable classes in the divided grouped variable class; determining the gradient with respect to the variable class for each variable class of the divided grouped variable class by comparing a first regression of mPFS or mOS on the product of ORR and the class variable value with a second regression of mPFS or mOS information on ORR information added to the class variable value; determining the gradient difference between the gradient of the second variable class and the gradient of the first variable class for each of the one or more second variable classes; and grouping the variable class if the gradient difference with respect to the variable class has a P value above a specified grouping P threshold or a second variable class sample size below a specified sample size threshold.

[0031] In some embodiments, determining a regression model for each subspace includes, for each variable, comparing a first regression of mPFS information or mOS information to ORR information attached to the class variable values ​​with a second regression of mPFS information or mOS information to ORR, and selecting variables having a P value less than a specified P threshold.

[0032] In some embodiments, the predictive model for mPFS on ORR is the logarithm of mPFS regressed on the logarithm of ORR over at least a portion of the subspace, or the predictive model for mOS on ORR is the logarithm of mOS regressed on the logarithm of OR over at least a portion of the subspace.

[0033] In some embodiments, the predictive model for at least some of the subspaces is further based on one or more of the labeling information and disease stage information.

[0034] In some embodiments, the treatment class information includes chemotherapy, a combination of chemotherapy and PD1 / PDL1, or a combination of two or more chemotherapy regimens, PD1 / PDL1, or targeted therapies.

[0035] In some embodiments, the indication information includes either or both non-small cell lung cancer data and melanoma data.

[0036] In some embodiments, the treatment line information includes either or both of the initial treatment (1L) and the initial treatment + (1L+) treatment.

[0037] In some embodiments, the methods described herein are performed by a computing system comprising at least one processor or one or more processors.

[0038] In some embodiments, the extracted or accessed clinical trial information is obtained from a quantitative clinical trial database. In some embodiments, the quantitative clinical trial database is generated by a method that includes: extracting or accessing information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information, relating to multiple clinical trials, using at least one processor; generating multiple structured datasets, including multiple structured datasets, including the extracted information, trial summary datasets, trial group datasets, baseline datasets, outcome datasets, and outcome comparison datasets, using at least one processor; applying natural language processing to at least a portion of the extracted or accessed information, including unstructured text, using at least one processor; and generating additional data, based on natural language processing, for at least a portion of the multiple structured datasets, corresponding to at least one of multiple data types, using at least one processor. [Brief explanation of the drawing]

[0039] [Figure 1] This is a flowchart of a process that may be performed according to some embodiments of this disclosure. [Figure 2] This disclosure schematically illustrates the logical information flow between various exemplary databases, modules, and computing environments in several embodiments of this disclosure. [Figure 3] This chart, according to several embodiments of the present disclosure, includes different datasets that can be incorporated into an integrated clinical trial database. [Figure 4] This is a logical diagram illustrating a process for generating clinical trial summary datasets for an integrated clinical trial database, according to several embodiments of the present disclosure. [Figure 5] This is a logical diagram illustrating a process for generating a clinical trial group dataset for an integrated clinical trial database, according to some embodiments of the present disclosure. [Figure 6] This is a logical diagram illustrating a process for generating a clinical trial baseline dataset for an integrated clinical trial database, according to several embodiments of the present disclosure. [Figure 7] This is a logical diagram illustrating a process for generating clinical trial outcome datasets for an integrated clinical trial database, according to some embodiments of the present disclosure. [Figure 8] This is a logical diagram illustrating a process for generating a clinical trial outcome comparison dataset for an integrated clinical trial database, according to some embodiments of the present disclosure. [Figure 9] This is a logical diagram illustrating the process of applying natural language processing to unstructured text stored in a database to extract clinical trial outcomes and clinical trial outcome comparison data, according to some embodiments of the present disclosure. [Figure 10] This is a schematic diagram showing an exemplary computer system that can be used to implement an embodiment of this disclosure. [Figure 11] This is a logical diagram illustrating the process of accessing clinical trial data on a host device from a client device using an application programming interface, according to some embodiments of the present disclosure. [Figure 12] This figure shows an example of the status of clinical trials organized according to different layers, according to some embodiments of the present disclosure. [Figure 13] This is a logical diagram illustrating a process, according to some embodiments of the present disclosure, for extracting clinical trial data from one or more external databases for an integrated clinical trial database and using the extracted clinical trial data as input to a trial simulator. [Figure 14] This figure shows a graphical user interface for a clinical trial simulator that demonstrates the regression of progression-free survival to overall response rate, according to some embodiments of the present disclosure. [Figure 15]This figure shows a graphical user interface for a clinical trial simulator that displays the probabilities of a drug therapy used in a clinical trial progressing from one development stage to the next or to approval with respect to the total number of progression-free survival events, and the probabilities of a drug therapy used in a clinical trial progressing from one development stage to the next or to approval with respect to a given size of an event, according to some embodiments of the present disclosure. [Figure 16] A flowchart illustrating the process of tree-based regression of patient-related progression-free survival in a clinical trial based on the overall response rate of patients in a clinical trial for a drug therapy for a specific disease, including a graph showing the results of the analysis, according to some embodiments of the present disclosure. [Figure 17] This is a table showing the distribution of predictors for the test groups included in some embodiments of the present disclosure. [Figure 18] This figure shows a tree-based machine learning model of overall response rate (ORR) regressed against median progression-free survival (mPFS) in some embodiments of the present disclosure. [Figure 19-1]Figures 19A-19E show graphs of ORRs regressed against heterogeneous mPFS in a tree-based machine learning model in an example, according to several embodiments of the present disclosure. Figure 19A shows a graph of ORRs regressed against mPFS with heterogeneity across treatment classes (all test groups) in an example. Figure 19B shows a graph of ORRs regressed against mPFS with heterogeneity across lines (treatment class group 1) in a tree-based machine learning model in an example. Figure 19C shows a graph of ORRs regressed against mPFS with heterogeneity across lines (treatment class group 2) in a tree-based machine learning model in an example. Figure 19D shows a graph of ORRs regressed against mPFS with heterogeneity across indications (treatment class group 1 1L / 1L+) in a tree-based machine learning model in an example. Figure 19E shows a graph of ORRs regressed against mPFS with heterogeneity across indications (treatment class group 2 remaining lines) in a tree-based machine learning model in an example. [Figure 19-2] Same as above. [Figure 19-3] Same as above. [Figure 20] This disclosure presents tree-based machine learning models of ORR that regress on objective median survival time (mOS) in some embodiments of the present disclosure. [Figure 21-1]Figures 21A-21C show graphs of ORRs regressed against heterogeneous mOS in a tree-based machine learning model in an example, according to several embodiments of the present disclosure. Figure 21A shows a graph of ORRs regressed against mOS with heterogeneity across treatment classes for all test groups in a tree-based machine learning model in an example. Figure 21B shows a graph of ORRs regressed against mOS across indications for treatment class group 2 in a tree-based machine learning model in an example. Figure 21C shows a graph of ORRs regressed against mOS with heterogeneity across lines for treatment class group 2 NSCLC / melanoma in a tree-based machine learning model in an example. [Figure 21-2] Same as above. [Figure 22] This figure includes a comparison of the mean squared error (MSE) of different regression models for ORR regression against mPFS and ORR regression against mOS in the examples. [Figure 23-1] Figures 23A-23D show the probabilities of survival estimation in Phase II and Phase III studies in the examples, using progression-free survival (PFS) as the endpoint and objective survival (OS) as the endpoint. Figure 23A is a graph of the success probability for Scenario 1, Figure 23B is a graph of the success probability for Scenario 2, Figure 23C is a graph of the success probability for Scenario 3, and Figure 23D is a graph of the success probability for Scenario 4. [Figure 23-2] Same as above. [Modes for carrying out the invention]

[0040] Some embodiments of this disclosure include computer implementations, systems, and non-temporary computer-readable storage media that provide standardized and quantified clinical trial data related to one or more drugs used for the treatment of diseases at various stages of approval and that are received or extracted from multiple database systems. Disclosures of some embodiments describe processes for standardizing and integrating clinical trial data obtained from various database systems that store similar clinical trial data using different data types and formats. Some of the clinical trial data stored in a particular conventional clinical trial database system may be unstructured text containing similar relevant outcome results as structured clinical trial data stored in other database systems, but the unstructured nature of the text makes it difficult to readily standardize any meaningful information related to the unstructured text in accordance with the clinical trial data stored in other databases. Some embodiments disclosed herein provide processes for obtaining clinical trial data related to specific measured results or outcome quantities, also known as variables, from multiple different database systems, generating standardized and integrated data based on the obtained clinical trial data, and storing the standardized and integrated clinical trial data in a disease (indication)-specific standardized database.

[0041] Some embodiments disclosed herein provide methods for generating an integrated clinical trial database, and systems that include or employ an integrated clinical trial database. Some embodiments disclosed herein provide methods for employing an integrated clinical trial database.

[0042] Some embodiments disclosed herein provide users (e.g., researchers, clinical trial sponsors, physicians, investors, etc.) with the probability or likelihood that a particular drug used in a clinical trial will progress through various stages of the clinical trial, and determine what factors, if any, significantly contribute to the likelihood that a particular drug will progress through a particular stage of the clinical trial to approval.

[0043] Several traditional individual database systems can provide some predictive analytics that generate success probabilities or likelihoods regarding whether a particular drug will progress to each stage of clinical trials. However, simply summing or combining these probabilities or likelihoods generated from multiple different individual traditional database systems only provides a summation or combination of the individual probabilities or likelihoods that the drug will progress through the different stages of clinical trials and ultimately reach approval. Furthermore, the methods used to calculate probabilities and / or likelihoods are likely to differ between different individual traditional database systems, given that the measures related to the clinical trial outcomes corresponding to the drug differ between the different databases storing the clinical trial data. As a result, any combination of individual probabilities or likelihoods may not provide a more accurate representation of the probability or likelihood that the drug will progress through the various stages of clinical trials and reach approval than the individual probabilities or likelihoods themselves.

[0044] Several methods, systems, and non-temporary computer-readable media described herein accurately provide the probability and likelihood of a drug progressing through various stages of a clinical trial and ultimately to approval, based on a normalized or standardized and structured set of clinical trial data. In addition, several methods, systems, and non-temporary computer-readable media described herein identify factors that significantly contribute to the probability and likelihood of a drug related to a clinical trial progressing through stages to approval. In some embodiments, the methods, systems, and non-temporary computer-readable media described herein can identify different therapeutic lines that contribute to the probability and likelihood of a drug related to a clinical trial progressing through stages to approval.

[0045] In some embodiments, safety and efficacy information stored across clinical trial database systems can be used to determine or model the likelihood that a drug will progress through different stages of clinical trials and ultimately reach approval. Safety and efficacy information can also, in some embodiments, be used to determine specific factors that contribute to the likelihood that a drug will be approved after progressing through different stages of clinical trials. This information is relevant to each clinical trial and provides, for each clinical trial, the likelihood that the drug will progress through various stages of the clinical trial and reach approval.

[0046] To this end, standardizing safety and efficacy data obtained from different clinical trial databases improves the ability to determine or model the likelihood of a drug progressing through various stages of clinical trials and ultimately reaching approval, and to determine specific factors that contribute to that progress through various stages of clinical trials and ultimately reaching approval.

[0047] In some embodiments, the described methods, systems, and non-temporal computer-readable media apply natural language processing (NLP) to unstructured text related to clinical trials obtained from a database system to identify relevant outcome events related to clinical trial outcomes corresponding to drugs administered during the clinical trial. In some embodiments, the methods, systems, and non-temporal computer-readable media detect sections of unstructured text, such as clinical trial design information and results related to the clinical trial. For example, in some embodiments, the methods, systems, and non-temporal computer-readable media apply NLP to unstructured text to detect information related to descriptions of how the study is conducted, including comparative group design and strategies for participant masking and assignment. In response to applying NLP to unstructured text, in some embodiments, the methods, systems, and non-temporal computer-readable media automatically annotate sentences related to clinical trial design information and corresponding results related to clinical trial outcomes, label sentences and phrases in unstructured text, and apply pattern matching techniques to identify and extract data related to clinical trial outcomes, such as treatment groups, biomarker types, efficacy estimates, confidence intervals, and p-values. In some embodiments, the extracted data is used by methods, systems, and non-temporary computer-readable media to determine the probability or likelihood of a drug related to a clinical trial progressing through various stages of approval, as well as any factors that significantly contribute to the probability or likelihood of success.

[0048] In some embodiments, the described methods, systems, and non-transient computer-readable media provide a graphical user interface (GU) that includes an application programming interface (API) enabling a user to access and interact with structured clinical trial data from one or more indication-specific integrated clinical trial databases. In some embodiments, the GUI also provides the user with the ability to find clinical trial data based on specific research area and project-specific needs. For example, the GUI may have an interface optimally designed for oncology research and another interface optimally designed for another research area. In some embodiments, the GUI receives input from the user, and accordingly, the system simulates clinical trial outcomes based on clinical trial data in one or more integrated clinical trial databases.

[0049] Figure 1 shows an exemplary method 100 that may be performed according to several implementations of the present disclosure. Method 100 provides a user with access to the analysis of structured clinical trial data from one or more databases (e.g., multiple clinical trial databases) via a graphical user interface. One or more steps of the method may be performed by one or more computing devices or computing systems (e.g., computing device 205 in Figure 2 or computing system 1000 in Figure 10). In 101, the method extracts or accesses clinical trial information related to one or more clinical trials from one or more databases (e.g., multiple clinical trial databases). For example, an account hosted on computing device 205 may request an access token by sending a client ID and password to an identity service and receive an access token from the identity service that verifies the account's identity. The computing device can then use the issued access token to extract or access information related to clinical trials stored or hosted in clinical trial databases 201 and 202. A further explanation of obtaining access to one or more clinical trial databases is given below in relation to Figure 11.

[0050] The clinical trial information obtained may include drug study design information, biomarker information, drug therapy information, and efficacy information related to one or more clinical trials from one or more databases (e.g., clinical trial databases 201 and 202). Drug study design information may include, but is not limited to, a description of how the study was, is, or will be conducted, including one or more of the comparative group design and strategies for participant masking and assignment. Biomarker information may include, but is not limited to, measurable biological parameters that can be evaluated throughout the clinical trial as indicators of a normal biological, pathogenic, and / or pharmacological response to a particular drug. Efficacy information may include, but is not limited to, efficacy endpoints. For some disease areas, e.g., cancer, efficacy endpoints may, in some embodiments, include overall response rate (ORR), progression-free survival (PFS), and overall survival (OS).

[0051] After extracting or accessing relevant information about clinical trials from clinical trial databases 201 and 202, the method can generate datasets relating to clinical trial summaries, clinical trial groups, and baselines corresponding to clinical trials, clinical trial outcomes, and outcome comparisons between clinical trials, at least some or all of them. For example, according to some embodiments, the computing device 205 can generate multiple structured datasets including a trial summary dataset 203a, a trial group dataset 203b (Figure 2), and at least some or all of a baseline dataset 203c (Figure 2), an outcome dataset 203d (Figure 2), and an outcome comparison dataset 203e (Figure 2). Some clinical trials may not have baseline information available for generating the baseline dataset 203c. Some clinical trials may not yet have outcome information available for generating the outcome dataset 203d. Some clinical trials may not yet have outcome comparison information available for generating the outcome comparison dataset 203e. If regulatory information is available, the computing device 205 can also generate a regulatory information dataset 203f (Figure 2) according to some embodiments.

[0052] The trial summary 203a dataset may include study-level information extracted from clinical trial databases 201 and / or 202. In some embodiments, this information may include raw datasets related to the study itself. For example, in some embodiments, the raw dataset may include one or more of the following: a new clinical trial identifier (NCT ID) related to the drug in the clinical trial, the industry related to the drug development, the number of groups related to the clinical trial, the start year of the clinical trial, and the completion year of the clinical trial. In some embodiments, the raw dataset may also include information related to the primary completion year, which is the date on which the end participant was tested or received an intervention aimed at final collection of data on the primary outcome, regardless of whether the clinical study was completed or terminated according to a given protocol. In some embodiments, the raw dataset may also include the name and / or type of sponsor (e.g., academic institution, industrial institution, government agency). In some embodiments, the trial summary dataset 203 may also include information related to the drug administered throughout the clinical trial or information related to the clinical trial.

[0053] The trial group dataset 203b may contain treatment information relevant to the clinical trial. The baseline dataset 203c may contain any or all of the raw datasets from clinical trial databases 201 and / or 202, including the results for the participant group in the clinical trial, baseline measurements relevant to the participants within the clinical trial group, and baseline counts relevant to the clinical trial.

[0054] Outcome dataset 203d may include any or all of the following: measures related to clinical trial outcomes, counts related to clinical trial outcomes, information about the population of groups participating in the clinical trial, and drug information related to clinical trial outcomes. This information may include any or all of the following: drug event information, drug phase, and drug approval information.

[0055] Outcome comparison dataset 203e may include analytical data that compares outcomes across clinical trials.

[0056] In some embodiments, multiple structured datasets may also include safety datasets. Outcome comparison datasets may include comparison results between treatments for either efficacy or safety endpoints within the study. Pairwise comparisons can be performed between different treatments, for different clinical endpoints, and for different analytical populations; therefore, it is necessary to standardize relevant comparisons for potential pooled analyses. Clinical safety measures include, but are not limited to, adverse events and safety laboratory values, which can be pooled together for analysis.

[0057] Next, method 100 proceeds to step 102 to determine whether the extracted or accessed clinical trial information contains unstructured text. If method determines that the extracted or accessed clinical trial information contains unstructured text (yes), method 100 proceeds to step 104 to apply natural language processing to the unstructured text.

[0058] In step 104, method 100 applies a natural language processing module, e.g., natural language processing module 204, to unstructured text relating to clinical trial information stored or hosted in clinical trial databases 201 and 202. The output of natural language processing module 204 may include structured data based on the unstructured text extracted or accessed from clinical trial databases 201 and 202 according to some embodiments. Method 100 proceeds to step 105, in which structured data corresponding to at least one of several data types is generated for at least a portion of several structured datasets based on the output of natural language processing to the unstructured text. A further description of an example of a natural language processing module is included below with respect to Figure 9.

[0059] The structured data is generated corresponding to at least one of several data types, and Method 100 provides the user with a graphical user interface (GUI) that receives input from the user regarding an analysis to be performed using the multiple structured datasets (Step 110).

[0060] Returning to step 102, if method 100 determines that the extracted or accessed clinical trial information does not contain unstructured text (no), then in step 103, method 100 generates data corresponding to at least one of several data types for at least a portion of several structured datasets, based on the extracted or acquired clinical trial information.

[0061] In some embodiments, method 100 generates one or more variables for some structured datasets based on at least some of the data types. For example, the method can combine information from clinical trials stored in clinical trial databases 201 and 202 related to trial summaries and then generate variables that can filter the outcomes of all clinical trials to determine which outcomes are associated with the clinical trial data stored in clinical trial database 201 or clinical trial database 202. In some embodiments, the generated variables based on the data types are stored in an integrated clinical trial database.

[0062] In 110, a graphical user interface (GUI) 205 is provided to the user for analysis to be performed on or performed on one or more structured datasets 203. In some embodiments, the GUI may receive from the user information or criteria regarding data to be copied or accessed from different structured datasets 205. As an example, the GUI may receive a request from the user via the GUI 205 to determine the probability of a drug succeeding through Phase 3 of a clinical trial. In some embodiments, the GUI may receive from the user information to identify the drug and / or information regarding a set of parameters to be included in the analysis.

[0063] Figure 2 is a schematic diagram illustrating the logical information flow between exemplary databases, modules, and computing devices for generating and providing an integrated clinical trial database and performing analysis on datasets stored in the database (for example, to simulate clinical trial outcomes), according to some embodiments of the present disclosure. Flow 200 shows the relationship between clinical trial data stored in clinical trial databases 201 and 202 and one or more datasets (trial summary dataset 203a, trial group dataset 203b, baseline dataset 203c, outcome dataset 203d, outcome comparison dataset 203e, and regulatory information dataset 203f) generated from clinical trial information obtained from clinical trial databases 201 and 202 and stored in the integrated clinical trial database.

[0064] Examples of clinical trial databases include, but are not limited to, the Aggregate Content of Clinical Trials (AACT) database, a publicly available relational database containing all information (e.g., protocol and outcome data elements) about all studies registered on clinicaltrials.gov, as well as Informa Pharma Intelligence Trialtrove (Trialtrove) and Biomedtracker, databases that provide clinical trial data, including sets, clinical research information, and data from FDA clinical trials. In some embodiments, one or more additional clinical trial databases may be used. In some embodiments, clinical trial databases other than AACT or Trialtrove may be employed. In some embodiments, information about clinical trials may also be obtained from one or more publicly available sources other than clinical trial databases (e.g., news articles). In some embodiments, information about clinical trials may also be obtained from one or more internal sources within the organization (e.g., internal reports, internal databases, etc.).

[0065] One or more datasets can be stored in one or more integrated, standardized clinical trial databases 203g, which, according to some embodiments, may be stored in a server 203. Clinical trial databases 201 and 202 include clinical trial data that can be used to create any one or more structured datasets, or that can be manipulated in response to the application of the NLP module 204 to clinical trial data stored in clinical trial databases 201 and 202. For example, some of the clinical trial data stored in either clinical trial database 201 or 202 may be unstructured text data, and the NLP module 204 can use the unstructured text data to create meaningful structured data that can then be added to one or more datasets. In some embodiments, the structured data generated by the NLP module 204 may be data relating to variables already present in one or more datasets. For example, the NLP module 204 can generate structured data relating to outcome dataset 203d and / or outcome comparison dataset 203f based on unstructured data stored in one of the clinical trial databases 201 or 202. Unstructured data may be data associated with the event type and date of one data point in a clinical trial, and according to some embodiments, the NLP module 204 can determine outcomes or outcome comparisons based on the event type and date. Outcomes and outcome comparisons associated with the event type and the date on which the event occurred may, according to some embodiments, be additional data added to outcome dataset 203d and outcome comparison dataset 203f. An example of natural language processing according to some embodiments is described in more detail below with reference to Figure 9.

[0066] The user can use the GUI 205a on the computing device 205 to retrieve, interact with, and / or analyze data in any of the datasets. In some embodiments, the user can create a dataset based on data requested by the user, a dataset that may include additional data not included in any of the individual datasets discussed herein, or a dataset that may not include any data included in any one of the individual datasets discussed herein. The user can enter commands via the GUI 205a to generate an analysis of one or more datasets. In some embodiments, the analysis includes determining the probability that a drug will successfully complete all stages of a clinical trial.

[0067] Figure 3 is a chart containing various datasets that can be included in an integrated clinical trial database 203g for use by the user via the GUI 205a of the computing device 205 (see Figure 2). In some embodiments, datasets obtained from the integrated clinical trial database are used to simulate clinical trial outcomes. The trial summary dataset 303 describes general study attributes. Attributes may include a clinical trial identifier (e.g., NCT_ID), the name of the trial or study (e.g., Trial_title), the phase of the trial or study (e.g., stage), the number of groups included in the trial or study, planned or actual enrollment information provided in the protocol section of the trial or study record (e.g., timing variables), the design of the trial or study, and subgroup information that can be curated from patient segment information in the trial or study record. Subgroup attributes may be variables generated based on a combination of patient-related information.

[0068] According to several embodiments, the server 203 creates a trial summary dataset 303 by first integrating or standardizing specific clinical trial data from one or more clinical trial databases (e.g., clinical trial database 201 and clinical trial database 202), as described below with reference to Figure 4.

[0069] The trial summary 303 may also, according to some embodiments, include primary investigational drug data used in the clinical trial, as well as data on other drugs used in combination with the primary investigational drug or as comparator drugs with the primary investigational drug (these are also referred to herein as other involved drugs). Class data and subclass data are, according to some embodiments, coded into variables relating to both the primary investigational drug (e.g., drugclass_primary, drugsubclass_primary) and the other involved drugs (e.g., either the combination ingredient or the comparator drug) (e.g., drugclass_other, drugsubclass_other). According to some embodiments, the server can generate one or more of the variables included in the trial summary dataset 303, as shown in Figure 3, from information obtained from the clinical trial database. Each of these variables in the trial summary dataset will be discussed in detail below in the description of the exemplary embodiment in Figure 4.

[0070] The treatment group dataset 306 may, according to some embodiments, be generated based on AACT data stored in the clinical trial database 201. The treatment group dataset 306 may, according to some embodiments, be used to curate all treatment group and treatment subgroup-related data. This includes treatment groups per study focusing on one or more drug components.

[0071] Baseline dataset 304 may also include clinical trial patient segment variables such as disease or illness stage, disease or illness type (e.g., cancer type), therapy line, or gene expression. Baseline dataset 304 may also include variables related to the treatments offered in each group of clinical trials. In some embodiments, variables may include data on drug combinations used to treat a disease or illness, classifications of drug combinations used to treat a disease or illness, or subclassifications of drug combinations used to treat a disease or illness. The variables employed in a combined, standardized clinical trial database for a structured dataset are specified by the data type. Thus, treatment groups can be curated using information related to the treatments in each group of clinical trials.

[0072] In some embodiments, additional subgroup and dosage data for each group in the study can be curated and added as supplemental data to further enrich the dataset. In some embodiments, drug coding may be based on the World Health Organization Drug Dictionary (WHODD) standard coding. In some embodiments, if accurate coding for a drug is not included in WHODD, drug coding can be generated to better classify the drug. According to some embodiments, the server can generate one or more variables included in the study group dataset 306, as shown in Figure 3, from information obtained from the clinical trial database. Each of these variables in the study group dataset 306 will be discussed in detail below with reference to the description of the exemplary embodiment in Figure 5.

[0073] The baseline dataset 304 can be constructed based at least in part on information obtained from a clinical trial database (e.g., AACT data). In some embodiments, the baseline dataset 304 may include demographic data and baseline measurements for the entire population of participants in the clinical trial, as well as data for each clinical trial group or comparison group. Some of the information included in the baseline data may include descriptions of each baseline or demographic feature measured in the clinical trial. For example, the baseline data may include age, sex / gender, race, ethnicity (if collected according to the protocol), and any other measurements assessed at baseline and used in the analysis of primary outcome measures. According to some embodiments, the server can generate one or more variables included in the baseline dataset 304, as shown in Figure 3, from information obtained from the clinical trial database. Each of these variables in baseline will be discussed in detail below in the description of the exemplary embodiment in Figure 6.

[0074] To integrate baseline variables reported across different clinical trials, baseline category (bl_cat), baseline subcategory (bl_subcat), baseline subcategory level (bl_subcat_level), and baseline subcategory sublevel (bl_subcat_subblevel) variables can be derived from information obtained from clinical trial data in clinical trial databases 201 and 202 to encode arbitrary baseline variables using a hierarchical structure. In some embodiments, baseline data may be obtained from only a single clinical trial database (e.g., only AACT).

[0075] Server 230 creates an outcome dataset 305 by first integrating or standardizing specific data related to clinical trial outcomes reported for each group of clinical trials obtained from clinical trial databases 201 and 202, as described below with reference to Figure 7. Some of the data related to clinical trial outcomes may, according to some embodiments, include efficacy data such as the overall response rate (ORR), progression-free survival (PFS), or overall survival (OS) endpoint variables observed in patients during the clinical trial.

[0076] ORR, PFS, or OS variables are exemplary efficacy endpoints used in oncology-related studies. For other disease areas, variables corresponding to different efficacy endpoints may be adopted. Relevant endpoints may be extracted and based on information obtained from one or more clinical trial databases (e.g., AACT clinical trial data or Biomedtracker clinical trial data). Other data that may be included in the outcome305 dataset may include dictionaries used to define the outcomes of the clinical trial. The outcome305 dataset may also include patient segment information of the trial group, which, according to some embodiments, can be based at least in part on information from clinical trial databases (e.g., AACT clinical trial data and Trialtrove clinical trial data) regarding group variables. In some embodiments, variables may include data on drugs or drug combinations used to treat a disease or disorder, classifications of drug combinations used to treat a disease or disorder, or subclassifications of drug combinations used to treat a disease or disorder. According to some embodiments, the server can generate one or more variables included in the outcome dataset 305, as shown in Figure 3, from information obtained from clinical trial databases. Each of these variables in the outcome dataset will be discussed in detail below in the description of the exemplary embodiment relating to Figure 7.

[0077] Server 230 creates an outcome comparison dataset 307 by first integrating or standardizing specific data related to the outcome comparison results between treatment groups in clinical trials from information obtained from clinical trial databases 201 and 202, as described below with reference to Figure 8. Some of the data may include variables related to one or more parameter value types (e.g., treatment difference, treatment rate, odds ratio, etc.), p-values ​​related to the clinical trial, confidence intervals related to the clinical trial, and statistical testing methods used in the clinical trial. According to some embodiments, the server can generate one or more variables included in the outcome comparison dataset 307, as shown in Figure 3, from information obtained from the clinical trial databases. Each of these variables in the outcome comparison dataset 307 will be discussed in detail below in the description of the exemplary embodiment in Figure 8.

[0078] As described above, the server 203 creates a trial summary dataset 303 by first integrating or standardizing clinical trial information obtained from one or more clinical trial databases (e.g., both clinical trial database 201 and clinical trial database 202) related to the drugs used in the clinical trial. The server 203 can initiate the process of standardizing the clinical trial information used to summarize the study by extracting data related to specific variables stored in clinical trial databases 201 and 202 that capture specific attributes of the study. In the exemplary embodiment shown in Figure 4, the server 203 generates a trial summary dataset (Trial_Summary402) by standardizing or integrating information on variables related to AACT data 301, Trialtrove data 302a, and BMT data 302b.

[0079] In some embodiments, the variables in the trial summary dataset may be a default set of variables based on information that can be accessed or extracted from clinical trial databases 201 and 202. These variables may be presented to the user via a GUI. In some embodiments, the GUI can receive input from the user instructing the server 203 to extract data from clinical trial databases 201 and 202 related to other variables not included in the list of variables related to the AACT301 data, Trialtrove302a data, and BMT302b data. In other embodiments, the GUI can receive input from the user instructing the server 203 to remove variables related to the AACT301 data, Trialtrove302a data, and BMT302b data from the trial summary.

[0080] Figures 4–8 illustrate data from the AACT, Trialtrove, and Biomedtracker (BMT) databases. In view of this disclosure, it will be understood by those skilled in the art that embodiments using other clinical trial databases with other data formats or data types are also within the scope of the present invention.

[0081] Figure 4 is a logical diagram 400 illustrating a process for generating a clinical trial summary 303 dataset according to several embodiments of the present disclosure. This process can begin with a server 203 connecting to clinical trial databases 201 and 202, and the server 203 extracting raw summary data related to AACT clinical trial data 301. Data related to AACT clinical trial data 301 may include the NCT ID related to the clinical trial, the number of arms related to the clinical trial, the start year of the clinical trial, the completion year of the clinical trial, the first year of completion of the clinical trial (primary completion year), the country, state, or region in which the clinical trial is being conducted, the name of the investigator conducting the clinical trial, the title of the clinical trial (trial title), and the overall status of the clinical trial (overall status).

[0082] The data associated with AACT clinical trial data 301 may also include the name of the sponsor (sponsor) and the type of sponsor (sponser_type). The type of sponsor may be an industry sponsor, an academic sponsor, or a government sponsor. AACT clinical trial data 301 may further include the time when enrollment was completed (enr_comp_time). The data associated with AACT clinical trial data 301 may also include the estimated total number of participants to be enrolled (target number) or the actual total number of participants enrolled in the clinical study (enrl_num).

[0083] AACT clinical trial data 301 may also include outcome-type variables that can be used to filter a set of obtained event types related to the clinical trial. Event types may include data related to clinical trial outcomes, data related to reported events in the clinical trial, and data related to the participant flow in the clinical trial. Data related to reported events may include summary information on reported adverse events (any unfavorable or undesirable medical events in a participant, including abnormal physical examination results, laboratory findings, symptoms, or diseases), including serious adverse events, other adverse events, and deaths. Data related to the participant flow in the clinical trial may include recruitment data related to the recruitment process and pre-assignment details (i.e., significant events in the study that occur after participant enrollment but before participant assignment). Data related to the participant flow includes data on the participant flow that applies to all milestones in the clinical trial.

[0084] Server 203 can set outcome variables to outcomes so that outcome data related to clinical trials are extracted from clinical trial databases 201 and 202 associated with NCT IDs.

[0085] Server 203 can also extract raw data related to Pharma Intelligence clinical trial data from clinical trial databases 201 or 202. The raw data may include Trialtrove data 302a and BMT data 302b. The raw data may include variables related to the clinical trial identifier (ID) (trialId), the clinical trial title (trialTitle), and the clinical trial status (trialStatus). The raw data may also include the clinical trial start date (trialStartDate), the clinical trial design (trialStudyDesign), and the clinical trial last modified date (trialLstModifiedDate).

[0086] Raw data may further include data related to the outcomes of the clinical trial (trialOutcomeDetails), data related to the results of the clinical trial (trialResults), and any notes related to the clinical trial (trialNotes).

[0087] The raw data may also include the primary endpoint of the clinical trial (trialPrimaryEndpoint) and any other endpoints related to the clinical trial (trialOtherEndpoint). In addition, the clinical trial data may include a list of the sponsors of the clinical trial (trialSponsors) and specific therapeutic areas (trialTherapeuticAreas) (e.g., oncology). In some embodiments, the clinical trial data may include the first primary drug set tested during the clinical trial (trialPrimaryDrugsTested). In some embodiments, the clinical trial data may also include one or more of the following: a second primary drug set tested during the clinical trial (bmtPrimaryDrugsTested), a first alternative drug set different from the first primary drug set tested during the clinical trial (trialOtherDrugsTested), and a second alternative drug set different from the second primary drug set tested during the clinical trial (bmtOtherDrugsTested).

[0088] Pharma Intelligence clinical trial data may also include a unique identifier (DrugID) used to identify specific drugs included in the Pharma Intelligence clinical trial data, as well as any brand name associated with the DrugID (brandName), and an indication group (drug indication + pivotal study (Yes / No), e.g., Breastcancer_Pivotal:Yes).

[0089] Server 203 can generate disease subtype data (e.g., data related to non-small cell lung cancers such as melanoma, squamous cell carcinoma, and adenocarcinoma) based at least in part on data related to the design of the clinical trial (TrialStudyDesign) and patient segment data related to the clinical trial. Therapy lines for treating the disease corresponding to the various disease subtypes identified in the disease subtype data. In addition to determining disease subtype data and data related to the design of the clinical trial, Server 203 can also generate gene expression data related to the disease corresponding to the various disease subtypes identified in the disease subtype data.

[0090] After server 203 extracts AACT clinical trial data and Pharma Intelligence clinical trial data from clinical trial databases 201 and 202, server 203 can apply a text mining keyword matching algorithm to extract data related to cancer subtype, gene expression, stage, and treatment line variables based on the patientSegment variable. For example, if the patientSegment column contains information related to various disease subtypes, such as "Adenocarcinoma," "Squamous Cell," and "Large Cell," server 203 can group the various disease subtypes with a single value, which may be represented as Adenocarcinoma|Squamous Cell|Large Cell. Similarly, if the patientSegment column contains information related to various disease stages, such as "Stage I" and "Stage II" of cancer, server 203 can group the various disease stages with a single value, which may be represented as Stage I|Stage II. Server 203 can also apply a text mining keyword matching algorithm to gene expression and therapy line variables to generate single values ​​for both gene expression and therapy line variables. In response to the application of the text mining keyword matching algorithm to the combined AACT clinical trial data and Pharma Intelligence clinical trial data, Server 203 can generate indicators that identify the data sources from which data related to cancer subtype, gene expression, stage, and therapy line variables were extracted. This can be done to track the source of the type of data associated with the patientSegment.

[0091] Because drug information is not integrated between AACT clinical trial data and Pharma Intelligence clinical trial data due to the complexity of drug naming and mechanism of action classification used by different entities conducting clinical trials, in some embodiments, a drug dictionary is created as a standard reference for coding drug names and classifying mechanisms of action.

[0092] Server 203 generates a set of variables that can be used to uniquely describe the drugs being administered in the clinical trial, based at least in part on the trialPrimaryDrugsTested variable, trialOtherDrugsTested variable, bmtprimarydrugstested variable, and bmtotherdrugtested variable.

[0093] Server 203 expands `trialPrimaryDrugsTested` and extracts variables such as `drugId`, `drugPrimaryName`, `drugName`, and `mechanismOfAction`, `mechanismSynonyms`, and `directMechanism` related to the primary drug administered during the clinical trial corresponding to the `Trialtrove` clinical trial data. Server 203 can also expand `trialSecondaryDrugsTested` and extract variables related to other drugs in the `Trialtrove` data. Server 203 can expand the `bmtprimarydrugstested` variable and extract variables such as `drug ID`, `brandname`, and `Target` related to the primary drug administered during the clinical trial corresponding to the `BMT` clinical trial data. Server 203 can also expand the `bmtotherdrugtested` variable and extract `drug ID` related to other drugs administered during the clinical trial corresponding to the `BMT` clinical trial data.

[0094] Server 203 can merge all four non-nested datasets using drugid. The result of the complete join of the Trialtrove dataset and the BMT dataset is a complete list of all drugs designed to address a disease (e.g., variable All.Names), which is included in the first subset of Pharma Intelligence clinical trial data and the second subset of Pharma Intelligence clinical trial data.

[0095] The Target, directMechanism, mechanismSynonyms variables, etc., can be used to assist in classifying drugs using drug dictionaries such as the World Health Organization (WHO) Drug Dictionary.

[0096] After a complete list of drugs is created and a predetermined list of variables is selected by the server 203, the list of drugs needs to be coded according to a drug dictionary. The drug dictionary may be stored locally on the server 203 according to some embodiments. The drug dictionary includes data related to Trialtrove clinical trial data and data related to BMT clinical trial data. Drug coding is then performed as drug classes and drug subclasses according to either WHODD or designated disease-specific rules.

[0097] Each drug can be classified into a specific class and subclass using an established drug dictionary having similar or identical names, according to some embodiments. Server 203 can extract trial group data related to each drug being studied in each clinical trial and, if multiple drugs are used in a group, can generate combination drugs used in each trial. In the merged drug dictionary, each therapeutic drug and drug combination is coded and classified into drug combo classes and drug combo subclasses. Logical diagram 500 in Figure 5 illustrates the process by which Server 203 creates clinical trial group datasets to add classifications and subclassifications to drugs used as treatment in a trial group.

[0098] Server 203 can initiate the process by loading study group data from the clinical trial database 201, which may, according to some embodiments, include information about various design groups, interventions, terms, phrases, or names synonymous with specific interventions, and interventions applicable to various design groups. Server 203 can extract design group data from the clinical trial database 201 related to study groups. The data may include information about protocol-designated groups, subgroups, or cohorts of participants in a clinical trial who are assigned to receive specific interventions or observations in accordance with a protocol that is followed for the administration of a specific drug to a participant.

[0099] Server 203 can extract intervention data related to test groups from the clinical trial database 201. The data may include, but are not limited to, specific medical interventions or exposures related to the clinical trial or to a test group or group within a test group, including drugs, medical devices, procedures, vaccines, and other notable products. Server 203 can extract terms, phrases, or names from the clinical trial database 201 that are synonymous with interventions applied to various design groups. Each term, phrase, or name is associated with one of the interventions related to the clinical trial. For example, Server 203 can extract three different names or phrases, each associated with a specific intervention related to a test group.

[0100] According to several embodiments, the server 203 can extract design group-specific intervention data from the clinical trial database 201 related to specific interventions applied to a particular group in a clinical trial. The design group-specific intervention data functions as a cross-reference between groups and the corresponding interventions applied to those groups. For example, if a clinical trial has multiple groups and multiple interventions, the design group-specific intervention data can, according to several embodiments, specify which intervention is associated with which group.

[0101] Server 203 can also be programmed to extract specific variables related to intervention data. For example, Server 203 can identify and extract data related to the NCT ID of the clinical trial, data describing the type of intervention applied to the trial group, and interventions may include specific procedures along with drugs, biological interventions, radiotherapy, etc. Additional intervention data that Server 203 can extract may include the names of drugs used during the clinical trial and descriptions of the drugs.

[0102] Server 203 can also be programmed to extract design group-specific variables, including the NCT ID, the group title, and the group description. The group description may include information identifying the role of the intervention received by the participants and / or may include, but not limited to, experimental groups, active control groups, placebo control groups, sham treatment control groups, and no-intervention groups.

[0103] Next, according to some embodiments, the server 203 combines data associated with various design groups, interventions, terms, phrases, and / or names used to refer to specific interventions, and information corresponding to interventions applied to various design groups, thereby creating a table containing all of the extracted variables described above. The server 203 can combine data by merging data associated with various design groups, interventions, terms, phrases, or names used to refer to specific interventions, and information corresponding to interventions applied to various design groups, using a common key variable that may include either an NCT ID, an identifier associated with an intervention, an identifier associated with an intervention, or an identifier associated with a design group. The resulting table (drug table) is then merged with a drug dictionary, thereby assigning drug classifications and subclassifications to the drugs administered to the study groups, as described below.

[0104] According to several embodiments, the server 203 can select variables associated with the NCT ID, names associated with drugs administered to the study groups, identifiers associated with each design group, drug descriptions, titles associated with study groups, study group descriptions, and the types of groups included in the study groups from the drug table, and can group the data related to the table resulting from the fusion of the drug table and the drug dictionary according to the selected variables. The types of groups included in the study groups may be either experimental or control groups. As seen in Figure 5, according to several embodiments, the AACT 301 may include variables such as NCT ID, group type, design group ID associated with each design group, name, title associated with study groups, intervention type, drug intervention associated with the intervention type, and drug description.

[0105] The drug dictionary 401 may contain a list of all different names (All_Names) associated with the drugs administered to the test group. The drug dictionary 401 may also contain classification information (drugclass) and subclass information (drugsubclass) associated with each drug included in the drug dictionary.

[0106] Next, according to some embodiments, the server 203 can tokenize the name variables in the drug table and merge the drug table and the drug dictionary based on the tokenized and recasted name variables in the drug dictionary. Then, the drugs in the drug table are coded into drug classes and drug subclasses.

[0107] After server 203 merges the revised drug table and the revised drug dictionary, the data in the table obtained from the merge of the revised drug table and the revised drug dictionary is grouped together by the variables associated with the NCT ID from the drug table, the names associated with the drugs administered to the study groups, the identifiers associated with each design group, the drug descriptions, the titles associated with the study groups, the study group descriptions, and the various study group types.

[0108] After the server 203 groups the data in the obtained table according to the aforementioned variables, it can generate a variable associated with the most commonly used drug name for the drug within the group, according to some embodiments. This variable can be represented as drugname_combo, as shown in Trial_Arm501, by combining each drug component of Trial_Arm. This drugname_combo variable can be coded as drugclass_combo and drugsubclass_combo, as shown in Trial_Arm501. Furthermore, Trial_Arm501 may include NCT ID (NCT_ID), trial group title (Ta_arm_title), and group type (group_type).

[0109] Each clinical trial may have corresponding baseline data or outcome data, depending on several embodiments. Baseline data may include data collected at the start of the clinical trial for all participants and each trial group. This data may include, but is not limited to, demographic information such as age, sex / gender, race and ethnicity, as well as study-specific measures (e.g., systolic blood pressure, previous antidepressant treatment, etc.). Server 203 can extract baseline data relevant to the clinical trial according to logical diagram 600 in Figure 6.

[0110] Server 203 can initiate the process of extracting baseline data relevant to a clinical trial by first determining whether baseline data exists for the clinical trial. For some clinical trials, baseline data may not exist, for example, because the clinical trial has been discontinued or has not yet started. According to some embodiments, Server 203 can determine whether any baseline data exists in the clinical trial database 201 by attempting to access the baseline data stored in the clinical trial database 201.

[0111] If the server 203 determines that baseline data for a clinical trial exists, it can extract baseline data from the clinical trial database 201. Baseline data related to a clinical trial may include outcome data related to one or more participant groups. For example, outcome data related to one or more groups may, according to some embodiments, include a combined or aggregated list of group titles and descriptions that can be used to report summary outcome information.

[0112] Outcome data associated with one or more participant groups may include an NCT ID, which is an outcome type variable that can indicate the outcome of the clinical trial. For example, according to some embodiments, the value of the outcome type variable may be "Baseline," "Outcome," "Reported Event," or "Participant Flow." In some embodiments, when the server 203 extracts data associated with the outcome type variable from the clinical trial database 201, it may be programmed to create a filter to extract only specific outcome type data, such as baseline outcome type data. Outcome data associated with one or more participant groups may also include, according to some embodiments, a title associated with the baseline of the study group containing one or more participant groups, and a description of the study group baseline. The description of the study group baseline may include data collected at the start of the clinical trial for all participants and each study group or comparison group. This data may include demographic information such as age, sex / gender, race and ethnicity, and study-specific measures (e.g., systolic blood pressure, previous antidepressant treatment, etc.).

[0113] As described above, according to some embodiments, the baseline data may further include a set of baseline measurements, which may include the NCT ID, a title associated with the baseline of the test group, a description of the baseline of the test group, a classification, a set of units, a parameter type, a value associated with the parameter type, a number associated with the parameter value, a variance type, a variance value associated with a particular type of variance, a lower or upper limit on the variance, or an upper or lower limit on the variance.

[0114] As described above, baseline data may also include a set of baseline count information, according to some embodiments. In some embodiments, the baseline may be generated from data obtained from a single clinical trial database (e.g., AACT). In some embodiments, baseline measurements and baseline counts are used to generate baseline data. The corresponding baseline data is shown in the variable list AACT301 in Figure 6.

[0115] In some embodiments, the server 203 can merge the variable list AACT301 with the study summary 402 to create a relationship between baseline data and disease subtype classification, as well as the disease stage in participants, the therapy line used to treat participants, and the expression of disease-related genes in participants. For example, the disease could be a type of cancer, the stage could include the stage of the cancer type for all participants, the therapy line could include a specific order in which different therapies are given to participants as the disease progresses through different stages, and the gene expression could include genes that are expressed or not expressed in relation to the disease. After merging the variable list AACT301 with the study summary 402, the server 203 can further merge the study group 501 into the merged variable list AACT301 and study summary 402 using the NCT ID, according to some embodiments. The server 203 merges the study group 501 into the merged variable list AACT301 and study summary 402 to determine whether there is similarity between the title associated with the study group 501 and the title associated with the baseline of the study group 501. Server 203 can compare the titles associated with study group 501 by tokenizing them with the baseline of study group 501, and then compare the number of tokenized words in the titles to determine whether the two titles are an exact match, a close match, or similar to each other. According to some embodiments, Server 203 compares the number of tokenized words in the baseline group title with the number of tokenized words in the same study group 501 title and determines that the number of tokenized words associated with study group 501 is the largest number of tokenized words that are the same as the tokenized words in the baseline study group title. Server 203 can then add information from the study group that has the largest number of tokenized words in common with the baseline study group title to the baseline dataset. For example, in some embodiments, patient segmentation information available in the summary data is then added to the baseline dataset for future patient subgroup identification.

[0116] Server 203 can also, according to some embodiments, determine whether the title of a study group includes phrases such as "all participants" or "total" that indicate whether all participants are included in the study group. Determining that the title includes phrases synonymous with these indicates whether the number of baselines includes all participants. If the title includes phrases that indicate all participants are included in the study group, Server 203 can create a flag indicating that the study group includes all participants. If the title does not include phrases that indicate all participants are included in the study group, Server 203 can, according to some embodiments, set a flag to a value that indicates the study group does not include all participants. This flag is sometimes called the total population flag and can be represented by (tot_pop_flg) in the baseline 602 dataset.

[0117] To unify the naming of baseline variables across various sources, a hierarchical structure is used to uniformly code them, according to some embodiments. Three variables can be exported from the AACT table, these being bl_variable, bl_description, and classification, from which bl_cat1, bl_cat2, bl_level, and bl_class_std are derived. This dataset may be named baseline_code_dictionary_v4. Furthermore, the exported variables baseline_cat1 can be renamed to bl_cat, bl_cat2 to bl_subcat, bl_level to bl_subcat_level, and bl_class_std to bl_subcat_sublevel.

[0118] After completing the various merges described above, server 203 can generate a baseline 602 dataset as shown in Figure 6. The server can receive requests from computing device 205 via GUI 205a for data contained in the baseline 602 dataset stored in server 203, and server 203 can send the requested data to computing device 205.

[0119] According to some embodiments, when the value of the result type variable is set to "Outcome", the server 203 may be programmed to extract only outcome result type data when extracting data associated with the result type variable from the clinical trial database 201 that matches the logical diagram 700 in Figure 7. Outcome data related to a clinical trial may include outcome data related to one or more participant groups. For example, outcome data related to one or more groups may include, according to some embodiments, a combined or aggregated list of group titles and descriptions that can be used to report summary outcome information. Summary outcome information may include a summary of the results related to the clinical trial. Outcome result type data may include an NCT ID associated with the clinical trial, an identifier (result_group_id) associated with the results of one or more participant groups observed in the clinical trial, a title (oc_arm_title) associated with the trial group corresponding to the outcome data, and a description (oc_arm_description) of the trial group corresponding to the outcome data.

[0120] Outcome data may further include, according to some embodiments, a set of outcome measures, which may include summary data relating to primary and secondary outcome measures for each group included in the clinical trial. Summary data may include parameter estimates (e.g., overall response rate, median progression survival, etc., in some embodiments) and measures of variance or precision. Outcome data may further include outcome count information, which may include sample size or counts included in the analysis of each outcome for each group in the clinical trial. The count or sample size may represent the number of participants, but may also represent other units of measurement, such as eye "lesions" or some other physical characteristics of the participants. Outcome data may further include outcome populations, which are patient populations related to the outcome measures. For example, in some embodiments, this includes patients with high PD1 expression. The aforementioned outcome data can be extracted from the clinical trial database 201 by the server 203.

[0121] Data related to a set of outcome measures can be represented through several variables. A set of outcome measures may be characterized by a title (oc_title), a description of the clinical trial outcome (oc_description), and the units used to capture during the clinical trial (for example, in some embodiments, this includes median progression-free survival in weeks, months, or years). A set of outcome measures may, according to some embodiments, be further characterized by a parameter type, a value associated with the parameter type, a number associated with the parameter value, a variance type, a variance value associated with a particular type of variance, a lower or upper limit on the variance, or an upper or lower limit on the variance.

[0122] After extracting different outcome data from the clinical trial database 201, the server 203 can, according to some embodiments, merge the different outcome data together using the NCT ID, an identifier associated with the results of a group of one or more participants observed in the clinical trial, and / or an identifier associated with the outcome.

[0123] Server 203 can extract additional outcome data from the clinical trial database 202, which may include information related to one or more drug efficacy endpoint measures corresponding to the clinical trial. The event data table, represented as the variable eventDataTable, may, according to some embodiments, include information about the group description or the treatment administered to the group of participants. This information can be represented as the variable (oc_arm_title). The reason the information can be represented as the variable oc_arm_title is that the group description is often included in the group title. The event data table may also include information about the number of participants who received the treatment. This information can be represented as the variable oc_sample size, also known as the count of participants. The results data table, represented as the variable dataTableResults, may include information related to the description of the efficacy endpoint for the group. The description can be represented as the variable oc_title.

[0124] Server 203 can combine the data extracted from clinical trial databases 201 and 202 by simply stacking the extracted data together. The resulting dataset can be represented as the variable result_sub.

[0125] The eventType variable may include various types of results related to the clinical trial, including Top Line Results, Final Results, Published Results, and Updated Results, which indicate the statistical significance of the trial results regardless of the stage or phase of the trial. Final Results may be results related to the completion of the clinical trial. Published Results may include results related to the completion of the clinical trial that have also been published. Updated Results may or may not be published, but may include new result data that did not exist before the creation of the Updated Results. Server 203 can extract one or more drug efficacy endpoints from any of the aforementioned results. For example, Server 203 can extract the overall response rate (ORR), progression-free survival (PFS), and / or overall survival (OS) efficacy endpoints from one or more of the aforementioned results.

[0126] Server 203 can identify clinical trials containing efficacy endpoints by determining whether the outcome title variable (oc_title) in result_sub for each clinical trial contained in result_sub contains keywords related to ORR, PFS, or OS, for example. Server 203 can identify and integrate outcomes by comparing words in each title corresponding to a clinical trial containing an outcome variable with keywords, such as the phrase "overall survival" and other relevant keywords. Server 203 can then determine, in some embodiments, that a clinical trial contains, for example, overall survival data related to that clinical trial. Similar algorithms can be applied to other trial outcomes, such as overall response rate, partial response rate, duration of response, etc.

[0127] After performing the keyword search described above, the server 203 can code efficacy endpoints across different clinical trials, according to several embodiments. The coding of efficacy endpoints corresponds to classifications and subclassifications. Classifications can be represented as outcome class variables (oc_class) and outcome subclassification variables (oc_subclass). The server 203 generates an outcome dictionary that includes both outcome class variables and outcome subclass variables.

[0128] The outcome dictionary can be generated, according to several embodiments, by excluding data related to the outcome time variable oc_title extracted from clinical trial databases 201 and 202, and by excluding data related to the outcome classification variable (oc_classification) extracted from clinical trial database 201. Outcome classes and outcome subclasses are defined by the needs of different disease areas and analytical needs, and can be updated when it is necessary to add new endpoints of note. In addition to comparing keywords related to the efficacy endpoints listed above with words in the outcome title oc_title, server 203 can also compare the same keywords with words included in the outcome explanation variable oc_description. Each efficacy endpoint corresponds to a classification in the outcome dictionary, and the class in the outcome dictionary can be represented as oc_class. The outcome dictionary also includes corresponding outcome subclassifications, and efficacy endpoints also correspond to outcome subclassifications.

[0129] In addition to the oc_title variable, the outcome dictionary may include the outcome explanatory variable oc_description and the outcome classification variable (oc_classification). After the outcome dictionary is generated by server 203, server 203 can merge the outcome dataset variable result_sub with the outcome dictionary for the oc_title, oc_description, and oc_classification variables. In Figure 7, the merge can be illustrated by joining the AACT301 and BMT302b data with outcome_variable_dic702 to create or add oc_class and oc_subclass variables to the outcome 701 dataset obtained from the merge, in order to classify the outcome variables by dictionary.

[0130] As shown in Figure 7, according to some embodiments, the list of variables included in the AACT301 data may include any or all of the following variables: oc_classification, oc_description, oc_dispersion_type, oc_dispersion_value_num, oc_dispersion_value_num, oc_dispersion_lower_limit, oc_dispersion_upper_limit, time_frame, and population (which may also be expressed as oc_population). According to some embodiments, the AACT301 data may further include NCT ID, sample_size, oc_units, oc_title, oc_param_value_num, oc_type, oc_arm_title, and oc_arm_desc. The AACT301 data is data extracted from the clinical trial database 201. The BMT302b data may include one or all of the following variables: eventType, eventDate, eventPhase, drugPhase, likelihoodofApproval, averageApprocal, changetoLikelihoodApproval, numberOfEvaluablePatients, and Endpoint_description, as well as treatment_descr (in Figure 7).

[0131] `numberOfEvaluablePatients` corresponds to the sample size or count of participants in the clinical trial. Since this value is the same as the count or sample size in the AACT301 data, server 203 can store the sample size, count, and the numerical value associated with the `numberOfEvaluablePatients` variable in the `sample_size` variable. The `Endpoint_description` variable can be associated with the `oc_title` variable, as the outcome title can contain words related to one or more efficacy endpoints, as described above. Therefore, server 203 can store the description provided in the `Endpoint_description` variable in the `oc_title` variable. The `treatment_descr` variable may contain a description of a treatment group similar to the description of a treatment group associated with the `oc_arm_desc` variable, and therefore, server 203 can store the description stored in the `treatment_descr` variable in the `oc_arm_desc` variable.

[0132] After the server 203 merges the outcome dictionary with the result_sub variable, according to some embodiments, the resulting dataset can be merged with the trial group 501 using the NCT ID to generate an outcome 701 dataset as shown in Figure 7. After merging the trial group 501 with the resulting dataset, the server 203 tokenizes the outcome trial group title variable (oc_arm_title) and the treatment or trial group title (arm_title), and the outcome trial group title and trial group title can be merged based on the similarity of the title names.

[0133] To extract genesub and cancersub, a program can be used to generate three functions (e.g., R functions) such as match_gene, match_sign, and match_cancer from oc_arm_title, trial_title, oc_description, oc_arm description, and oc_classification.

[0134] PopulationSub can be extracted, according to several embodiments, using a keyword search algorithm on variables such as (in order) group title, group description, outcome title, title, classification, and description.

[0135] In response to a user request for outcome data related to one or more relevant clinical trials, the server 203 can display the outcome 701 dataset on the display of the computing device 205 via the GUI 205a.

[0136] Figure 8 is a logical diagram illustrating a process 800 for generating a clinical trial outcome comparison dataset according to several embodiments of the present disclosure. The outcome comparison dataset may include statistical analysis results such as odds ratios, hazard ratios, response rate differences, risk differences, risk ratios, or other estimated parameters for comparisons between different treatments. Server 203 can extract comparison results from clinical trial database 201, extract comparison results from clinical trial database 202, and stack them vertically.

[0137] Server 203 can send requests to the clinical trial database 201 to access and extract data related to statistical outcome analyses performed on primary and secondary clinical trial outcomes. According to some embodiments, the data may include results, confidence intervals, and other measures of variance related to treatment effect estimations, as well as p-values. This data may be referred to as outcome analysis data and may be represented in variable form as outcome_analysis. Server 203 can also send requests to the clinical trial database 201 to access and extract data identifying the comparison groups involved in each statistical outcome analysis. This data may be referred to as outcome analysis group data and may be represented in variable form as outcome_analysis_group. Server 203 can also send requests to the clinical trial database 201 to access and extract data related to the combined or aggregated list of group clinical trial titles and descriptions used to report summary result information. Server 203 can also send requests to the clinical trial database 202 to access and extract data related to outcomes of clinical trial data stored in the clinical trial database 202.

[0138] Outcome analysis group data may include NCT IDs associated with different clinical trials, identifiers associated with outcome analyses of different clinical trials, and identifiers associated with different participant groups involved in comparing outcome results of different participant groups.

[0139] Outcome analysis data may include an NCT ID, an identifier associated with the outcome analysis which may be represented as the variable outcome_analysis_id, and the type of statistical test applied to the clinical trial data to analyze the clinical trial data. The statistical test may be one or all of the following, depending on the embodiment: a superiority test, a non-inferiority test, or an equivalence test. The outcome analysis data may also include a description of the statistical test.

[0140] Outcome analysis data may also include, according to some embodiments, a dispersion variable that measures the variance of clinical trial group data for a particular group relative to another group. One example is the standard error of the mean. The dispersion variable can be represented as dispersion_type. Outcome analysis data may also include, according to some embodiments, the corresponding value of the dispersion variable. Outcome analysis data may also include, according to some embodiments, a modifier, symbol, or sign that indicates whether the p-value related to a statistical test is less than or greater than a certain value. The modifier can be represented as the variable p_value_modifier. Outcome analysis data may also include a p-value variable related to a statistical test, which can be represented as p_value.

[0141] Outcome analysis data may also include information related to the confidence intervals of statistical tests according to some embodiments. For example, outcome analysis data may include a variable associated with the number of sides (1 or 2) of the confidence interval, which can be represented as ci_n_sided. Additional information related to the confidence interval may include the level associated with the confidence interval, which can be represented as a percentage and captured by the variable ci_percent. A lower limit, which can be represented as the variable ci_lower_limit, and an upper limit, which can be represented as the variable ci_upper_limit, related to the confidence interval, are also included in the outcome analysis data according to some embodiments.

[0142] Outcome analysis data may also include, according to some embodiments, the statistical methods used to analyze clinical trial group data. For example, the methods may be statistical tests used to calculate p-values. This can be expressed as a variable method and may include any of the following tests: ANCOVA, ANOVA, chi-squared, corrected chi-squared, Cochran-Mantel-Haenszel, Fisher Exact, Kruskal-Wallis, LogRank, Mantel-Haenszel, McNemar, mixed model analysis, regression, Cox regression, linear regression, logistic regression, signed test, one-sided t-test, two-sided t-test, Wilcoxon (Mann-Whitney), or any other statistical test.

[0143] Outcome analysis data may also include information about the procedures used to estimate the effect of the intervention, which, according to some embodiments, may be, for example, statistical tests of hypotheses. Information about the procedures used to estimate the effect of the intervention can be represented as the variable estimate_description. Outcome analysis data may also include group descriptions, which, in some embodiments, can be represented as the variable groups_description, for example, the intended treatment group, the safety group, or the high PD1 expression group.

[0144] Server 203 can also select variables included in outcome data 70: nct_id, oc_arm_title, oc_geneSub, oc_geneSign, oc_cancerSub, populationSub, drug_name_combo, drug_intervention, drugclass_combo, and drugsubclass_combo. According to some embodiments, Server 203 can also extract outcome analysis-related data from clinical trial database 202 and NCT_ID, Title, result_type, and group_type from clinical trial database 201.

[0145] For BMT outcome analysis, server 203 can consider the result_sub raw data and filter the result_sub raw data with respect to the variables result_type="BMT_outcome" and group_type="difference". In some embodiments, the BMT and AACT comparison dictionaries are generated separately using the outcome dictionary generation approach summarized above. The bmt comparison is joined with the bmt comparison dictionary for outcome_title. The bmt comparison dataset is then joined with the outcome table for nct_id to revert the test group titles and group variables. For AACT, analysis_group.rdata and analysis rdata data are merged using NCT_ID, ctgov_group_code, and result_group_id, and then merged with result_sub for NCT_ID to obtain oc_arm title. In some embodiments, the outcome_bmt comparison data is stacked with the aact comparison data.

[0146] The server merges the outcome comparison dataset with the final outcome dataset for oc_title and NCT_DI, and obtains arm title and arm title2 as important variables for obtaining drug_level information.

[0147] Figures 4-8 are used for illustrative purposes only, and those skilled in the art will understand that any other variable names may be used in light of this disclosure.

[0148] Figure 9 is a logical diagram illustrating the process of applying natural language processing to unstructured text stored in a database to extract clinical trial outcomes and clinical trial outcome comparison data, according to some embodiments of the present disclosure. The natural language processing may include one or more computer-readable instructions, which, when executed by the server 203, cause the server to send a prompt to a clinical trial database or databases 201 and / or 202 requesting clinical trial data related to outcomes or results corresponding to the clinical trial. The requested clinical trial data may include unstructured text describing the outcomes or results of the clinical trial. When the server 203 executes the computer-readable instructions of natural language processing, it can detect sections in the unstructured text corresponding to design and result information related to the clinical trial. The server 203 may enter authentication information, such as a user's username and password, into a web page and request raw data or pre-configured datasets disclosed herein from a database (e.g., Trialtrove901) that hosts the raw data or pre-configured datasets stored in file 902.

[0149] Natural language processing can extract requested data from database 901 by detecting sections in a file that correspond to the requested data. For example, when the server executes a command related to natural language processing, it can detect specific sections of raw data, such as results sections, or sections related to the design of a clinical trial. The server can also clean or standardize the data in relation to the data format used to generate one or more pre-configured datasets discussed herein. A natural language processing command can further cause server 203 to generate a file 910 containing the cleaned text, sections containing data related to the requested data, and even specific sentences related to the requested data. File 910 can be used by server 202 when executing a natural language processing command and can be automatically annotated 907 with sentences having words and / or labels related to the requested data.

[0150] When the configuration function 904 is executed by the server 203, it causes the server 203 to use one or more named entity models 905 to detect specific data in the cleaned text, such as disease, cell type, dose, dose strength, endpoint, drug name, gene expression data, etc., while executing instructions related to natural language processing.

[0151] For example, natural language processing may include a command that, in some embodiments, causes server 203 to annotate unstructured text using labels that identify specific named entities, such as disease, cell type, dosage of drugs administered to participants during a clinical trial, dose strength, efficacy endpoint, type of drug used, statistics collected throughout the clinical trial, and the number of patients or the number of outcomes or results of the clinical trial, or any or all of these. Server 203 can annotate unstructured text 907 by adding labels to sentences, phrases, or words that identify any of the aforementioned entities, or to any other relevant information that can be used to identify data corresponding to each of the datasets stored in server 203.

[0152] After completing automatic annotation 907, server 203 can perform entity pattern matching 908 between the entities identified above and the entities recognized in the unstructured text. For example, natural language processing can supply the aforementioned list of entities, and natural language processing can cause server 203 to perform entity association, allowing server 203 to recognize specific information such as biomarker levels, treatment types, tumor types, treatment doses, treatment or study groups, and groups, all relevant to the clinical trial in the unstructured text. After completing pattern matching, server 203 can generate a database 909 along with mappings between the recognized information and the automatically annotated sentences or phrases. The extracted information can then be used by server 203 to create one or more datasets to be stored in server 203. After the database (e.g., combined clinical database) is generated, server 203 can also output a file containing one or more structured datasets according to data requested by the user.

[0153] In some embodiments, information or data is retrieved periodically (e.g., daily, weekly, etc.) from a clinical trial database, new or modified data is processed, and incorporated into a combined, standardized clinical trial database. In some embodiments, the combined, standardized clinical trial database supports and enables rapid and comprehensive meta-analysis using up-to-date and unbiased data compared to conventional systems for analyzing clinical trial data. In some embodiments, information curation and incorporation can be coordinated and guided by knowledge of the therapeutic field and drug development experience. In some embodiments, the combined, standardized clinical trial database can better support robust decisions in clinical development and planning.

[0154] Figure 10 schematically shows an exemplary computing device or system 1000 (for example, computing device 205 in Figure 2). System 1000 may be used to perform operations described in relation to one or more implementations by any of the methods of this disclosure. For example, system 1000 may be included in any or all of the server components or other computing devices discussed herein. System 1000 may include one or more processors 1010, one or more memories 1020, one or more storage devices 1030, and one or more input / output (I / O) devices 1040. Components 1010, 1020, 1030, and 1040 may be interconnected using a system bus 1050.

[0155] The processor 1010 may be configured to execute instructions within the system 1000. The processor 1010 may include a single-threaded processor or a multi-threaded processor. In some embodiments, one or more processors 1000 may include one or more graphical processing units. The processor 1010 may be configured to execute or process instructions stored in either or both of the memory 1020 or the storage device 1030. By executing the instructions, graphical information may be displayed on the I / O device 1040 via a user interface or otherwise presented. In some embodiments, one or more processors may include one or more graphical processing units.

[0156] Memory 1020 can store information within system 1000. In some implementations, memory 1020 is a computer-readable medium. In some implementations, memory 1020 may include one or more volatile memory units. In some implementations, memory 1020 may include one or more non-volatile memory units.

[0157] The storage device 1030 may be configured to provide high-capacity storage for the system 1000. In one implementation, the storage device 1030 is a computer-readable medium. The storage device 1030 may include a floppy disk device, a hard disk device, an optical disk device, a tape device, or other types of storage devices. The I / O device 1040 may provide I / O operations for the system 1000. In some implementations, the I / O device 1040 may include a keyboard, a pointing device, or other devices for data input. In some implementations, the I / O device 1040 may include an output device such as a display unit for displaying a graphical user interface or other types of user interfaces.

[0158] Figure 11 is a logical diagram illustrating a process 1100 that, according to some embodiments of the present disclosure, accesses clinical trial data on a host device from a client device 1101 using an application programming interface (API) 1104. This process may be employed to retrieve information and data from a clinical trial database for generating a combined, standardized clinical trial database. Process 1100 illustrates the process of exchanging authentication information between a client 1101, such as a computing device 205, and an API 1104 hosted on a server 203. The client 1101 can request an access token from an identity service 1102 that stores access tokens by executing a command in accordance with an access token request 1111. The client 1101 can send authentication information to the identity service 1102, including a client ID, password, grant type, and scope. The identity service 1102 returns an access token in response to the execution of the command and returns an access token 1112 in response to acknowledgment of the authentication information submitted by the client 1101. The identity service 1102 may be hosted by a server of a third-party provider. Client 1101 can use the access token in API request 1113 when attempting to request data from API 1104. API 1104 can return the requested data 1114. Process 1100 is executed by computing device 205 in response to the user entering authentication information used to request data from server 203.

[0159] Figure 12 is a table of clinical trial statuses organized according to different tiers, according to several embodiments of the present disclosure. Table 1200 shows how different clinical trials are structured according to their status. Table 1200 includes multiple tiers, including a first tier (tier 1) for completed clinical trials, and including final results such as outcome data included in outcome 203d. Table 1200 also includes a second tier (tier 2) for clinical trials that were terminated or are ongoing without final results. Table 1200 also includes a third tier (tier 3) for clinical trials with no results at all. Table 1200 may include a fourth tier (tier 4) which may be a summary tier for clinical trials whose status is unknown. Endpoints associated with each tier may be the efficacy or safety of the drug administered throughout the clinical trial.

[0160] Layer 1 contains the final results related to the clinical trial, so the data related to the final results can be used to structure the clinical trial endpoints. Different combinations of layers may be used by users to design future clinical trials based on data recorded across different layers. Table 1200 also includes the percentage of all clinical trials that fall into one of the various layers, as well as the percentage of clinical trials supported or conducted by a certain number of pharmaceutical companies.

[0161] Figure 13 is a logical diagram illustrating a process, according to several embodiments of the present disclosure, for extracting clinical trial data from one or more external databases to generate a combined standardized clinical trial database, and using the clinical trial data from the combined standardized clinical trial database as input to a trial simulator. Process 1300 can be initiated by server 203 extracting or accessing data stored in clinical trial databases 201 and 202, which are logically represented as external information 1215. The data stored in external information 1215 is data collected by a third party and relating to one or more clinical trials. External information 1215 may be standardized, combined, or integrated and stored in the combined standardized clinical trial database 1211 in one or more forms of the various datasets described above (e.g., trial summaries 203a, trial groups 203b, baselines 203c, outcomes 203d, outcome comparisons 203e, and / or regulatory information 203f). The combined standardized clinical trial database may also be referred to herein as a quantitative knowledge database.

[0162] In some embodiments, the internal information 1216 (e.g., confidential information or information generated within the organization) may be data associated with one or more clinical trials that are accessible to a user requesting access to the combined standardized clinical trial database 1211 but are not included in the external information 1215. The server 203 can extract or access the internal information 1216 in response to a request from a computing device 205 to the server storing the internal information 1216 so that the server 203 can access it. Once the external information 1215 and the internal information 1216 are combined and integrated, the combined clinical trial database 1211, the study population specification 1201 is applied to the data stored in the combined clinical trial database 1210 and stored in the study-specific database 1210. The study-specific database and the control group history database may be project-specific databases curated from the original combined standardized clinical trial database. For example, a chemotherapy dataset in oncology may be further extracted from the combined standardized clinical trial database and curated to focus on trials with chemotherapy treatments. Established chemotherapy datasets can be used as a control group for clinical trial design and decision-making. An example of a study population might be patients with metastatic non-small cell lung cancer receiving first-line treatment, including chemotherapy. For example, a treatment ORR->PFS prediction model 1202 can be investigated using data stored in a study-specific database 1210, which may be stored on server 203 or another server, or locally, to generate a meta-analysis model 1213. As an example, a tree-based meta-analysis model 1213 can be generated according to process 1600 in Figure 16. The database 1210 stores baseline factors specific to the treatment, treatment outcome, and study design, as well as the disease and population of interest, enabling the establishment of a notable prediction model.One or more control group assumptions, such as chemotherapy in first-line treatment of non-small lung cancer metastases, can be applied to data stored in the study-specific database 1210, and the resulting data can be stored in the control group history study database 1212. For example, a control group assumption might be that chemotherapy is used in first-line treatment of non-small lung cancer metastases. Using the data stored in the control group history study database 1212 and the meta-analysis linear model 1213, the results and success probabilities of ongoing clinical trials or clinical trials designed from scratch by the user can be simulated. Database 1212 can also be derived from 1211, but focuses on a notable control group, namely chemotherapy in first-line treatment of non-small lung cancer metastases.

[0163] In some embodiments, a customized user interface (e.g., a GUI) can be implemented to access data in a combined, standardized clinical trial database and perform analysis or simulation. In some embodiments, the GUI is customized based on the therapeutic area and therapeutic needs of interest. For example, an R-shiny-based interface for oncology is implemented, as shown in Figure 14. In some embodiments, the GUI allows non-modelers to interactively explore the database and test various modeling and simulation scenarios through interface design. Figure 14 is a screenshot of a graphical user interface for a clinical trial simulator showing the regression of progression-free survival to overall response rate according to some embodiments of the present disclosure. Various trial simulators can be created according to user needs, and this embodiment is simulated to predict late-phase trial outcomes by inputting early-phase trial outcomes (i.e., proof-of-concept studies (POCs)) using a meta-analysis model established after POC study results are available (Post to POC). Other predictive modeling can be performed prior to POC studies (Prior to POC). Additional modeling (sensitivity analysis) can also be performed at each stage to ensure the robustness of the analysis based on relevant data in the database. A GUI interface for test simulations can be implemented using a GUI for accessing data. In some embodiments, the GUI may be designed based on the user's needs or requirements.

[0164] Figure 15 is a screenshot of the graphical user interface of a clinical trial simulator, showing the probabilities of a drug therapy used in a clinical trial progressing from one development stage to the next, or to approval for a total number of progression-free survival events, and the probabilities of a drug therapy used in a clinical trial progressing from one development stage to the next, or to approval for an event of a given size, according to several embodiments of the present disclosure. The user can use the simulator to predict the success of the next trial and optimize the next trial design.

[0165] Some embodiments provide estimates of drug approval probabilities based, at least in part, on structured datasets within an integrated clinical trial database. In some embodiments, the determination of drug approval probabilities can be performed using machine learning approaches. For example, the analysis may be based on the method described in the publication *Machine Learning with Statistical Imputation for Predicting Drug Approvals* by Lo et al., *Harvard Data Science Review*, Issue 1.1, Oct. 4, 2019, pp. 1–42. The aforementioned publication is incorporated herein by reference in its entirety.

[0166] Lo's publication describes a predictive model for evaluating the probability of approval of a drug candidate in two scenarios: after Phase 2 and after Phase 3 trials. Lo's publication states that statistical imputation is used to handle missing data, e.g., k-nearest neighbors (kNN) (e.g., 5NN) imputation. Lo states that machine learning techniques are used to generate predictions, including cross-validation for training and a set of pending tests for performance evaluation, and that model performance is measured using the standard "Area Under Receiver Operating Characteristic Curve" (AUC) metric. AUC is the estimated probability that the classifier ranks positive outcomes higher than negative outcomes. Lo's publication states that a Random Forest (RF) classifier model is used to predict approval.

[0167] Some embodiments include methods and applications for analysis based on structured datasets within an integrated clinical trial database. Figure 16 schematically illustrates a tree-based regression analysis model for predicting progression-free survival (PFS) related to treatment for a particular disease based on the overall response rate (ORR) of patients in a clinical trial, according to some embodiments of this disclosure. Further explanation of exemplary tree-based regression analysis is provided below. PFS can be determined based on in silico trial simulations.

[0168] An integrated clinical trial database for cancer was constructed based on clinical trial information obtained from the AACT, trialtrove, and BMT databases. A graphical user interface was generated for analysis based on the structured datasets within the integrated clinical trial database (see GUI in Figure 14).

[0169] To predict progression-free survival (PFS) based on proof-of-concept (POC) objective response rates (ORR) for PD1 checkpoint response inhibitors, analyses were performed on structured datasets. Only company-funded intervention trials conducted since 2010 were included. Late-stage PD1-related clinical trials enrolling late-stage patients were included. Groups were categorized into the following treatment classes: PD1 monotherapy, chemotherapy monotherapy, other targeted therapy monotherapy, PD1 in combination with chemotherapy, PD1 in combination with other targeted therapy, chemotherapy in combination with other targeted therapy, and PD1 in combination with chemotherapy and other targeted therapy. Only trials with paired ORR / PFS groups were included. Only primary analysis population (PAS) ORR / PFS was included. The analysis included approximately 200 groups with paired ORR / PFS. The key variables are as follows: ORR; line: 1L / 1L+ and 2L / 2L+; stage: III / IV and IV; PAS PDL1 status: positive, unspecified; cancer type: melanoma, NSCLC and others; and treatment class: PD1 monotherapy, chemotherapy monotherapy, other targeted therapy monotherapy, PD1 + chemotherapy, PD1 + other targeted therapy, chemotherapy + other targeted therapy, PD1 + chemotherapy + other targeted therapy. For each therapy development, it was assumed that the underlying Phase II and Phase III populations / ORR were the same.

[0170] Figure 16 schematically illustrates a tree-based model of PFS based on ORR, including a graph of Log(ORR) versus Log(PFS) for splitting, according to several embodiments of the present disclosure. The first splitting in the tree-based regression analysis is by line, where the ORR / PFS gradient is most heterogeneous across the line compared to stage, cancer class, and PAS PDL1 status. The second splitting in the tree-based regression is by treatment class, where the ORR / PFS gradient is most heterogeneous across treatment class (chemotherapy-related or unrelated) compared to stage, cancer class, and PAS PDL1 status within "1L / 1L+". The third splitting in the tree-based regression analysis is by stage, where the ORR / PFS gradient is most heterogeneous across stage compared to cancer class and PAS PDL1 status within "1L / 1L+, and unchemotherapy-related". Splitting is stopped if no clinically significant ORR / gradient heterogeneity is found, or if the same size of subgroups after splitting is 5 or less. The predictive model for each of the splits is Log(PFS) ~ Log(ORR) + U, where U is the variable with a significant intercept. The final tree-based regression model can be compared with other regression-based models to evaluate and compare model performance (e.g., mean squared error (MSE) of model predictions in training and test split settings).

[0171] The functions described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly embodied in an information carrier (e.g., a machine-readable storage device) for execution by a programmable processor, and the method steps may be executed by a programmable processor executing a program of instructions for performing the functions of the described implementation by manipulating input data and producing outputs. The functions described may, advantageously, be implemented in one or more computer programs executable in a programmable system including a data storage system, at least one input device, and at least one output device, and at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to them. A computer program is a set of instructions that can be used directly or indirectly within a computer to perform a particular activity or produce a particular result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed as a standalone program or in any form such as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0172] Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. The elements of a computer may include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may also include one or more mass storage devices for storing data files, or may be operably coupled to communicate with them; such devices include magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and all forms of non-volatile memory, including CD-ROMs and DVD-ROM disks. Processors and memory may be complemented by or incorporated into application-specific integrated circuits (ASICs).

[0173] To facilitate user interaction, the functionality can be implemented on a computer having a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer.

[0174] The functionality may be implemented in a computer system that includes backend components such as data servers, middleware components such as application servers or internet servers, or frontend components such as client computers with a graphical user interface or internet browser. The system components may be connected by any digital data communication format or medium, such as a communication network. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the internet.

[0175] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network, such as those mentioned above. The client-server relationship is established by computer programs running on each computer that have a client-server relationship with each other.

[0176] Furthermore, the logical flow shown in the diagram does not require a specific order or sequence to obtain the desired result. Additionally, other steps can be added to or removed from the described flow, and other components can be added to or removed from the described system. [Examples]

[0177] Example: Predicting progression-free survival and overall survival using objective response rates from PD1 / PDL1 combination therapy development. In the development of PD1 / PDL1 combination therapies for solid tumors, objective response rate (ORR) is a commonly used clinical endpoint in early-phase studies, while progression-free survival (PFS) and overall survival (OS) are widely used in late-phase studies. Early-phase clinical outcomes such as ORR can inform the optimization of late-phase trial design and the evaluation of probability of success (POS) by developing predictive models for median PFS (mPFS) and median OS (mOS). Existing literature includes ORR / mPFS / mOS correlations and surrogate tests, but the number of clinical trials included is limited. In this study, without establishing surrogates, mPFS and mOS were predicted based on early efficacy in ORR, optimizing late-phase trial design for PD1 / PDL1 combination therapy development. To include appropriate and eligible clinical trials, a comprehensive quantitative clinical trial landscape database (QLD), also known as an integrated clinical trial database, was constructed by combining information from various sources, including clinicaltrial.gov, publications, and company press releases regarding relevant indications and treatments. For scientific accuracy and completeness, a generalizable algorithm was developed to systematically extract and manually curate structured data. Finally, over 150 late-stage clinical trials were identified for the development of the ORR / mPFS (ORR / mOS) predictive model, compared to at most 40 in existing literature. A tree-based machine learning regression model was derived to augment the information, account for heterogeneity in the ORR / mPFS (ORR / mOS) relationship, and simultaneously ensure that the predictive model is robust and has an explicit structure for clinical interpretation. In over 1000 cross-validations, the predicted mean squared error of the proposed model is comparable to random forests and extreme gradient boosting, and far superior to commonly used additive or interaction regression models by cross-validation. An exemplary application of the proposed ORR / mPFS (ORR / mOS) predictive model to late-stage trial POS evaluation for PD1 / PDL1 combination therapy is illustrated.

[0178] 1. Introduction Anti-PD1 / anti-PDL-1 therapies, such as pembrolizumab (FDA approved in 2014), nivolumab (FDA approved in 2014), and atezolizumab (FDA approved in 2016), have been developed over the past decade to treat various cancers, including NSCLC and melanoma. More recently, PD1 / PDL-1 combination therapies, such as PD1 / PDL-1 in combination with other targeted therapies, have become increasingly popular due to their superior clinical efficacy. In clinical trials of PD1 / PDL-1 combination therapy in solid tumors, ORR is a commonly used clinical endpoint in early-phase studies, while PFS / OS is widely used in late-phase studies. ORR is an attractive early clinical endpoint because it allows for shorter trial periods, the use of smaller patient cohorts, and even single-arm trial designs (Clarke JM, Wang X, and Ready NE (2015). Surrogate clinical endpoints to predict overall survival in non-small cell lung cancer trials - are we in a new era? Translational Lung Cancer Research, 4, 804-808). The correlation or surrogacy of ORR with PFS / OS is a central consideration in determining its efficacy, and therefore, there is a variety of literature studying the relationship between ORR / mPFS / mOS. For example, Zhu et al. (Zhu Andrew X, Lin Yong, Ferry David Raymond, Widau Ryan C and Saha Abhijoy (2022). Meta-analysis of surrogate endpoints for survival in patients with unresectable hepatocellular carcinoma treated with immune checkpoint inhibitor-based regimens. American Society of Clinical Oncology) studied the relationship between ORR / mPFS / mOS and immune checkpoint inhibitors, and Ye et al.(Ye Jiabu, Ji Xiang, Dennis Phillip A, Abdullah Hesham and Mukhopadhyay Pralay (2020). Relationship between progression-free survival, objective response rate, and overall survival in clinical trials of PD-1 / PD-L1 immune checkpoint blockade: a meta-analysis, Clinical Pharmacology and Therapeutics, 108(6), 1274-1288), Nie et al. (Nie Runcong, Chen Foping, Yuan Shuqiang, Luo Yingshan, Chen Shi, Chen Yongming, Chen Xiaojiang, Chen Yingbo, Li Yuanfang and Zhou Zhiwei (2019). Evaluation of objective response, disease control and progression-free survival as surrogate endpoints for overall survival in anti-programmed death-1 and anti-programmed death ligand 1 trials, European Journal of Cancer, 106, 1-11), and Ito et al. (Ito Kentaro, Miura Satoru, Sakaguchi Tadashi, Murotani Kenta, Horita Nobuyuki, Akamatsu Hiroaki, Uemura Kohei, Morita Satoshi and Yamamoto Nobuyuki (2019). The impact of high PD-L1 expression on the surrogate endpoints and clinical outcomes of anti-PD-1 / PD-L1 antibodies in non-small cell lung cancer.Lung Cancer,128,113-119) It is possible that PD1 / PDL1 deletion can significantly increase ORR / mPFS / mOS (Blumenthal et al.(Blumenthal Gideon M,Karuri Stella). Overall response rate, progression-free survival, and overall survival with targeted and standard therapies in advanced non-small-cell lung cancer: US Food and Drug Administration trial-level and Journal of Clinical Oncology, 33(9), 1008. (Goring Sarah, Varol Nebibe, Waser Nathalie, Popoff Evan, Lozano-Ortega Greta, Lee Adam, Yuan Yong, Eccles Laura, Tran Phuong and Penrod John R. (2022). Lung Cancer, 170, 122-132.(Hua Tiantian, Gao Yuan, Zhang Ruyang, Wei Yongyue and Chen Feng(2022).Lung Cancer,170,122-132).The study "Validating ORR and PFS as surrogate endpoints in phase II and III clinical trials for NSCLC patients: difference exists in the strength of surrogacy in various trial settings" (BMC Cancer, 22(1), 1-13) investigates the ORR / mPFS / mOS relationship for NSCLC. However, all existing literature focusing on the ORR / mPFS / mOS relationship contains conflicting results. While some tumor types or therapies showed ORR surrogacy for PFS or OS, some studies failed to establish an association. One reason may be that only a limited number of clinical studies were included in the investigation, leading to heterogeneity of the relationship. Therefore, predictive models for all tumor types or all therapies have not been clearly developed to date. In this study, without establishing endpoint surrogates, we integrated an appropriate number of clinical trials across various tumor types, treatment classes, etc., to establish a comprehensive predictive model for the development of PD1 / PDL1 combination therapy. Thus, the efficacy observed in early-phase studies can be used to better predict the success of late-phase studies or to better plan late-phase studies.

[0179] To include a sufficient number of clinical trials to capture ORR / mPFS (ORR / mOS) heterogeneity in predictive model development, a comprehensive quantitative landscape database (QLD) was derived by including all PD1 / PDL1, NSCLC, and melanoma-related clinical trials from clinicaltrial.gov and Informa. Since the inventors focused on establishing a predictive model for PD1 / PDL1 combination therapy development, all PD1 / PDL1-related clinical trials were included. To facilitate predictive model development, the PD1 / PDL1-related trials were enriched with less common treatment classes (chemotherapy, other targeted therapies, chemotherapy + other targeted therapies, which are systematically defined in Section 2.2). NSCLC and melanoma were selected as the two most numerous solid tumors, each with more clinical trials than other tumor types. To generate the QLD, all PD1 / PDL1, NSCLC, and melanoma-related trials available on clinicaltrial.gov and Informa up to July 2022, September 2022, and December 2022, respectively, were included. Specifically, the inventors first utilized relevant clinical studies from clinicaltrial.gov and efficacy outcomes from sponsor-uploaded studies. In addition, Informa was incorporated to provide additional data from publications, conference abstracts and presentations, and corporate press releases.

[0180] Next, we selected all eligible clinical trials from QLD and constructed a tree-based regression predictive model for PD1 / PDL1 combination therapy development. A key idea of ​​the tree-based regression machine learning model is to forward partition the predictor space into several simple subspaces until ORR / mPFS (ORR / mOS) heterogeneity is eliminated. Compared with existing simple predictive models such as linear regression models, the proposed method is more flexible and takes into account ORR / mPFS (ORR / mOS) heterogeneity. Compared with existing machine learning predictive models such as random forests and limit gradient boosting models, the proposed method has an explicit structure for clinical interpretation. The predictive performance of the proposed method was compared with variable-selected multiple linear regression models with and without interaction, random forest, and limit gradient boosting. We demonstrate an exemplary application of the developed tree-based machine learning predictive model to late-phase trial POS evaluation. Also, the primary application in this embodiment was to predict late-phase mPFS / mOS based on early ORR to inform late-phase trial POS. We selected late-phase studies using both ORR and mPFS / mOS from the same study.

[0181] 2. Method 2.1 QLD The objective of creating the QLD, which could also be called an integrated clinical trial database, was to curate historical clinical research-related information into an indication or mechanism of action (MOA) specific database. The integrated clinical trial database QLD was used to prioritize development plans, optimize projects and studies to mitigate risks, and ultimately generate insights to facilitate critical investigations and timely decision-making. The primary focus was on efficacy and safety profiling and their correlation to model success probabilities, as well as the identification of influential factors. The input data sources here were clinicaltrial.gov and Informa. Specifically, in this embodiment, the QLD was generated by including all PD1 / PDL1, NSCLC, and melanoma-related clinical trials from clinicaltrial.gov and Informa that had structured information. The QLD included at least three types of information, including trial summaries, outcomes, and outcome comparisons.

[0182] Study Summary: This dataset included general study-level attributes such as study ID, study name, phase, number of groups, study start / end date, study design, cancer type, therapy line, cancer stage, primary investigational drug / drug class / drug subclass, and other involved drugs / drug classes / drug subclasses (either combination ingredients or comparator drugs).

[0183] Outcomes: This dataset included clinical outcomes reported for each study group. For each outcome record, the study ID, study name, group title, drug / drug combo, outcome title and description, outcome value, and number of patients were provided. Corresponding drug / drug combo classes / subclasses and endpoint classes (e.g., ORR, PFS, OS, etc.) were curated using a combination of manual and algorithmic methods.

[0184] Outcome Comparison: This dataset contained outcome comparison results between treatment groups. For each outcome comparison record, the study ID, study name, outcome comparison p-value, confidence interval, statistical test method, and primary and comparative drug / drug combo were provided. Corresponding primary and comparative drug / drug combo classes / subclasses, endpoint classes (e.g., ORR, PFS, OS, etc.), and comparison classes (e.g., difference, odds ratio, hazard ratio, etc.) were curated using a combination of manual and algorithmic methods.

[0185] In this embodiment, a predictive model was developed using information from the trial summary and outcomes.

[0186] 2.2 Data Selection and Predictors The following inclusion and exclusion criteria were applied to filter eligible trial groups from the QLD and develop an ORR / mPFS (ORR / mOS) predictive model for PD1 / PDL1 combination therapy development: (1) All clinical trials initiated after 2010 were included due to their more recent and popular MOA. (2) Unknown or early-phase (Phase I, Phase I / II, Phase II) clinical trials were excluded due to potential immature PFS and OS. (3) Trials including non-advanced cancer patients (cancer stage I / II) were excluded. (4) Trial groups that did not include paired ORR / mPFS (ORR / mOS) were excluded.

[0187] For each included trial group, only paired ORR / mPFS and ORR / mOS from the primary analysis population were included in the predictive model development. Clinical outcomes from the primary analysis population were selected because they are the most important indicators influencing the trial POS and study design. In addition to ORR, mPFS, and mOS, treatment class, indication, treatment line, cancer stage, and PD1 / PDL1 status were extracted for each included trial group in the primary analysis population. Note that all information was either at the study level or treatment group level, and patient-level data could not be used. Treatment classes were defined into seven categories, including chemotherapy, PD1 / PDL1 therapy, other targeted therapies, PD1 / PDL1 + chemotherapy, PD1 / PDL1 + other targeted therapy, other targeted therapies + chemotherapy, and PD1 / PDL1 + other targeted therapies + chemotherapy. Here, other targeted therapies refer to all targeted therapies other than PD1 / PDL1 therapy. Since the primary objective of this embodiment is the development of PD1 / PDL1 combination therapy, all targeted therapies other than PD1 / PDL1 therapy were grouped together as other targeted therapies, and therefore the number of groups included for each target therapy class other than PD1 / PDL1 was relatively small. Therefore, it is realistic to combine them and assume that they share the same ORR / mPFS (ORR / mOS) relationship. Indications were grouped into three categories: NSCLC, melanoma, and others (Hua et al. 2022; Nie et al. 2019). Treatment lines were classified into three categories: 1L, 1L+, and the rest. Cancer stages were grouped into two categories: stage IV only and stage III / IV mixed. PD1 / PDL1 status was grouped into two categories: positive and unspecified. The status of the remaining biomarkers (other than PD1 / PDL1) for the primary analysis population was not extracted for analysis. This is because their associated therapies were not sufficiently included, and therefore the biomarker-positive prevalence in the analysis dataset was relatively low. In addition, in this embodiment, since the primary objective was the development of PD1 / PDL1-related therapies, heterogeneity in the ORR / mPFS (ORR / mOS) relationship across biomarkers (other than PD1 / PDL1) was not addressed.Therefore, we assumed that the ORR / mPFS (ORR / mOS) relationship is uniform across biomarker statuses (other than PD1 / PDL1).

[0188] 2.3 Tree-based machine learning regression models This section describes the development of a tree-based machine learning regression model to predict ORR for mPFS / mOS. In addition to ORR, treatment class, indication, treatment line, PD1 / PDL1 status, and cancer stage were also included as predictors to better explain the homogeneity / heterogeneity of ORR / mPFS (ORR / mOS). Compared to conventional linear predictive models such as additive models, the proposed model considers the heterogeneity of ORR / mPFS (ORR / mOS). Compared to modern black-box machine learning models such as random forests and extreme gradient boosting, the proposed model has a clear structure for clinical interpretation. A key idea of ​​the proposed model was to forward partition the predictor space into several simple regions until significant heterogeneity of the ORR / mPFS (ORR / mOS) relationship was eliminated. The proposed tree-based machine learning regression model is summarized in the following steps.

[0189] Step 1 (Group variable class with uniform gradient): Notation

number

number

number

[0190] Step 2 (Splitting by the grouped variable classes in Step 1, assuming a significant heterogeneous gradient): logmPFS(mOS)~logORR×VG b is compared with L b >1, and the gradient difference is evaluated by comparing logmPFS(mOS)~logORR+VG for each variable V that satisfies b b . V b* is selected, and (1) if the ANOVA P-value<0.01, (2) if the ANOVA P-value is the most significant among all candidate variables, the branch is split by VG b* .

[0191] Step 3: Case 1: If there is a split in Step 2, Step 1 and Step 2 are repeated with the remaining variables Θ\VG for each branch of VG b* b* .

[0192] Case 2 (Potential heterogeneous intercept adjustment): If there is no split in Step 2, logmPFS(mOS)~logORR×V b is for each variable V b ​​The intercept effect is evaluated by comparing logmPFS(mOS) ~ logORR with respect to ∈Θ. Here,

number

[0193] Specifically, in steps 1-3, all regression models are weighted regressions, with each test group sample size as the weight. In the development of the ORR / mPFS (ORR / mOS) predictive model, Θ includes treatment class, indication, treatment line, PD1 / PDL1 status, and cancer stage.

[0194] 3.Results For the ORR predictive model for mPFS, 154 trials (321 trial groups) were ultimately included. For the ORR predictive model for mOS, 123 trials (249 trial groups) were ultimately included. The number of late-stage clinical trials included was far greater than the existing literature which included at most 40 (Ye et al., 2020). This demonstrated the comprehensiveness of the QLD and ensured the generalizability and reproducibility of the developed predictive model. A summary of the distribution of each predictor is shown in Table 1. For continuous variables such as ORR, the mean and standard deviation are summarized. For categorical variables such as treatment class, indication, treatment line, PD1 / PDL1 status, and cancer stage, the quantity and proportion of each category are provided.

[0195] [Table 1]

[0196] A tree-based machine learning prediction model for ORR / mPFS is summarized in Figure 18. Specifically, for treatment classes, chemotherapy, combination therapy with other targeted therapies + chemotherapy, and combination therapy with PD1 / PDL1 + other targeted therapies + chemotherapy were grouped together based on a uniform ORR / mPFS relationship within these three treatment classes (Therapy Class Group 1). Similarly, PD1 / PDL1 therapy and combination therapy with PD1 / PDL1 + other targeted therapies were grouped together (Therapy Class Group 2) (Figure 19A). (Note: C represents chemotherapy, P represents PD1 / PDL1 therapy, and T represents other targeted therapies in Figure 19A). The ORR / mPFS relationships across Therapy Class Group 1, Therapy Class Group 2, other targeted therapies, and combination therapy with PD1 / PDL1 + chemotherapy were heterogeneous and were left as tree branches. This indicates that the same ORR can result in different mPFS across different treatment classes. Furthermore, within treatment class group 1 and treatment class group 2, different treatment lines result in heterogeneous ORR / mPFS relationships (Figures 19B and 19C). Additionally, within the remaining 1L / 1L+ lines of treatment class group 1 and treatment class 2, different indications result in inconsistent ORR / mPFS relationships (Figures 19D and 19E). The tree-based regression approach allows for the enrichment of information in ORR / mPFS (ORR / mOS) relationships across various therapy classes, tumor types, and treatment lines, when homogeneity is identified among these influential factors.

[0197] A tree-based machine learning prediction model for ORR / mOS is summarized in Figure 20. Specifically, chemotherapy, chemotherapy + other targeted therapies, and PD1 / PDL1 + other targeted therapies + chemotherapy were grouped together by a homogeneous ORR / mOS relationship (Therapy Class Group 1), and similarly, other targeted therapies, PD1 / PDL1 + chemotherapy, and PD1 / PDL1 + other targeted therapies were grouped together (Therapy Class Group 2) (Figure 21A) (Note: C indicates chemotherapy, P indicates PD1 / PDL1 therapy, and T indicates other targeted therapies in Figure 21A). The ORR / mOS relationships across Therapy Class Group 1, Therapy Class 2, and PD1 / PDL1 therapy were heterogeneous. Furthermore, within Therapy Class Group 2, different indications resulted in heterogeneous ORR / mOS relationships (Figure 21B). Within NSCLC and melanoma in Therapy Class 2, different treatment lines resulted in heterogeneous ORR / mOS relationships (Figure 21C).

[0198] Next, the predictive performance of the proposed tree-based machine learning regression model was evaluated in comparison with other commonly used weighted regression models and black-box machine learning models. M1: mPFS(mOS)~ORR, M2: mPFS(mOS)~ORR + treatment class + indication + treatment line + PD1 / PDL1 status + cancer stage, M3: variable selection from M2, M4: mPFS(mOS)~ORR × (treatment class + indication + treatment line + PD1 / PDL1 status + cancer stage), M5: variable selection from M4, M6: random forest, M7: extreme gradient boosting, M8: tree-based machine learning regression model proposed by the inventors. Predictive performance is evaluated by 80%~20% cross-validation. The entire dataset is divided into 80% and 20% over 1000 iterations. In each replication, 80% of the data is used to train the model and 20% is used to test the mean squared error (MSE) of the prediction. Specifically, we searched for various combinations of hypertuning parameters for random forests and extreme gradient boosting, and reported those with the smallest mean squared prediction error (MSE) over 1000 iterations. For random forests, we searched for the number of trees in the forest ∈{500, 1000}, the maximum number of nodes in each tree ∈{10, 20, 30}, and the minimum number of observations at each tree terminal node ∈{10, 20, 30} for the hypertuning parameters. For extreme gradient boosting, we searched for the maximum number of iterations ∈{25, 50, 100}, the maximum depth of each tree ∈{1, 2, 3, 4, 5}, and the minimum number of observations at each tree terminal node ∈{10, 20, 30} for the hypertuning parameters.

[0199] As shown in Figure 22, the predicted MSE of the tree-based machine learning model is comparable to that of M6 and M7 (black-box machine learning models) and significantly smaller than that of M1, M2, M3, M4, and M5 (regression models). This demonstrates the robustness of the model and structure developed by tree-based machine learning regression. Specifically, M1, M2, and M3 did not consider ORR / mPFS (ORR / mOS) heterogeneity, M4 and M5 did not augment the information, and M6 and M7 lacked clear clinical interpretation. In general, compared to other equivalents, the proposed tree-based machine learning regression model was robust, augmented the information, simultaneously detected ORR / mPFS (ORR / mOS) heterogeneity, and had a clear structure for clinical interpretation.

[0200] 4. Application This section describes how the developed tree-based machine learning predictive model is applied to late-stage trial POS evaluation. Two scenarios were considered: Scenario 1: A 1:1 randomized Phase III clinical trial with a target population of previously untreated PD1 / PDL1-positive advanced / metastatic NSCLC patients, comparing PD1 / PDL1 + targeted therapy (other than PD1 / PDL1) as the treatment group and Keytruda as the control group. Scenario 2: A 1:1 randomized Phase III clinical trial with a target population of previously untreated advanced / metastatic NSCLC patients, comparing PD1 / PDL1 + chemotherapy as the treatment group and chemotherapy as the control group. The sample size was N for each group. For simplicity, a censorship rate of 0% was assumed. Regarding the treatment group, the Phase II study was a single-group study with a sample size of n. For Scenario 1, the mPFS and mOS for Keytruda were assumed to be 5.40 months (standard deviation 0.48 months) and 16.7 months (standard deviation 1.48 months), respectively (Mok et al. 2019). For Scenario 2, the mPFS and mOS for chemotherapy were assumed to be 5.00 months (standard deviation 0.48 months) and 13.0 months (standard deviation 1.07 months), respectively (Gogishvili et al., 2022). In addition, it was assumed that the survival curves for both the treatment group and the control group were exponentially distributed. The Phase III trial POS was evaluated using PFS and OS as clinical endpoints, respectively, as follows. Step 1: Sample the ORR of the treatment group through a Phase II study. Step 2: Sample 100% of patients using draw replacement. For the treatment group, develop an ORR / mPFS (ORR / mOS) tree-based machine learning prediction model based on the ORR sampled in Step 1, and predict mPFS (mOS). Step 3: Sample mPFS (mOS) from the control group. Step 4: Derive survival curves for each group using the mPFS (mOS) sampled in Steps 2 and 3. Step 5: Using the survival curves from Step 4, simulate N patients for each group and derive the hazard ratio P-value using a proportional hazards model. Step 6: Repeat Steps 1-5 5000 times to evaluate the percentage of simulated trials with a P-value < 0.05.

[0201] POS was summarized in Figures 23A–23D across different Phase II / III study sample sizes n and N. In Scenario 1, using OS as the endpoint, POS was relatively small when the ORR was 40% (Figure 23A). This is because the predicted mean mOS is 18 months, similar to the mOS of its counterpart, Keytruda. In such a situation, the likelihood of moving to a Phase III study is low. For the remainder, POS increased as Phase II / III study sample sizes n and N increased, due to the small variance between the mPFS(mOS) prediction and the simulated trial hazard ratio estimate. In addition, the magnitude of the POS increase decreased as n and N increased. For example, for a Phase III study sample size N, POS increased more when N changed from 100 to 300 than when N changed from 300 to 500. For a Phase II study sample size n, POS increased more for n changing from 10 to 30 than for n changing from 30 to 50. POS could not be significantly improved because the sample size in the Phase II / III study was sufficient, which may help justify the desirable sample size in the Phase II / III study design.

[0202] 5. Discussion In this embodiment, a comprehensive QLD and tree-based machine learning model were established to enhance the prediction of mPFS / mOS by ORR for the development of PD1 / PDL1 combination therapy. The QLD integrated information from different sources such as clinicaltrial.gov and Informa to include an appropriate number of past clinical trials. A generalizable algorithm was proposed to systematically extract trial-level summaries and clinical outcomes from structured data from clinicaltrial.gov and Informa, supplemented by manual curation for scientific accuracy and completeness. Next, a tree-based machine learning regression model was developed to augment the information and account for heterogeneity in the ORR / mPFS (ORR / mOS) relationship while simultaneously providing an explicit structure for clinical interpretation. The model and structure developed by the proposed method are more robust than common existing machine learning predictive models (e.g., random forest and extreme gradient boosting) determined by cross-validation. Through the predictive model, the observed ORR in early-phase studies can be used to better predict the success of late-phase studies or to better plan late-phase studies.

[0203] One potential extension of this embodiment is to incorporate past clinical trials from other tumor types beyond NSCLC and melanoma for the development of ORR / mPFS (ORR / mOS) predictive models in some embodiments. This could enrich each targeted therapy class and allow for a systematic exploration of potential ORR / mPFS (ORR / mOS) heterogeneity across each targeted therapy class and the target biomarker status in the primary analysis population. Thus, in some embodiments, predictive models derived from more tumors can support various targeted therapy-related late-phase study designs, and the embodiments are not limited to use in PD1 / PDL1 combination therapy development. In addition, some embodiments may include other potential predictors beyond treatment class, tumor type, therapy line, cancer stage, and future biomarker status in the primary analysis population, depending on their availability in data sources (e.g., clinicaltrial.gov and Informa).

[0204] Several implementations of this disclosure have been described. Nevertheless, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are also within the scope of the following claims.

Claims

1. It is a method, Extracting or accessing information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information related to multiple clinical trials, using at least one processor; The process involves generating a plurality of structured datasets, each containing a plurality of data types, based on the extracted information, using at least one of the processors, wherein the plurality of structured datasets include a trial summary dataset, a trial group dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset. The at least one processor applies natural language processing to at least a portion of the extracted or accessed information, which includes unstructured text. The at least one processor generates additional data corresponding to at least one of the multiple data types for at least a portion of the multiple structured datasets based on the natural language processing, To provide a graphical user interface for receiving user input regarding the analysis performed using the aforementioned multiple structured datasets, A method that includes this.

2. The method according to claim 1, wherein the plurality of structured datasets also include a regulatory event dataset.

3. The method according to claim 1 or 2, wherein at least a portion of the extracted information, which includes unstructured data, includes recent information that is not yet available as structured clinical trial information from the one or more databases.

4. Extracting or accessing second information from at least one of the one or more databases, including one or more of drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial, using at least one processor; below: The at least one processor generates second data corresponding to at least one of the plurality of data types based on the second information, and adds it to at least one of the plurality of datasets, or updates at least a portion of the data in at least one of the plurality of datasets, and The at least one processor applies natural language processing to at least a portion of the second information, which includes unstructured text, and the at least one processor generates, based on the natural language processing, second data or revised data corresponding to at least one of the multiple data types for at least a portion of the multiple structured datasets. One or both of the following, The method according to any one of claims 1 to 3, further comprising:

5. At least one processor periodically extracts or accesses additional information from at least one of the following databases: drug research design information, biomarker information, drug therapy information, and efficacy information relating to at least one clinical trial. below: The at least one processor generates additional data corresponding to at least one of the plurality of data types based on the additional information and adds it to at least one of the plurality of datasets, or updates at least a portion of the data in at least one of the plurality of datasets, and The at least one processor applies natural language processing to at least a portion of the additional information, including unstructured text, and the at least one processor generates additional or revised data corresponding to at least one of the multiple data types for at least a portion of the multiple structured datasets based on the natural language processing. One or both of the following, The method according to any one of claims 1 to 3, further comprising:

6. The at least one processor executes a machine learning module based on the multiple structured datasets to determine the probability of success of a clinical drug trial progressing to the next clinical trial stage. The method according to any one of claims 1 to 5, further comprising:

7. The at least one processor executes a machine learning module based on the multiple structured datasets to determine the probability of success in obtaining regulatory approval for a drug in the current clinical trial. The method according to any one of claims 1 to 6, further comprising:

8. The processor executes a machine learning module to determine, for each of the multiple regulatory routes, the probability of success in obtaining regulatory approval for the drug in the current clinical trial, based on the multiple structured datasets. The method according to any one of claims 1 to 7, further comprising:

9. The at least one processor identifies one or more scientific factors, design factors, regulatory factors, or operational factors that contributed to the likelihood of past approval for at least a portion of the plurality of clinical drug trials. The method according to any one of claims 1 to 8, further comprising:

10. The method according to claim 9, wherein the one or more scientific factors include a mechanism of action related to each of the one or more clinical drug studies, the one or more design factors include an endpoint related to each of the one or more clinical studies, and the one or more regulatory factors include the designation of the one or more clinical drug studies as a breakthrough.

11. The method according to any one of claims 1 to 10, further comprising assigning a trial status category to each clinical trial in the plurality of structured datasets, wherein the trial status category includes one or more of completed with final results, terminated or continued without final results, and no results.

12. The method according to any one of claims 1 to 11, further comprising using the at least one processor to simulate one or more Phase 3 clinical trial outcomes based on one or more proof-of-concept study outcomes.

13. The aforementioned GUI, A first field for receiving a first input containing disease-related information, the first field specifying a variable, a combination of variables of interest, or both; A second field to identify the output to be generated and A method according to any one of claims 1 to 12, including the method described in any one of claims 1 to 12.

14. The method according to claim 13, wherein the first input includes or identifies a disease-specific database or a subset of disease-specific databases, and the disease-specific database is at least partially based on standardized clinical trials related to the disease.

15. The method according to claim 13 or 14, wherein the identified generated output includes one or more of the following: a description of one or more first clinical trials relating to a disease; features or outcomes relating to one or more first clinical trials; or predictions of outcomes of one or more first clinical trials relating to a drug used to treat a disease.

16. The method according to any one of claims 1 to 15, wherein the test group dataset includes data types for relating test outcomes to treatment information.

17. The method according to any one of claims 1 to 16, further comprising generating a predictive model or in silico test simulation for predicting test outcomes.

18. The method according to any one of claims 1 to 16, wherein the extracted or accessed trial outcome information includes endpoints related to each of the clinical trials.

19. The method of claim 18, further comprising generating a predictive model or in silico test simulation for predicting the endpoint, wherein the predictive model or in silico test simulation is at least partially based on a machine learning model.

20. A method for determining a predictive model for median progression-free survival (mPFS) or median objective survival (mOS) in a late-phase clinical trial, based on the objective response rate (ORR) in an early-phase clinical trial, Extracting or accessing clinical trial information, including ORR information, mPFS information, mOS information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information, from one or more databases relating to multiple clinical trials. Based on ORR information, treatment class information, indication information, treatment line information, disease stage information, and biomarker status information, define a variable class that includes at least some or all of the variables, This includes determining a tree-based regression machine learning model for mPFS or for mOS based on ORR, wherein the determination is Perform forward segmentation of the predictor space into several subspaces based on the variable class until no mPFS based on ORR heterogeneity exists, or until no mOS based on ORR heterogeneity exists. Determining regression models for each subspace, Methods that include...

21. One or more of the aforementioned treatment classes include anti-PD1 and / or anti-PLD1 therapy (PD1 / PDL1 therapy), The method according to claim 20, wherein the biomarker status information includes PD1 and / or PDL1 status information.

22. Performing forward segmentation of the predictor space into several subspaces until there are no more mPFS based on ORR heterogeneity, or until there are no more mOS based on ORR heterogeneity, The gradient for each variable class is determined, and the variable classes are grouped by a uniform gradient to form the grouped variable classes. For each grouped variable class, if there is a significant heterogeneous gradient within the grouped variable class, the grouped variable class is divided. The method according to claim 20 or 21, including the method described in claim 20 or 21.

23. The gradient for each variable class is determined, and the variable classes are grouped by a uniform gradient to form the grouped variable classes. Determine the first variable class that has the largest sample size compared to the sample sizes of one or more second variable classes, For each variable class, The first regression of the mPFS or mOS on the product of the ORR and the value of the class variable, A second regression of the mPFS information or mOS information to the ORR information added to the value of the class variable and By comparing these, the gradient with respect to the aforementioned variable class is determined. For each of the one or more second variable classes mentioned above, The gradient difference between the gradient of the second variable class and the gradient of the first variable class is determined, If the gradient difference for the aforementioned variable class has a p-value that exceeds a specified grouping threshold or a second variable class sample size that falls below a specified sample size threshold, the variable class is grouped. The method according to claim 22, including the method described in claim 22.

24. The method according to claim 23, further comprising performing the steps described in claim 23 for each of the remaining ungrouped second variable classes.

25. For each grouped variable class, if there is a significant heterogeneous gradient within the grouped variable class, the grouped variable class may be split. For each variable with a grouped class number greater than 1, The first regression of the mPFS or mOS on the product of the ORR and the value of the grouped class variable, A second regression of the mPFS information or mOS information to the ORR information added to the value of the grouped class variable and Comparing and If the ANOVA p-value is less than the specified splitting threshold, and the ANOVA p-value is the most significant for that variable compared to all other variables, then the grouped variable class is selected for splitting. The method according to claim 23 or 24, including the method described in claim 23 or 24.

26. For each divided and grouped variable class, To determine the first variable class of the divided grouped variable class that has the largest sample size compared to the sample sizes of other variable classes within the divided grouped variable class, which is one or more second variable classes, For each variable class of the divided and grouped variable classes, The first regression of the mPFS or mOS on the product of the ORR and the value of the class variable, A second regression of the mPFS information or mOS information to the ORR information added to the value of the class variable and By comparing these, the gradient with respect to the aforementioned variable class is determined. For each of the one or more second variable classes mentioned above, The gradient difference between the gradient of the second variable class and the gradient of the first variable class is determined, If the gradient difference for the aforementioned variable class has a P value that exceeds the specified grouping P threshold or a second variable class sample size that falls below the specified sample size threshold, then the variable class is grouped. The method according to claim 25, further comprising carrying out the following.

27. The method according to any one of claims 20 to 26, wherein the extracted or accessed trial outcome information includes endpoints related to each of the clinical trials.

28. The method according to any one of claims 20 to 27, wherein the treatment class information includes chemotherapy, a combination of chemotherapy and PD1 / PDL1, or a combination of two or more chemotherapy regimens, PD1 / PDL1, or targeted therapies.

29. It is a system, Storage configured to store an integrated clinical trial database, Memory for storing one or more instructions, At least one processor operably coupled to the storage and A system equipped with, The at least one processor is configured or programmed to read the one or more instructions stored in the memory, Extracting or accessing information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information related to multiple clinical trials, using at least one processor; The process involves generating a plurality of structured datasets, each containing a plurality of data types, based on at least a portion of the extracted information, using the aforementioned at least one processor, wherein the plurality of structured datasets include a trial summary dataset, a trial group dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset. The at least one processor applies natural language processing to at least a portion of the extracted or accessed information, which includes unstructured text. The at least one processor generates additional data corresponding to at least one of the multiple data types for at least a portion of the multiple structured datasets based on the natural language processing, The storage of the plurality of structured datasets, wherein the integrated clinical trial database includes the plurality of structured datasets. To provide a graphical user interface for receiving user input regarding analyses performed using the integrated clinical trial database, A system that causes at least one of the processors to perform the above.

30. A non-temporary computer-readable medium containing instructions, wherein when the instructions are executed by a processing device, Extracting or accessing information from one or more databases, including drug research design information, biomarker information, drug therapy information, and efficacy information related to multiple clinical trials, using at least one processor; The process involves generating a plurality of structured datasets, each containing a plurality of data types, based on at least a portion of the extracted information, using the aforementioned at least one processor, wherein the plurality of structured datasets include a trial summary dataset, a trial group dataset, a baseline dataset, an outcome dataset, and an outcome comparison dataset. The at least one processor applies natural language processing to at least a portion of the extracted or accessed information, which includes unstructured text. The at least one processor generates additional data corresponding to at least one of the multiple data types for at least a portion of the multiple structured datasets based on the natural language processing, The storage of the plurality of structured datasets, wherein the integrated clinical trial database includes the plurality of structured datasets. To provide a graphical user interface for receiving user input regarding analyses performed using the integrated clinical trial database, A non-temporary computer-readable medium that causes the processing device to perform the aforementioned processing.

Images