Computerized systems, methods, media, and computer program products for improving computational efficiency of evaluation and selection tendency scoring models

Abstract

Computerized systems, methods, media, and computer program products are provided for improving the computational efficiency of evaluating and selecting a tendency scoring model. In an example method, a computer system accesses first data representing a plurality of characteristics of a test subject and second data representing a plurality of candidate tendency score models that may be used to estimate effectiveness of a treatment. The system selects a set of tendency score models by sequentially evaluating at least some of the candidate tendency score models until one or more stop criteria are met. The effectiveness of the treatment may be estimated by utilizing the selected set of tendency score models. Further, the system stores a data structure representing the set of tendency score models, and outputs the data structure.

Description

Technical Field

[0001] The present description generally relates to systems and methods for automatically evaluating and selecting propensity score models to improve the computational efficiency of computer systems. Background Art

[0002] Generally, a clinical trial is a prospective biomedical or behavioral research investigation designed to answer a specific question related to a biomedical or behavioral intervention. For example, a clinical trial can be conducted to evaluate the safety and / or efficacy of a vaccine, a drug, a dietary option, a dietary supplement, and / or a medical device. Summary of the Invention

[0003] The computer system can be configured to automatically evaluate and select propensity score models, such as those used in analyzing the results of clinical trials.

[0004] Typically, propensity score models are used to generate propensity scores that can be used in matching or weighting, which are statistical techniques that attempt to estimate the effect of a treatment, policy, or other intervention by accounting for covariates that are thought to be confounding. For example, propensity score methods can be used to reduce confounding bias, that is, to select experimental and control subjects in a way that causes the subject characteristics to differ systematically between treatment groups, where subjects with a potentially better prognosis are preferentially given a treatment over other subjects.

[0005] In an example implementation, a computer system can create a list of a large number of propensity score models that are utilized to analyze the results of a clinical trial. The various propensity score models may differ in their suitability and effectiveness in estimating the association between a treatment state (e.g., "exposure") and an observed subject characteristic (e.g., "covariate"). The computer system can evaluate at least some of the propensity score models and select one or more propensity score models that are particularly suitable or effective in evaluating a particular set of results. In addition, information related to the selected propensity score models can be stored for future retrieval and / or presentation to a user (e.g., to facilitate analysis of the clinical trial).

[0006] In some implementations, the computer system can be configured to automatically evaluate and select propensity score models by prioritizing the evaluation of some propensity score models over other propensity score models, and to halt the evaluation when one or more suitable propensity score models are identified. For example, the computer system can be configured to evaluate the propensity score models sequentially (e.g., in order of complexity, such as from least complex to most complex, etc.) until one or more stopping criteria are met (e.g., indicating that one or more suitable propensity score models have been identified). In addition, the computer system can output one or more data structures representing the identified propensity score models.

[0007] The implementations described herein can provide various technical advantages. As an example, the implementations described herein enable a computer system to automatically evaluate and select a propensity score model in a particularly efficient manner. For example, in typical techniques, a computer system can evaluate each available propensity score model and select one or more propensity score models based on the evaluation. However, this technique may consume a large amount of computer resources (e.g., processing utilization, memory utilization, data storage utilization, etc.) due to the brute force nature of the evaluation. In contrast, as described herein, a computer system can alternatively evaluate the propensity score models sequentially and stop evaluating the propensity score models when one or more stopping criteria are met (e.g., indicating that one or more suitable propensity score models have been identified). Thus, in at least some implementations, the computer system does not have to evaluate each available propensity score model, and thus can reduce the computer resources consumed (e.g., compared to the computer resources that would be consumed according to traditional techniques).

[0008] As another example, the implementations described herein can be used to improve the analysis of clinical trial results (e.g., by selecting and utilizing a propensity score model that is particularly suitable for analyzing the results). This enables researchers to better evaluate the effectiveness of medical interventions, thereby improving the safety and / or efficacy of treatments of subjects. For example, the techniques described herein can be used to better understand the effectiveness of treatments and modify the treatments to further improve their efficacy. In addition, this reduces the likelihood that clinical trial results will be misinterpreted or otherwise inappropriately analyzed, thereby improving the efficiency of research and development treatments (e.g., by reducing the amount of resources that would be expended in pursuing research and development goals based on misinterpretations of clinical trial studies and / or inappropriate analyses of clinical trial results).

[0009] As another example, the implementations described herein can be used to provide a structured data file for storing information related to the evaluation of propensity score models and presenting such information to a user in an organized and easily understandable manner. Thus, the user can intuitively determine the suitability of each evaluated propensity score model and select one or more propensity score models to facilitate analysis of a clinical investigation.

[0010] In one aspect, a system includes: a user interface circuit system for generating a user interface, the user interface comprising one or more visual representations of a tabular structured data file when drawn on a display device, the tabular structured data file comprising a grid of data cells; a memory for storing: first data representing multiple characteristics of each of a plurality of test subjects, and second data representing multiple candidate propensity score models for estimating the effectiveness of a treatment performed on one or more of the test subjects; a processor communicatively coupled to the at least one memory, wherein the processor is configured to: access the first data and the second data from the memory; select a propensity score model set from the plurality of candidate propensity score models, wherein selecting the propensity score model set comprises: sequentially evaluating the propensity score models in the candidate propensity score models at least some of the candidate propensity score models until one or more stopping criteria are satisfied, wherein evaluating each of the candidate propensity score models comprises: obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and determining whether the output of the candidate propensity score model satisfies the one or more stopping criteria; and selecting the set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models; and using the memory to store the tabular structured data file representing the set of propensity score models; and outputting the tabular structured data file, wherein outputting the tabular structured data file comprises using the user interface circuit system and the display device so that the user interface is presented to the user, wherein the user interface includes an indication of the set of propensity score models.

[0011] In one aspect, a method includes: accessing, by a computer system, from one or more hardware storage devices: first data representing multiple characteristics of each test subject in a plurality of test subjects, and second data representing multiple candidate propensity score models that can be used to estimate the effectiveness of a treatment for one or more test subjects in the plurality of test subjects; and selecting a set of propensity score models from the multiple candidate propensity score models. Selecting the set of propensity score models includes: sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, wherein evaluating each of the candidate propensity score models includes: obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and determining whether the output of the candidate propensity score model meets the one or more stopping criteria. Selecting the set of propensity score models also includes: selecting the set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models. The method also includes: using the one or more hardware storage devices to store, through the computer system, a data structure representing the propensity score model set; and outputting the data structure through the computer system, wherein outputting the data structure includes: causing a user interface to be presented to a user through the computer system, wherein the user interface includes an indication of the propensity score model set.

[0012] Implementations of this aspect may include one or more of the following features.

[0013] In some implementations, the method may further include estimating the effectiveness of the treatment based on the set of propensity score models.

[0014] In some implementations, the method may further include performing a clinical investigation based on the data structure.

[0015] In some implementations, the method can also include modifying treatment of one or more additional subjects based on the data structure.

[0016] In some implementations, the set of propensity score models may include only a single propensity score model.

[0017] In some implementations, the set of propensity score models can include multiple propensity score models.

[0018] In some implementations, sequentially evaluating at least some of the candidate propensity score models may include determining a sequential order of the candidate propensity score models, and evaluating at least some of the candidate propensity score models based on the sequential order.

[0019] In some implementations, the sequence order may be determined based on the complexity of each of the candidate propensity score models.

[0020] In some implementations, the sequence order may be determined based on the statistical complexity of each of the candidate propensity score models.

[0021] In some implementations, the output of the candidate propensity score model can represent a standardized difference for each of the one or more covariates in the candidate propensity score model.

[0022] In some implementations, the one or more stopping criteria can include determining that a standard deviation of each of the one or more covariates of the candidate propensity score model is less than a threshold.

[0023] In some implementations, the one or more stopping criteria can include determining that a number of evaluated candidate propensity score models is greater than or equal to a threshold.

[0024] In some implementations, the data structure may include a tabular structured data file comprising a grid of data cells, wherein the data cells are arranged in multiple rows and columns, and wherein the data cells represent: at least one of the candidate propensity score models, and at least one of the standard deviations corresponding to the at least one of the candidate propensity score models.

[0025] In some implementations, the method can include receiving user input representing one or more selection criteria, and selecting at least some of the candidate propensity score models based on the one or more selection criteria.

[0026] In some implementations, each of the candidate propensity score models can be configured to generate a corresponding propensity score based on at least a portion of the first data, and wherein the propensity score represents the probability that the treatment has been administered to the specific test subject based on the characteristics of the test subject.

[0027] In some implementations, estimating the effectiveness of a treatment under clinical investigation can include minimizing selection bias associated with an estimate of the effect of the treatment based on at least one propensity score model from the set of propensity score models.

[0028] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of each of the chat agent, triage agent, and retrieval agent. The one or more computer systems can be configured to perform the actions by means of software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform the specific actions during operation. The one or more computer programs can be configured to perform the specific actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0029] The details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and the description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a diagram of an example system for automatically evaluating and selecting propensity score models.

[0031] Figure 2 is a diagram of an example evaluation and selection engine.

[0032] Figure 3 is a flowchart of an example process for automatically evaluating and selecting propensity score models.

[0033] Figure 4 is a flowchart of an example process for automatically evaluating and selecting propensity score models.

[0034] Figure 5 is a diagram of an example computer system.

[0035] Like reference numbers and designations in the various drawings represent like elements.

[0036] Specific implementation method

[0037] Figure 1 An example system 100 is shown for automatically evaluating and selecting a propensity score model, such as a propensity score model for use in analyzing the results of a clinical trial. For example, the system 100 may receive clinical trial data (e.g., baseline characteristics) and automatically evaluate and select one or more propensity score models based on the clinical trial data (e.g., one or more propensity score models that are particularly suitable for balancing baseline characteristics). Furthermore, the system 100 may output a data structure representing the selected propensity score model.

[0038] The system 100 includes an evaluation and selection engine 150 implemented on the computer system 102a. Generally, the engine 150 is configured to obtain data representing propensity score models available to the system 100 and evaluate at least some of the propensity score models (e.g., to determine the suitability of each evaluated propensity score model for analyzing a particular clinical trial dataset). Furthermore, in at least some implementations, the engine 150 can select one or more of the evaluated propensity score models (e.g., a propensity score model that is particularly suitable for analyzing the clinical trial dataset) and present the selected propensity score model to a user.

[0039] For example, during an example operation of the system 100, the engine 150 obtains a clinical trial dataset. As an example, the clinical trial dataset may represent characteristics of each subject of the clinical trial (e.g., the subject's age, medical condition, medical history, demographic information, etc.). As another example, the clinical trial dataset may represent the treatment status of each subject (e.g., whether each subject has been selected for treatment (and if so, the type of treatment), whether each subject has been selected for placebo treatment, etc.). In some implementations, the clinical trial dataset may be implemented at least in part as one or more portions of structured data.

[0040] In addition, the engine 150 can obtain model data representing a propensity score model that can be used to analyze clinical trial data sets. Generally, a propensity score model is used to generate a propensity score that can be used in matching or weighting, where matching or weighting is a statistical technique that attempts to estimate the effect of a treatment, policy, or other intervention by accounting for covariates that are believed to be confounding. For example, a propensity score approach can be used to reduce confounding bias, that is, to select experimental and control subjects in a way that causes the subject characteristics to differ systematically between the treatment groups, where subjects with a potentially better prognosis are preferentially given a treatment over other subjects.

[0041] In at least some implementations, the likelihood of bias in clinical trials increases due to differences in treatment outcomes (e.g., such as mean treatment effects) between treated and untreated groups, where the differences may be due to factors that predict treatment rather than the treatment itself. For example, in a randomized trial, randomization enables an unbiased estimate of the treatment effect. For each covariate, randomization means that the mean values ​​of the treatment groups will be balanced according to the randomization principle. Unfortunately, for observational studies, the allocation of treatments to subjects is typically not random. Propensity score matching and weighting attempt to reduce treatment allocation bias and simulate randomization by creating a sample of treated units that are comparable to the sample of untreated units on all observed covariates.

[0042] In particular, the "propensity score" generated by the propensity score model represents the probability that a subject has been treated, given the value of its covariates. When the covariates are unbalanced between the treatment groups, the stronger the confounding effect of the covariates (that is, the stronger the association between the covariates and whether the subject was treated and the outcome), the greater the bias in the analysis of the naive treatment effect. By having similar units for the propensity scores in both the treatment and control groups, this confounding is reduced.

[0043] In some implementations, the model data can include portions of one or more mathematical functions, equations, computer macros, and / or computer code to generate one or more propensity scores using a clinical trial dataset as input. In some implementations, the model data can be implemented at least in part as one or more portions of structured data.

[0044] In some implementations, at least some of the clinical trial data and / or model data can be retrieved from one or more hardware data storage devices 160a local to the computer system 102a. In some implementations, at least some of the clinical trial data and / or model data can be retrieved from one or more hardware data storage devices 160b remote from the computer system 102a (e.g., one or more remote computer systems 102b, such as server computers, communicatively coupled to the computer system 102a via the network 106). In some implementations, at least some of the clinical trial data and / or model data can be manually input by a user (e.g., using a graphical user interface (GUI) 154 represented by the computer system 102a).

[0045] Various propensity score models may differ in their suitability and effectiveness in estimating the association between treatment states (e.g., "exposures") and observed subject characteristics (e.g., "covariates"). Engine 150 may evaluate at least some of the propensity score models (e.g., based on clinical trial data and model data) and select one or more propensity score models that are particularly suitable or effective in evaluating a particular set of outcomes.

[0046] In addition, data related to the evaluation of the propensity score model and / or the indication of the selected propensity model can be stored for future retrieval and / or presentation to a user (e.g., to facilitate analysis of a clinical trial). For example, in some implementations, at least some of the data related to the evaluation of the propensity score model and / or the indication of the selected propensity model can be stored locally on hardware data storage 160a and / or remotely on hardware data storage 160b. As another example, in some implementations, at least some of the data related to the evaluation of the propensity score model and / or the indication of the selected propensity model can be presented to a user via GUI 154.

[0047] In some implementations, the engine 150 can be configured to automatically evaluate and select propensity score models by prioritizing the evaluation of some propensity score models over others, and suspending evaluation when one or more suitable propensity score models are identified.

[0048] For example, the engine 150 can determine a sequence of propensity score models to evaluate based on the model data. In some implementations, the sequence can be arranged in order of complexity (e.g., from the least complex propensity score model to the most complex propensity score model). In some implementations, the complexity of the propensity score model can be determined based on the type of operation performed using the propensity score model (e.g., statistical operation), the type of statistical relationship being modeled (e.g., statistical relationship between covariates), and / or the amount of computer resources that will be consumed by employing the propensity score model.

[0049] Furthermore, the engine 150 can be configured to evaluate the propensity score models according to the determined sequence until one or more stopping criteria are met (eg, indicating that one or more suitable propensity score models have been identified).

[0050] For example, the engine 150 can be configured to evaluate the propensity score model until the standard deviation of some or all of the evaluation covariates for the propensity score model is less than a threshold value (e.g., indicating that the balance of baseline characteristics between the different treatment groups has reached a sufficient degree after propensity score matching or weighting). This can be beneficial, for example, in enabling the engine 150 to identify one or more suitable propensity score models without requiring the engine 150 to evaluate every available propensity score model (e.g., thereby reducing consumed computer resources).

[0051] As another example, engine 150 can be configured to evaluate propensity score models until a specified maximum number of propensity score models have been evaluated. After completing the evaluation process, engine 150 can select one or more of the best propensity score models tested. This can be beneficial, for example, in enabling engine 150 to limit the amount of computer resources consumed.

[0052] In general, the output of engine 150 can be used to analyze data from clinical trials. For example, based on the output of engine 150, researchers can use a specific propensity score model to account for covariates that predict the acceptance of a treatment being evaluated in a target clinical trial, allowing them to better estimate the safety and / or efficacy of the treatment (e.g., to reduce confounding factors in subsequent analyses). In addition, this enables researchers to improve their understanding of the safety and / or efficacy of the treatment of interest. For example, the techniques described herein can be used to better understand the effectiveness of a treatment and modify the treatment to further improve its efficacy.

[0053] In general, each computer system in computer systems 102a and 102b can include any number of electronic devices configured to receive, process, and transmit data. Examples of computer systems include client computing devices (e.g., desktop computers or laptop computers), server computing devices (e.g., server computers or cloud computing systems), mobile computing devices (e.g., cellular phones, smartphones, tablet computers, personal data assistants, laptop computers with networking capabilities), wearable computing devices (e.g., smart watches), and other computing devices capable of receiving, processing, and transmitting data. In some implementations, the computer systems can include computing devices that operate using one or more operating systems (e.g., Microsoft Windows, Apple macOS, Linux, Unix, Google Android, and Apple iOS, etc.) and one or more architectures (e.g., x86, PowerPC, and ARM, etc.). In some implementations, one or more of the computer systems need not be located locally with respect to the rest of system 100, and one or more of the computer systems can be located in one or more remote physical locations.

[0054] Each of computer systems 102a and 102b may include a corresponding user interface (e.g., GUI 154) that enables a user to interact with the computer system, other computer systems, and / or engine 150. Example interactions include viewing data, transferring data from one computer system to another computer system, and / or issuing commands to a computer system. Commands may include, for example, any user instructions to one or more of the computer systems to perform a specific operation or task. In some implementations, a user may install software applications on one or more of the computer systems to facilitate the performance of these tasks.

[0055] exist Figure 1, computer system 102a is shown as a single component. However, in practice, computer system 102a may be implemented on one or more computing devices (e.g., each computing device includes at least one processor such as a microprocessor or microcontroller). As an example, computer system 102a may be a single computing device connected to network 106, and engine 150 may be maintained and operated on the single computing device. As another example, computer system 102a may include multiple computing devices connected to network 106, and engine 150 may be maintained and operated on some or all of the computing devices. For example, computer system 102a may include several computing devices, and engine 150 may be distributed across one or more of the computing devices.

[0056] The network 106 may be any communication network through which data can be transmitted and shared. For example, the network 106 may be a local area network (LAN) or a wide area network (WAN) such as the Internet. The network 106 may be implemented using various networking interfaces, such as wireless networking interfaces (such as Wi-Fi, Bluetooth, or infrared) or wired networking interfaces (such as Ethernet or serial connections). The network 106 may also include a combination of more than one network and may be implemented using one or more networking interfaces.

[0057] Figure 2 Various aspects of the evaluation and selection engine 150 are shown in more detail. Generally, the engine 150 includes several operating modules for performing specific functions related to the operation of the engine 150. For example, the engine 150 includes a database module 210, a communication module 220, a processing module 230, and a user interface module 240. The operating modules can be provided as one or more computer-executable software modules, hardware modules, or a combination thereof. For example, one or more of the operating modules can be implemented as a software code block having instructions that cause one or more processors of the engine 150 to perform the operations described herein. Additionally or alternatively, one or more of the operating modules can be implemented in an electronic circuit system (such as, for example, a programmable logic circuit, a field programmable logic array (FPGA), or an application-specific integrated circuit (ASIC)).

[0058] The database module 210 maintains information relevant to evaluating and selecting propensity score models.

[0059] As an example, the database module 210 may store input data 210a that is used as input for evaluating and selecting a propensity score model.

[0060] As an example, the input data 210a may include (e.g., as referenced Figure 1The clinical trial data (described above) such as data indicating characteristics of each subject of the clinical trial and treatment status of each subject, etc.

[0061] As another example, the input data 210a may include model data representing a propensity score model that may be used to analyze a clinical trial dataset (e.g., Figure 1 described), such as data representing one or more mathematical functions, equations, and / or computer code portions used to generate one or more propensity score models.

[0062] As another example, input data 210a may include instructions from a user related to the evaluation and selection of a propensity score model. For example, input data 210a may include instructions from a user to evaluate a specific subset of propensity score models available for use with engine 150. Furthermore, input data 210a may include instructions from a user to evaluate propensity score models in a particular sequential order. Furthermore, input data 210a may include instructions from a user to terminate the evaluation of a propensity score model when a certain stopping criterion is met.

[0063] In some implementations, at least a portion of the input data 210a can be retrieved from one or more local hardware data storage devices (e.g., hardware data storage device 160a) and / or remote hardware data storage devices (e.g., hardware data storage device 160b). In some implementations, at least a portion of the input data 210a can be received from a user (e.g., using a GUI 154 generated by a user interface module 240).

[0064] In addition, the database module 210 can store output data 210b generated by the engine 150. As an example, for each propensity score model evaluated, the output data 210b can include data representing the evaluation results (e.g., one or more metrics indicating the effectiveness of the propensity score model in balancing characteristics between a group of subjects who received the treatment of interest and a group who did not receive the treatment). For example, the output data 210b can represent one or more covariates of the clinical trial data and metrics associated with each covariate (e.g., standard deviation).

[0065] As another example, the output data 210b may include data indicating a selection of one or more propensity score models. For example, the output data 210b may include data indicating that one or more propensity score models have met one or more criteria (e.g., indicating that one or more suitable propensity score models have been identified).

[0066] As another example, the output data 210b can filter or sort the data based on the results of the evaluation. For example, the output data 210b can indicate the various propensity score models that have been evaluated and can be sorted (e.g., from highest to lowest based on the number of subjects retained after matching). As another example, the output data 210b can indicate a subset of the propensity score models that have been evaluated (e.g., the N best propensity score models, where N is 1 or greater than 1), and omit the remaining propensity score models.

[0067] Additionally, the database module 210 may store processing rules 210 c for specifying how the data in the database module 210 may be processed to evaluate and select a propensity score model.

[0068] As an example, processing rules 210c may include one or more rules for identifying propensity score models for evaluation by engine 150 and the order in which they are evaluated.

[0069] As another example, processing rules 210c may indicate how to input data into various available propensity score models and how to generate output data 210b (eg, representing an evaluation of the propensity score model) based on the input data.

[0070] As another example, processing rules 210c may indicate one or more rules for determining whether to terminate evaluation of a propensity score model (eg, one or more rules for specifying that evaluation of a propensity score model be terminated when a certain stopping criterion is met).

[0071] As another example, processing rules 210c may specify that generated output data 210b be presented to a user and / or stored (eg, using database module 210) for future retrieval and / or processing.

[0072] Example data processing techniques are described in further detail below.

[0073] As described above, the engine 150 also includes a communication module 220. The communication module 220 enables data to be transmitted to the engine 150. For example, the communication module 220 can be communicatively connected to the network 106 so that it can transmit data to and receive data from the computer system 102b. Information received from the computer system 102b can be processed (e.g., using the processing module 230) and stored (e.g., using the database module 210).

[0074] As described above, the engine 150 also includes a processing module 230. The processing module 230 processes data stored by the engine 150 or otherwise accessible to the engine 150. For example, the processing module 230 can be used to perform one or more of the operations described herein (e.g., operations associated with evaluating and selecting a propensity score model).

[0075] The user interface module 240 is configured to present information to a user and / or receive input from a user. As an example, the user interface module 240 may include one or more display devices (e.g., a display screen, a touch screen, etc.) configured to present a user interface (e.g., a GUI 154) that enables a user to interact with the computer system 102a and / or the engine 150. Example interactions include viewing data, transferring data from one component to another, and / or issuing commands to the computer system 102a and / or the engine 150. Commands may include, for example, any user instructions to one or more of the computer system 102a and / or the engine 150 to perform a specific operation or task.

[0076] In some implementations, a software application can be used to facilitate the performance of the tasks described herein. As an example, the application can be installed on the computer system 102a. In addition, a user can interact with the application to input data and / or commands to the engine 150 and view the data generated by the engine 150.

[0077] As described above, a computer system (e.g., using engine 150) can be configured to evaluate and select propensity score models by prioritizing the evaluation of certain propensity score models over other propensity score models, and to discontinue evaluation when one or more suitable propensity score models are identified. As an example, Figure 3 A process 300 for evaluating a propensity score model is shown in FIG.

[0078] In process 300, a computer system accesses data related to propensity score models available for evaluation (e.g., candidate propensity score models) and evaluates (302) any of the following propensity score models, wherein a set of covariates of the clinical trial data is represented in its original form in the propensity score model.

[0079] While performing the evaluation, the computer system determines whether one or more stopping criteria have been met (304). Generally, the stopping criteria can specify conditions that, when met, indicate that one or more suitable propensity score models have been identified.

[0080] In some implementations, the one or more stopping criteria may include a criterion that is met when the standard deviation of all evaluated covariates in the evaluated propensity score model is less than a threshold value (eg, "STDDIFF_CUT").

[0081] In some implementations, the one or more stopping criteria may include a criterion that is met when the total number of evaluated propensity score models is greater than or equal to a threshold number (eg, "MOD_MAX_NUM").

[0082] In some implementations, one or more stopping criteria may include a combination of multiple criteria, and the stopping criteria may be considered satisfied when at least one of the criteria is satisfied. Figure 3 As shown, the stopping criteria can be considered to be met when (i) the standard deviation of all evaluated covariates in the evaluated propensity score models is less than a threshold value (e.g., "STDDIFF_CUT"), or (ii) the total number of evaluated propensity score models is greater than or equal to a threshold number (e.g., "MOD_MAX_NUM").

[0083] If one or more stopping criteria are met, process 300 terminates (306).

[0084] At the termination of process 300, the computer system generates one or more data structures representing the evaluated propensity score models. For example, the computer system can generate a report summarizing the covariates for each of the evaluated propensity score models. As another example, the computer system can select one or more propensity score models with the best performance (e.g., having the lowest standard deviation for its covariates) and indicate those propensity score models in the report.

[0085] In some implementations, the data structure can include one or more spreadsheets. For example, the data structure can include a tabular structured data file that includes a grid of data cells arranged in rows and columns. In addition, the data cells can include information such as (i) one or more of the propensity score models evaluated, and (ii) information related to each of those propensity score models (e.g., standard deviations of covariates for those propensity score models, etc.). In some implementations, the data cells can be sorted and / or filtered to facilitate analysis of the propensity score models (e.g., as described above).

[0086] When process 300 terminates, the computer system may also use the selected propensity score model to generate the number and percentage of matched subjects in each treatment group if propensity score matching is selected, or generate the weighted number of subjects in each treatment group if propensity score weighting is selected.

[0087] If one or more stopping criteria are not met, the computer system continues to evaluate additional propensity score models until one or more stopping criteria are met. For example, as shown in Figure 3 FIG. 6, the computer system can evaluate, in sequence:

[0088] (i) any propensity score model with (one or more) variables having a square root transformation for (one or more) non-negative numerical covariates (i.e., covariate(s) with values > 0) (308),

[0089] (ii) any propensity score model with (one or more) variables having a log transformation for (one or more) positive numerical covariates (i.e., covariate(s) with values > 0) (310),

[0090] (iii) any propensity score model with (one or more) variables having a logit transformation for (one or more) percentage numerical covariates (i.e., covariate(s) with values between 0 and 100) (312),

[0091] (iv) any propensity score model with (one or more) variables having a square root transformation for (one or more) non-negative numerical covariates (314),

[0092] (v) any propensity score model with (one or more) variables having a log transformation for (one or more) positive numerical covariates (316),

[0093] (vi) any propensity score model with (one or more) variables having a logit transformation for (one or more) percentage numerical covariates (318),

[0094] (vii) any propensity score model with combinations of differently transformed numerical covariates (e.g., two combinations for variables X and Y, such as sqrt(X) + log(Y), log(X) + sqrt(Y), etc.) (320),

[0095] (viii) any propensity score model with untransformed (one or more) interaction variables (322), and

[0096] (ix) any propensity score model with transformed (one or more) interaction variables (324).

[0097] At each evaluation stage 308, 310, 312, 314, 316, 318, 320, and 322, the computer system determines whether one or more stopping criteria have been met (304). If so, the process 300 terminates (306). If not, the process 300 proceeds to the next evaluation stage in the sequence.

[0098] Upon completion of the evaluation phase 324, the process 300 terminates even if one or more stopping criteria have not been met.

[0099] In some implementations, a user can specify that the computer system automatically generate and evaluate the propensity score model after all available transformations (e.g., "NUM_AUTO_TRANS") (326). If this option is turned off by the user, the computer system can skip evaluating the transformed propensity score model (evaluation stages 308, 310, 312, 314, 316, 318, and 320) and proceed directly from evaluation stage 302 to evaluation stage 322.

[0100] In some implementations, a user can specify a specific subset of available propensity score models for the computer system to evaluate (e.g., "INTERACT_NO") (328). If this option is specified by the user as any non-negative integer, the computer system can evaluate propensity score models with two-way interactions that have at most the specified number of two-way interactions in each propensity score model (evaluation stages 322 and 324). For example, if this value is set to 1, each model can have at most one interaction term.

[0101] Example Processing

[0102] Figure 4 An example process 400 is shown for automatically evaluating and selecting a propensity score model, such as a propensity score model used in analyzing the results of a clinical trial. In some implementations, process 400 can be performed by system 100 described in this disclosure (eg, using engine 150).

[0103] In process 400, the system accesses from one or more hardware storage devices (i) first data representing multiple characteristics of each of a plurality of test subjects, and (ii) second data representing multiple candidate propensity score models that can be used to estimate the effectiveness of a treatment for one or more of the plurality of test subjects (402).

[0104] Furthermore, the system selects a set of propensity score models from a plurality of candidate propensity score models (404). In some implementations, each candidate propensity score model can be configured to generate a corresponding propensity score based on at least a portion of the first data. Furthermore, the propensity score can represent a probability that a treatment has been administered to a particular test subject based on characteristics of the test subject.

[0105] The set of propensity score models is selected by sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met (404a). Evaluating each of the candidate propensity score models includes: (i) obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and (ii) determining whether the output of the candidate propensity score model meets one or more stopping criteria.

[0106] In some implementations, sequentially evaluating at least some of the candidate propensity score models can include (i) determining a sequence order of the candidate propensity score models and (ii) evaluating at least some of the candidate propensity score models based on the sequence order.

[0107] In some implementations, the sequence order can be determined based on a complexity of each of the candidate propensity score models.

[0108] In some implementations, the sequence order can be determined based on a statistical complexity of each of the candidate propensity score models.

[0109] The set of propensity score models is selected by sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met (404a). Evaluating each of the candidate propensity score models includes: (i) obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and (ii) determining whether the output of the candidate propensity score model meets one or more stopping criteria.

[0110] In some implementations, the set of propensity score models can include only a single propensity score model.

[0111] In some implementations, the set of propensity score models can include multiple propensity score models.

[0112] In some implementations, the output of a candidate propensity score model can represent a standard deviation of each of one or more covariates of the candidate propensity score model.

[0113] In some implementations, the one or more stopping criteria can include a determination that a standard deviation for each of one or more covariates of a candidate propensity score model is less than a threshold value.

[0114] In some implementations, the one or more stopping criteria can include a determination that a number of evaluated candidate propensity score models is greater than or equal to a threshold value.

[0115] The system stores, using one or more hardware storage devices, a data structure representing the set of propensity score models (406).

[0116] The system outputs the data structure (408). For example, the system can cause a user interface to be presented to a user, the user interface including an indication of the set of propensity score models.

[0117] Further, effectiveness of the treatment can be estimated (410) based on the set of propensity score models. In some implementations, estimating the effectiveness of the treatment of the clinical investigation can include minimizing selection bias associated with an estimate of an effect of the treatment based on at least one propensity score model of the set of propensity score models.

[0118] In some implementations, the process 400 can further include conducting the clinical investigation based on the data structure (e.g., based on the selected propensity score model(s) and / or the propensity scores generated by those model(s)).

[0119] In some implementations, the process 400 can further include modifying a treatment of one or more additional subjects based on the data structure (e.g., based on the selected propensity score model(s) and / or the propensity scores generated by those model(s)).

[0120] In some implementations, the process 400 can further include receiving user input representing one or more selection criteria and selecting at least some of the candidate propensity score models based on the one or more selection criteria.

[0121] Example computer system

[0122] Figure 5 An example computing system in accordance with implementations of the present disclosure is depicted. The system 500 can be used for any of the operations described for the various implementations discussed herein. The system 500 can include one or more processors 510, memory 520, one or more storage devices 530, and one or more input / output (I / O) devices 560, which can be controlled by one or more I / O interfaces 540. The various components 510, 520, 530, 540, or 560 can be interconnected by at least one system bus 550, which can enable data transfer between the various modules and components of the system 500.

[0123] The processor(s) 510 can be configured to process instructions for execution within the system 500. The processor(s) 510 can include single-threaded processor(s), multi-threaded processor(s), or both. The processor(s) 510 can be configured to process instructions stored in the memory 520 or stored on the storage device(s) 530. The processor(s) 510 can include hardware-based processor(s) each including one or more cores. The processor(s) 510 can include general-purpose processor(s), special-purpose processor(s), or both.

[0124] Memory 520 can store information within system 500. In some implementations, memory 520 includes one or more computer-readable media. Memory 520 can include any number of volatile memory units, any number of non-volatile memory units, or both volatile and non-volatile memory units. Memory 520 can include read-only memory, random access memory, or both. In some examples, memory 520 can be used as active or physical memory by one or more executing software modules.

[0125] The storage device(s) 530 can be configured to provide (e.g., permanent) mass storage to the system 500. In some implementations, the storage device(s) 530 can include one or more computer-readable media. For example, the storage device(s) 530 can include a floppy disk drive, a hard disk drive, an optical disk drive, or a magnetic tape drive. The storage device(s) 530 can include read-only memory, random access memory, or both. The storage device(s) 530 can include one or more of an internal hard drive, an external hard drive, and a removable drive.

[0126] One or both of memory 520 and storage device(s) 530 may include one or more computer-readable storage media (CRSMs). A CRSM may include one or more electronic storage media, magnetic storage media, optical storage media, magneto-optical storage media, quantum storage media, mechanical computer storage media, and the like. The CRSM may provide storage for computer-readable instructions describing data structures, processes, applications, programs, other modules, or other data for the operation of system 500. In some implementations, the CRSM may include a data storage device that provides storage of computer-readable instructions or other information in a non-transitory format. The CRSM may be incorporated into system 500 or may be external to system 500. The CRSM may include read-only memory, random access memory, or both. The one or more CRSMs suitable for tangibly embodying computer program instructions and data may include any type of non-volatile memory, including, but not limited to, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices); magnetic disks (such as internal hard disks and removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. In some examples, processor(s) 510 and memory 520 may be supplemented by, or incorporated into, one or more application specific integrated circuits (ASICs).

[0127] The system 500 can include one or more I / O devices 560. The I / O device(s) 560 can include one or more input devices, such as a keyboard, mouse, pen, game controller, touch input device, audio input device (e.g., microphone), gesture input device, tactile input device, image or video capture device (e.g., camera), or other devices. In some examples, the I / O device(s) 560 can also include one or more output devices, such as a display, LED(s), audio output device (e.g., speaker), printer, tactile output device, or the like. The I / O device(s) 560 can be physically incorporated in the computing device(s) of the system 500, or can be external with respect to the computing device(s) of the system 500.

[0128] The system 500 can include one or more I / O interfaces 540 to enable components or modules of the system 500 to control, interface with, or otherwise communicate with the I / O device(s) 560. The I / O interface(s) 540 can enable information to be transmitted to or between components of the system 500 with respect to the system 500 by serial, parallel, or other type of communication. For example, the I / O interface(s) 540 can conform to a version of the RS-232 standard for serial ports, or a version of the IEEE 1284 standard for parallel ports. As another example, the I / O interface(s) 540 can be configured to provide connectivity through a Universal Serial Bus (USB) or Ethernet. In some examples, the I / O interface(s) 540 can be configured to provide serial connectivity compliant with a version of the IEEE 1394 standard.

[0129] The I / O interface(s) 540 can also include one or more network interfaces that enable communication between computing devices in the system 500 or between the system 500 and other computing systems connected to a network. The network interface(s) can include one or more network interface controllers (NICs) or other types of transceiver devices configured to send and receive communications over one or more networks using any network protocol.

[0130] The computing devices of system 500 can communicate with each other using one or more networks. Such networks can include public networks such as the Internet, private networks such as an institutional or personal intranet, or any combination of private and public networks. The networks can include any type of wired or wireless network, including but not limited to local area networks (LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs (WLANs), and mobile telecommunications networks (e.g., 3G, 4G, Edge, etc.), among others. In some implementations, communications between computing devices can be encrypted or otherwise secured. For example, communications can employ one or more public or private keys, ciphers, digital certificates, or other credentials supported by a security protocol such as any version of the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocol.

[0131] System 500 can include any number of any type of computing device. The computing device(s) can include, but are not limited to, personal computers, smart phones, tablet computers, wearable computers, implantable computers, mobile gaming devices, e-book readers, automobile computers, desktop computers, laptop computers, notebook computers, game consoles, home entertainment devices, network computers, server computers, mainframe computers, distributed computing devices (e.g., cloud computing devices), microcomputers, systems on a chip (SoCs), and systems in a package (SiPs), among others. Although examples herein can describe the computing device(s) as physical device(s), implementations are not so limited. In some examples, a computing device can include one or more of a virtual computing environment, a hypervisor, an emulation, and a virtual machine executing on one or more physical computing devices. In some examples, two or more computing devices can comprise a cluster, cloud, farm, or other grouping of devices that cooperate to provide load balancing, failover support, parallel processing capability, shared storage resources, shared networking capability, or other aspects.

[0132] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0133] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be machine-readable storage, machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0134] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0135] A computer program which can also be referred to or described as a program, software, software application, app, module, software module, script, or code can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. The

[0136] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, an index database can include multiple collections of data, which can be organized differently and accessed differently.

[0137] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components that are installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0138] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0139] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0140] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0141] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device of the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0142] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0143] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the device), and the client device transmits data (e.g., data generated by the user interacting with the device) back to the server. Data generated at the client device, including a page view or other user interaction may

[0144] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions, but rather as descriptions of features that can be specific to certain embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0145] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0146] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A system comprising: user interface circuitry for generating a user interface that, when rendered on a display device, comprises one or more visual representations of a tabular structured data file comprising a grid of data cells; Memory for storing: first data representing a plurality of characteristics of each of a plurality of test subjects, and second data representing a plurality of candidate propensity score models for estimating the effectiveness of a treatment for one or more of the test subjects; a processor communicatively coupled to the at least one memory, wherein the processor is configured to: accessing the first data and the second data from the memory; Selecting a propensity score model set from the plurality of candidate propensity score models, wherein selecting the propensity score model set comprises: Sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, wherein evaluating each of the candidate propensity score models comprises: obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and determining whether an output of the candidate propensity score model satisfies the one or more stopping criteria; and selecting the set of propensity score models based on sequential evaluation of at least some of the candidate propensity score models; and using the memory to store the tabular structured data file representing the set of propensity score models; and Outputting the tabular structured data file, wherein outputting the tabular structured data file comprises causing the user interface to be presented to a user using the user interface circuitry and the display device, wherein the user interface includes an indication of the propensity score model set.

2. A method comprising: Accessed by a computer system from one or more hardware storage devices: first data representing a plurality of characteristics of each of a plurality of test subjects, and second data representing a plurality of candidate propensity score models for estimating the effectiveness of a treatment for one or more of the plurality of test subjects; Selecting a propensity score model set from the plurality of candidate propensity score models, wherein selecting the propensity score model set comprises: Sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, wherein evaluating each of the candidate propensity score models comprises: obtaining an output of the candidate propensity score model based on at least a portion of the first data as input, and determining whether an output of the candidate propensity score model satisfies the one or more stopping criteria; and selecting the set of propensity score models based on sequential evaluation of at least some of the candidate propensity score models; and Using the one or more hardware storage devices to store, by the computer system, a data structure representing the propensity score model set; and The data structure is outputted, by the computer system, wherein outputting the data structure comprises causing, by the computer system, a user interface to be presented to a user, wherein the user interface comprises an indication of the set of propensity score models. The method of claim 2 , further comprising estimating the effectiveness of the treatment based on the set of propensity score models.

4. The method according to claim 3, wherein: Estimating the effectiveness of the treatment in clinical investigations includes: Selection bias associated with estimating the effect of the treatment is minimized based on at least one propensity score model in the set of propensity score models.

5. The method according to claim 2, further comprising: A clinical investigation is performed based on the data structure.

6. The method according to claim 2, further comprising: Treatment of one or more additional subjects is modified based on the data structure.

7. The method according to claim 2, wherein: The propensity score model set consists of one propensity score model.

8. The method according to claim 2, wherein: The propensity score model set includes multiple propensity score models.

9. The method according to claim 2, wherein: Sequentially evaluating at least some of the candidate propensity score models includes: determining the sequential order of the candidate propensity score models, and At least some of the candidate propensity score models are evaluated based on the sequential order.

10. The method according to claim 9, wherein: The sequence order is determined based on the complexity of each candidate propensity score model among the candidate propensity score models.

11. The method according to claim 9, wherein: The sequence order is determined based on the statistical complexity of each candidate propensity score model among the candidate propensity score models.

12. The method according to claim 2, wherein: The output of the candidate propensity score model represents the standard deviation of each of the one or more covariates of the candidate propensity score model.

13. The method according to claim 11, wherein: The one or more stopping criteria include determining that a standard deviation of each of the one or more covariates of the candidate propensity score model is less than a threshold value.

14. The method according to claim 11, wherein The one or more stopping criteria include determining that the number of evaluated candidate propensity score models is greater than or equal to a threshold.

15. The method according to claim 13, wherein The data structure comprises a tabular structured data file comprising a grid of data cells, wherein the data units are arranged in multiple rows and columns, and The data unit represents: at least one candidate propensity score model among the candidate propensity score models, and At least one standard deviation of the standard deviations corresponding to at least one of the candidate propensity score models.

16. The method according to claim 2, further comprising: receiving user input indicating one or more selection criteria, and At least some of the candidate propensity score models are selected based on the one or more selection criteria.

17. The method according to claim 2, wherein: Each of the candidate propensity score models is configured to generate a respective propensity score based on at least a portion of the first data, and wherein the propensity score represents a probability that the treatment has been administered to a specific test subject based on characteristics of the test subject.

18. A system comprising: at least one processor; as well as A memory is communicatively coupled to the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 2 to 17.

19. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 2 to 17.

20. A computer program product comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 2 to 17.

Images