Computerized system for increasing computational efficiency in evaluation and selecting propensity score models

Abstract

To provide a system and method for automatically evaluating and selecting propensity score models to increase computational efficiency of a computer system.SOLUTION: In an example method, a computer system accesses first data representing a plurality of characteristics of test subjects and second data representing a plurality of candidate propensity score models that can be used for estimating an effectiveness of a treatment. The system selects a set of propensity score models by sequentially evaluating at least some of the candidate propensity score models until one or more stop criteria are satisfied. The effectiveness of the treatment can be estimated by utilizing the selected set of propensity score models. Furthermore, the system stores a data structure representing the set of propensity score models and outputs the data structure.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This description generally relates to systems and methods for automatically evaluating and selecting propensity score models to increase the computational efficiency of computer systems. [Background technology]

[0002] Generally, a clinical trial is a prospective biomedical or behavioral research investigation designed to answer a specific question about a biomedical or behavioral intervention. For example, clinical trials can be conducted to evaluate the safety and / or efficacy of vaccines, medicines, dietary choices, nutritional supplements, and / or medical devices. Summary of the Invention

[0003] The computer system can be configured to automatically evaluate and select propensity score models, such as those used in analyzing the results of clinical trials.

[0004] Generally, propensity score models are used to generate propensity scores that can be used for matching or weighting, which is a statistical technique that attempts to estimate the effect of a treatment, policy, or other intervention by accounting for covariates that are considered confounds. For example, propensity score methods can be used to reduce confounding bias by selecting experimental and control subjects who complete a study so that subject characteristics systematically differ between treatment groups, perhaps leading to subjects with a better prognosis receiving one treatment over another.

[0005] In one exemplary implementation, a computer system may generate a list of multiple propensity score models for analyzing the results of a clinical trial. Each of the propensity score models may vary in suitability and effectiveness in estimating the association between a treatment situation (e.g., an "exposure") and a subject's observed characteristics (e.g., a "covariate"). The computer system may evaluate at least some of the propensity score models and select one or more that are particularly appropriate or effective in evaluating a particular set of outcomes. Furthermore, information about the selected propensity score models may be stored for future retrieval and / or presented to a user (e.g., to facilitate analysis of the clinical trial).

[0006] In some implementations, the computer system can be configured to automatically evaluate and select propensity score models by prioritizing the evaluation of some propensity score models over others, and to stop the evaluation when one or more suitable propensity score models are identified. For example, the computer system can be configured to sequentially evaluate propensity score models (e.g., in order of complexity, such as least complex to most complex) until one or more stopping criteria are met (e.g., indicating that one or more suitable propensity score models have been identified). Furthermore, the computer system can output one or more data structures representing the identified propensity score models.

[0007] The implementations described herein can provide various technical advantages. As an example, the implementations described herein enable a computer system to automatically evaluate and select propensity score models in a particularly efficient manner. For example, in a common technique, a computer system may evaluate all available propensity score models and select one or more propensity score models based on the evaluation. However, this technique can consume a large amount of computer resources (e.g., processing usage, memory usage, data storage usage, etc.) because the evaluation is brute force. In contrast, as described herein, the computer system can instead sequentially evaluate propensity score models and cease evaluating the propensity score models when one or more stopping criteria are met (e.g., indicating that one or more suitable propensity score models have been identified). Thus, in at least some implementations, the computer system does not need to evaluate all available propensity score models, and thus can consume fewer computer resources (e.g., compared to those consumed according to conventional techniques).

[0008] As another example, implementations described herein can be used to improve the analysis of clinical trial results (e.g., by selecting and utilizing a specifically appropriate propensity score model to analyze the results). This allows researchers to better assess the effectiveness of medical interventions, thereby improving the safety and / or efficacy of treatments for subjects. For example, the techniques described herein can be used to better understand the effectiveness of treatments and to modify treatments to further improve their efficacy. This further reduces the likelihood that clinical trial results will be misinterpreted or improperly analyzed, improving the efficiency with which treatments are researched and developed (e.g., by reducing the amount of resources consumed in pursuing research and development goals that are based on misinterpretation of clinical trial studies and / or improper analysis of clinical trial results).

[0009] As another example, implementations described herein can be used to provide a structured data file for storing information regarding the evaluation of propensity score models and presenting such information to a user in an organized and easy-to-understand manner, thereby enabling the user to intuitively determine the suitability of each of the evaluated propensity score models and select one or more of the propensity score models to facilitate the analysis of a clinical study.

[0010] In an aspect, a system includes a user interface circuit for generating a user interface that, when rendered on a display device, includes one or more visual representations of a tabular structured data file including a grid of data cells; a memory for storing first data representing a plurality of characteristics of each of a plurality of test subjects and second data representing a plurality of candidate propensity score models for estimating the effectiveness of a treatment administered to one or more of the test subjects; and a processor communicatively coupled to the at least one memory, the processor accessing the first data and the second data from the memory and selecting a set of propensity score models from among the plurality of candidate propensity score models, wherein the processor sequentially evaluates at least some of the candidate propensity score models until one or more stopping criteria are met. the sequential evaluation of each of the candidate propensity score models includes obtaining an output of the candidate propensity score model based at least in part on the first data as input and determining whether the output of the candidate propensity score model satisfies one or more stopping criteria; selecting a set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models; storing, using a memory, a tabular structured data file representing the set of propensity score models; and outputting the tabular structured data file, the outputting including causing a user interface to be presented to a user using a user interface circuit and a display device, the user interface including a display of the set of propensity score models.

[0011] In one aspect, a method includes accessing, by a computer system, from one or more hardware storage devices, first data representing a plurality of characteristics of each of a plurality of test subjects and second data representing a plurality of candidate propensity score models that can be used to estimate the effectiveness of a treatment for one or more of the plurality of test subjects, and selecting a set of propensity score models from among the plurality of candidate propensity score models. Selecting the set of propensity score models includes sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, where evaluating each of the candidate propensity score models includes obtaining an output of the candidate propensity score model based at least in part on the first data as input, and determining whether the output of the candidate propensity score model satisfies the one or more stopping criteria. Selecting the set of propensity score models also includes selecting the set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models. The method also includes storing, by the computer system, using one or more hardware storage devices, a data structure representing the set of propensity score models; and outputting, by the computer system, the data structure, the outputting including causing the computer system to present a user interface to a user, the user interface including a display of the set of propensity score models.

[0012] Implementations of this aspect can include one or more of the following features.

[0013] In some implementations, the method can also include estimating the effectiveness of the treatment based on the set of propensity score models.

[0014] In some implementations, the method can also include conducting a clinical study based on the data structure.

[0015] In some implementations, the method can also include modifying treatment of one or more additional subjects based on the data structure.

[0016] In some implementations, the set of propensity score models can include only a single propensity score model.

[0017] In some implementations, the set of propensity score models can include multiple propensity score models.

[0018] In some implementations, sequentially evaluating at least some of the candidate propensity score models can include determining a sequential order of the candidate propensity score models and evaluating at least some of the candidate propensity score models based on the sequential order.

[0019] In some implementations, the sequential order can be determined based on the complexity of each of the candidate propensity score models.

[0020] In some implementations, the sequential order can be determined based on the statistical complexity of each of the candidate propensity score models.

[0021] In some implementations, the output of the candidate propensity score model may represent the standardized difference for each of the one or more covariates of the candidate propensity score model.

[0022] In some implementations, the one or more stopping criteria may include determining that the standardized difference for each of the one or more covariates of the candidate propensity score model is less than a threshold value.

[0023] In some implementations, the one or more stopping criteria may include a determination that the number of candidate propensity score models evaluated is greater than or equal to a threshold value.

[0024] In some implementations, the data structure can include a tabular structured data file including a grid of data cells, the data cells arranged according to a plurality of rows and a plurality of columns, the data cells representing at least one of the candidate propensity score models and the standardized difference corresponding to at least one of the candidate propensity score models.

[0025] In some implementations, the method may include receiving user input representing one or more selection criteria and selecting at least some of the candidate propensity score models based on the one or more selection criteria.

[0026] In some implementations, each of the candidate propensity score models can be configured to generate a respective propensity score based at least in part on the first data, the propensity score representing the probability that a particular test subject has been administered the treatment based on characteristics of the test subject.

[0027] In some implementations, estimating the effectiveness of a treatment in a clinical study can include minimizing selection bias associated with estimating the effect of the treatment based on at least one propensity score model of the set of propensity score models.

[0028] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of a chat agent, a triage agent, and a retrieval agent. One or more computer systems can be configured to perform particular actions by installing software, firmware, hardware, or combinations thereof on the system that cause the system to perform the actions during operation. One or more computer programs can be configured to perform particular actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0029] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 illustrates an exemplary system for automatically evaluating and selecting propensity score models. [Figure 2] FIG. 1 illustrates an exemplary rating and selection engine. [Figure 3] FIG. 1 is a flowchart diagram of an exemplary process for automatically evaluating and selecting a propensity score model. [Figure 4] FIG. 1 is a flowchart diagram of an exemplary process for automatically evaluating and selecting a propensity score model. [Figure 5] FIG. 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE INVENTION

[0031] Like reference numbers and designations in the various drawings indicate like elements.

[0032] 1 illustrates an exemplary system 100 for automatically evaluating and selecting propensity score models, such as those used in analyzing the results of a clinical trial. For example, system 100 can receive clinical trial data (e.g., baseline characteristics) and automatically evaluate and select one or more propensity score models (e.g., one or more propensity score models that are particularly suitable for balancing the baseline characteristics) based on the clinical trial data. Furthermore, system 100 can output a data structure representing the selected propensity score models.

[0033] System 100 includes an evaluation and selection engine 150 implemented on computer system 102a. Generally, engine 150 is configured to obtain data representing propensity score models available to system 100 and evaluate at least some of the propensity score models (e.g., to determine the suitability of each of the evaluated propensity score models in analyzing a particular set of clinical trial data). Further, in at least some implementations, engine 150 can select one or more of the evaluation propensity score models (e.g., propensity score models that are particularly suitable for analyzing the set of clinical trial data) and present the selected propensity score models to a user.

[0034] For example, during exemplary operation of system 100, engine 150 obtains a set of clinical trial data. As one example, the set of clinical trial data may represent characteristics of each of the subjects in the clinical trial (e.g., the subject's age, health status, medical history, demographic information, etc.). As another example, the set of clinical trial data may represent the treatment status of each of the subjects (e.g., whether each of the subjects was selected for treatment, and if so, the type of treatment, whether each of the subjects was selected for placebo treatment, etc.). In some implementations, the set of clinical trial data may be embodied, at least in part, as one or more pieces of structured data.

[0035] Additionally, engine 150 can retrieve model data representing a propensity score model that can be used to analyze a set of clinical trial data. Generally, propensity score models are used to generate propensity scores that can be used for matching or weighting, which are statistical techniques that attempt to estimate the effect of a treatment, policy, or other intervention by accounting for covariates that are considered confounding. For example, propensity score methods can be used to reduce confounding bias by selecting experimental and control subjects who complete a study such that subject characteristics systematically differ between treatment groups, perhaps leading to subjects with a better prognosis receiving one treatment over another.

[0036] In at least some implementations, the potential for bias in clinical trials arises from differences in treatment outcomes (e.g., as average treatment effects) between treated and untreated groups that may be caused by factors predicting treatment rather than the treatment itself. For example, in randomized trials, randomization allows for unbiased estimates of treatment effects. For each covariate, randomization implies that treatment groups are balanced on average by the laws of randomization. Unfortunately, in observational studies, the assignment of treatments to study subjects is often not random. Propensity score matching and weighting attempt to mimic randomization, reducing treatment assignment bias, by creating a sample of treated units that is comparable to a sample of untreated units across all observed covariates.

[0037] In particular, the "propensity score" generated by a propensity score model represents the likelihood that a subject will receive a treatment, given the values ​​of the covariates. The stronger the confounding effect of a covariate, i.e., the more strongly the covariate is associated with whether a subject receives a treatment and with the outcome, the greater the bias in analyzing the naive treatment effect when the covariate is imbalanced between treatment groups. By having units with similar propensity scores in both the treatment and control groups, such confounding is mitigated.

[0038] In some implementations, the model data can include one or more mathematical functions, equations, computer macros, and / or portions of computer code for generating one or more propensity scores using a set of clinical trial data as input. In some implementations, the model data can be embodied, at least in part, as one or more portions of structured data.

[0039] In some implementations, at least a portion of the clinical trial data and / or model data can be retrieved from one or more hardware data storage devices 160a local to the computer system 102a. In some implementations, at least a portion of the clinical trial data and / or model data can be retrieved from one or more hardware data storage devices 160b remote from the computer system 102a (e.g., one or more remote computer systems 102b, such as server computers communicatively coupled to the computer system 102a via the network 106). In some implementations, at least a portion of the clinical trial data and / or model data can be manually entered by a user (e.g., using a graphical user interface (GUI) 154 presented by the computer system 102a).

[0040] Each of the propensity score models may vary in suitability and effectiveness in estimating the association between a treatment situation (e.g., an "exposure") and a subject's observed characteristics (e.g., a "covariate"). Engine 150 may evaluate at least some of the propensity score models (e.g., based on clinical trial data and model data) and select one or more propensity score models that are particularly appropriate or effective in evaluating a particular set of outcomes.

[0041] Additionally, data regarding the evaluation of the propensity score model and / or a representation of the selected propensity model can be stored for future retrieval and / or presented to a user (e.g., to facilitate analysis of a clinical trial). For example, in some implementations, at least a portion of the data regarding the evaluation of the propensity score model and / or a representation of the selected propensity model can be stored locally on hardware data storage device 160a and / or remotely on hardware data storage device 160b. As another example, in some implementations, at least a portion of the data regarding the evaluation of the propensity score model and / or a representation of the selected propensity model can be presented to a user via GUI 154.

[0042] In some implementations, engine 150 can be configured to automatically evaluate and select propensity score models by prioritizing the evaluation of some propensity score models over others, and to cease evaluation when one or more suitable propensity score models are identified.

[0043] For example, engine 150 can determine a sequence for evaluating propensity score models based on the model data. In some implementations, the sequence can be ordered by complexity (e.g., from the least complex propensity score model to the most complex propensity score model). In some implementations, the complexity of a propensity score model can be determined based on the type of operation (e.g., statistical operation) performed using the propensity score model, the type of statistical relationship (e.g., statistical relationship between covariates) modeled, and / or the amount of computer resources consumed by utilizing the propensity score model.

[0044] Further, engine 150 can be configured to evaluate the propensity score models according to the determined sequence until one or more stopping criteria are met (e.g., indicating that one or more suitable propensity score models have been identified).

[0045] For example, engine 150 can be configured to evaluate a propensity score model until the standardized difference among some or all of the evaluated covariates in the propensity score model is below a threshold (e.g., indicating that a sufficient degree of balance in baseline characteristics between different treatment groups has been achieved after propensity score matching or weighting). This can be beneficial, for example, in that it allows engine 150 to identify one or more suitable propensity score models without having to evaluate all available propensity score models (e.g., thereby reducing consumed computer resources).

[0046] As another example, engine 150 can be configured to evaluate propensity score models until a certain maximum number of propensity score models have been evaluated. Once the evaluation process is complete, engine 150 can select one or more of the best propensity score models tested. This can be beneficial, for example, in limiting the amount of computer resources consumed by engine 150.

[0047] Generally, the output of engine 150 can be used to analyze clinical trial data. For example, based on the output of engine 150, researchers can use certain propensity score models to account for covariates that predict receipt of a treatment evaluated in a clinical trial of interest, allowing them to better estimate the safety and / or efficacy of the treatment (e.g., to reduce confounding factors in subsequent analyses). This, in turn, can allow researchers to gain a better understanding of the safety and / or efficacy of a treatment of interest. For example, the techniques described herein can be used to better understand the effectiveness of a treatment and to modify the treatment to further improve its efficacy.

[0048] Generally, each of computer systems 102a and 102b can include any number of electronic devices configured to receive, process, and transmit data. Examples of computer systems include client computing devices (e.g., desktop computers or notebook computers), server computing devices (e.g., server computers or cloud computing systems), mobile computing devices (e.g., cellular phones, smartphones, tablets, personal digital assistants, network-enabled notebook computers), wearable computing devices (e.g., smart watches), and other computing devices capable of receiving, processing, and transmitting data. In some implementations, computer systems can include computing devices that operate using one or more operating systems (e.g., Microsoft Windows, Apple macOS, Linux, Unix, Google Android, and Apple iOS, among others) and one or more architectures (e.g., x86, PowerPC, ARM, among others). In some implementations, one or more of the computer systems need not be located locally with respect to the rest of system 100; one or more of the computer systems can be located at one or more remote physical locations.

[0049] Each of computer systems 102a and 102b may include a respective user interface (e.g., GUI 154) that allows a user to interact with the computer system, other computer systems, and / or engine 150. Exemplary interactions include viewing data, sending data from one computer system to another, and / or issuing commands to the computer systems. Commands may include, for example, any user instructions to one or more of the computer systems to perform a particular operation or task. In some implementations, a user may install software applications on one or more of the computer systems to facilitate the performance of these tasks.

[0050] 1, computer system 102a is depicted as a single component. However, in practice, computer system 102a may be implemented on one or more computing devices (e.g., each computing device includes at least one processor, such as a microprocessor or microcontroller). As an example, computer system 102a may be a single computing device connected to network 106, and engine 150 may be maintained and operated on this single computing device. As another example, computer system 102a may include multiple computing devices connected to network 106, and engine 150 may be maintained and operated on some or all of these computing devices. For example, computer system 102a may include multiple computing devices, and engine 150 may be distributed across one or more of these computing devices.

[0051] Network 106 can be any communications network capable of transferring and sharing data. For example, network 106 can be a local area network (LAN) or a wide area network (WAN) such as the Internet. Network 106 can be implemented using various network interfaces, such as, for example, a wireless network interface (such as Wi-Fi, Bluetooth, or infrared) or a wired network interface (such as an Ethernet or serial connection). Network 106 can also include a combination of multiple networks and can be implemented using one or more network interfaces.

[0052] 2 illustrates various aspects of evaluation and selection engine 150 in more detail. Engine 150 generally includes multiple computing modules that perform specific functions related to the operation of engine 150. For example, engine 150 includes a database module 210, a communications module 220, a processing module 230, and a user interface module 240. The operational modules may be provided as one or more computer-executable software modules, hardware modules, or a combination thereof. For example, one or more of the operational modules may be implemented as blocks of software code that include instructions that cause one or more processors of engine 150 to perform the operations described herein. Additionally or alternatively, one or more of the operational modules may be implemented in an electronic circuit, such as, for example, a programmable logic circuit, a field programmable logic array (FPGA), or an application-specific integrated circuit (ASIC).

[0053] The database module 210 maintains information related to the evaluation and selection of propensity score models.

[0054] As an example, the database module 210 can store input data 210a that is used as input for evaluating and selecting propensity score models.

[0055] As an example, input data 210a may include clinical trial data (e.g., as described with reference to FIG. 1), such as data describing characteristics of each of the clinical trial subjects and the treatment status of each of the subjects.

[0056] As another example, input data 210a may include model data representing propensity score models that can be used to analyze a set of clinical trial data (e.g., as described with respect to FIG. 1 ), such as data representing portions of one or more mathematical functions, equations, and / or computer code for generating one or more propensity score models.

[0057] As another example, input data 210a may include instructions from a user regarding the evaluation and selection of propensity score models. For example, input data 210a may include instructions from a user to evaluate a particular subset of propensity score models available to engine 150. Furthermore, input data 210a may include instructions from a user to evaluate propensity score models in a particular sequential order. Furthermore, input data 210a may include instructions from a user to stop evaluating propensity score models when particular stopping criteria are met.

[0058] In some implementations, at least a portion of the input data 210a may be retrieved from one or more local hardware data storage devices (e.g., hardware data storage device 160a) and / or remote hardware data storage devices (e.g., hardware data storage device 160b). In some implementations, at least a portion of the input data 210a may be received from a user (e.g., using the GUI 154 generated by the user interface module 240).

[0059] Additionally, database module 210 can store output data 210b generated by engine 150. As an example, output data 210b can include, for each of the evaluated propensity score models, data representing the results of the evaluation (e.g., one or more metrics indicating the effectiveness of the propensity score model in balancing characteristics between a group of subjects receiving a treatment of interest and a group not receiving the treatment). For example, output data 210b can represent one or more covariates of the clinical trial data and a metric (e.g., standardized difference) associated with each of the covariates.

[0060] As another example, output data 210b can include data representing a selection of one or more propensity score models. For example, output data 210b can include data indicating that one or more propensity score models met one or more criteria (e.g., data indicating that one or more suitable propensity score models were identified).

[0061] As another example, the output data 210b can filter or sort the data based on the results of the evaluation. For example, the output data 210b can present each of the evaluated propensity score models and can be sorted (e.g., from largest to smallest based on the number of subjects retained after matching). As another example, the output data 210b can present a subset of the evaluated propensity score models (e.g., the N-best propensity score models, where N is one or more) and omit the remaining propensity score models.

[0062] Additionally, the database module 210 can store processing rules 210c that specify how the data in the database module 210 can be processed to evaluate and select propensity score models.

[0063] As an example, processing rules 210c may include one or more rules for identifying propensity score models for evaluation by engine 150 and the order in which they are to be evaluated.

[0064] As another example, processing rules 210c may indicate how to input data into each of the available propensity score models and how to generate output data 210b (e.g., representing an evaluation of the propensity score models) based on the input data.

[0065] As another example, processing rules 210c may indicate one or more rules for determining whether to stop evaluating a propensity score model (e.g., one or more rules specifying that evaluation of a propensity score model is to stop when certain stopping criteria are met).

[0066] As another example, the processing rules 210c may specify that the generated output data 210b be presented to a user and / or stored for future retrieval and / or processing (e.g., using the database module 210).

[0067] Exemplary data processing techniques are described in further detail below.

[0068] As mentioned above, engine 150 also includes a communications module 220. Communications module 220 enables data to be sent to and received from engine 150. For example, communications module 220 may be communicatively coupled to network 106 to send data to and receive data from computer system 102b. Information received from computer system 102b may be processed (e.g., using processing module 230) and stored (e.g., using database module 210).

[0069] As mentioned above, engine 150 also includes processing module 230. Processing module 230 processes data stored in or otherwise accessible to engine 150. For example, processing module 230 can be used to perform one or more of the operations described herein (e.g., operations associated with evaluating and selecting a propensity score model).

[0070] User interface module 240 is configured to present information to and / or receive input from a user. As an example, user interface module 240 may include one or more display devices (e.g., display screens, touch screens, etc.) configured to present a user interface (e.g., GUI 154) that allows a user to interact with computer system 102a and / or engine 150. Exemplary interactions include viewing data, transmitting data from one component to another, and / or issuing commands to computer system 102a and / or engine 150. Commands may include, for example, any user instructions to one or more of computer system 102a and / or engine 150 to perform a particular operation or task.

[0071] In some implementations, a software application can be used to facilitate the performance of the tasks described herein. As an example, an application can be installed on computer system 102a. Furthermore, a user can interact with the application to input data and / or commands to engine 150 and to review data generated by engine 150.

[0072] As described above, a computer system (e.g., using engine 150) can be configured to evaluate and select propensity score models by prioritizing the evaluation of some propensity score models over others, and to cease evaluation when one or more suitable propensity score models are identified. As an example, a process 300 for evaluating propensity score models is shown in FIG. 3.

[0073] In process 300, a computer system accesses data regarding propensity score models available for evaluation (e.g., candidate propensity score models) and evaluates any of the propensity score models in which a set of covariates from the clinical trial data is represented in its original form in the propensity score model (302).

[0074] Having performed the evaluation, the computer system determines whether one or more stopping criteria have been met 304. Generally, the stopping criteria may specify conditions that, when met, indicate that one or more suitable propensity score models have been identified.

[0075] In some implementations, the one or more stopping criteria can include a criterion that is met when the standardized differences of all evaluated covariates in the evaluated propensity score model are less than a threshold value (e.g., "STDDIFF_CUT").

[0076] In some implementations, the one or more stopping criteria may include a criterion that is met if the total number of propensity score models evaluated is greater than or equal to a threshold number (e.g., "MOD_MAX_NUM").

[0077] In some implementations, the one or more stopping criteria can include a combination of multiple criteria, and the stopping criteria can be considered met if at least one of the criteria is met. For example, as shown in Figure 3, the stopping criteria can be considered met if (i) the standardized differences of all evaluated covariates in the evaluated propensity score models are less than a threshold value (e.g., "STDDIFF_CUT"), or (ii) the total number of evaluated propensity score models is greater than or equal to a threshold number (e.g., "MOD_MAX_NUM").

[0078] If one or more stopping criteria are met, the process 300 ends (306).

[0079] At the end of process 300, the computer system generates one or more data structures representing the evaluated propensity score models. For example, the computer system can generate a report summarizing the covariates of each of the evaluated propensity score models. As another example, the computer system can select one or more best-performing propensity score models (e.g., smallest standardized differences in covariates) and present those propensity score models in the report.

[0080] In some implementations, the data structure can include one or more spreadsheets. For example, the data structure can include a tabular structured data file including a grid of data cells, the data cells arranged according to rows and columns. Further, the data cells can include information such as (i) one or more of the propensity score models evaluated and (ii) information about each of those propensity score models (e.g., standardized differences of covariates for those propensity score models). In some implementations, the data cells can be sorted and / or filtered to facilitate analysis of the propensity score models (e.g., as described above).

[0081] At the end of process 300, the computer system may also generate the number and percentage of matched subjects in each treatment group using the selected propensity score model if propensity score matching is selected, or the weighted number of subjects in each treatment group if propensity score weighting is selected.

[0082] If one or more stopping criteria are not met, the computer system continues evaluating additional propensity score models until one or more stopping criteria are met. For example, as shown in Figure 3, the computer system can sequentially evaluate the following: (i) Any propensity score model (308) with square-root transformed variables (one at a time) for non-negative numeric covariates (i.e., covariates with values ​​>= 0); (ii) any propensity score model (310) with log-transformed variables (one at a time) for positive numeric covariates (i.e., covariates with values ​​>0); (iii) any propensity score model (312) with logit-transformed variables (one at a time) for percentage-numeric covariates (i.e., covariates with values ​​between 0 and 100); (iv) any propensity score model with square-root transformed variables (multiple times) for nonnegative numeric covariates (314); (v) any propensity score model with log-transformed variables (multiple times) for positive numeric covariates (316); (vi) any propensity score model with logit-transformed variables (multiple times) for percentage-numerical covariates (318); and (vii) any propensity score model with combinations of different transformed numerical covariates (e.g., two combinations of variables X and Y, e.g., sqrt(X) + log(Y), log(X) + sqrt(Y), etc.) (320); (viii) any propensity score model with interaction variables without transformation (322), and (iX) Any propensity score model with interaction variables with transformation (324).

[0083] At each evaluation stage 308, 310, 312, 314, 316, 318, 320, and 322, the computer system determines whether one or more stopping criteria have been met 304. If so, the process 300 ends 306. If not, the process 300 proceeds to the next evaluation stage in the sequence.

[0084] Once the evaluation stage 324 is complete, the process 300 ends, even if one or more stopping criteria have not been met.

[0085] In some implementations, a user can specify that the computer system automatically generate and evaluate all available propensity score models using a transformation (e.g., "NUM_AUTO_TRANS") 326. If this option is turned off by the user, the computer system can skip the evaluation of propensity score models with transformations (e.g., evaluation steps 308, 310, 312, 314, 316, 318, and 320) and proceed directly from evaluation step 302 to evaluation step 322.

[0086] In some implementations, a user can specify that the computer system evaluate a specific subset of available propensity score models (e.g., "INTERACT_NO") (328). When this option is specified by the user as any non-negative integer, the computer system can evaluate propensity score models with two-way interactions (evaluation steps 322 and 324) using up to the specified number of two-way interactions in each propensity score model. For example, if this value is set to 1, each model can have at most one interaction term.

[0087] Example Process 4 shows an example process 400 for automatically evaluating and selecting propensity score models, such as those used in analyzing the results of clinical trials. In some implementations, process 400 can be performed by system 100 (e.g., using engine 150) described in this disclosure.

[0088] In process 400, the system accesses (402) from one or more hardware storage devices: (i) first data representing a plurality of characteristics of each of a plurality of test subjects; and (ii) second data representing a plurality of candidate propensity score models that can be used to estimate the effectiveness of a treatment for one or more of the plurality of test subjects.

[0089] Additionally, the system selects a set of propensity score models from among the plurality of candidate propensity score models (404). In some implementations, each of the candidate propensity score models can be configured to generate a respective propensity score based at least in part on the first data. Furthermore, the propensity score can represent a probability that a particular test subject has been administered the treatment based on the characteristics of the test subject.

[0090] Selecting the set of propensity score models includes sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met (404a). Evaluating each of the candidate propensity score models includes (i) obtaining an output of the candidate propensity score model based at least in part on the first data as input, and (ii) determining whether the output of the candidate propensity score model satisfies the one or more stopping criteria.

[0091] In some implementations, sequentially evaluating at least some of the candidate propensity score models can include (i) determining a sequential order of the candidate propensity score models and (ii) evaluating at least some of the candidate propensity score models based on the sequential order.

[0092] In some implementations, the sequential order can be determined based on the complexity of each of the candidate propensity score models.

[0093] In some implementations, the sequential order can be determined based on the statistical complexity of each of the candidate propensity score models.

[0094] Selecting the set of propensity score models also includes selecting the set of propensity score models based on sequential evaluation of at least some of the candidate propensity score models (404b).

[0095] In some implementations, the set of propensity score models can include only a single propensity score model.

[0096] In some implementations, the set of propensity score models can include multiple propensity score models.

[0097] In some implementations, the output of the candidate propensity score model may represent the standardized difference for each of the one or more covariates of the candidate propensity score model.

[0098] In some implementations, the one or more stopping criteria may include determining that the standardized difference for each of the one or more covariates of the candidate propensity score model is less than a threshold value.

[0099] In some implementations, the one or more stopping criteria may include a determination that the number of candidate propensity score models evaluated is greater than or equal to a threshold value.

[0100] The system uses one or more hardware storage devices to store (406) a data structure representing the set of propensity score models.

[0101] The system outputs the data structure 408. For example, the system can present a user interface to the user, the user interface including a display of the set of propensity score models.

[0102] Further, the efficacy of the treatment can be estimated based on the set of propensity score models 410. In some implementations, estimating the efficacy of the treatment for the clinical study can include minimizing selection bias associated with estimating the effect of the treatment based on at least one propensity score model of the set of propensity score models.

[0103] In some implementations, process 400 may also include conducting a clinical study based on the data structure (e.g., based on selected propensity score models and / or propensity scores generated by those models).

[0104] In some implementations, process 400 may also include modifying the treatment of one or more additional subjects based on the data structure (e.g., based on the selected propensity score models and / or the propensity scores generated by those models).

[0105] In some implementations, process 400 may also include receiving user input representing one or more selection criteria and selecting at least some of the candidate propensity score models based on the one or more selection criteria.

[0106] Exemplary Computer System 5 illustrates an exemplary computing system according to an implementation of the present disclosure. System 500 may be used for any of the operations described with respect to the various implementations discussed herein. System 500 may include one or more processors 510, memory 520, one or more storage devices 530, and one or more input / output (I / O) devices 560 controllable via one or more I / O interfaces 540. The various components 510, 520, 530, 540, or 560 may be interconnected through at least one system bus 550, enabling data transfer between the various modules and components of system 500.

[0107] The processor 510 may be configured to process instructions for execution within the system 500. The processor 510 may include a single-threaded processor, a multi-threaded processor, or both. The processor 510 may be configured to process instructions stored in the memory 520 or the storage device 530. The processor 510 may include hardware-based processors, each including one or more cores. The processor 510 may include a general-purpose processor, a special-purpose processor, or both.

[0108] The memory 520 may store information within the system 500. In some implementations, the memory 520 includes one or more computer-readable media. The memory 520 may include any number of volatile memory units, any number of non-volatile memory units, or both volatile and non-volatile memory units. The memory 520 may include read-only memory, random access memory, or both. In some examples, the memory 520 may be used as active or physical memory by one or more executing software modules.

[0109] The storage device 530 may be configured to provide (e.g., persistent) mass storage for the system 500. In some implementations, the storage device 530 may include one or more computer-readable media. For example, the storage device 530 may include a floppy disk device, a hard disk device, an optical disk device, or a tape device. The storage device 530 may include read-only memory, random access memory, or both. The storage device 530 may include one or more of an internal hard drive, an external hard drive, or a removable drive.

[0110] Either or both of memory 520 or storage device 530 may include one or more computer-readable storage media (CRSM). The CRSM may include one or more of electronic storage media, magnetic storage media, optical storage media, magneto-optical storage media, quantum storage media, mechanical computer storage media, etc. The CRSM may provide storage of computer-readable instructions describing data structures, processes, applications, programs, other modules, or other data for operation of system 500. In some implementations, the CRSM may include a data store providing non-transitory storage of computer-readable instructions or other information. The CRSM may be incorporated into system 500 or may be external to system 500. The CRSM may include read-only memory, random-access memory, or both. One or more CRSMs suitable for tangibly embodying computer program instructions and data may include any type of non-volatile memory, including, but not limited to, semiconductor memory devices such as EPROM, EEPROM, flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. In some examples, processor 510 and memory 520 may be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs).

[0111] System 500 may include one or more I / O devices 560. I / O devices 560 may include one or more input devices such as a keyboard, mouse, pen, game controller, touch input device, audio input device (e.g., microphone), gesture input device, haptic input device, image or video capture device (e.g., camera), or other device. In some examples, I / O devices 560 may also include one or more output devices such as a display, LEDs, audio output device (e.g., speaker), printer, haptic output device, etc. I / O devices 560 may be physically incorporated into one or more computing devices of system 500 or may be external to one or more computing devices of system 500.

[0112] The system 500 may include one or more I / O interfaces 540 to enable components or modules of the system 500 to control, interface with, or otherwise communicate with I / O devices 560. The I / O interfaces 540 may allow information to be transferred within or out of the system 500, or between components of the system 500, through serial, parallel, or other types of communication. For example, the I / O interfaces 540 may conform to a version of the RS-232 standard for serial ports or a version of the IEEE 1284 standard for parallel ports. As another example, the I / O interfaces 540 may be configured to provide connections via Universal Serial Bus (USB) or Ethernet. In some examples, the I / O interfaces 540 may be configured to provide serial connections conforming to a version of the IEEE 1394 standard.

[0113] I / O interface 540 may also include one or more network interfaces that enable communication between computing devices within system 500 or between system 500 and other computing systems connected to a network. A network interface may include one or more network interface controllers (NICs) or other types of transceiver devices configured to send and receive communications over one or more networks using any network protocol.

[0114] The computing devices of system 500 may communicate with each other and with other computing devices using one or more networks. Such networks may include public networks such as the Internet, private networks such as organizational or personal intranets, or any combination of private and public networks. The networks may include any type of wired or wireless network, including, but not limited to, local area networks (LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs (WLANs), mobile communication networks (e.g., 3G, 4G, edge, etc.), etc. In some implementations, communications between computing devices may be encrypted or otherwise secured. For example, communications may use one or more public or private encryption keys, ciphers, digital certificates, or other credentials supported by a security protocol such as the Secure Sockets Layer (SSL) or any version of the Transport Layer Security (TLS) protocol.

[0115] System 500 may include any number of any type of computing devices. Computing devices may include, but are not limited to, personal computers, smartphones, tablet computers, wearable computers, embedded computers, mobile gaming devices, e-readers, in-vehicle computers, desktop computers, laptop computers, notebook computers, game consoles, home entertainment devices, network computers, server computers, mainframe computers, distributed computing devices (e.g., cloud computing devices), microcomputers, systems-on-chips (SoCs), systems-in-packages (SiPs), etc. Although examples herein describe computing devices as physical devices, implementations are not limited in this respect. In some examples, computing devices may include one or more virtual computing environments, hypervisors, emulations, or virtual machines running on one or more physical computing devices. In some examples, two or more computing devices may include a cluster, cloud, farm, or other group of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, shared network capabilities, or other aspects.

[0116] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0117] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or additionally, the program instructions can be encoded in an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus.

[0118] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0119] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any style of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any style, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple linked files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a data communications network.

[0120] The term "database" is used broadly herein to refer to any collection of data. The data need not be structured in any particular way, or even structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an index database can contain multiple collections of data, each of which can be organized and accessed in a different way.

[0121] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer or multiple identical computers.

[0122] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0123] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic elements of a computer are a central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from and / or transfer data to these storage devices. However, a computer need not necessarily include such devices. Furthermore, a computer may be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0124] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROMs, EEPROMs, flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0125] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer that includes a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0126] Embodiments of the subject matter described herein can be implemented in a computing system including a back-end component, e.g., as a data server, or a middleware component, e.g., as an application server, or a front-end component, e.g., as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0127] A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server acts as a client, sending data, e.g., HTML pages, to a user device for the purpose of displaying data to a user interacting with the device and receiving user input. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0128] While this specification contains details of many specific implementations, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0129] Similarly, while the figures may depict operations in a particular order, and the claims may describe operations in a particular order, this should not be understood as requiring that such operations be performed in the particular order depicted, or in the sequential order depicted, or that all of the depicted operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0130] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0131] 100 systems 102a Computer Systems 106 Network 150 Rating and Selection Engine 150 Engine 154 GUI 160a Hardware Data Storage Devices 160b Hardware Data Storage Devices 210 Database Module 210a Input data 210b Output data 210c Processing Rules 220 Communication Module 230 Processing Module 240 User Interface Module 300 processes 400 processes 500 Systems 510 processor 520 memory 530 Storage Devices 540 I / O interface 550 System Bus 560 Input / Output (I / O) Devices

Claims

1. a user interface circuit for generating a user interface that, when rendered on a display device, comprises one or more visual representations of a tabular structured data file including a grid of data cells; A memory, first data representing a plurality of characteristics of each of a plurality of test subjects; second data representing a plurality of candidate propensity score models for estimating the effectiveness of treatments administered to one or more of the test subjects; a memory for storing the a processor communicatively coupled to the at least one memory; 1. A system comprising: accessing the first data and the second data from the memory; selecting a set of propensity score models from among the plurality of candidate propensity score models; sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, wherein evaluating each of the candidate propensity score models comprises: obtaining an output of the candidate propensity score model based at least in part on the first data as an input; determining whether the output of the candidate propensity score model satisfies the one or more stopping criteria; and sequentially evaluating the selecting the set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models; selecting, using the memory to store the tabular structured data file representing the set of propensity score models; outputting the tabular structured data file, comprising using the user interface circuitry and the display device to present the user interface to a user, the user interface comprising a display of the set of propensity score models; and A system configured to:

2. by the computer system from one or more hardware storage devices, first data representing a plurality of characteristics of each of a plurality of test subjects; second data representing a plurality of candidate propensity score models for estimating the efficacy of treatment for one or more of the plurality of test subjects; accessing the selecting a set of propensity score models from among the plurality of candidate propensity score models, sequentially evaluating at least some of the candidate propensity score models until one or more stopping criteria are met, wherein evaluating each of the candidate propensity score models comprises: obtaining an output of the candidate propensity score model based at least in part on the first data as an input; determining whether the output of the candidate propensity score model satisfies the one or more stopping criteria; a sequentially evaluating step comprising: selecting the set of propensity score models based on the sequential evaluation of at least some of the candidate propensity score models; a selecting step comprising: storing, by the computer system, a data structure representing the set of propensity score models using the one or more hardware storage devices; outputting the data structure by the computer system, the outputting comprising causing the computer system to present a user interface to a user, the user interface comprising a display of the set of propensity score models; A method comprising:

3. The method of claim 2 , further comprising estimating the efficacy of the treatment based on the set of propensity score models.

4. estimating the efficacy of the treatment in the clinical study, 4. The method of claim 3, comprising minimizing selection bias associated with the estimate of treatment effect based on the at least one propensity score model of the set of propensity score models.

5. The method of claim 2 , further comprising conducting a clinical study based on the data structure.

6. 3. The method of claim 2, further comprising modifying the treatment of one or more additional subjects based on the data structure.

7. The method of claim 2 , wherein the set of propensity score models consists of one propensity score model.

8. The method of claim 2 , wherein the set of propensity score models comprises a plurality of propensity score models.

9. sequentially evaluating at least some of the candidate propensity score models, determining a sequential order of the candidate propensity score models; evaluating at least some of the candidate propensity score models based on the sequential order; The method of claim 2 , comprising:

10. The method of claim 8 , wherein the sequential order is determined based on the complexity of each of the candidate propensity score models.

11. The method of claim 8 , wherein the sequential order is determined based on the statistical complexity of each of the candidate propensity score models.

12. 3. The method of claim 2, wherein the output of the candidate propensity score model represents a standardized difference for each of one or more covariates of the candidate propensity score model.

13. 12. The method of claim 11 , wherein the one or more stopping criteria comprise a determination that the standardized difference for each of the one or more covariates of the candidate propensity score model is less than a threshold.

14. The method of claim 11 , wherein the one or more stopping criteria comprises a determination that the number of evaluated candidate propensity score models is greater than or equal to a threshold.

15. the data structure comprises a tabular structured data file including a grid of data cells; the data cells are arranged according to a plurality of rows and a plurality of columns; The data cell is at least one of the candidate propensity score models; at least one of the standardized differences corresponding to at least one of the candidate propensity score models; The method of claim 13, wherein

16. receiving user input representing one or more selection criteria; selecting at least some of the candidate propensity score models based on the one or more selection criteria; The method of claim 2 further comprising:

17. 3. The method of claim 2, wherein each of the candidate propensity score models is configured to generate a respective propensity score based at least in part on the first data, the propensity score representing a probability that the treatment was administered to a particular test subject based on the characteristics of the test subject.

18. at least one processor; a memory communicatively coupled to the at least one processor; wherein said memory stores instructions that, when executed by said at least one processor, cause said at least one processor to perform a method according to any one of claims 2 to 17.

19. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 2 to 17.

Images