Systems and methods for measurement-efficient randomized controlled trials using transformers with input masking

Abstract

In various aspects a method for reducing a number of measurements needed by a phase- 3 clinical randomized controlled trial may be provided. The method may include receiving initial data. The method may also include pre-processing the initial data. The method may also include generating candidate planned missing designs. Generating candidate planned missing designs may include training initial imputer models across a range of masking efficiency levels. Generating candidate planned missing designs may also include using the initial imputer models to learn anew set of model-optimized PMDs. The method may also include identifying optimal imputer-PMD pairs. Identifying optimal imputer-PMD pairs may include calculating aPMD's efficiency and its imputer performance. Identifying optimal imputer-PMD pairs may also include comparing a performance of each learned imputer-PMD pair and each initial imputer-PMD pair. The method may also include identifying eligible candidate pairs. The method may also include selecting a stored imputer-PMD pair.

Description

[0001] SYSTEMS AND METHODS FOR MEASUREMENT-EFFICIENT RANDOMIZED CONTROLLED TRIALS USING TRANSFORMERS WITH INPUT MASKING

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] The present application claims priority to U.S. Provisional Patent Application 63 / 662.467, filed June 21, 2024, the contents of which are incorporated by reference herein in its entirety.

[0004] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0005] This invention was made with government support under Grant No. CNS-1907381 awarded by the National Science Foundation. The government has certain rights in the invention.

[0006] TECHNICAL FIELD

[0007] The present disclosure is drawn to systems and methods for measurement-efficient randomized controlled trials using transformers with input masking.

[0008] BACKGROUND

[0009] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of the present disclosure that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate abetter understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0010] Clinical randomized controlled trials (RCTs) collect hundreds of measurements spanning various metric types (e.g., laboratory' tests, cognitive / motor assessments, etc.) across lOOs-lOOOs of subjects to evaluate the effect of a treatment, but do so at the cost of significant trial expense. To reduce the number of measurements, trial protocols can be revised to remove metrics extraneous to the study’s objective, but doing so requires additional human labor and limits the set of hypotheses that can be studied with the collected data. In contrast, a planned missing design (PMD) can reduce the amount of data collected without removing any metric by imputing the unsampled data. Standard PMDs randomly sample data to leverage statistical properties of imputation algorithms, but are ad hoc, hence suboptimal. Methods that can learn PMDs produce more sample-efficient PMDs, but are not suitable for RCTs because they require ample prior data (150+ subjects) to model the data distribution.

[0011] BRIEF SUMMARY

[0012] Various deficiencies in the prior art are addressed below by the disclosed systems and methods for reducing a number of measurements needed by a phase-3 clinical randomized controlled trial.

[0013] In various aspects, a method for reducing a number of measurements needed by a phase- 3 clinical randomized controlled trial may be provided. The method may include receiving initial data including data from a study containing fewer subjects than a complete conventional RCT, a protocol schedule, and a subject missingness matrix. The method may also include preprocessing the initial data by handling native missingness, normalizing the initial data, and dividing the initial data into training and validation sets.

[0014] The method may also include generating candidate planned missing designs (PMDs). Generating candidate planned missing designs may include training initial imputer models across a range of masking efficiency levels. Each efficiency level may be defined as a fraction of measurements collected under a study protocol that are masked out by the candidate PMD. Losses may be evaluated over eligible masked elements. Generating candidate PMDs may also include using the initial imputer models to leam a new set of models and model-optimized PMD by using a mask learning algorithm to leam a binary mask by modeling it as a differentiable layer of logits and fitting it with atask objective along with regularization applied to the logits, where, during training, imputer weights are not frozen, and gradients are zeroed out for ineligible measurements under a user-specified constraint matrix.

[0015] The method may further include identifying optimal imputer-PMD pairs. Identifying optimal imputer-PMD pairs may include calculating a PMD’s efficiency and its imputer performance. Identifying optimal imputer-PMD pairs may include comparing a performance of each learned imputer-PMD pair and each initial imputer-PMD pair, and ranking the learned imputer-PMD pairs based on performance differences and design objective.

[0016] Identifying optimal imputer-PMD pairs may also include identifying eligible candidate pairs among the learned imputer-PMD pairs by determining if each imputer-PMD pair has an efficiency above an efficiency threshold and imputation performance above an imputation performance threshold, where thresholds are determined by a reference imputer-PMD pair (i.e., a given initial imputer-PMD pair), pruning the eligible candidate pairs by storing the eligible candidate pairs with comparable efficiency and lowest imputation error or. if no eligible candidate pairs exist, storing a reference imputer-PMD pair.

[0017] The method may also include selecting a stored imputer-PMD pair.

[0018] In some embodiments, a selected candidate imputer-PMD pair may have an efficiency closest to a target efficiency.

[0019] In some embodiments, the method may further include applying a selected candidate imputer-PMD pair to an RCT study. The method may also include imputing measurements omitted from the RCT study using the at least one of the eligible candidate pairs.

[0020] In some embodiments, the method may further include choosing a stored imputer-PMD pair with desired performance characteristics and performing a phase-3 clinical RCT where measurements are only collected based on the PMD of the stored imputer-PMD pair, and imputing corresponding dropped measurements using the imputer of the stored imputer-PMD pair.

[0021] In some embodiments, each imputer may be trained using an MTSIT algorithm. In some embodiments, generating candidate PMDs may include using an RSD-based PMD generator to generate random binary matrices of shape ntx nmusing a Bernoulli distribution parametrized by an efficiency level while being constrained by the user constraints.

[0022] In some embodiments, learning the new set of models and model-optimized PMD may include generating a set of diverse PMDs per imputation model by sampling different hyperparameter combinations of mask regularization weights and learning rates used to train a mask-modulated model.

[0023] In some embodiments, for each sampled mask regularization weight and learning rate, a mask layer may be attached to a bottom of each initial imputer, where a same Transformer architecture may be used for the imputer with weights initialized from an initially trained imputer. A weight mask may be initialized according to a distribution used to generate an RSD- based PMD of a given efficiency. The masked imputer may be trained on an imputation task induced by a parametrized PMD using an MTSIT algorithm, pilot study dataset, and sampled hyperparameters.

[0024] In some embodiments, calculating a PMD’s efficiency and its imputer performance may include, for continuous metrics, reporting a value representing a normalized root-meansquare-deviation (nRMSD) over masked elements. In some embodiments, calculating a PMD’s efficiency and its imputer performance may include, for categorical metrics, calculating accuracy and macro Fl over masked elements per metric and reporting a median of scores across metrics to yield a single score per performance metric.

[0025] In some embodiments, comparing a performance may include calculating confidence intervals per performance metric and using the confidence intervals to lowerbound a performance gain of a given learned pair over a reference pair.

[0026] In some embodiments, the user constraints may include at least one of a target budget or efficiency level. In some embodiments, the eligible masked elements may be determined by the user constraints.

[0027] In some embodiments, comparing the performance of each learned imputer-PMD pair and each initial imputer-PMD pair may be based on confidence intervals.

[0028] In various aspects, a non-transitory computer-readable storage medium may be provided. The non-transitory computer-readable storage medium may include instructions that, when executed by one or more processing units, cause the one or more processing units to perform a method as disclosed herein.

[0029] In various aspects, a system may be provided. The system may include one or more processing units. The system may also include a non-transitory computer-readable storage medium as disclosed herein. The non-transitory computer-readable storage medium may be operably coupled to the one or more processing units.

[0030] BRIEF DESCRIPTION OF FIGURES

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with a general description of the present disclosure given above, and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0032] Figure 1 shows a depiction of a flow diagram of an embodiment of a method.

[0033] Figure 2 shows a schematic illustration of a system.

[0034] Figure 3A shows a depiction of a flowchart for generating candidate planned missing designs (PMDs).

[0035] Figure 3B shows a depiction of a flowchart for choosing PMDs.

[0036] Figure 3C shows a depiction of a flowchart of the selection of the PMD-imputer pair.

[0037] Figure 4 show's a depiction of a flowchart of an embodiment of the METRIK framework. Figure 5 shows a depiction of a flowchart of an embodiment of an initial imputer learning step.

[0038] Figure 6 shows a depiction of a flowchart of an embodiment for the generation of candidate PMD-imputer pairs.

[0039] Figure 7A shows a depiction of an algorithm for generating candidate PMDs.

[0040] Figure 7B shows a depiction of an algorithm for choosing PMDs.

[0041] Figure 8 shows a depiction of a Table of Dataset properties.

[0042] Figure 9 shows a depiction of a Table of Dataset properties.

[0043] Figures 10A and 10B show graphs of performance gains obtained by METRIK over baseline PMD algorithms for a setting that maximizes efficiency.

[0044] Figures 11A. 11B, and 11C show graphs of performance gains under METRIK over baseline PMD algorithms across baseline efficiency levels.

[0045] Figures 12A and 12B show graphs of performance gains under METRIK over baseline PMD algorithms across baseline efficiency levels.

[0046] Figures 13 A, 13B, and 13C show graphs of performance gains obtained by METRIK over an RSD baseline under two design objectives, i.e., maximize efficiency or maximize imputation performance.

[0047] Figures 14A and 14B show graphs of results comparing the effect of the two design objectives, i.e., maximize efficiency or maximize imputation performance.

[0048] Figure 15 shows a depiction of sample PMDs produced by RSD (top two rows) and a sample planned missing design generated by METRIK (bottom row).

[0049] Figures 16A, 16B, 16C, and 16D show depictions of sample PMDs produced by RSD and METRIK (bottom row) across datasets and different metric types for a setting where the design objective is to maximize efficiency.

[0050] Figures 17A, 17B, and 17C show graphs of performance gains under METRIK and an ablated version that replaces the candidate pool of learned PMDs with random ones generated by an MF design.

[0051] Figures 18A and 18B show graphs of results comparing performance gains under METRIK against an ablated version.

[0052] Figures 19A, 19B, and 19C show graphs of results comparing performance gains under METRIK against an ablated version.

[0053] Figures 20A and 20B show graphs of results comparing performance gains under METRIK against an ablated version. Figures 21 A, 21B, and 21C show graphs of performance gains under METRIK and an ablated version that does not use confidence intervals for performance estimation during PMD selection.

[0054] Figures 22 A and 22B show graphs comparing performance gains under METRIK against an ablated version that does not use confidence intervals for performance estimation during PMD selection.

[0055] Figures 23 A. 23B, and 23C show graphs comparing performance gains under METRIK against an ablated version that does not use confidence intervals for performance estimation during PMD selection.

[0056] Figures 24A and 24B show graphs comparing performance gains under METRIK against an ablated version that does not use confidence intervals for performance estimation during PMD selection.

[0057] Figures 25A, 25B, and 25C show boxplots of the distribution in the solution characteristics.

[0058] Figure 26A shows a scatterplot of the comparison of the imputation performance between METRIK and the baseline-generated solution across the NN 102 dataset.

[0059] Figure 26B shows a boxplot of the performance difference between METRIK and the baseline across the NN102 dataset.

[0060] Figure 27A shows a scatterplot of the comparison of the imputation performance between METRIK and the baseline-generated solution across the LSI dataset.

[0061] Figure 27B shows a boxplot of the performance difference between METRIK and the baseline across the NN102 dataset.

[0062] Figure 28A shows a scatterplot of the comparison of the imputation performance between METRIK and the baseline-generated solution across the LSI dataset.

[0063] Figure 28B shows a boxplot of the performance difference between METRIK and the baseline across the NN102 dataset.

[0064] Figure 29A shows a scatterplot of the absolute performance of METRIK and the RSD baseline across the NN 102 dataset.

[0065] Figure 29B shows a boxplot of the performance difference between METRIK and the RSD baseline on the NN102 dataset.

[0066] Figure 30A shows a scatterplot of the absolute performance of METRIK and the RSD baseline across the NN102 dataset. Figure 30B shows a boxplot of the performance difference between METRIK and the RSD baseline on the NN 102 dataset.

[0067] Figure 31 A shows a scatterplot of the absolute performance of METRIK and the RSD baseline across the NN102 dataset.

[0068] Figure 3 IB shows a boxplot of the performance difference between METRIK and the RSD baseline on the NN 102 dataset.

[0069] Figure 32A shows a scatterplot of the absolute performance of METRIK and the RSD baseline across the NN102 dataset.

[0070] Figure 32B shows a boxplot of the performance difference between METRIK and the RSD baseline on the NN 102 dataset.

[0071] Figure 33 A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the LSI dataset.

[0072] Figure 33B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the LSI dataset.

[0073] Figure 34A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the LSI dataset.

[0074] Figure 34B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the LSI dataset.

[0075] Figure 35 A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the LSI dataset.

[0076] Figure 35B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the LSI dataset.

[0077] Figure 36A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the LSI dataset.

[0078] Figure 36B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the LSI dataset.

[0079] Figure 37A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the CEF dataset.

[0080] Figure 37B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the CEF dataset.

[0081] Figure 38A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the CEF dataset. Figure 38B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the CEF dataset.

[0082] Figure 39 A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the CEF dataset.

[0083] Figure 39B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the CEF dataset.

[0084] Figure 40 A shows a scatterplot of the absolute performance of METRIK and the RSD- generated solutions across different statistical parameters on the CEF dataset.

[0085] Figure 40B shows a boxplot of the performance difference between METRIK and the RSD-generated solutions across different statistical parameters on the CEF dataset.

[0086] Figure 41 depicts a high-level block diagram of a computing device suitable for use within the context of the various embodiments.

[0087] It should be understood that the appended drawings are not necessarily to scale, presenting a somewhat simplified representation of various features illustrative of the basic principles of the present disclosure. The specific design features of the sequence of operations as disclosed herein, including, for example, specific dimensions, orientations, locations, and shapes of various illustrated components, will be determined in part by the particular intended application and use environment. Certain features of the illustrated embodiments have been enlarged or distorted relative to others to facilitate visualization and clear understanding. In particular, thin features may be thickened, for example, for clarity or illustration.

[0088] DETAILED DESCRIPTION

[0089] The following description and drawings merely illustrate the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within its scope. Furthermore, all examples recited herein are principally intended expressly to be only for illustrative purposes to aid the reader in understanding the principles of the present disclosure and the concepts contributed by the inventor(s) to furthenng the art and are to be construed as being without limitation to such specifically recited examples and conditions. Additionally, the term, "or," as used herein, refers to a nonexclusive or, unless otherwise indicated (e.g., “or else” or “or in the alternative”). Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0090] The numerous innovative teachings of the present application will be described with particular reference to the presently preferred exemplary embodiments. However, it should be understood that this class of embodiments provides only a few examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the claims. Moreover, some statements may apply to some features but not to others. Those skilled in the art and informed by the teachings herein will realize that the present disclosure is also applicable to various other technical areas or embodiments. Disclosed is a novel framework that leams a planned missing design for a Randomized Control Trial from a small pilot study (i.e., initial study) by leveraging a state-of-the-art Transformer-based imputation framework with mask weight learning and implementing a sample strategy that generates diverse candidate PMDs and a selection strategy that chooses PMDs satisfying the design objective.

[0091] The clinical randomized controlled trial (RCT) is the gold-standard approach for evaluating the efficacy of a medical intervention (e.g.. drug). The trial is designed to statistically establish efficacy based on a single outcome, also called the primary endpoint, examples of which include overall survival in cancer trials or changes in physiological measures of quality-of-life scores in patients with heart disease.

[0092] Besides the primary outcome, trial investigators monitor many additional exploratory outcomes to generate new hypotheses through post hoc analyses, given the intense implementation effort and investment made to conduct the RCT; however, doing so increases trial expenses (e.g., by 1.7 M USD) without immediate return since they are non-essential to the RCT’s objective. Consequently, trial designers have devised a couple of strategies to reduce trial expenses incurred by these exploratory measures. One strategy implements a governance committee that simplifies the data collection protocol by omitting extraneous outcomes at the cost of undermining their prospective scientific value. Another strategy implements a planned missing design (PMD), which removes the collection of certain measurements at certain timepoints rather than removing entire outcomes from the study protocol. The missing data are tackled using statistical methods. While PMDs based on random sampling are widely used, given their simplicity and statistical properties, they have low efficiency since they7do not leverage the correlational structure within the data to inform PMD construction. For example, prior work demonstrated that optimizing the PMD could substantially reduce the data collection budget (e.g., by up to 75%) while matching the variance level measured under an ad hoc PMD based on random sampling. Despite the marked performance advantage under optimized designs, some proposed algorithms optimized the PMD for a statistical parameter of interest (e.g., the variance of the mean slope parameter in a linear growth curve model), which is not possible for exploratory outcomes, given the various types of statistical analyses (e.g., effects of interactions, effects of nonlinear terms) that could be explored in these unspecified post hoc analyses.

[0093] To create a PMD that is optimal for a dataset and generalizes to unspecified downstream statistical tasks, a framework called METRIK (Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing). METRIK learns the PMD by modeling it as a differentiable function and optimizing it for the task of imputation using gradient-based search to generate candidates followed by a selection strategy that compares the generated candidates against baseline PMDs to find high-quality candidates. The candidate PMDs are fit on data collected form an internal pilot study implementing the complete data collection protocol for the RCT of interest. The superiority of METRIK's PMD has been empirically validated compared to a design based on random sampling across several downstream statistical tasks and real-world RCT datasets. By constructing PMDs with substantially better statistical performance than a random-sampling based PMD, METRIK makes the PMD a feasible tool for collecting data on exploratory outcomes under a budget while allowing trial designers to save at least several hundreds-of-thousands of US dollars per trial on large-scale clinical RCTs.

[0094] According to various aspects of the present disclosure, a method may be provided. Referring to FIG. 1, a flowchart of an embodiment of a method (100) for reducing a number of measurements needed by a phase-3 clinical randomized controlled trial (RCT) is shown. The method may include receiving (110) initial data including data from a study containing fewer subjects than a complete conventional RCT (e.g., ~60 subjects), a protocol schedule, user constraints, and a subject missingness matrix. The method may also include pre-processing (120) the initial data by handling native missingness, normalizing the initial data, and dividing the initial data into training and validation sets.

[0095] The method may also include generating (130) candidate planned missing designs (PMDs). Generating candidate PMDs may include training (134) initial imputer models across a range of masking efficiency levels. Each efficiency level may be defined as a fraction of measurements collected under a study protocol that are masked out by the candidate PMD, where losses are evaluated over eligible masked elements. Generating candidate PMDs may also include using (138) the initial imputer models to learn a new set of models and model- optimized PMD by using a mask learning algorithm to leam a binary mask by modeling it as a differentiable layer of logits and fitting it with a task objective along with regularization applied to the logits. During training, imputer weights may not be frozen, and gradients are zeroed out for ineligible measurements under a user-specified constraint matrix.

[0096] The method may also include identifying (140) optimal imputer-PMD pairs. Identifying optimal imputer-PMD pairs may include calculating (142) a PMD’s efficiency and performance. Identifying optimal imputer-PMD pairs may also include comparing (144) a performance of each learned imputer-PMD pair and each initial imputer-PMD pair, and ranking the learned imputer-PMD pairs based on performance differences and design objective. Comparing the performance of each learned imputer-PMD pair and each initial imputer-PMD pair may be based on calculated confidence intervals.

[0097] Identifying optimal imputer-PMD pairs may also include identifying (146) eligible candidate pairs among the learned imputer-PMD pairs by determining if each imputer-PMD pair has an efficiency above an efficiency threshold and imputation performance above an imputation performance threshold, where thresholds are determined by a reference imputer- PMD pair (i.e., a given initial imputer-PMD pair), pruning the eligible candidate pairs by storing the eligible candidate pairs with comparable efficiency and lowest imputation error or, if no eligible candidate pairs exist, storing a reference imputer-PMD pair.

[0098] The method may further include selecting (150) a stored imputer-PMD pair.

[0099] In some embodiments, the method may also include applying (160) a selected imputer- PMD pair to an RCT study. The method may also include imputing measurements omitted from the RCT study using the selected imputer-PMD pair.

[0100] In some embodiments, the selected imputer-PMD pair may have an efficiency closest to a target efficiency.

[0101] In some embodiments, the method may also include choosing a stored imputer-PMD pair with desired performance characteristics and performing a phase-3 clinical RCT where measurements are only collected based on the PMD of the stored imputer-PMD pair, and imputing corresponding dropped measurements using the imputer of the stored imputer-PMD pair.

[0102] In some embodiments, each imputer may be trained using a Multivariate Time Series Imputation with Transformers (MTSIT) algorithm. In some embodiments, generating candidate PMDs may include using an RSD-based PMD generator to generate random binary matrices of shape ntx nmusing a Bernoulli distribution parameterized by an efficiency level while being constrained by the user constraints. In some embodiments, learning the new set of models and model-optimized PMD may include generating a set of diverse PMDs per imputation model by sampling different hyperparameter combinations of mask regularization weights and learning rates used to train a mask-modulated model.

[0103] In some embodiments, for each sampled mask regularization weight and learning rate, a mask layer may be attached to a bottom of each initial imputer, where a same Transformer architecture may be used for the imputer with weights initialized from an initially trained imputer. A mask weight may be initialized according to a distribution used to generate an RSD- based PMD of a given efficiency. The masked imputer may be trained on an imputation task induced by a parametrized PMD using an MTSIT algorithm, pilot study dataset, and sampled hyperparameters.

[0104] In some embodiments, calculating a PMD’s efficiency and its imputer performance may include, for continuous metrics, reporting a value representing a normalized root-mean-square deviation (nRMSD) over masked elements. Calculating a PMD’s efficiency and its imputer performance may include, for categorical metrics, calculating accuracy and macro Fl over masked elements per metric and reporting a median of scores across metrics to yield a single score per performance metric. Comparing a performance may include calculating confidence intervals per performance metric and using the confidence intervals to lowerbound a performance gain of a given learned pair over a reference pair.

[0105] In some embodiments, the user constraints may include at least one of a target budget or efficiency level. In some embodiments, the eligible masked elements may be determined by the user constraints.

[0106] In some embodiments, comparing the performance of each learned imputer-PMD pair and each initial imputer-PMD pair may be based on confidence intervals.

[0107] According to various aspects of the present disclosure, a non-transitory computer- readable storage medium may be provided. The non-transitory computer-readable storage medium may include instructions that, when executed by one or more processing units, causes the one or more processing units to perform a method as disclosed herein.

[0108] According to various aspects of the present disclosure, a system may be provided. The system may include one or more processing units. The system may also include a non-transitory computer-readable storage medium as disclosed herein. Referring to FIG. 2, an embodiment of a system (200) is shown. The system (200) may include a first system (210) for reducing a number of measurements needed by a phase-3 clinical randomized controlled trial. The first system (210) may include one or more processing units (211) operably coupled to a memory (212), a non-transitory computer-readable storage device (213), a communications interface (214), and one or more input / output devices (215) (e.g., a display, a mouse, a keyboard, etc.). The processing unit(s) may be operably coupled to additional components as needed to perform the various tasks.

[0109] As used herein, the term '‘processing unit” may refer to any CPU, GPU, core hardware thread, or other processing construct. As used herein, the term “thread” refers to any software or processing unit or arrangement thereof that is configured to support the concurrent execution of multiple operations.

[0110] The first system (210) may be configured to operably communicate with one or more first remote processing units (220) that may be used by one or more researchers (221), scientists, or other users who w ould like to design an appropriate RCT

[0111] The first system (210) may be configured to operably communicate with one or more second remote processing units (230) that may be used by one or more researchers (231). scientists, clinicians, etc., who would or intend to conduct an appropriate RCT involving a plurality of subjects (232), which are preferably human subjects, who may have one or more conditions. The plurality of subjects is preferably less than that of a complete conventional RCT (e.g., ~60 subjects). Though it is envisioned there may be greater than or less than 60 subjects. The second remote processing unit(s) may be configured to send communications to the system. The second remote processing unit(s) may be configured to send communications to first processing unit(s). This may include, e.g., test data or data related to the subjects.

[0112] The first system (210) may be configured to operably communicate with one or more third remote processing units (240). The third remote processing units may include, e.g., one or more databases or applications accessible via one or more application programming interfaces (APIs).

[0113] Strategies for PMD Generation

[0114] The random sampling design (RSD) is a type of PMD that randomly samples each measurement across timepoints and subjects. The random sampling of measurements enables inference methods to provide valid estimates of statistical parameters. RSD variants have been applied across clinical studies, for example, for investigating discrepancies between pediatric patient and parent-reported responses on lengthy questionnaires and for analyzing patterns of change from longitudinal studies.

[0115] Methods for constructing PMDs fall under RSDs and optimization-based approaches. RSDs leverage statistical properties of imputation algorithms to implement PMDs that yield unbiased parameter estimates. While the matrix sampling design randomly samples each measurement per subject, tailored randomized PMDs can further reduce the variance of certain types of parameter estimates. For example, the Multiform (MF) design yields more efficient estimates of correlations by dividing measurements or questions into sets, creating multiple forms that pool questions across random pairs of sets, and randomly administering forms across subjects. However, because the MF design reduces the variance of the estimates unequally across the pairs of variables, it requires users to allocate variables across sets to ensure that all interactions are measured efficiently. While ad hoc allocation strategies have been suggested, some studies have shown that the allocation can be optimized, but these approaches require fitting a data-generating model, which requires large prior datasets (e.g., 500 subjects), making these methods inefficient and unsuitable for RCTs.

[0116] Search Algorithms

[0117] Various search algorithms have been devised to find solutions in high-dimensional spaces (i.e., with more than 100 dimensions), including Bayesian Optimization (BO), population-based methods (e.g., genetic algorithms), and gradient-based methods. While standard BO does not scale well, high-dimensional variants can improve its scalability but impose assumptions on the underlying data structure that would require domain knowledge to justify. In contrast, population-based methods like genetic algorithms do not impose assumptions on the data structure and perform better than BO in practice but are computationally intense owing to the iterative evaluation and adaptation of a large pool of candidate solutions.

[0118] In practice, gradient-based search finds solutions faster; how ever, it is not applicable to solutions defined over discrete search spaces, such as a binary mask. One work devised a method for differentiating over a binary mask. Specifically, the mask is represented by the Gumbel-sigmoid distribution, which instantiates the reparameterization trick to enable differentiation over the parameters of the distribution. The soft mask is then binarized using the threshold operator and passed through a straight-through estimator to enable differentiation through the thresholding step. The study used the mask to identify task-specific subset of weights within a pre-trained neural network by parametrizing the mask over the entire weight space and applying the mask to the pre-trained frozen neural network. The mask is learned by optimizing it on the task of interest that the neural network was trained on and penalizing its sparsity. The study validated the mask learning algorithm by demonstrating that the subnetwork parametrized by the learned mask maintains performance on test splits with similar distribution as the train splits but suffers drops in performance when evaluated on out-of-distribution splits.

[0119] Imputation Algorithms

[0120] Basic imputation techniques, e.g., mean filling or univariate approaches, are unable to model complex relationships over variables in multivariate datasets. Hence, imputation algorithms tailored to the multivariate setting have been developed. For example, the MICE algorithm uses a Markov Chain Monte Carlo approach to leam the joint distribution over variables, thereby removing the need to pre-specify the form of the distribution that is needed with joint modeling. Specifically, at each iteration, each missing variable is regressed using models (e.g.. linear regressors) fit on the observed and imputed variables from a previous iteration. 3D-MICE extends MICE to integrate temporal information present over the variables as in clinical data collection settings. Specifically, it uses Gaussian processes to model temporal correlations and combines predictions under the Gaussian process model with predictions under MICE using a weighted average, where weights are determined by the variance of the predictions. Later work has shown that machine-leaming-based approaches, e.g.. gradient boosting, outperform 3D-MICE on clinical imputation tasks, but these methods use hand- designed features. In contrast, approaches based on deep learning architectures, e.g., arecurrent neural network (RNN), can perform competitively to machine-learning approaches and demonstrate state-of-the-art performance on various clinical imputation tasks without requiring features to be pre-specified. Specifically, one study demonstrates that a Transformer-based architecture developed for multivariate time-series applications (MVTS) outperforms RNN- based imputation frameworks on a clinical imputation task, given the Transformer’s abilities to model long-range dependencies and the use of masked autoencoding to pre-train the entire model.

[0121] Given the challenges with designing a good feature set, an alternative is to use artificial neural networks for automated feature learning. Recent work has demonstrated the viability of learning such models for imputation of temporal datasets. Specifically, studies have demonstrated that the Transformer architecture, designed to model long-range dependencies using attention mechanisms, empirically ranks the top amongst various machine learning algorithms across various missingness mechanisms and rates on various datasets, including clinical ones. One such framework, called Multivariate Time-Series Imputation with

[0122] Transformers (MTSIT), achieves this high performance by implementing a model with a Transformer-based encoder and a linear decoder and training the model with the masked autoencoding objective, in which several elements from the input sequence are masked out (i.e., set to 0) and the decoder is tasked to predict the values of these elements. The specific training loss is shown in Eq. 1 , where X denotes the predicted values, X denotes the ground truth, and M denotes the set of indices within the tensor that have been masked out.

[0123] Framework

[0124] METRIK is a framework for learning a PMD for an RCT from a small internal pilot study (i.e., initial study).

[0125] Referring to FIG. 3 A and FIG. 3B, a flowchart of an embodiment of METRIK is show n. METRIK may require a user to first conduct an internal pilot RCT under a complete data collection protocol pc, which is abinary matrix of size nttimepoints x nmmetrics, that encodes whether a measurement for a specific timepoint-metric pair is collected. Given the data collected form the pilot Dpilot, METRIK generates candidate PMDs by first pre-processing Dpuot Io make it suitable for training, fitting initial imputation models using an algorithm (e.g., MTSIT algorithm), and then training these models further with mask weight learning to generate a diverse set of PMDs, as shown in FIG. 3A. Then, it chooses PMDs and their associated imputers among the candidates by ranking them according to the user’s design objective, i.e., whether to maximize efficiency or imputation performance, as shown in FIG. 3B.

[0126] FIG. 4 show s another flowchart of an embodiment of the METRIK framework. In matrix pcwhite squares indicate the measurements that are collected. The patterned squares indicate the measurements that are not collected. In matrix E white squares indicate the measurements that are eligible to be omitted. The patterned squares indicate the measurements that are not eligible to be omitted. In matrix pmdsoi, black indicates measurements that are omitted (white indicates measurements that are not omitted). Given the data collected from the pilot, Dpiiot. METRIK then creates a PMD-imputer pair (pmdsoi, mso[) that satisfies the user’s constraints, which include the target budget or efficiency level etarget, which is the desired fraction of measurements under the complete protocol that should be omitted under the PMD, and eligibility mask E, which is a matrix encoding that determines which measurements are eligible to be omitted. These would include outcomes that are essential to the statistical analysis defining the RCT question, including the primary outcome and baseline features (e.g.. demographics, medical history, etc.). The generated PMD pmdsoi may be applied to the remaining study population of the RCT (N - nPUoi) and the correspondingly omitted measures may be imputed using msot. Finally, the data from the pilot and remaining RCT study may be merged to yield the final dataset.

[0127] Data Collection and Pre-processing

[0128] METRIK requires that the trial investigator implement an internal pilot study, which collects data using a relatively small sample size (e.g., 60 vs. a typical RCT size of 100-1000 patients, though it is envisioned the sample size may be greater than or less than 60) using the same data collection protocol as the planned RCT. / v. This strategy avoids collecting additional data beyond the final population size N while providing a path forward for optimizing the data collection protocol for the rest of the RCT.

[0129] After data collection. METRIK pre-processes the pilot study dataset DPuot, by handling native missingness, normalizing it, and dividing it into training and validation sets. Native missingness stems from patient dropout and sampling irregularities in the data collection protocol across metrics. METRIK addresses it through mean / mode imputation. The missingness is determined by P G {0,l}ntX"m, a binary mask indicating which measurements from the data collection protocol are eligible for masking, where ntis the study duration and nmis the number of metrics monitored over the study (this mask is provided by the trial designer), and S G {0,l}”' P,!"tXnt / "m. a binary' mask indicating which measurements were collected per subject, where ns puotis the number of subjects monitored over the pilot study (this mask is also provided by the trial designer). After imputation, it performs min-max normalization per metric and then divides the same dataset into training and validation sets at the patient level. Generating PMDs

[0130] Next, a walkthrough is provided for the flowchart for generating candidate PMDs , as shown in FIG. 3A (pseudocode shown in FIG. 7A). To generate candidate PMDs, METRIK seeds PMD learning by fitting initial imputers on the pilot study dataset. Specifically, METRIK trains an imputer mefor a given efficiency level e, defined as the fraction of measurements collected under study protocol P (e.g., protocol schedule) that are masked out by the PMD. meis trained using the MTSIT algorithm, given its state-of-the-art performance on time-senes based applications. Specifically, it uses an RSD-based PMD generator, given by pmdRso(e), which generates random binary matrices of shape ntX nmusing a Bernoulli distribution parametrized by efficiency level e while being constrained by P. METRIK trains imputers across masking efficiencies e ranging between 0 and 100%. During training, losses are only evaluated over eligible masked elements, determined by P and S. METRIK stores the trained models along with samples of corresponding RSD-based PMDs in set M.

[0131] METRIK may also leam initial imputers using the MTSIT framework, given its state- of-the-art performance on tabular datasets. The flowchart for this step is shown in FIG. 5. To apply MTSIT, METRIK first pre-processes the pilot study dataset, DPuot, by filling in natively missing data, which is encoded by S G [0,l]nPi!ot><nt><"m, a tensor indicating which measurements were collected per subject (this includes measurements not collected under protocol pcMETRIK does this to train the model since the model cannot handle null entries; hence, for simplicity; METRIK uses mean-based imputation as a placeholder. After imputation, METRIK performs min-max normalization per feature and divides the dataset into training and validation sets at the patient level.

[0132] After pre-processing the data, METRIK may apply MTSIT to train imputers for different efficiencies e spanning the range given by 0 and 100%. To train an imputer mefor some efficiency e, it samples a mask per subject under the RSD, given by function pmdRso(e where the probability of some measurement being omitted or masked out is given by a Bernoulli distribution parameterized by constraint matrix E, have 0 masking probability. During training, losses are only evaluated over masked elements that do not correspond to natively missing data (determined by E) to avoid learning from entries that lack ground truth. After training, METRIK stores the initial imputation models along with the RSD function in Set Mref

[0133] METRIK then uses the initial imputation models to leam a new set of models and model-optimized PMDs. METRIK achieves this using the mask learning algorithm, which 1 earns a binary mask by modeling it as a differentiable layer of logits and fitting it with the task objective along with regularization applied to the logits (mask weights). METRIK applies the mask learning algorithm to each initial imputer and generates a set of diverse PMDs per imputation model, me, by sampling different hyperparameter combinations of mask regularization weights and learning rates used to train the mask-modulated model. Specifically, for each sampled mask regularization weight 7^^, and learning rate 17, METRIK attaches a mask layer to the bottom of each initial imputer, where the same Transformer architecture is used for the imputer with weights initialized from me. The weight mask is initialized according to the distribution used to generate an RSD-based PMD of efficiency e in order to mitigate potential effects from distribution shifts during training. The masked imputer is then trained on the imputation task induced by the parameterized PMD using the MTSIT algorithm, pilot study dataset, and sampled hyperparameters. In contrast to other works, imputer weights are not free since the goal is to optimize performance. In addition, during training, no mask noise is added, and gradients are zeroed out for measurements excluded from the data protocol to ensure that the derived PMD satisfies constraints set by P. After training, the learned PMD pmd* (i.e., represented by the mask layer) and imputer m get stored in set M*.

[0134] Said alternatively, METRIK may generate a candidate set of new PMD-imputer pairs, as shown in FIG. 6. To do this, METRIK may use gradient-based search to identify good candidates, given its computational advantage relative to other search algorithms. Specifically, to leam a candidate pair, denoted by PMD pmd* and m*. METRIK may first initialize a new imputation model using the weights of some reference model meand prepend a differentiable binary mask layer to the model. METRIK may initialize this layer according to the distribution used to generate an RSD of efficiency e to mitigate effects from distribution shifts during training. The model with the masking layer is trained on Dpiiot using the MTSIT algorithm, where the PMD is modeled by the distribution parameterized by the mask layer. Furthermore, to control the mask’s efficiency, METRIK incorporates the mask sparsity loss, which is weighted against the imputation objective Amw. To leam a valid PMD, METRIK zeroes out gradients for ineligible measurements under the user constraint set by E. Using this generation algorithm, METRIK leams diverse candidates by sampling different hyperparameter configurations defined by the choice of initial imputer me, mask sparsity weight Amw, and learning rate 77. The candidate pairs are stored in set kf. Choosing PMDs

[0135] Given the sets of initial and learned imputation models and their associated PMDs, i.e., Al and M*, respectively, METRIK chooses imputer-PMD pairs that are optimal based on the design criterion, as shown in the flowchart in FIG. 3B (pseudocode given in FIG. 7B). METRIK determines optimality by calculating the PMD s efficiency and the imputer’s performance. For continuous metrics, the normalized root-mean-square deviation (nRMSD) over masked elements (see Eq. (1)) is reported, which is negated for notational convenience under the performance maximization setting. For categorical metrics, accuracy and macro Fl are calculated over masked elements per metric (for example, see Eq. (2)) and, analogous to the averaging operation used for nRMSD. the median of the scores across metrics is reported to yield a single score per performance metric, which is referred to as pooled accuracy (pACC) and pooled macro Fl (pMFl).

[0136] Given metrics for assessing optimality, METRIK compares the performance of each learned imputer-PMD pair (m*, pmd*) and each initial imputer-PMD pair (mref, pmclre^ on the validation set of the pilot study dataset, i.e., Dpiiotvai, and ranks the learned imputer-PMD pairs based on the performance differences and design objective. These pairs are ranked based on performance differences since the set of initial imputer-PMD pairs serves as a reference for attainable operating points, i.e., pairs of sampling efficiency and imputation performance. To estimate performance differences accurately, given the small validation set, METRIK calculates confidence intervals per performance metric and uses them to lowerbound the performance gain of the learned pair over the reference pair. Specifically, it calculates confidence intervals for each pair’s performance, given by pm* and pref, sets the performance of the learned pair to the lower limit of the confidence interval, given by pmiOWer,ancl estimates the performance of the reference pair as the upper limit of the confidence interval, given by pmref, upper- It also records the PMD’s efficiency associated with each pair, given by em* for the learned and reference imputer-PMD pairs, respectively.

[0137] Then to identify optimal imputer-PMD pairs, METRIK ranks candidate pairs based on their improvements in efficiency and imputation performance relative to each reference imputer-PMD pair. Specifically, METRIK first identifies eligible candidate pairs, i.e., those with higher efficiency (em* > emre^ and imputation performance and stores them in set M*ligible. Then, METRIK uses stable sorting to first sort the eligible candidate pairs by efficiency and then sort by imputation performance when the design objective is to maximize efficiency and vice versa when the objective is to maximize imputation performance. It concludes by storing the top candidate imputer-PMD pair Pmdtop) Perreference imputer-PMD pair in solution set Msot. If eligible candidates do not exist, the algorithm stores the reference pair instead.

[0138] Given Msoi, the user chooses the imputer-PMD pair with desired performance characteristics and uses it during the execution of the Phase-3 trial. Specifically, the Phase-3 study only collects measurements based on the PMD and then imputes the corresponding dropped measurements using the imputer.

[0139] Said alternatively, METR1K identifies the best PMD-imputer pair of desired efficiencyetarget using a search process shown if FIG. 3C.

[0140] Given the reference and candidate PMD-imputer pairs stored in sets Mreand M* respectively, METRIK chooses between each candidate pair (pmd*, m’) and each reference pair ipmdret. mref) based on their performance, which is estimated using the pilot data. METRIK determines performance using a suitable measure, such as normalized root-meansquare deviation (nRSMD), which is given in Eq. (2). nRMSD averages the squared error across masked measurements indexed by the indicator liym, which indicates if the measurement associated with subject i at timepoint j for metric m has been masked out. The squared error compares the model prediction yi]magainst the ground truth yi]mand is normalized to the range of the metric defined by ym, the matrix of measurements across subjects and timepoints for metric m. The formula presented here differs from others in that it normalizes across subjects to avoid cases of dividing by zero and averages across all metrics to report a single score.

[0141] To obtain robust performance estimates, given the small size of the validation split of DpUot, METRIK draws many samples from each PMD to calculate the error and calculates confidence intervals over the errors using bootstraps. Then, METRIK identifies eligible candidates as pairs with higher efficiency (i.e., e* > eref) and lower imputation error (i.e., Pupper < Pref, lower) compared to the reference pair. For the latter criterion, METRIK uses the upper and lower limits of the confidence intervals for the candidate and reference pairs, respectively, to identify solutions that generalize. If no eligible candidates exist. METRIK selects the reference PMD- imputer pair. Otherwise, when multiple eligible candidates exist, METRIK prunes the set by identifying candidates with comparable efficiency and selecting the one with the lowest imputation error, yielding the set M*ligible pruned. The identified solutions are stored in Msoi. Finally, since METRIK has generated solutions spanning efficiencies between 0% and 100%. METRIK returns the PMD-imputer pair given by pmdsol, msol) whose efficiency is closest tO tar get-

[0142] Given the selected PMD-imputer pair (pmdsoi, msoi), the trial investigator may apply this PMD to a remaining number of subjects N-nPnot) in the study population. Once the data for these subjects has been collected under pmdsoi, the measurements omitted from pmdsoi are imputed using msoito yield the dataset DRCT. The final dataset is obtained by concatenating the pilot study data with this dataset.

[0143] First Experimental Setup

[0144] The datasets, baseline methods, and performance measures used for the experiments are described here. Implementation details (e.g., hyperparameter selection) are also described herein.

[0145] Datasets

[0146] METRIK is evaluated on five real-world clinical RCT datasets, which were obtained from NINDS. These RCTs are Phase-2 / Phase-3 trials that compare the effect of 1-2 experimental drugs against a control condition or standard-of-care in treating various neurological disorders, e.g., myasthenia gravis. For each RCT, a dataset is extracted (details of convenience sample generation can be found herein), whose characteristics, i.e., the total number of subjects ns itotai, number of visits during trial nt, number of metrics nmalong with the percentage of measurements labels as part of the trial protocol and percentage of metrics that are continuous are reported for each dataset in FIG. 8. For all datasets, ns piiotis set to 60, the recommended pilot study size for standard RCTs. The remainder of the dataset is used for testing. Baseline Algorithms

[0147] PMDs learned by METRIK are compared to those generated by several ad hoc strategies. A matrix sampling design is implemented, in which each measurement is randomly sampled based on a Bernoulli distribution. This baseline is referred to as RSD. The MF design and its variant called the Multiform Longitudinal (MFL) design, which randomizes form assignments over time across study participants, are both implemented. Comparisons are also made against the Wave design, which randomly samples the timepoints (rather than the metrics) at which measurements are taken. Specifically, one version is considered that includes the endpoints from the study (Wave+) and another version that excludes the endpoints (Wave), given that prior work demonstrates the advantage of this design. Additional implementation details are provided herein. By design, certain PMD generation strategies are not able to achieve specific baseline efficiency levels; the MF and MFL designs have a minimum efficiency of about 30% while the Wave designs are restricted by the total number of timepoints for a given RCT

[0148] Performance Evaluation

[0149] Changes in efficiency (e) and imputation performance (nRMSD, pACC, and pMFl) are reported relative to the baseline algorithms. Performance is evaluated using baseline efficiencies of {5%. 10%. 30%, 50%, 70%, 90%}, 5-fold cross validations per dataset are performed, and the distribution of the performance measures on the test set across the folds and datasets is shown using boxplots. To denoise performance estimates under the random sampling designs, 10K masked elements are sampled from the test set. PMDs are also visualized to analyze the masking patterns learned by METRIK.

[0150] Results and Discussion

[0151] First, results that demonstrate the performance advantage of METRIK over the baseline algorithms are presented. Then, results from ablation studies that demonstrate the key ingredients underlying METRIK’s effectiveness are presented. METRIK vs. Baselines

[0152] METRIK outperforms the baselines based on random sampling across different efficiency levels and under both design objectives: maximize efficiency or maximize imputation performance.

[0153] FIG. 10A and FIG. 10B present results for a design that aims to increase sampling efficiency over categorical metrics (changes in pMFl are omitted for visual simplicity and are shown in FIG. 11A, FIG. I1B, and FIG. 1 IC). Specifically, FIGS. 10A and I0B show performance gains obtained by METRIK over baseline PMD algorithms for a setting that maximizes efficiency. For MF and MFL, PMDs are only feasible for efficiency levels > 30%. Compared to RSD with 5% efficiency, METRIK outperforms RSD across higher baseline efficiencies, although efficiency gains diminish (e.g., efficiency increases by a median of 10%, IQR: [4%, 14%] relative to RSD with 30% efficiency) because the maximum achievable efficiency is 100%. In addition, since categorical metrics tend to dominate the dataset, its hypothesized that at higher baseline masking efficiencies, more uncorrelated metrics (e.g., metrics across different forms) are necessarily sampled, thereby reducing imputation performance and making it difficult for METRIK to find better PMDs. In these settings, METRIK performs no worse than the baseline because it chooses the baseline PMD.

[0154] METRIK also outperforms other baseline algorithms, although gains in efficiency can be higher than those for RSD. For example, when the baseline efficiency is 30%, METRIK increases efficiency by a median of 10%, IQR: [4%, 14%] for RSD, 29% (IQR: [19%, 46%]) for MF, 22% (IQR: [16%, 40%]) for MFL, 9% (IQR: [1%, 15%]) for Wave+, and 32% (IQR: [13%, 56%]) for Wave (all methods generally improve pACC and pMFl). It's hypothesized that gains across other baseline algorithms are comparable or higher because these algorithms sample a subset of random designs, making it difficult for the imputer to leam more complex interactions across measurements.

[0155] Similar performance gains under METRIK manifest over continuous metrics as shown in FIG. 12A and FIG. 12B, which shows results over continuous metrics.

[0156] For a design that aims to maximize imputation performance at a given efficiency level, METRIK produces PMDs that appropriately trade off efficiency with imputation performance. FIG. 13 A, FIG. 13B, and FIG. 13C compare the modes for the RSD baseline (only the results for the RSD baseline are shown since similar trends hold across other baselines). For example, at a baseline efficiency of 5%, the version of METRIK that maximizes performance increases efficiency by a median of 18% (IQR: [15%, 22%]), pACC by a median of 16% (IQR: [11%, 19%]), and pMFl by a median of 18% (IQR:[8%, 30%]), while the version of METRIK that maximizes efficiency increases by a median of 38% (IQR:[30%, 44%]), pACC by a median of 7% (IQR: [4%, 8%]), and pMFl by a median of 4% (IQR: [0%, 13%]). Similar tradeoffs hold across higher baseline efficiencies and over continuous metrics (see FIG. 14A and FIG. 14B which show results comparing the effect of the two design objectives, i.e., maximize efficiency or maximize imputation performance, on performance gains obtained by METRIK over the RSD baseline for a setting involving continuous metrics).

[0157] METRIK’s ability to increase efficiency and imputation performance stems from its ability to identify correlated measurements over time and across metrics. To assess this, the masking patterns under RSD-based PMDs are compared to those under METRIK. Specifically, FIG. 15 shows two RSD-generated PMDs (first two charts from left) along with one generated by METRIK (right-most chart) on the FSZONE dataset when maximizing efficiency, where the RSD-based PMDs have worse efficiency and imputation performance compared to the METRIK-based PMD. Compared to the RSD-based PMDs, the METRIK-based PMD is less fragmented, and masks out longer sequences along the metric or temporal dimensions. For example, it masks out metrics indexed 6 through 11 and 51 through 54. These ranges correspond to metrics from the same form. Specifically, 6-11 are assessments of abnormalities of the cranial nerv es and 51-54 are assessments of abnormalities in muscle strength, both conducted in a neurological exam. METRIK also masks out time segments for certain metrics; for example, indices 70-75, which assess the patient’s capacity to perform daily activities, are masked out for most visits past the baseline visit. Similar patterns emerge among METRIK- based PMDs for other RCT datasets and metric types (see FIGS. 16A-16D which show sample PMDs produced by RSD and METRIK across datasets and different metric types for a setting where the design objective is set to maximize efficiency), demonstrating that METRIK leverages correlations over time and metrics to improve the quality of the PMD.

[0158] Ablation Studies

[0159] To understand key components underlying METRIK’s effectiveness, ablations are conducted on the PMD selection step since design choices underlying the initial imputation fitting and PMD learning steps, e g., mask weight initialization, are based on tuning. For PMD selection, the effect of two design decisions is investigated: the algorithm used to generate candidate PMDs and the use of confidence intervals in estimating the imputation performance associated with a given PMD. Source for Candidate PMDs

[0160] Instead of using PMDs generated through mask weight learning, a candidate pool is filled with PMDs generated by random sampling baselines to demonstrate that random sampling is unlikely to yield solutions since the masked metrics are less correlated. FIG. 17A, FIG. 17B, and FIG. 17C show results from this ablation for categorical metrics when the trial objective is to maximize efficiency. Specifically. FIG. 17A. FIG. 17B, and FIG. 17C show performance gains under METRIK and an ablated version that replaces the candidate pool of learned PMDs wi th random ones generated by the MF design. Gains are measured with respect to an MF-based PMD baseline for a setting that maximizes efficiency. Results are shown for 30% baseline efficiency, as the ablated method shows no gains at other efficiencies. For simplicity, results under the MF design at a baseline efficiency of 30% are only reported since the MFL design gives comparable results, and the remaining designs show no improvements across baseline efficiencies. Compared to METRIK, the ablated framework yields poorer solutions. At a baseline efficiency of 30%, using a random candidate pool based on the MF design improves efficiency over an MF-based PMD by a median of 0% (IQRL [0%, 19%]) compared to METRIK, which improves efficiency over an MF-based PMD by 29% (IQR: [19%, 46%]) while obtaining larger gains in pACC and pMFl. The performance drop under the ablated framework extends to continuous metrics (see FIG. 18A and FIG. 18B) and the performance maximization setting of METRIK (see FIG. 19A, FIG. 19B and FIG. 19C and FIG. 20A and FIG. 20B). Specifically, FIG. 1 A and FIG. 18B, FIG 19A, FIG. 19B, and FIG. 19C and FIG. 20 A and FIG. 20B show graphical results comparing performance gains under METRIK against an ablated version that replaces the candidate pool of learned PMDs with random ones generated by the MF design. Gains are measured with respect to an MF-based PMD baseline. For simplicity, results are only shown for certain baseline efficiencies since the ablated method yields no performance gains at other baseline efficiencies.

[0161] Confidence Intervals for Performance Estimation

[0162] It is demonstrated that using confidence intervals improves METRIK’s performance by removing them from the performance estimation step. Results from this ablation for categorical metrics under the efficiency maximization setting are show n in FIG. 21 A, FIG. 21 B, and FIG. 21 C, where it's shown that performance changes relative to the RSD baseline since RSD baseline is competitive among the other random sampling baselines. At 5% baseline efficiency, METRIK, with and without confidence intervals, boosts efficiency by a median of 38% (IQR: [30%, 44%]) and 64% (IQR: [52%, 73%]), respectively. However, METRIK without confidence intervals achieves higher efficiency by hurting imputation performance. Specifically, it degrades pACC by a median of 0% (IQR: [-2%, 1%]) and degrades pMFl (median change in pMFl is 1%, IQR: [-8%, 6%]), while METRIK does not (median change in pACC is 7%, IQR: [4%, 8%] and median change in pMFlis 4%, IQR: [0%, 13%]). Similar effects from this ablation are observed at higher baseline efficiencies, as well as for continuous metrics (see FIG. 22A and FIG. 22B) and under the performance maximization setting (see FIG. 23A, FIG. 23B, and FIG. 23C and FIG. 24Aand FIG. 24B). Specifically, FIGS. 22A-22B, FIGS. 23A-23C, and FIGS. 24A-24B show results comparing performance gains under METRIK against an ablated version that does not use confidence intervals for performance estimation during PMD selection. Gains are measured with respect to the RSD baseline. The ablated framework shows that using confidence intervals is an effective strategy7at denoising performance estimates obtained on small validation datasets, which thereby enables METRIK to yield solutions with better performance guarantees.

[0163] Second Experimental Setup

[0164] For the second experimental setup, METRIK was evaluated on three real-world clinical RCT datasets, which were obtained from NINDS. These RCTs are Phase-2 / Phase-3 trials that compare the effect of 1-2 experimental drugs against a control condition or standard-of-care in treating various neurological disorders. For each RCT, a convenience sample is extracted from the dataset (details of this procedure can be found herein). Only continuous features are considered to simplify the evaluation step. Furthermore, due to the large size of the CEF and LSI datasets, the set of subjects and features included in each dataset are sampled. The characteristics of the final datasets are reported in FIG. 9, which include the number of timepoints ntand metrics nmto demonstrate the scale of data collection. The table also reports the total number of measurements eligible for masking, which is a subset of the possible ntx nmmeasurements because the data are sampled irregularly and because the primary metric and baseline measurements are ineligible for masking, given that they are essential for the RCT’s primary objective. Finally, the total number of subjects in the dataset is given by ns.npiiot subjects are used by METRIK while the remaining subjects are used for testing. Performance Measures

[0165] To demonstrate the framework's utility, its ability to generate new and diverse PMDs is characterized. Specifically, the total number of solutions generated by METRIK and the percent of solutions that are new or distinct relative to the baseline are reported. To measure diversity, the size of the range in efficiencies of the newly generated solutions is reported.

[0166] Next, statistical performance over the newly generated PMD-imputer pairs across different sampling budgets in the 5-95% range are reported. Specifically, the imputation performance over the PMD-masked elements using nRMSD is calculated. In addition, to obtain denoised estimates of imputation performance, the nRMSD is reported over 10K masked elements obtained by sampling masks under each PMD and applying them to the data samples.

[0167] Downstream statistical analyses are also performed using the imputed datasets. Specifically, Generalized Estimating Equations (GEE) are used. GEE is a common type of statistical model used for calculating population-average treatment effects from longitudinal data, given its ability to account for correlations and robustness to misspecification of covariance structure. Several common statistical analysis tasks are considered to test the framework's generalizability, including testing for the effect of the treatment and the effect of treatment-time interactions. Specifically, a model is fitted per metric, where for each model, other post-treatment variables are excluded to avoid biasing the analysis per standard guidance. Baseline predictors are also excluded to test the model’s sensitivity to imputed measures. In addition, to handle natively missing data, a common strategy is applied in which the model is fit using only observed outcomes. Following examples from prior work, the form of the treatment effect model is given by Eq. 3 and the form of the time-interaction effects model is given by Eq. 4, where Y is the metric value, X is the treatment indicator, t is the time index, and model parameters (intercepts and coefficients) are given by ?,. where f>treat corresponds to the treatment effect and f>treatxtime corresponds to the effect of the interaction between treatment and time.

[0168] Besides calculating the parameters of interest, i.e., fttreat and Ptreatxttme , the GEE model also calculates their variances, i.e., Var( / ?treat) and Var( / ?t).eatxt[me). The parameter estimates and their variances are calculated using the PMD-imputed dataset against the estimates calculated using the ground truth dataset. The error is measured using absolute percentage error. To test the effect of the imputations on parameter estimates, it’s assumed that the PMD-imputed dataset has the same natively missing data patterns as the ground truth dataset so that the GEE models are fit on corresponding pairs of imputed and ground truth measurements. To yield a single error measure from error estimates across the different metrics. the absolute percentage errors are pooled by taking their median, as shown in Eq. (5):

[0169] (5)

[0170] Median 100

[0171] In addition to calculating absolute performance levels, the performance measures under METRIK are compared against those under the baseline RSD strategy;

[0172] To obtain robust performance estimates, their distribution is characterized using fivefold cross validation per dataset and the results are visualized.

[0173] Results

[0174] Here, the experimental results that demonstrate METRIK’s superiority over the baseline RSD strategy are reported.

[0175] First, it’s demonstrated that METRIK generates many diverse PMD-imputer pairs that are distinct from the baseline solutions. This tendency is captured by characterizing the distribution in the number of generated solutions, the percent of solutions that are new or distinct from the baseline, and the range in the efficiency of the newly generated solutions, which is shown in FIG. 25 A, FIG. 25B, and FIG. 25C. For example, on the NN102 dataset, METRIK generates a median of 29 solutions (interquartile range [IQR] : 8) with a median of 79% of solutions being new (IQR: 10%). Furthermore, the range in efficiency among the newly generated solutions is a median of 60% (IQR: 5%). Similar statistics are observed across the other datasets as observed in the remaining boxplots in FIGS. 25A-25C.

[0176] Next, it’s demonstrated that the newly generated PMD-imputer pairs under METRIK have superior performance to the baseline ones across imputation and downstream statistical tasks. First, this is established on the imputation task since the solutions were optimized for this task. As shown in FIG. 26A, FIG. 26B, FIG. 27 A, FIG. 27B. FIG. 28A and FIG. 28B. the imputation error under the newly generated solutions is generally lower than the error under the baseline solutions across different efficiencies and datasets. For example, on the NN 102 dataset, all of the baseline PMD-imputer pairs have nRMSD above 0.20 while METRIK’s newly generated pairs have nRMSDs no worse than 0.20. The median improvement (reduction) in the nRMSD across efficiencies is 0.017 (IQR: 0.007). Similar performance advantages are observed across other datasets. The improvements in test imputation performance demonstrate that METRIK’s selection strategy identifies PMD-imputer pairs that generalize despite being selected on a small validation dataset.

[0177] It’s demonstrated that solutions under METRIK generalize to downstream statistical tasks for which they were not optimized. This is demonstrated by evaluating the error in estimates of sample statistical parameters (i.e. average and variance of treatment effect and treatment-time interactions) against ground truth estimates. Across a range of efficiencies. METRIK’s newly generated solutions have mean absolute percentage errors (“MAPEs”) that are lower or competitive to those under the baseline, as shown for the NN102 dataset in FIGS. 29A-29B, FIGS. 30A-30B, FIGS. 31A-31B, and FIGS. 32A-32B. For example, the MAPE in treatment effect coefficient (FIG. 29A and FIG. 29B) typically lies below 10% for efficiencies less than 30% under METRIK’s newly generated solutions while the MAPE typically lies above 10% for the baseline solutions. The reduction in the MAPE across efficiencies is a median of 7% (IQR: 4%). Similar trends are observed across other statistical parameters (see FIGS. 29A-29B. FIGS. 30A-30B, FIGS. 31A-31B, and FIGS. 32A-32B) and for the other datasets shown in FIGS. 33A-33B, FIGS. 34A-34B, FIGS. 35A-35B, and FIGS. 36A-36B as well as FIGS. 37A-37B, FIGS. 38A-38B, FIGS. 39A-39B, and FIGS. 40A-40B. The performance improvement across diverse statistical evaluation tasks under METRIK demonstrates that improvements on imputation generalize to improvements on unseen downstream statistical tasks.

[0178] Limitations

[0179] Having demonstrated METRIK’s effectiveness, one shortcoming is that METRIK may compromise performance over certain metrics while improving aggregate performance across the metrics. This performance differential can be problematic if the compromised metrics are important for the study. The safeguard against this shortcoming is for users of METRIK to analyze the results at a per-metric level (on the validation set), to ensure that changes in efficiency / errors over metncs of interest are in a favorable direction. Future w ork can also incorporate additional user constraints into METRIK to mitigate this issue. Another issue is that certain operating points may not be feasible to attain (e.g., high imputation performance) under imputation models trained on small datasets and will therefore require an increase in the training (pilot study) dataset size, but at the cost of reducing the total number of measurements that can be saved by the framework since fewer subjects will be subjected to the PMD. Future work will address this limitation by developing an adaptive strategy’ that increases the training dataset size to yield operating points with better imputation performance while accounting for the tradeoff in the total number of measurements collected across subjects.

[0180] The retrospective nature of the PMD design process under METRIK does not allow one to evaluate its impact on certain feasibility criteria for the generated study design, namely, participant adherence, engagement, and risk of dropout, but a subsequent internal pilot study can be done to assess these criteria. Incorporating feasibility' constraints (e.g., caps on unplanned missingness) into METRIK’s design and selection process remains future work. Besides these methodological improvements, the future work will also examine METRIK’s performance under other types of statistical analyses to assess its generalizability and develop methodological extensions.

[0181] The METRIK framework learns a PMD from an RCT from a small pilot study by leveraging a state-of-the-art Transformer-based imputation framework with mask weight learning and then implements a sampling strategy that generates diverse candidate PMDs and a selection strategy that chooses PMDs satisfying the design objective. Results across several real-world RCT datasets demonstrate that METRIK is effective at discovering PMD-imputer pairs with higher efficiency and better imputation performance compared to ad hoc random sampling designs. Given its performance, METRIK can be used by trial designers to automatically’ design PMDs for RCTs without requiring additional data overhead, thereby solving the problem of improving the efficiency' (costs) of the data collection process in RCTs without needing to remove any metric.

[0182] Conclusion

[0183] The METRIK framework is introduced which designs PMDs that yield substantially less biased estimates for any statistical parameter of interest compared to other studies and frameworks, thereby ensuring that the data collected under the PMD will be useful for unspecified future post hoc analyses while making data collection efficient. METRIK achieves this by generating candidate PMDs and associated imputation models using a differentiable framework and selecting among candidates based on imputation performance obtained on an internal pilot RCT dataset implementing the full data collection protocol. Results across imputation and several downstream statistical tasks across real-world RCT datasets demonstrate that METRIK is effective at discovering PMD-imputer pairs with substantially better statistical performance (i.e.. with reductions in nRMSD and MAPE ranging up to 0.03 and 52%, respectively, across datasets and efficiencies) compared to the RSD baseline. By constructing high-quality PMD-imputer pairs, METRIK obtains estimates on non-primary outcomes, thereby preserving the scientific, exploratory value of the study while offering study designers an opportunity to strategically invest the savings into higher-priority clinical studies.

[0184] Equations

[0185] Eq. (El) shows the formula for calculating nRMSD, where nsis the number of subjects, ntis the number of timepoints, nmis the number of metrics, is an indicator stating whether the measurement associated with subject i at timepoint J for metric m has been masked out, is the model’s prediction, yt>mis the ground truth, and ymis the matrix of measurements across subjects and timepoints. The formula presented here differs from others in that it normalizes across subjects to avoid cases of dividing by zero and averages across all metrics to report a single score.

[0186] Eq. (E2) shows the formula for calculating the accuracy per metric, adopting the same notation as used in Eq. (El). For simplicity, only the calculation for accuracy is shown, but the same approach, i.e., for measuring performance over the masked elements only, is used for calculating the macro Fl per metric.

[0187] Implementation Details

[0188] A review of the implementation details for the METRIK framework and the baseline algorithms, along with the computational resources are provided below.

[0189] METRIK

[0190] The initial imputer was trained using the MTSIT algorithm, where the architecture includes a Transformer encoder and linear decoder; specifically, a Transformer encoder was adopted with three encoder blocks and eight heads, with the dimensionality of the model set to 64 and dimensionality of the feed-forward layer set to 256, following similar configurations from the original work. For simplicity, imputers were trained for continuous and categorical covariates separately, where imputers for continuous variables are trained using the means squared error loss over masked elements while imputers for categorical variables are trained using the cross-entropy loss over masked elements. Each model is trained for 6K epochs using the Radam optimizer with the learning rate set to 1 x 10-3(continuous metrics) or 1 x 10-4(categorical metrics) with early checkpointing.

[0191] During mask weight training, the following variables are set to, Aranfle= {1 x IO-7, 1 x IO-6, 1 x 10-5} and v\range= {0.1, 0.5, 1, 5, 10} (these ranges were found sufficient to yield solutions based on the training / validation set), and the same optimization settings as used for training the initial imputer are otherwise adopted, although the last checkpoint is used since the mask stabilizes towards the end of training.

[0192] For PMD selection, a two-sided 95% confidence interval is calculated using the bootstrap algorithm and the number of bootstraps is set to IK since that was sufficient for intervals to converge.

[0193] Baselines

[0194] The implementation for each baseline is described herein.

[0195] The RSD baseline is implemented by having it sample each measurement under protocol P based on a Bernoulli distribution determined by the baseline efficiency level. For the MF design, the total number of items used to control the forms are varied (maximum set to 20). This parameter determines the efficiency of the forms. The metrics are randomly and equally divided across item sets, following a suggested strategy, and then a set of forms are generated by obtaining every' pairwise combination over item sets. For a given form, measurements not included in the chosen pair of item sets are masked out. The MFL design differs from MF in that at each timepoint it randomly chooses a form among the ones generated under the MF design.

[0196] For Wave designs, the number of timepoints that can be dropped are varied (the maximum is determined by nt, the number of timepoints for a given RCT). This parameter determines the efficiency of the form. Given the number of timepoints to drop, all resulting combinations over eligible timepoints are generated and measurements across all selected timepoints are masked out. For the Wave+ design, the set of eligible timepoints includes the endpoints while for the Wave design, the set of eligible timepoints excludes the endpoints.

[0197] Computer Environments

[0198] The first experiments were conducted using one CPU core for model training and mask weight learning (per trained model) and five CPU cores for PMD selection, using 2.6 GHz Intel Skylake / 2.8 GHz Intel Cascade Lake processors with a total memory of at most 6GB. Each experimental run (i.e., defined by a complete run of the algorithm for a given dataset and seed) requires at most 444 CPU hours. The total compute time for the experiments across datasets and seeds, including the baseline evaluations, is at most 15K CPU hours.

[0199] Per dataset, a convenience sample was obtained from the raw clinical dataset by applying a sequence of steps. First, any data collection forms that are not prescribed according to the trial protocol, comprises measurements that have no associated study visit, or inconveniently formatted data entries were removed. Afterwards, entries including duplicate rows. NaN indices, duplicate columns or metrics, subjects not assigned to any treatment group in the RCT, and timepoints or visit IDs not part of the trial protocol were removed. Next, the data was post-processed by standardizing NaN placeholders, removing metrics with mixed data types (e.g., string and numerical), and removing metrics that are constant or entirely NaN. The type of each metric is automatically determined (i.e., continuous vs. categorical) based on the data type and range of values and also drop metrics that have very low variance to avoid issues with normalization. The protocol mask is also automatically determined by identifying measurements (i.e., specific metric-timepoint pairs) with sufficiently low missingness rates. Subject masks S are automatically determined based on NaN entries.

[0200] The second experiments were conducted using CPU cores. Each step within METRIK, i.e., training a single initial imputer, generating a single PMD-imputer pair, and selecting among the candidates, requires at most 4 GB of memory.

[0201] All experiments were implemented using standard numerical Python packages and PyTorch.

[0202] In the second experimental setup, per dataset, a convenience sample was obtained from the raw clinical dataset by applying a sequence of steps. First, any data collection forms that are not prescribed according to the trial protocol, comprised measurements that have no associated study visit, or were inconveniently formatted, were removed. Then, entries including duplicate rows, NaN indices, duplicate columns or metrics, subjects not assigned to any treatment group in the RCT, and timepoints or visit ids not part of the trial protocol were removed. Next, the data was post-processed by standardizing NaN placeholders, removing metrics with mixed data types (e.g., string and numerical), and removing metrics that were constant or entirely NaN. The type of each metric (e.g., continuous vs. categorical) was automatically determined based on the data t pe and range of values. Metrics that have very low variance were also dropped to avoid issues with normalization. The complete data collection protocol, pcwas also automatically determined by identifying measurements (i.e.. specific metric-timepoint pairs) with sufficiently low missingness rates. Subject masks S are automatically determined based on NaN entries. The treatment assignment and list of timepoints considered part of the baseline data collection were also obtained per trial participant.

[0203] It is contemplated that the various systems and methods may be implemented via a computing device. As depicted in FIG. 41, a computing device (4100) includes a processor element (4103) (e.g., a central processing unit (CPU) and / or other suitable processor(s)), a memory (4104) (e.g.. random access memory (RAM), read only memory (ROM), and the like), a cooperating module / process (4105), and various input / output device (4106) (e.g., a user input device (such as a keyboard, a keypad, a mouse, and the like), a user output device (such as a display, a speaker, and the like), an input port, an output port, a receiver, a transmitter, and storage devices (e.g.. a persistent solid state drive, a hard disk drive, a compact disk drive, and the like)).

[0204] It will be appreciated that the functions depicted and described herein may be implemented in hardware and / or in combination of software and hardware, e.g., using a general -purpose computer, one or more application specific integrated circuits (ASIC), and / or any other hardware equivalents. In one embodiment, the cooperating process (4105) can be loaded into memory (4104) and executed by processor (4103) to implement the functions as discussed herein. Thus, cooperating process (4105) (including associated data structures) can be stored on a computer readable storage medium, e.g., RAM memory, magnetic or optical drive, or diskette, and the like.

[0205] It is contemplated that some of the steps discussed herein may be implemented within hardware, for example, as circuitry that cooperates with the processor to perform various method steps. Portions of the functions / elements described herein may be implemented as a computer program product wherein computer instructions, when processed by a computing device, adapt the operation of the computing device such that the methods and / or techniques described herein are invoked or otherwise provided. Instructions for invoking the methods may be stored in tangible and non-transitory computer readable medium such as fixed or removable media or memory, and / or stored within a memory within a computing device operating according to the instructions.

[0206] Thus, various embodiments for reducing a number of measurements needed by a phase- 3 clinical randomized controlled trial (RCT) may be implemented via code stored on a nontransient medium in or suitable for use with a receiver (e.g.. a special purpose receiver or decoding portion therein, computing device implementing a receiver function or decoding function, and so on), by a receiver or decoding portion thereof configured to perform the method such as by executing such code, by a special purpose device configured for performing the method and so on.

[0207] Various modifications may be made to the systems, methods, apparatus, mechanisms, techniques, and portions thereof described herein with respect to the various figures, such modifications being contemplated as being within the scope of the present disclosure. For example, while a specific order of steps or arrangement of functional elements is presented in the various embodiments described herein, various other orders / arrangements of steps or functional elements may be utilized within the context of the various embodiments. Further, while modifications to embodiments may be discussed individually, various embodiments may use multiple modifications contemporaneously or in sequence, compound modifications and the like.

[0208] Although various embodiments which incorporate the teachings of the present disclosure have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings. Thus, while the foregoing is directed to various embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof. As such, the appropriate scope of the present disclosure is to be determined according to the claims.

Claims

What is claimed is:

1. A method for reducing a number of measurements needed by a phase-3 clinical randomized controlled trial (RCT), comprising: receiving initial data including data from a study containing fewer subjects than a complete conventional RCT, a protocol schedule, user constraints, and a subject missingness matrix; pre-processing the initial data by handling native missingness, normalizing the initial data, and dividing the initial data into training and validation sets; generating candidate planned missing designs (PMDs) by: training initial imputer models across a range of masking efficiency levels, each efficiency level defined as a fraction of measurements collected under a study protocol that are masked out by the candidate PMDs, where losses are evaluated over eligible masked elements; and using the initial imputer models to learn a new set of models and model- optimized PMD by using mask learning algorithm to learn a binary mask by modeling it as a differentiable layer of logits and fitting it with an MTSIT algorithm along with regularization applied to the logits, where, during training, imputer weights are not frozen, and gradients are zeroed out for ineligible measurements under a user-specified constraint matrix; identifying optimal imputer-PMD pairs by: calculating a PMD’s efficiency and its imputer performance; comparing a performance of each learned imputer-PMD pair and each initial imputer-PMD pair, and ranking the learned imputer-PMD pairs based on performance differences and design objective; and identifying eligible candidate pairs among the learned imputer-PMD pairs by determining if each imputer-PMD pair has an efficiency above an efficiency threshold and imputation performance above an imputation performance threshold, where thresholds are determined by a reference imputer-PMD pair (i.e., a given initial imputer-PMD pair), pruning the eligible candidate pairs by storing the eligible candidate pairs with comparable efficiency and lowest imputation error or, if no eligible candidate pairs exist, storing a reference imputer-PMD pair; and selecting a stored imputer-PMD pair.

2. The method of claim 1, wherein a selected imputer-PMD pair has an efficiency closest to a target efficiency.

3. The method of any one of the preceding claims, further comprising applying a selected imputer-PMD pair to an RCT study.

4. The method of claim 3, further comprising imputing measurements omitted from the RCT study using the selected imputer-PMD pair.

5. The method of any one of the preceding claims, further comprising choosing a stored imputer-PMD pair with desired performance characteristics and performing a phase-3 clinical RCT where measurements are only collected based on the PMD of the stored imputer-PMD pair, and imputing corresponding dropped measurements using the imputer of the stored imputer-PMD pair.

6. The method of any one of the preceding claims, wherein each imputer is trained using an MTSIT algorithm.

7. The method of any one of the preceding claims, wherein generating candidate PMDs includes using an RSD-based PMD generator to generate random binary matrices of shape nt x nmusing a Bernoulli distribution parametrized by an efficiency level while being constrained by the user constraints.

8. The method of any one of the preceding claims, wherein learning the new set of models and model-optimized PMD includes generating a set of diverse PMDs per imputation model by sampling different hyperparameter combinations of mask regularization weights and learning rates used to train a mask-modulated model.

9. The method of claim 8. wherein, for each sampled mask regularization weight and learning rate, a mask layer is attached to a bottom of each initial imputer, where a same Transformer architecture is used for the imputer with weights initialized from an initially trained imputer.

10. The method of claim 9. wherein a weight mask is initialized according to a distribution used to generate an RSD-based PMD of a given efficiency.

11. The method of claim 10, wherein the masked imputer is trained on an imputation task induced by a parametrized PMD using an MTSIT algorithm, pilot study dataset, and sampled hyperparameters.

12. The method of any one of the preceding claims, wherein calculating a PMD’ s efficiency and its imputer performance includes, for continuous metrics, reporting a value representing a normalized root-mean-square deviation (nRMSD) over masked elements.

13. The method of any one of the preceding claims, wherein calculating a PMD’ s efficiency and its imputer performance includes, for categorical metrics, calculating accuracy and macro Fl over masked elements per metric and reporting a median of scores across metrics to yield a single score per performance metric.

14. The method of any one of the preceding claims, wherein comparing the performance includes calculating confidence intervals per performance metric and using the confidence intervals to lowerbound a performance gain of a given learned pair over a reference pair.

15. The method of any one of the preceding claims, wherein the user constraints include at least one of a target budget or efficiency level.

16. The method of any one of the preceding claims, wherein the eligible masked elements are determined by the user constraints.

17. The method of any one of the preceding claims, wherein comparing the performance of each learned imputer-PMD pair and each initial imputer-PMD pair is based on confidence intervals.

18. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processing units, causes the one or more processing units to perform a method of any one of claims 1-17.

19. A system, comprising: one or more processing units; and a non-transitory computer-readable storage medium of claim 18, operably coupled to the one or more processing units.

20. The method of any claim 1, further comprising choosing a stored imputer-PMD pair with desired performance characteristics and performing a phase-3 clinical RCT where measurements are only collected based on the PMD of the stored imputer-PMD pair, and imputing corresponding dropped measurements using the imputer of the stored imputer-PMD pair.

21. The method of claim 1 , wherein each imputer is trained using an MTSIT algorithm.

22. The method of claim 1, wherein generating candidate PMDs includes using an RSD- based PMD generator to generate random binary matrices of shape nt x nmusing a Bernoulli distribution parametrized by an efficiency level while being constrained by the user constraints.

23. The method of claim 1, wherein learning the new set of models and model-optimized PMD includes generating a set of diverse PMDs per imputation model by sampling different hyperparameter combinations of mask regularization weights and learning rates used to train a mask-modulated model.

24. The method of claim 23, wherein, for each sampled mask regularization weight and learning rate, a mask layer is attached to a bottom of each initial imputer, where a same Transformer architecture is used for the imputer with weights initialized from an initially trained imputer.

25. The method of claim 24, wherein a weight mask is initialized according to a distribution used to generate an RSD-based PMD of a given efficiency.

26. The method of claim 25, wherein the masked imputer is trained on an imputation task induced by a parametrized PMD using an MTSIT algorithm, pilot study dataset, and sampled hyperparameters.

27. The method of claim 1, wherein calculating a PMD’s efficiency and its imputer performance includes, for continuous metrics, reporting a value representing a normalized rootmean-square deviation (nRMSD) over masked elements.

28. The method of claim 1, wherein calculating a PMD’s efficiency and its imputer performance includes, for categorical metrics, calculating accuracy and macro Fl over masked elements per metric and reporting a median of scores across metrics to yield a single score per performance metric.

29. The method of claim 1, wherein comparing the performance includes calculating confidence intervals per performance metric and using the confidence intervals to lowerbound a performance gain of a given learned pair over a reference pair.

30. The method of claim 1, wherein the user constraints include at least one of a target budget or efficiency level.

31. The method of claim 1 , wherein the eligible masked elements are determined by the user constraints.

32. The method of claim 1, wherein comparing the performance of each learned imputer- PMD pair and each initial imputer-PMD pair is based on confidence intervals.

33. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processing units, causes the one or more processing units to perform a method of claim 1.

34. A system, comprising: one or more processing units; and a non-transitory computer-readable storage medium of claim 33. operably coupled to the one or more processing units.

Images