Automated Randomized Validation of RTSM Systems

Abstract

A method for managing a pharmaceutical clinical trial includes determining a randomized patient list and a randomized kit list for a clinical trial, automatically validating the randomization list, and configuring a RTSM system based on the validated list. Validating the randomized patient list can include verifying that the randomized patient list includes a balance above a balance threshold or a spread above a spread threshold, and verifying that the randomized patient list has randomness above a predetermined randomness threshold. Validating the randomized kit list can include verifying that the randomized kit list has randomness above a predetermined randomness threshold.

Description

[Technical field]

[0001] (Related Applications) This application claims the benefit under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 300,589, filed January 18, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to automated validation of randomization lists in clinical trial management and setting up a Randomization and Trial Supply Management (RTSM) system based on such validated randomization lists. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a schematic diagram of an example of a system for managing pharmaceutical trials. [Diagram 2] 1 is a flow chart illustrating an example of a method for configuring a RTSM system for a clinical trial. [Diagram 3] 1 is a flow chart illustrating an example of a method for validating a randomized patient list. [Figure 4] 1 is a flow chart illustrating an example of a method for validating a randomized kit list. [Diagram 5] 1 is a flow chart illustrating an example of a method for interpreting a clinical trial specification using natural language processing. [Figure 6] FIG. 1 is a schematic diagram of a natural language processing based test specification interpreter. [Figure 7] FIG. 13 illustrates an example of a graphical display of a randomized patient list that may be output to a user in connection with validation of a randomized patient list. [Figure 8] FIG. 13 illustrates an example of a graphical display of a randomization kit list that may be output to a user in connection with validation of a randomization patient list. [Figure 9]FIG. 13 illustrates another example of a graphical display of a randomization kit list that may be output to a user in connection with validation of a randomization patient list. [Figure 10] FIG. 1 is a schematic diagram of an exemplary embodiment of a user computing environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Randomization and clinical trial supply management (RTSM) systems typically execute a clinical trial supply plan based on a randomization patient list and a randomization kit list, either of which may be generated according to clinical trial parameters provided by the clinical trial sponsor. Validating these randomization lists via automated processes and computer simulations allows for efficient setup of a RTSM system with statistically sound randomization that has been found to be more reliable than known manual processes.

[0005] Referring to the drawings, in which like reference numbers represent the same or similar features in the various views, Figure 1 is a schematic diagram of an exemplary system 100 for managing a clinical trial, such as a randomized drug trial. System 100 can include a randomized list generation and validation system 102 and a RTSM system 104, either or both of which can be in electronic communication with a number of user computing devices 106, one of which is shown in Figure 1.

[0006] In general, the randomization list generation and validation system 102 and the RTSM system 104 may be involved in various aspects of the design and management of a pharmaceutical clinical trial. As described in more detail below, the RTSM system 104 may generally guide and manage the clinical trial. In contrast, the randomization list generation and validation system 102 may generate randomization lists, including randomized patient lists and randomized kit lists, and validate the content and randomness of those lists.

[0007] The RTSM system 104 may include one or more computing devices and, in embodiments, may perform a number of operations to organize and orchestrate pharmaceutical clinical trials, such as blinding, dispensing medications, and automated resupply of medications. The RTSM system 104 may be in electronic communication with one or more user computing devices 106 to receive input from users (e.g., clinical trial parameters) and provide output to users (e.g., reports on validated patient and kit lists, ordering one or more medications, status of patients participating in the clinical trial, information about the clinical trial site, etc.).

[0008] The RTSM system 104 may be configured to communicate and exchange information with a variety of different user types (i.e., with one or more user computing devices corresponding to such user types). For example, in a given clinical trial, the RTSM system 104 may provide an input / output interface for a clinical trial sponsor 108 (e.g., a pharmaceutical company). The clinical trial sponsor 108, in embodiments, may input all parameters of the clinical trial or clinical trial phase of the RTSM system 104. All such parameter sets may include, for example, patient visits or other actions related to patients, lots and lot operations, inventory and inventory operations at the site or depot level, shipments and shipment operations, temperature excursions and their management, approvals and parameters associated with returns of dispensed or undispensed drugs, inventory disposal status, responses to alerts or notifications, adding and parameterizing sites, depots, and countries of implementation, setting forecast parameters such as reliability dials and long-term time frame dials, setting enrollment rates for groups of sites, setting users and user access rights, setting cohorts, loading kit lists and randomization lists, and unblinding patients or kits. The RTSM system 104 can then determine what resupply actions must be taken and output the results to the clinical trial sponsor. The RTSM system 104 can also track and record the status of the clinical trial based on data from one or more depots, sites, physicians, and patients, as described below, and can make such information available to the clinical trial sponsor.

[0009] The RTSM system 104 can be in electronic communication with one or more clinical trial sites 110, such as, for example, hospitals or clinics. The RTSM system 104 can output instructions, such as, for example, medication orders for a particular patient, to the clinical trial site 110. The RTSM system 104 can receive data, such as patient information, medication records, etc., from the clinical trial site 110 and maintain data indicating which patients have taken which medications involved in the clinical trial, at which sites, and when.

[0010] The RTSM system 104 may further communicate with one or more distribution facilities 112, such as, for example, depots, warehouses, or other facilities involved in the distribution of active, placebo, and / or control drugs involved in the clinical trial to the clinical trial sites 110. The RTSM system 104 may, for example, issue shipping instructions to such distribution facilities 112. In embodiments, such shipping instructions may be generated and / or sent automatically by the RTSM system 104 based on a supply plan. The RTSM system may also receive data, such as shipping records and inventory information, from such facilities 112, such that the RTSM system 104 has data indicating the distributed and remaining inventory of drugs involved in the clinical trial and the location of those inventories. Drugs (e.g., active drug kits, control drug kits, and placebo kits) may be distributed and supplied to patients according to instructions from the RTSM system, and in particular, according to randomization patient lists and randomization kit lists generated and validated according to the present disclosure.

[0011] One or more investigators (e.g., physicians) 114 may also be in electronic communication with the RTSM system 104. The RTSM system 104 may output information about drugs involved in clinical trials and about the clinical trials to such investigators 114 to enable the investigators to inform patients accordingly about the clinical trials. The RTSM system 104 may receive enrollment information from such investigators (e.g., the number of patients to whom the investigators recommended the clinical trial, characteristics of those patients, the particular patients enrolled, etc.) and may maintain data indicating enrollment numbers and characteristics of enrolled patients.

[0012] The randomization list generation and validation system 102 may include a processor 118 and a non-transitory computer readable memory 120. The non-transitory computer readable memory 120 includes instructions that, when executed by the processor 118, cause the processor 118 to perform one or more methods, algorithms, processes, etc. of the present disclosure. The memory 120 may store one or more modules in the form of executable instructions (e.g., software) that are executed by the processor 118. Example modules are described below.

[0013] In an embodiment, the memory 120 can store a specification interpretation module 122. A user can submit information indicating parameters of a clinical trial (e.g., via a user computing device, which can be associated with a clinical trial sponsor, as described above), and the supply forecasting system 102 can receive this information. The specification interpretation module 122 can process and parse the received information to extract and determine the clinical trial parameters.

[0014] Memory 120 may also store a randomization list validation module 124 configured to automatically generate and validate randomization lists, such as randomization patient lists and randomization kit lists, based on the clinical trial parameters determined by specification interpreter 122. Examples of methods for automatically validating randomization lists are described with respect to Figures 3 and 4.

[0015] The randomization list generation and validation system 102 can provide an electronic user interface for use by a clinical trial sponsor. For example, the validation system can output a validation report for one or more patient lists and / or kit lists. The validation report can include, for example, a graphical representation of the patient list and / or kit list (one example of which is shown and described with respect to FIG. 7), among other information.

[0016] In embodiments, the RTSM system 104 and / or the randomization list generation and validation system 102 may be implemented on a Software-as-a-Service (SaaS) basis. Thus, the validation system and / or the RTSM system 104 may be embodied in one or more servers, databases, etc. that are electronically accessible to users via the Internet. Additionally or alternatively, the randomization list generation and validation system 102 and / or the RTSM system 104, or one or more aspects or functionality of the randomization list generation and validation system 102 and / or the RTSM system 104, may be implemented, for example, on a local server of a clinical trial sponsor or in a local copy stored on a user computing device of a clinical trial sponsor, clinical trial site, depot, etc.

[0017] Although the RTSM system 104 and the randomization list generation and validation system 102 are illustrated and described as separate systems, in embodiments, the RTSM system 104 and the randomization list generation and validation system 102, or some of their functions, may be implemented on a single system or common computing resources.

[0018] 2 is a flow chart illustrating an example method 200 for configuring a RTSM system with an automatically validated randomization list. Method 200, or one or more aspects of method 200, may be performed by a combination of a validation system and a RTSM system, such as, for example, randomization list generation and validation system 102 and RTSM system 104 of FIG.

[0019] The method 200 may include receiving one or more specification files from a user at block 202. The user may be, for example, a clinical trial sponsor (e.g., a pharmaceutical company). The specification file may include one or more file types. For example, in one embodiment, the specification file may include one or more text files and one or more spreadsheet files. In an embodiment, text may be entered in a free or natural format rather than in response to structured prompts. The specification file may include information available to the sponsor about the proposed drug trial, such as block size, treatment arms, ratios, treatment kits, comparators and placebos, kit number ranges and sequence number ranges, cohorts, strata, geographic information, or demographic information.

[0020] Method 200 may further include converting and consolidating the specification file to extract or otherwise determine parameters of the clinical trial, at block 204. These parameters include desired patient randomization list parameters and / or desired kit randomization list parameters contained in the specification file. Exemplary methods for performing block 204 are described with respect to FIGS. 5 and 6.

[0021] The method 200 may further include, at block 206, generating a randomized patient list and a randomized kit list based on the parameters of the clinical trial determined at block 206. For example, the randomized patient list may have a randomization sequence of patients assigned to an active or placebo group of the clinical trial, or to an active or control group, or to any one of a number of different active strength groups. The randomized patient list may be based on individual randomization in some embodiments (e.g., in embodiments where each patient has an equal likelihood of being assigned to any given group, such as an active group, a placebo group, a control group, an active strength group, etc., as all other patients). In other embodiments, the randomized patient list may be based on randomization blocks, which may have variable or random sizes, or another suitable randomization technique may be employed. Similarly, the kit list may be individually randomized or based on randomization blocks, which may have variable or random sizes, or another suitable randomization technique may be employed.

[0022] Method 200 may further include, at block 208, validating the patient list and kit list generated at block 206. Exemplary operations of block 208 are described with respect to Figures 3 and 4. Block 208 may, in some embodiments, be performed automatically once the patient list and kit list are generated.

[0023] Validating the patient list and kit list in block 208 may include generating a report including validation results. For example, the report may include one or more graphical displays of patient and / or kit distribution, as described with respect to Figures 7-9. The report may additionally or alternatively include information or tables regarding the patient list or kit list, and / or information regarding one or more simulations or mathematical analyses regarding the patient list or kit list, such as the number and sequence of patient enrollments in one or more simulations.

[0024] If the patient list or kit list is invalid, method 200 may output an error in block 212 and return to block 206 to generate a new randomized patient list and / or randomized kit list to replace the invalid list. The error output in block 212 may identify the validation step or portion (e.g., one or more aspects of method 300 of FIG. 3 and method 400 of FIG. 4) that was not successful. The error output in block 212 may include results of one or more simulations, such as one or more patient enrollment simulations, and an indication of whether the simulations met one or more thresholds for the randomization list.

[0025] If the patient list and kit list are valid, the method 200 may include generating kit labels according to the validated lists at block 210. Block 210 may further include printing kit labels and applying the kit labels according to the randomization list such that kit distribution and delivery is efficient based on the user's randomization and other parameters.

[0026] If the patient list and kit list are valid, method 200 may include, at block 212, configuring a RTSM system according to the validated randomized patient list and the validated randomization kit list and the remaining clinical trial parameters determined at block 206. The RTSM system may be configured to execute and monitor the clinical trial plan incorporating the randomized patient list and the randomization kit list. Additionally, active drug kits, placebo kits, and / or control drug kits may be dispensed to patients according to the validated randomized patient list and the validated randomization kit list.

[0027] By including automated document translation and interpretation to determine clinical trial parameters, automated generation of randomization lists, and automated validation of the randomization lists, system 100 and method 200 can enable automated configuration of a RTSM system for a particular clinical trial, requiring only free-form text and spreadsheet files from the clinical trial initiator, and no manual validation of the randomization list. Additionally, the automated validation of the randomization list described in methods 200, 300, 400 provides a more statistically sound validation than current approaches, enabling more reliable clinical trials.

[0028] 3 is a flow chart illustrating an example method 300 for automatically validating a randomized patient list. The method 300, or one or more portions thereof, may be performed by the randomization list generation and validation system 102 of FIG.

[0029] Method 300 may include validating the validity of a patient list file at block 302. Validating the validity of a patient list file may generally include verifying that the list contains valid content, such as, for example, that the randomized patient list matches or meets a user's study description. For example, block 302 may include verifying that group proportions, block sizes, number of blocks per stratum, and other proportions and quantities in the patient list meet the study description. Block 302 may include one or more of validating the total number of rows (i.e., patients) and total number of blocks for the patient list, validating the number of rows and blocks for one or more treatment groups, validating the number of rows and blocks for one or more cohort and stratum combinations, or validating the size and quantity of each of a plurality of block types.

[0030] The method 300 may further include verifying the quality of patient assignment in the randomized patient list at block 304. Verifying the quality of patient assignment may include, for example, verifying that the patient list includes a proper balance of active to placebo patients, or active to control patients, or patients receiving a first strength of active to patients receiving a second strength of active, etc., for the entire list and / or within each defined stratum or cohort. In some embodiments, the balance in each grouping (e.g., the ratio of active to all patients) may be compared to a predefined threshold, and the balance may be deemed valid if the balance exceeds the threshold or is within a certain range of the predefined threshold. For example, if a clinical trial aims to include ⅔ of the patients in the active portion of the clinical trial, a simulation in which about 60% to about 72% of the patients are included in the active treatment group is considered valid. The exact boundaries of validity depend on several aspects of the clinical trial, such as the number of patients, stratification, and treatment effect. In some embodiments, the balance for each stratum or cohort may be calculated and the entire patient list may be deemed invalid if the balance for any stratum or cohort does not meet a threshold, i.e., a patient list may have a valid balance if the balance for each stratum or cohort meets a balance threshold.

[0031] Block 304, in some embodiments, may include computing one or more simulations of patient enrollment, computing one or more balances for each simulation, and comparing the balance(s) for each simulation to a predefined threshold. In some simulation embodiments, each patient may have an equal probability for each of two or more blocking factors in the clinical trial. In other simulation embodiments, different probabilities may apply to different strata.

[0032] In embodiments where a simulation is performed, the balance of the patient list may be calculated multiple times during the simulation (e.g., after enrollment of 100 patients, then after enrollment of 200 patients, etc.) and each time the balance may be compared to a threshold. As a result, the patient balance may be evaluated over the course of the simulation for changes or progression. If the balance does not meet the threshold at any of the calculation points in the simulation, i.e., if the number of patients in the treatment arms of the clinical trial is either too high or too low compared to a desired ratio or proportion, then in some embodiments the patient list may be deemed invalid. In some embodiments, the time points in the simulation at which the balance is evaluated and / or performed may be selected by the user, including either the end of the simulated clinical trial and / or one or more intermediate points in the clinical trial.

[0033] In addition to or instead of calculating one or more balances for a patient list, in some embodiments, a spread for the patient list may be calculated and the spread may be compared to a threshold. For example, a spread may be calculated for a patient list having multiple block types. In such embodiments, the number of patients in the treatment group in each block type may be calculated and the spread (e.g., standard deviation) of those numbers may be calculated and compared to a threshold. If the spread is below the threshold, the patient list may be valid at block 304.

[0034] In some embodiments, multiple individual patient lists can be generated from the clinical trial specification, block 304 is performed for each of those patient lists, and the results of block 304 across the multiple patient lists can indicate the quality of the clinical trial specification parameters provided by the clinical trial sponsor, i.e., the results of block 304 across the multiple patient lists can indicate whether the clinical trial parameters are in line with the desired patient randomization technique (e.g., by the number or percentage of valid or invalid patient lists based on those parameters).

[0035] The method 300 may further include verifying randomness of patient assignment in the randomized patient list at block 306. For example, block 306 may include determining that the block pattern is uniformly distributed across the patient list. In some embodiments, determining that the block pattern is uniformly distributed may include determining the distribution of the treatment pattern across the blocks and calculating a p-value for that distribution, where the p-value indicates the probability of obtaining a distribution at least as extreme as the subject's distribution. The p-value may be calculated using a chi-square test in some embodiments. In other embodiments, another suitable mathematical method for calculating the p-value may be used. The p-value may be compared to a predefined threshold (e.g., 0.05, i.e., 5%) in some embodiments, and if the p-value exceeds the threshold, the list may be deemed to be valid for randomization.

[0036] As an example of determining the distribution of treatment patterns, consider an exemplary clinical trial with a block size of three patients. Each block is intended to include two active patients and one placebo patient. In this example, there are three treatment patterns: AAP (i.e., the first and second patients in the block receive the active drug, and the third patient receives a placebo), APA, and PAA. Thus, in this example, block 306 may include determining the number of blocks with each of these three patterns and calculating a p-value for the distribution of those block / pattern counts.

[0037] In method 300, if the patient list is determined to be invalid at any of blocks 302, 304, and 306, the patient list may be deemed invalid. Conversely, if the patient list is determined to be valid at all of blocks 302, 304, and 306, the patient list may be deemed valid.

[0038] 4 is a flow chart illustrating an example of a method for automatically validating a randomized kit list. The method 400, or one or more portions thereof, may be performed by the randomization list generation and validation system 102 of FIG.

[0039] The method 400 may include validating the validity of the kit list file at block 402. Validating the validity of the kit list file may generally include verifying that the list contains valid content, such as, for example, that the randomized kit list matches or meets a user's test description. For example, block 402 may include one or more of validating the total number of rows and the total number of blocks in the kit list, comparing the use (or non-use) of kit blocks to user requirements for the use (or non-use) of kit blocks, or validating the number of rows and blocks for one or more kit types. In some embodiments, each section of the kit list may contain different kit types (e.g., as if each section were its own kit list) and may use different block sizes. Thus, in some embodiments, block 402 may include evaluating the kit types and block sizes of each of the multiple sections to verify that the kit types, block types, block counts, and sequence ranges found in each section match expectations.

[0040] Method 400 may further include verifying randomness of kit assignment in the randomized kit list at block 404. For example, block 404 may include calculating the mean sequence of each kit type and verifying that kit types in the same section have mean sequences close to the median of the section. Block 304 may further include calculating the distribution of run lengths (i.e., consecutive kit numbers) for unblocked lists and comparing this distribution to the expected number of runs of each length. For example, block 406 may include determining that the run lengths are uniformly distributed. In some embodiments, determining that the run lengths are uniformly distributed may include calculating a p-value for the run length distribution. The p-value may be calculated using a chi-square test in some embodiments. The p-value may be compared to a predefined threshold (e.g., 0.05, i.e., 5%) in some embodiments, and if the p-value exceeds the threshold, the list may be deemed to be valid for randomization.

[0041] In some embodiments, block 404 may include, for blocked kit lists, determining that the block patterns are uniformly distributed across the kit list. In some embodiments, determining that the block patterns are uniformly distributed may include determining the distribution of the kit patterns across the blocks and calculating a p-value for that distribution, where the p-value indicates the probability of obtaining a distribution at least as extreme as the subject's distribution. The p-value may be calculated using a chi-square test in some embodiments. In other embodiments, another suitable mathematical method for calculating the p-value may be used. The p-value may be compared to a predefined threshold (e.g., 0.05, i.e., 5%) in some embodiments, and if the p-value exceeds the threshold, the list may be deemed to be suitable for randomization.

[0042] 5 is a flow chart illustrating an example of a method 500 for determining parameters for a proposed pharmaceutical clinical trial, including patient parameters, kit parameters, and randomization parameters. Method 500, or one or more portions of method 500, may be performed, in embodiments, by the RTSM system 104, by the randomization list generation and validation system 102, and / or by another system, such as a forecasting system, in communication with the RTSM system.

[0043] In some embodiments, the portions of method 500 that relate solely to randomization parameters may be performed by randomization list generation and validation system 102, independent of other clinical trial parameters, in a process separate from the processes performed to identify other parameters of the clinical trial. For example, randomization list generation and validation system 102 may be implemented as a system separate from RTSM system 104 to generate randomized patient lists and / or randomization kit lists separately from the generation of the rest of the clinical trial. Thus, while the following description addresses randomization parameters together with other clinical trial parameters, it should be understood that the portions related to randomization may be performed alone.

[0044] The method 500 may include receiving a spreadsheet file and one or more text files from a user at block 502. These files may include information about studies to be conducted that is currently available to the user. The text files may be, for example, HTML files. The spreadsheet file and the text files may be received, for example, via file upload through a website provided by the supply forecasting system. In an embodiment, the supply forecasting system may provide an interface through which a user may enter free-form text to describe the proposed study (which may be or may result in one or more text files of block 502), and through which the user may upload one or more spreadsheets.

[0045] Method 500 may further include converting the spreadsheet into one or more text-based tables at block 504. The spreadsheet may, in embodiments, be converted into, for example, one or more HTML-based tables, or into any format in which the text-based file was provided.

[0046] The method 500 may further include inserting the text-based table into one or more text files at block 506, and performing natural language processing on the one or more text files to determine parameters of the clinical trial at block 508. An example of natural language processing that may be performed at block 508 is described with respect to FIG. 6. Further details regarding natural language processing are provided in U.S. Patent Application Publication No. 2017 / 0154168, entitled "Generating RTSM Systems for Clinical Trials," inventor Edward A. Tourtellotte, assigned to 4G Clinical, LLC, which is incorporated herein by reference in its entirety. Using natural language processing (NLP), a user can describe their clinical trial in plain English, regardless of the level of detail of the information available, and the system can surface the available details of the clinical trial for further processing.

[0047] 6 is a schematic diagram of an exemplary electronic specification interpreter 600 that applies natural language processing. The specification interpreter 600 may, in an embodiment, perform one or more portions of the method 500 of FIG. 5, particularly block 508. The specification interpreter 600 may include a master interpreter 602, one or more specialized interpreters 604, one or more information extractors 606, one or more specialized natural language processing (NLP) tools 608, and one or more core NLP libraries 610. The specification interpreter 600 may, in an embodiment, further include additional components.

[0048] A core NLP library, such as core NLP library 610, can include natural language (e.g., English) information such as vocabulary and syntax, corpora, taggers, tokenizers, stemmers, lemmatizers, learning algorithms, etc. The core NLP library 610 can be used by the specification interpreter 600 to inspect the specification entered by the specification author. For example, the specification interpreter 600 can recognize words and stems, flag incorrect spellings, analyze grammar and semantics, detect logical operations and data definitions, and / or suggest alternative words based on the core NLP library 610. In some circumstances, the core NLP library 610 can be provided by a third party, such as from a development toolkit (e.g., Natural Language Toolkit, or NLTK) for a particular programming language (e.g., Java or Python).

[0049] Specialized NLP tools, such as specialized NLP tools 608, can provide system-specific extensions to the core NLP library 610. For example, specialized NLP tools 608 can include additional information about words, phrases, terms, syntax, reserved words / phrases, keywords, and operators as commonly used in the context of a particular field (e.g., clinical trials). Specialized NLP tools 608 can also include advanced language analysis functions such as tokenizing, stemming, grammar, collocation, language corpus, sentence structure, and semantic analysis, among other natural language processing functions. Specialized NLP tools 608 can also include artificial neural networks.

[0050] The specialized NLP tools 608 may also be used by the specification interpreter 600 to check the specification entered by the specification creator. For example, the specification interpreter 600 may recognize system-specific words, flag incorrect spellings, and / or suggest alternative words based on the specialized NLP tools 608. The specialized NLP tools 608 may be provided by a third party or may be developed internally. An information extractor, such as the information extractor 606, may extract specific information from the specification. One type of information extracted may include matrices (e.g., information structured in a tabular format). Matrices may be used in specification creation when it is convenient to organize data in a matrix format rather than in natural language. For example, matrices may be used to define patient visit schedules and associated medication regimens or other visit events. Matrices may also be used to define patient kit types, supply chain relationships and lead times, or any other data that is normally defined in a matrix.

[0051] Another type of information to be extracted may include a particular style of text, such as a definition. The definition may be structured, for example, as [SUBJECT OF DEFINITION] [VERB PROPOSITION] [ITEM or LIST OF ITEMS], although the structure may be different or more flexible to allow the relevant information to be written and extracted using a free English style. Other text styles may be defined as needed.

[0052] A master interpreter 602 may receive a clinical trial specification. The master interpreter 602 may perform a top-level parsing of the specification and organize interpretations in a logical sequence of themes. A theme may include one or more defined clinical trial topics, which may continue across one or more sections of the specification. Interpretation of a particular theme may be supported by a specialized interpreter 604 configured to parse information related to the particular theme across the entire specification (in one or more sections). The master interpreter 602 may invoke various specialized interpreters to process various themes of the clinical trial through the specification.

[0053] The specialized interpreters 604 may include a study level interpreter, a randomization interpreter, an action interpreter, a drug supply interpreter, a manufacturing interpreter, an enrollment interpreter, or other interpreters as needed. The study level interpreter may interpret high level information in the specification. For example, the high level information may include specific study level information such as the total number of patients desired to be screened and randomized, the randomization scheme and method, and other required parameters. The randomization interpreter may interpret the randomization information in the specification. The randomization interpreter can recognize the size of the rand list, the treatment arms used, the proportions of treatment arms, the stratification and cohorts, the assignment of strata to each part of the list (e.g., for females assign rand numbers [101,200], for males assign random numbers [201,300]), the randomization method (use of minimization or use of rand lists, with or without dynamic blocking), the type of blocks (size and kit ratios used in each block) and counts of each block type, the layout of the list (column headers and contents of each column), filename conversions, and the delimiters used for the generated files.

[0054] The action interpreter can interpret the event action information in the specification. For example, the event action can include information such as which medications can be dispensed in the context of each patient visit. The drug supply interpreter can interpret the drug supply information in the specification. For example, the drug supply information can include supply chain, shipping rules, supply kit information, etc. The manufacturing interpreter can interpret the various production steps and intermediate manufacturing lead times that convert the clinical trial components into kits that can be supplied to patients. The enrollment interpreter can interpret the proportion of patients participating in the study at various levels (e.g., study level, regional level, etc.). The list of specialization interpreters is not exhaustive. Additional specialization interpreters can be added as needed.

[0055] In some situations, the master interpreter 602 may be the input point to the specification interpreter 600. For example, the master interpreter 602 may receive the specification and split the interpretation into different sections or themes. The master interpreter 602 may then invoke (launch) the appropriate specialized interpreter 604 to analyze each section or theme. Each specialized interpreter 604 may then invoke one or more information extractors 606 as needed to extract information, such as matrices and definitions, from each section or theme of the specification. Each information extractor 606 may then invoke one or more specialized NLP tools 608 as needed. Each specialized NLP tool 608 may then invoke one or more core NLP libraries 610 to assist in processing each section or theme of the specification.

[0056] The output of the core NLP library 610 may be returned to the specialized NLP tool 608 that called the core NLP library 610. The output of the specialized NLP tool 608 may be returned to the information extractor 606 that called the specialized NLP library 610. The output of the information extractor 606 may be returned to the specialized interpreter 604 that called the information extractor 606. The output of the specialized interpreter 604 may be returned to the master interpreter 602 that called the specialized interpreter 604.

[0057] In some embodiments, the output of the specification interpreter 600 can include basic data structures and driving logic. The basic data structures and driving logic can include logic containers and other necessary or related structures. For example, the data structures can hold planned patient visit and dosing information, data driving information (e.g., BMI for use in dosing), data related to supply chain organization, supply shipments and / or patient kits, or any other data desired to be collected or processed during a clinical trial. If necessary, the logic containers can hold specific processing instructions for the predictive system to comply with the specification.

[0058] FIG. 7 shows an example of a graphical display 700 of a randomized patient list that may be generated and output by the validation system of FIG. 1 after validating (or attempting to validate) the randomized patient list. The display may include an ordered set of coded adjacent squares or other shapes, each corresponding to an individual patient or kit. The shapes may be arranged in the same order as the patient list. The coding of the squares may be color coded, pattern coded, or other coding that allows visual differentiation of each type of square. In the illustrated embodiment, the squares are color coded, with patients assigned to the active group being a first color and patients assigned to the placebo group being a second color. Thus, by manual inspection, a user may quickly assess the distribution of patients in the entire clinical trial or in a portion of the clinical trial, and better understand the numerical and statistical validation information generated by the validation system.

[0059] 8 and 9 show examples of graphical displays 800, 900 of randomized kit lists that may be generated and output by the validation system of FIG. 1 after validating (or attempting to validate) the randomized kit list. The displays 800, 900 are of the same kit list, but ordered differently. The first display 800 is ordered by kit order (i.e., grouping all kits of the same type), and the second display is ordered by kit number. Like the patient list display of FIG. 7, the kit list displays 800, 900 may be color-coded, pattern-coded, or otherwise coded to allow visual differentiation of the different types of squares. In the illustrated embodiment, the squares are color-coded, with each of the four kit types having a unique color. As a result, the display 800 allows the user to quickly visually verify that the relative quantities of the various kit types are appropriate. Display 900 allows a user to quickly assess the distribution of kits across or within a portion of a clinical trial, including run lengths, run sequences, etc., and to better understand the numerical and statistical validation information generated by the validation system.

[0060] 10 is a schematic diagram of an exemplary embodiment of a user computing environment including a general-purpose computing system environment 1000, e.g., a desktop computer, laptop, smartphone, tablet, or any other such device capable of executing instructions stored on a non-transitory computer-readable medium, etc. Additionally, while described and illustrated in the context of a single computing system 1000, those skilled in the art will appreciate that various tasks described below may be performed in a distributed environment having multiple computing systems 1000 linked via a local or wide area network, in which executable instructions may be associated with and / or executed by one or more of the multiple computing systems 1000.

[0061] In its most basic configuration, the computing system environment 1000 typically includes at least one processing unit 1002 and at least one memory 1004, which may be linked via a bus 1006. Depending on the exact configuration and type of computing system environment, the memory 1004 may be volatile (such as RAM 1010), non-volatile (such as ROM 1008, flash memory, etc.), or some combination of the two. The computing system environment 1000 may have additional features and / or functionality. For example, the computing system environment 1000 may also include additional storage devices (removable and / or non-removable), including, but not limited to, magnetic or optical disks, tape drives, and / or flash drives. Such additional memory devices may be accessible to the computing system environment 1000, for example, by a hard disk drive interface 1012, a magnetic disk drive interface 1014, and / or an optical disk drive interface 1016. As will be appreciated, these devices that will be linked to the system bus 1006 enable reading from and writing to the hard disk 1018, reading from or writing to a removable magnetic disk 1020, and / or reading from or writing to a removable optical disk 1022, such as a CD / DVD ROM or other optical media. The drive interfaces and their associated computer-readable media enable non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing system environment 1000. Those skilled in the art will appreciate that other types of computer-readable media capable of storing data may also be used for the same purposes.Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory, nano drives, memory sticks, other read / write and / or read-only memory and / or any other method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Any such computer storage media may be part of the computing system environment 1000.

[0062] Many program modules may be stored in one or more of the memory / media devices. For example, a basic input / output system (BIOS) 1024, containing the basic routines that help to transfer information between elements within the computing system environment 1000, such as during start-up, may be stored in ROM 1008. Similarly, the RAM 1010, hard drive 1018, and / or peripheral memory devices may be used to store computer-executable instructions including an operating system 1026, one or more application programs 1028 (which may include, for example, functionality for the randomization list generation and validation system 102 and / or the RTSM system 104 of FIG. 1), other program modules 1030, and / or program data 1022. Additionally, computer-executable instructions may be downloaded to the computing environment 1000 as needed, for example, via a network connection.

[0063] An end user may enter commands and information into the computing system environment 1000 through input devices such as a keyboard 1034 and / or a pointing device 1036. Although not shown, other input devices may include a microphone, joystick, game pad, scanner, etc. These and other input devices are typically connected to the processing unit 1002 by a peripheral interface 1038, which in turn is coupled to the bus 1006. The input devices may be connected directly or indirectly to the processor 1002 through an interface such as, for example, a parallel port, game port, firewire, or universal serial bus (USB). For viewing information from the computing system environment 1000, a monitor 1040 or other type of display device may also be connected to the bus 1006 through an interface such as a video adapter 1032. In addition to the monitor 1040, the computing system environment 1000 may also include other peripheral output devices (not shown), such as speakers and printers.

[0064] The computing system environment 1000 may also utilize logical connections to one or more computing system environments. Communication between the computing system environment 1000 and a remote computing system environment may be performed via a further processing device responsible for network routing, such as a network router 1052. Communication with the network router 1052 may be performed via a network interface component 1054. Thus, it will be understood that within such a networked environment, such as the Internet, the World Wide Web, a LAN, or other similar types of wired or wireless networks, program modules depicted relative to the computing system environment 1000, or portions thereof, may be stored in the memory storage device(s) of the computing system environment 1000.

[0065] The computing system environment 1000 may also include localization hardware 1056 to determine the location of the computing system environment 1000. In an embodiment, the localization hardware 1056 may include, by way of example only, a GPS antenna, an RFID chip or reader, a WiFi antenna, or other computing hardware that may be used to capture or transmit signals that may be used to determine the location of the computing system environment 1000.

[0066] Although the present disclosure has described certain specific embodiments, it will be understood that the claims are not intended to be limited to these embodiments, unless expressly set forth in the claims. On the contrary, the disclosure is intended to encompass alternatives, modifications, and equivalents that may be included within the spirit and scope of the disclosure. Furthermore, in the detailed description of the present disclosure, numerous specific details are described in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that systems and methods consistent with the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure various aspects of the present disclosure.

[0067] Some portions of the detailed description of this disclosure have been presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer or digital system memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, logic block, process, or the like, is herein, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, but not necessarily, these physical manipulations take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system or similar electronic computing device. For reasons of convenience and in accordance with common usage, such data will be referred to as bits, values, elements, symbols, characters, terms, numbers, or the like in reference to the various embodiments disclosed herein. It should be borne in mind, however, that these terms are merely convenient labels to be interpreted to refer to physical operations and quantities and should be further interpreted in the context of commonly used terms in the art. Unless otherwise specifically mentioned, and as will be apparent from the discussion herein, throughout the discussion of the present embodiments, descriptions utilizing terms such as "identifying / determining / seeking / determining / discriminating" or "outputting" or "transmitting" or "recording" or "locating" or "storing" or "displaying" or "receiving" or "recognizing" or "utilizing" or "generating" or "providing" or "accessing" or "checking" or "notifying" or "communicating" are understood to refer to actions and processes of manipulating and transforming data in a computer system or similar electronic computing device. Data is represented as physical (electronic) quantities in the registers and memory of a computer system, and is transformed to other data that is similarly represented as physical quantities in the memory or registers of a computer system, or other such information storage, transmission, or display device, as described herein or as would be understood by one of ordinary skill in the art.

Claims

1. 1. A method for managing a clinical trial of a pharmaceutical agent, comprising: a server receiving one or more electronic files from a user containing parameters of a clinical trial; the server performing natural language processing on the one or more electronic files to determine the parameters; generating a randomized patient list based on the determined parameters; The server, verifying that the randomized patient list contains a balance above a balance threshold or a spread above a spread threshold; and verifying that the randomized patient list has randomness above a predetermined randomness threshold; validating the randomized patient list by the server configuring the RTSM system according to the validated randomized patient list; A method comprising:

2. 10. The method of claim 1, further comprising dispensing the kit of drugs to the patient according to an established RTSM system.

3. validating the total number of rows and the total number of blocks in the randomized patient list; Validating the number of rows and blocks for one or more treatment arms; Validating the number of rows and blocks for one or more combinations of cohorts and strata; or Validating sizes and quantities of multiple block types; 13. The method of claim 1, further comprising verifying that the randomized patient list contains valid content by one or more of the following:

4. Verifying that the randomized patient list includes a balance or spread above a predetermined threshold includes: Computing a plurality of simulations of patient enrollment; calculating said balance or said spread according to said simulation; The method of claim 1 , comprising:

5. 5. The method of claim 4, wherein calculating a plurality of simulations of patient enrollment comprises assuming equal likelihood of patient enrollment in each of a plurality of strata or cohorts of the clinical trial.

6. Verifying that the randomized patient list has randomness above a predetermined randomness threshold comprises: determining a block pattern distribution within the randomized patient list; Calculating the probability of obtaining a distribution of block patterns at least as extreme as the observed block pattern distribution; comparing said probability to said predetermined randomness threshold; The method of claim 1 , comprising:

7. 1. A method for managing a clinical trial of a pharmaceutical agent, comprising: a server receiving one or more electronic files from a user containing parameters of a clinical trial; the server performing natural language processing on the one or more electronic files to determine the parameters; generating a randomized kit list based on the determined parameters; The server, verifying that the randomized kit list has randomness above a predetermined randomness threshold; validating the randomized kit list by: the server configuring an RTSM system according to the validated randomization kit list; A method comprising:

8. 10. The method of claim 7, further comprising dispensing the kit of drugs to the patient according to an established RTSM system.

9. Validating the randomized kit list includes: Validating the total number of rows and the total number of blocks in the randomized kit list; or Validating the number of rows and blocks for one or more kit types; 8. The method of claim 7, further comprising verifying that the randomized kit list contains valid content by one or more of the following:

10. Verifying that the randomized kit list has randomness above a predetermined randomness threshold includes: For unblocked lists, determining that the probability of the distribution of run lengths in the randomized kit list exceeds the randomness threshold; or Regarding blocked lists, determining a block pattern distribution within the randomized kit list; Calculating the probability of obtaining a distribution of block patterns at least as extreme as the observed block pattern distribution; comparing said probability to said predetermined randomness threshold; The method of claim 7, comprising:

11. 1. A method for managing a clinical trial of a pharmaceutical agent, comprising: a server receiving one or more electronic files from a user containing parameters of a clinical trial; the server performing natural language processing on the one or more electronic files to determine the parameters; generating a randomized patient list and a randomized kit list based on the determined parameters; The server, verifying that the randomized patient list contains a balance above a balance threshold or a spread above a spread threshold; and verifying that the randomized patient list has randomness above a predefined patient list randomness threshold; validating the randomized patient list by The server, verifying that the randomized kit list has randomness above a predefined kit list randomness threshold; validating the randomized kit list by: the server configuring an RTSM system according to a validated randomization patient list and a validated randomization kit list; A method comprising:

12. 12. The method of claim 11, further comprising dispensing the kit of drugs to the patient according to an established RTSM system.

13. Validating the randomized patient list comprises: validating the total number of rows and the total number of blocks in the randomized patient list; Validating the number of rows and blocks for one or more treatment arms; Validating the number of rows and blocks for one or more combinations of cohorts and strata; or Validating sizes and quantities of multiple block types; 12. The method of claim 11, further comprising verifying that the randomized patient list contains valid content by one or more of the following:

14. Verifying that the randomized patient list includes a balance or spread above a predetermined threshold includes: Computing a plurality of simulations of patient enrollment; calculating said balance or said spread according to a simulation; The method of claim 11 , comprising:

15. 15. The method of claim 14, wherein computing a plurality of simulations of patient enrollment comprises assuming equal likelihood of patient enrollment in each of a plurality of strata or cohorts of the clinical trial.

16. Verifying that the randomized patient list has randomness above a predetermined patient list randomness threshold comprises: determining a block pattern distribution within the randomized patient list; Calculating the probability of obtaining a distribution of block patterns at least as extreme as the observed block pattern distribution; comparing said probability to said predetermined patient list randomness threshold; The method of claim 11 , comprising:

17. Validating the randomized kit list includes: validating the line counts and block counts based on the one or more electronic files; or Validating the number of rows and blocks for one or more kit types; 12. The method of claim 11, further comprising verifying that the randomized kit list contains valid content by one or more of the following:

18. Verifying that the randomized kit list has randomness above a predetermined randomness threshold includes: determining, for an unblocked list, that the probability of the distribution of run lengths within the randomized kit list exceeds the randomness threshold; or Regarding blocked lists, determining a block pattern distribution within the randomized kit list; Calculating the probability of obtaining a distribution of block patterns at least as extreme as the observed block pattern distribution; comparing said probability to said predetermined randomness threshold; The method of claim 11 , comprising:

19. the randomized patient list is a second randomized patient list and the randomized kit list is a second randomized kit list; generating a first randomized patient list and a first randomized kit list based on the determined parameters; the server attempting to validate the first randomized patient list; the server attempting to validate the randomized kit list; In response to the server determining that the randomized patient list or the randomized kit list is invalid, the server outputting an error; the server determining the second randomized patient list and the second randomized kit list; The method of claim 11 further comprising:

20. 12. The method of claim 11, further comprising generating and outputting a report of the validation of the randomized patient list and the randomized kit list, the report including a graphical representation of one or more of the randomized patient list or the randomized kit list.

Images