Apparatus for facilitating development of software, method of generating target pattern, hardware storage device, and computer program product

Abstract

The present application provides apparatuses, methods of generating target patterns, hardware storage devices, and computer program products that facilitate development of software. An apparatus for facilitating development of software to use a neural network in generating output indicative of near real-time predictions of clinical study enrollment includes a processor; and a hardware storage device storing instructions executable by the processor to perform one or more first operations comprising receiving instructions executable by the processor to perform one or more second operations comprising receiving, from a user interface, input data corresponding to data input into one or more portions of the user interface; inputting, into the neural network, a target pattern, received input data specific to a given clinical trial, enrollment data, and historical data; and receiving, from the neural network, prediction data specifying a near real-time prediction of enrollment for the clinical trial.

Description

Technical Field

[0001] This specification generally relates to systems and methods for developing software for calibrating neural networks. Background Technology

[0002] In the existing technology, spreadsheets can be used as an internal tool to track the progress of study enrollment, but this adds extra work to collecting and entering study enrollment data. Summary of the Invention

[0003] The disclosed implementation includes an apparatus for facilitating software development to use a neural network in generating near real-time predictions of clinical study enrollment. The apparatus includes: one or more processors; and one or more hardware storage devices storing instructions executable by the processor to perform one or more first operations. The one or more first operations include: receiving one or more instructions in a development environment, the instructions executable by the processor to perform one or more second operations. The one or more second operations include: causing one or more user interfaces to be drawn having one or more input portions for inputting data for a specified clinical trial; receiving data from the one or more user interfaces and inputting the one or more first operations into the user interfaces. The data in each part corresponds to input data, wherein the received input data specifies a given clinical trial; a target pattern for the given clinical trial is received from one or more user interfaces; historical data and enrollment data for the given clinical trial are accessed from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease area associated with the clinical trial, and wherein the enrollment data specifies the actual enrollment in the clinical trial; the target pattern, the received input data specifying the given clinical trial, the enrollment data, and the historical data are input into a neural network; one or more near real-time prediction data specifying the enrollment for the clinical trial are received from the neural network; and one or more visualizations of the received prediction data are output to one or more user interfaces.

[0004] In some implementations, the clinical trial is associated with one or more sites, wherein the accessed enrollment data is for a given site within the one or more sites, and wherein the prediction data specifies one or more near-real-time predictions of enrollment for the clinical trial at the given site within the one or more sites. The target pattern specifies the enrollment target, activation date, and / or completion date. The neural network is a recurrent neural network. One or more visualizations provide a first view at the research level or a second view at the country level. The research level may include a clinical trial research level.

[0005] In another implementation, a method for generating a target pattern, implemented by a data processing system, is used in conjunction with a neural network to generate near real-time predictions indicative of enrollment in a clinical study. The method includes: rendering one or more user interfaces having one or more input portions for inputting data to a specified clinical trial; receiving input data from the one or more user interfaces corresponding to data input to the one or more portions of the user interfaces, wherein the received input data specifies a given clinical trial; and generating a target pattern for the given clinical trial by the data processing system based on the received input data, wherein the target pattern... Specify a target enrollment and completion date for the given clinical trial; access historical and enrollment data for the given clinical trial from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease domain associated with the clinical trial, and wherein the enrollment data specifies the actual enrollment in the clinical trial; input the target pattern, the enrollment data, and the historical data into a neural network; receive one or more near real-time predictions from the neural network specifying enrollment for the clinical trial; and output one or more visualizations of the received prediction data to one or more user interfaces.

[0006] In these implementations, the clinical trial is associated with one or more sites, wherein the accessed enrollment data is for a given site within the one or more sites, and wherein the prediction data specifies one or more near-real-time predictions of enrollment for the clinical trial at the given site within the one or more sites. The target pattern specifies the enrollment target, activation date, and / or completion date. Actions include calibrating the neural network by: accessing clinical trial data from a hardware storage device; segmenting the clinical trial data into test data and training data; identifying one or more features of the training data; encoding the one or more features; transmitting at least a portion of the one or more encoded features to one or more hidden layers of the neural network; determining one or more values ​​of one or more hyperparameters of the neural network based on the output of the one or more hidden layers; and repeating the transmission and determination until the neural network output satisfies one or more values ​​of one or more thresholds.

[0007] In these implementations, the actions include: training the neural network using supervised learning techniques based on training data to predict one or more near-real-time predictions, wherein the neural network is configured to receive the target pattern, input data specifying the given clinical trial, the enrollment data, and the historical data as inputs, and to process the inputs to generate an output specifying one or more near-real-time predictions for the enrollment of the given clinical trial, wherein the neural network includes a plurality of artificial neurons connected by edges and aggregated into a plurality of neural network layers including at least an input layer and an output layer, wherein each edge is configured to transmit a signal from one artificial neuron to another, and wherein the output of each artificial neuron is calculated based on the input of that artificial neuron according to a plurality of weights.

[0008] In these implementations, the actions include: the data processing system processing new data using the plurality of artificial neurons in a trained neural network according to the values ​​of the plurality of weights to identify one or more near-real-time predictions of enrollment for the given clinical trial, wherein the artificial neurons in the input layer are configured to receive the new data as input, and the artificial neurons in the output layer are configured to generate new outputs identifying the near-real-time predictions of enrollment, and wherein the new data includes the target pattern and the received input data. The actions also include training the neural network to predict enrollment, including: receiving training data, wherein one or more portions of the training data are associated with labels specifying enrollment counts; defining a first squared error function that quantifies the prediction error of the neural network based on the training data; and setting the weights of the neural network to minimize the first prediction error.

[0009] In these implementations, the neural network is a recurrent neural network. One or more visualizations provide a research-level first view or a national-level second view. Actions also include: determining an enrollment amount less than a threshold for one or more near-real-time predictions; in response to the determination, generating functional data that, upon receipt by a clinical trial management system, causes the clinical trial management system to perform one or more operations, wherein at least one of the one or more operations is allocating memory and processing resources for generating one or more communications and transmitting them over a network to one or more client devices, each client device associated with the clinical trial study location, and the communications informing users of the client devices that have received the communications that the enrollment amount is less than the threshold and prompting the user to perform one or more actions; and transmitting the functional data to the clinical trial management system. The functional data is an alarm.

[0010] In another implementation, one or more machine-readable hardware storage devices are used to generate target patterns for use with neural networks to generate near real-time predictions of clinical study enrollment. The machine-readable hardware storage devices store instructions executable by one or more processors to perform operations including: causing one or more user interfaces to draw one or more input portions for inputting data for a specified clinical trial; receiving input data from the one or more user interfaces corresponding to data input to the one or more portions of the user interfaces, wherein the received input data specifies a given clinical trial; and generating a target pattern by a data processing system based on the received input data. The system comprises: a target pattern for a given clinical trial, wherein the target pattern specifies a target enrollment and completion date for the given clinical trial; accessing historical data and enrollment data for the given clinical trial from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease domain associated with the clinical trial, and wherein the enrollment data specifies actual enrollment in the clinical trial; inputting the target pattern, the enrollment data, and the historical data into a neural network; receiving from the neural network one or more near real-time predictions specifying enrollment for the clinical trial; and outputting one or more visualizations of the received prediction data to one or more user interfaces.

[0011] In these implementations, clinical trials are associated with one or more sites, wherein the accessed enrollment data is for a given site within one or more sites, and wherein prediction data specifies one or more near-real-time predictions of enrollment for a given site within one or more sites. Target patterns specify enrollment targets, activation dates, and / or completion dates. The operation also includes calibrating the neural network by: accessing clinical trial data from a hardware storage device; segmenting the clinical trial data into test data and training data; identifying one or more features of the training data; encoding the one or more features; transmitting at least a portion of the one or more encoded features to one or more hidden layers of the neural network; determining one or more values ​​of one or more hyperparameters of the neural network based on the output of the one or more hidden layers; and repeating the transmission and determination until the neural network output satisfies one or more values ​​of one or more thresholds.

[0012] Other embodiments of the foregoing aspects include corresponding computer systems (e.g., data processing systems), devices, and computer programs recorded on one or more computer storage devices, each computer system, device, and computer program configured to perform the actions or operations described herein. One or more computer systems may be configured to perform specific actions by installing software, firmware, hardware, or combinations thereof on the system, which in operation cause the system to operate. One or more computer programs may be configured to perform specific actions by including instructions that, when executed by a data processing device, cause the device to operate.

[0013] In the fast-paced world of clinical trials, real-time monitoring of ongoing research is crucial. This continuous surveillance allows investigators to stay on top of the research's progress, identify potential bottlenecks or areas of concern, and take timely action to ensure research objectives are met. Since these objectives often include specific enrollment targets within a given timeframe, it is essential to have accurate and dynamic tools to predict future enrollment based on data accumulated throughout the study. Thus, a key area of ​​this technology is using machine learning models (Gated Recurrent Unit (GRU)-based Recurrent Neural Networks (RNNs)) in combination with historical clinical trial enrollment data and actual patient enrollment observed in ongoing clinical trials to date to generate future enrollment predictions for the study. Clients are provided with site-specific enrollment predictions, as well as aggregated study and country-level predictions to evaluate research enrollment progress and identify potential countries or sites requiring intervention.

[0014] Other solutions for study enrollment tracking and monitoring may fall short in the following ways. First, alternative solutions do not integrate with systems that track patient enrollment in studies. The system described in this paper integrates with direct data feeds from ongoing study subject enrollment, such as via electronic data capture systems (EDC) or clinical trial management systems (CTMS). Second, alternative solutions only provide tracking / monitoring functionality. Users can only see enrollments from studies to date and use their own judgment regarding whether studies are still meeting enrollment targets on time. Alternatively, alternative systems require users to input their own predictions or projections for future enrollments. The system described in this paper automatically generates a predicted enrollment trajectory based on actual enrollments to date and historical data.

[0015] Third, alternative solutions utilize extrapolation-based enrollment prediction methods that lack accuracy. For example, simple extrapolation methods based on observed enrollment rates to date lack accuracy in enrollment prediction. In validation exercises using the technique described herein, the technique described herein achieves an accuracy improvement of between 20% and 50% compared to such simple extrapolation methods.

[0016] The technology described in this paper addresses the technical challenge of accurately predicting future enrollment in clinical trials as research progresses and provides the flexibility to generate updated predictions at various time points for ongoing monitoring purposes. The technology offers both a study-level aggregated view and the ability to zoom in to specify participating countries or sites for diagnostic analysis. Depending on the specific study design, users can apply various customizations to the predictions, such as specifying the start-up activation time of a site or representing a site as closed for enrollment once a certain number of patients have been enrolled.

[0017] The technique described in this paper utilizes a data-driven deep learning model that predicts the future number of patients enrolled in a given study and at a given location for each month up to a user-specified maximum number of months. The granularity of the predictions (study-location-month) allows users to view the predictions in an aggregated study view as well as more detailed views for each country and location. RNNs are utilized because they are best suited for sequence generation. This approach allows for the flexibility of generating predictions at various time points as the study progresses. The data processing system described in this paper takes historical data from past studies conducted within the investigated disease domain and historical data from participating countries / locations, combined with actual enrollments observed to date, as input to generate predictions; thus, the model can learn from historical patterns and can also be adapted to any details within the ongoing study, such as rare patient populations, restrictive study eligibility criteria, and different macro-environmental contexts. The model performance has been validated on a test set with over 2,000 studies and has demonstrated superior accuracy relative to various baseline methods. Furthermore, several preprocessing and post-processing algorithms have been developed to handle potential user-customized needs, as mentioned in this paper.

[0018] The technology described in this paper offers numerous advantages, including system integration with EDC and / or CTMS data, where EDC and / or CTMS data serve as a source of actual enrollment data under periodic data refresh. Whenever a user requests an updated forecast based on the most recently received available data, the system automatically generates the latest predicted enrollment trajectory. Users can compare the predicted trajectories of study enrollment timelines with their planned times and determine whether any intervention is needed to meet the target study enrollment timeline. Predictions are based on historical data from studies conducted within the same disease domain and adjusted for observed actual enrollment behavior in each specific study. The utilization of deep learning models provides the flexibility to generate predictions at various time points and improved accuracy. The model has been validated on a large clinical trial test set and has demonstrated excellent accuracy and generalizability.

[0019] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1A A software development environment for calibrating neural networks to output near real-time predictions of enrollment in clinical trial studies is shown.

[0021] Figure 1B A data processing system for controlling the operation of a clinical trial management system is shown.

[0022] Figure 2 An example data processing system is shown for using neural networks to output near real-time predictions of enrollment in clinical trial studies.

[0023] Figure 3 An example calibration is shown.

[0024] Figure 4 An example of training a neural network is depicted.

[0025] Figure 5 The workflow of the example system is described.

[0026] Figure 6 The inputs and outputs of the example system are described.

[0027] Figures 7A-7C Each example graphical user interface is shown for adding and / or editing studies that include location attributes.

[0028] Figure 8A A sample graphical user interface for viewing aggregated study enrollment forecasts is shown.

[0029] Figure 8B A sample graphical user interface for viewing national-level forecasts is shown.

[0030] Figure 8C A sample graphical user interface for viewing site-level forecasts is shown.

[0031] Figure 9 An example processing method for developing and validating study enrollment prediction models is shown.

[0032] Figure 10 The model performance relative to the baseline is illustrated.

[0033] Figure 11 An example processing is shown for generating target patterns used in conjunction with neural networks to produce near real-time predictions of clinical study enrollment.

[0034] Figure 12An example computing system based on an implementation of this disclosure is described. Detailed Implementation

[0035] refer to Figure 1A The diagram illustrates a data processing environment 10. The data processing environment 10 includes a software development environment 12 for developing and programming software. As described herein, the software development environment 12 includes various software development tools and APIs, for example, to enable the writing of software to perform the functionality of the data processing system 11. That is, the software development environment 12 includes a workspace with a set of processing and programming tools for developing source code for applications or software products. The data processing environment 10 includes client devices 14 for transmitting data to and receiving data from the data processing system 11. The data processing environment 10 also includes a data source 16 for transmitting data to the data processing system 11 as described herein. The data source 16 may be a CTMS, EDC, etc. In this example, the data source 16 is integrated with the data processing environment 10 to enable the automatic retrieval of real-time data 48 from the data source 16.

[0036] Data processing system 11 includes a target schema generator 18 for generating target schemas. Typically, a target schema includes data of a specified type, data in a specified format, data including specified content (e.g., values), and / or a data structure structured with fields for storing the specified type of data. For example, a target schema may specify a particular study (or a specific location within a study), target enrollment and completion dates (e.g., when the target schema is at the study level), and activation date (e.g., when the target schema is at the location level). In this example, data source 16 transmits study data 44 to the target schema generator 18. Study data 44 specifies available studies for which enrollment data can be predicted. Study data 44 includes information about the study (e.g., a clinical trial study), such as start date, end date, study type, target enrollment, and geographic location for one or more locations within the study. Study data 44 also includes real-time data, such as real-time data specifying actual enrollment performance (e.g., such as...). Figure 6(As shown). Using study data 44, target pattern generator 18 generates graphical user interface (GUI) data that, when plotted on client device 14, displays one or more GUIs for selecting a specific study for predicted enrollment. Target pattern generator 18 transmits this GUI data to client device 14. In response, client device 14 plots one or more GUIs and receives input data through one or more GUIs specifying or otherwise indicating the specific study for which enrollment should be predicted. Through one or more user interfaces, such as by specifying study enrollment targets, site activation and / or completion dates, users can add or edit study and site attributes as needed. In this example, study enrollment targets are study attributes. Typically, site attributes specify the characteristics (e.g., geographic location) of a specific site where a clinical trial is being conducted. For example, various types of site attributes exist, such as activation date, completion date, country, cohort (e.g., the cohort in that country for that site), and historical performance metrics and competing trials on site. Some site attributes are part of the research data, and some site attributes are part of the historical data. For example, country, population, and activation date are part of the research data. Historical performance metrics and competing field trials are part of the historical data.

[0037] Client device 14 transmits attribute data 34 to target pattern generator 18. Attribute data 34 specifies various attributes of the selected study. These attributes include, for example, the target completion date. Using attribute data 34, target pattern generator 18 generates target pattern 36. In this example, attribute data 34 only specifies the study enrollment target. Thus, in order to generate target pattern 36, target pattern generator 18 retrieves other relevant information from study data 44 required to generate target pattern 36. This other relevant information includes, for example, the study activation date / study start date and study completion date. Target pattern 36 also includes information specifying a specific location of the study. Target pattern generator 18 transmits target pattern 36 to prompter 20. Prompter 20 then generates prompt 38 and transmits prompt 38 to neural network 22. Prompt 38 includes, for example, content specifying a request for predicted enrollment via a specific date, and also provides target pattern 36, thereby providing neural network 22 with the information needed to predict enrollment at a specific location for a specific clinical study up to the target date.

[0038] In response to prompt 38, neural network 22 generates prediction data 40. In this example, prediction data 40 specifies the predicted enrollment count up to a target date for a specific clinical setting at a specific location. Neural network 22 transmits prediction data 40 to post-processor 24. Post-processor 24 provides the execution of various processing operations on prediction data 40, including, for example, generating relevant key performance indicators (KPIs) and various visualizations representing predicted enrollment. Post-processor 24 outputs real-time prediction 42, which includes data specifying the customizations made to prediction data 40. As illustrated herein, these customizations include data formatting and the generation of visualizations to provide indications of predicted enrollment. Data storage 26 receives and stores real-time prediction data 42 from post-processor 24.

[0039] While generating prediction data 40, neural network 22 also receives real-time data 48 from data source 16 and historical data 46 from data storage 28. Historical data 46 specifies historical enrollments for a given clinical trial study, a given location, and / or historical enrollments for similar or comparable locations. Historical data 46 may also specify enrollments for the clinical trial study in the same or similar disease domain as the clinical trial study for which predictions are being generated. To achieve this, study data item 44 specifies the disease domain, enabling neural network 22 to retrieve historical data in associated (e.g., same or similar) disease domains. Real-time data 48 includes data specifying actual enrollments to date for a specific study at a specific location. Providing neural network 22 with historical data 46 and real-time data 48 enables neural network 22 to provide more accurate predictions. Typically, real-time data 48 includes data received by data processing system 11 in real-time, near real-time, or live relative to the time the data was initially received by data source 16. Typically, near real-time refers to the timeliness of data or information that has been delayed by electronic communication and the time required for automated data processing. The term "real-time" as used herein includes real-time and / or near real-time.

[0040] In this example, the data processing system 11 also includes a trainer 30, a calibrator 32, and a data storage 28. The calibrator 32 is used to tune the neural network and determine its hyperparameters. Once the neural network is fine-tuned, the trainer 30 uses training data to train the neural network. In this example, the data storage 28 transmits a modeling dataset 50 to the trainer 30 and the calibrator 32. The modeling dataset 50 includes trial-grade and trial site-grade data for various clinical trials. As will be described in detail below, the calibrator 32 uses the modeling dataset 50 to calibrate or fine-tune the neural network. The calibrator 32 outputs the calibrated model 52 to the trainer 30. As will be described in detail herein, the trainer 30 then uses the modeling dataset 50 to train the calibrated model 52 to produce a trained model 54. The trainer 30 outputs the trained model 54 to the neural network 22. In this example, the neural network 22 is or includes the trained model 54.

[0041] exist Figure 1A In variations, neural network 22 can more generally be a machine learning model. Various types of machine learning models exist, including, for example, generative artificial intelligence (AI) models with one or more large language models (LLMs). Example LLMs include models with one or more generative pre-trained transformers (GPTs), such as models implemented using one or more artificial neural networks, etc.

[0042] Generally, machine learning can encompass a variety of different techniques used to train machines to perform specific tasks without being specifically programmed to do so. Machines can be trained using various machine learning techniques, including, for example, supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the machine is given an input of interest and a corresponding output. The machine adjusts its functions to produce the desired output when given input. Supervised learning is often used to teach computers to solve problems with deterministic outcomes; for example, a training set can be used to train a machine learning model. In contrast, in unsupervised learning, input is given without providing a corresponding desired output. Reinforcement learning describes an algorithm in which a machine makes decisions using trial and error. Feedback informs the machine when a good or bad choice is made. The machine then adjusts its algorithm accordingly. For example, a trained learning model can be embodied as a generalized linear model (GLM). Different types of GLMs may be suitable for various scenarios. Zero-inflated negative binomial GLMs can be used because they model discrete counts of events (such as transitions) that occur over a given time period.

[0043] In another example, the trained machine learning model can be represented as an artificial neural network, such as... Figure 1AAs shown. Artificial neural networks (ANNs), or connectionist systems, are computational systems inspired by the biological neural networks that make up the animal brain. ANNs are based on collections of connected units or nodes (called artificial neurons). Similar to synapses in the biological brain, each connection can transmit a signal from one artificial neuron to another. The receiving artificial neuron can process the signal and then use it to notify any additional artificial neurons connected to it.

[0044] In common ANN implementations, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated as a nonlinear function of the sum of its inputs. The connections between artificial neurons are called "edges." Artificial neurons and edges can have weights that adjust as learning progresses (e.g., the individual inputs to an artificial neuron can be weighted individually). Weights increase or decrease the strength of the signal at the connection. Artificial neurons can have thresholds such that a signal is only sent if the aggregated signal exceeds that threshold. The transfer functions along the edges typically have an sigmoid shape, but they can also take the form of other nonlinear functions, piecewise linear functions, or step functions. Typically, artificial neurons are aggregated into layers. Different layers can perform different kinds of transformations on their inputs. The signal may travel from the first layer (e.g., the input layer) to the last layer (e.g., the output layer) after multiple traversals of layers.

[0045] Typically, supervised learning techniques are used to train neural networks based on training data. The neural network is configured to receive training data as input (e.g., input data records) and process the input to generate an output, for example, indicating whether a PII has been detected. In this example, the neural network includes multiple artificial neurons connected by edges and aggregated into multiple neural network layers, each comprising at least an input layer and an output layer. Each edge is configured to transmit a signal from one artificial neuron to another, and the output of each artificial neuron is computed based on its input according to multiple weights. In this example, new data is processed using the multiple artificial neurons in the trained neural network to detect PIIs based on the values ​​of the multiple weights. The artificial neurons in the input layer are configured to receive new data as input, and the artificial neurons in the output layer are configured to generate a new output indicating whether a PII has been detected.

[0046] refer to Figure 1B This illustrates data processing environment 10'. Data processing environment 10' is data processing environment 10 ( Figure 1A A version of ). Data processing environment 10' includes data processing system 11' (which is connected to ). Figure 1A The data processing system 11 is the same as or Figure 1AA version of the data processing system 11), the data processing system 11' has a target pattern generator 18' (which is connected to... Figure 1A The target pattern generator 18 is the same as or Figure 1A A version of the target pattern generator 18), prompter 20' (which is related to...) Figure 1A The prompt 20 is the same or Figure 1A A version of the prompter 20), neural network 22' (which is related to...) Figure 1A The neural network 22 in the middle is the same or Figure 1A A version of neural network 22), postprocessor 24' (which is related to...) Figure 1A The post-processor 24 is the same or Figure 1A A version of the post-processor 24), trainer 30' (which is in conjunction with...) Figure 1A The trainer 30 is the same or Figure 1A A version of the trainer 30) and calibrator 32' (which is with Figure 1A The calibrator 32 in the middle is the same or Figure 1A (A version of the calibrator 32 in the data processing environment 10'). The data processing environment 10' also includes data storage 28' (which is connected to...). Figure 1A The data storage in 28 is the same or Figure 1A The data storage 28 (a version) and data processing system 17 are included. The data processing environment 10' includes a network 57 and client devices 19a...19n.

[0047] Data processing system 11' includes a controller 55 pre-programmed with logic to detect when a predicted intake (specified by real-time prediction 42') falls below a threshold or specified amount. In this example, post-processor 24' transmits information specifying a target intake, which is the threshold or specified amount, to controller 55. As described below, when controller 55 detects that the amount specified by real-time prediction 42' is less than the target amount, controller 55 generates functional data 56, which controls the operation of data processing system 17 and causes data processing system 17 to perform one or more operations. In some examples, functional data 56 includes an alarm specifying that the intake specified by real-time prediction 42' is less than the target intake.

[0048] The data processing system 17 includes a data source 16', a processing device 21, a memory 27, a resource allocator 58, and a communication engine 59. The data source 16' is the same as or a version of the data source 16 in Figure 1. The resource allocator 58 includes logic to allocate the memory and processing resources of the data processing system 17, for example, based on various tasks required by the data processing system 17. In this example, the memory resources are the memory 27 and / or the data source 16'. The processing resources are the processing device 21. The communication engine 59 is programmed with logic to generate communications and transmit them to external systems, such as client devices 19a…19n.

[0049] Data processing system 17 operates both when enrollment is predicted to fall short of the target enrollment and when enrollment is predicted to exceed the target enrollment. In this example, data processing system 17 receives functional data 56. Based on the content of functional data 56, data processing system 17 determines that enrollment will not meet the target enrollment. When this occurs, data processing system 17 is programmed with logic to perform the following operations: Data processing system 17 notifies communication engine 59 that communications need to be sent to client devices 19a…19n. In this example, the client devices are associated with the clinical trial study site, and the communications inform users of the client devices that have received the communications that enrollment is less than the threshold amount and may prompt the users to perform one or more actions. Data processing system 17 also notifies resource allocator 58 that communication engine 59 needs to generate and transmit these communications and requests resource allocator 58 to provide sufficient available processing and memory resources for the communications to be generated and transmitted. In this example, resource allocator 58 is programmed with logic that specifies how much processing and memory resources communication engine 59 needs to generate and transmit communications. By executing this programming logic, the resource allocator 58 determines whether it needs to shut down, pause, or delay other tasks and / or operations being performed by the data processing system 17 to ensure that there are sufficient memory and processing resources available for the generation and transmission of communication.

[0050] In this example, once the resource allocator 58 has determined that sufficient memory and processing resources are available, it notifies the communication engine 59. The communication engine 59 then generates communications 23a…23n and transmits them to client devices 19a…19n to prompt the user or client devices 19a…19n to perform one or more actions.

[0051] Figure 2Various aspects of the data processing system 11 are illustrated. Typically, the data processing system 11 includes several operational modules that perform specific functions related to the operation of the data processing system 11. For example, as previously described, the data processing system 11 includes a target pattern generator 18, a prompter 20, a neural network 22, a post-processor 24, a trainer 30, and a calibrator 32. Furthermore, the data processing system 11 includes a database module 60, a communication module 62, and a processing module 64. The operational modules can be provided as one or more computer-executable software modules, hardware modules, or combinations thereof. For example, one or more operational modules can be implemented as software code blocks having instructions that cause one or more processors of the data processing system 11 to perform the operations described herein. Alternatively or concurrently, one or more operational modules can be implemented in electronic circuitry, such as programmable logic circuits, field-programmable arrays (FPGAs), or application-specific integrated circuits (ASICs).

[0052] Database module 60 maintains information related to real-time enrollment prediction using neural network 22. As an example, database module 60 may store training data 60a used to train or prompt neural network 22. In some implementations, training data 60a may include data representing historical clinical trial studies and associated enrollment counts for those clinical trial studies. The data representing clinical trial studies may specify attributes of the clinical trial studies, which may also specify the disease domain studied by the clinical trial study (or otherwise associated with the clinical trial study). Such attributes may include name, location of the clinical trial study, start date, completion date, target enrollment, etc.

[0053] As another example, database module 60 may store input data 60b used as input to neural network 22. As an example, input data 60b may include commands or instructions provided by the user, including information about specific desired outputs of data processing system 11, information input into the graphical user interface, etc. For example, input data 60b may specify the specific clinical trial study for which enrollment prediction is requested. Input data 60b may also include target enrollment, start date, completion date, and information about the individual location of participation in the clinical trial study, as well as the disease domain of the clinical trial study, and / or any other information about the clinical trial study.

[0054] In addition, input data 60b may include historical data, such as site historical performance metrics derived from historical studies. Historical studies include start dates, completion dates, actual enrollments for each month, etc. Input data 60b also includes generating real-time enrollment data for the targeted clinical trial study (and / or a specific site of the clinical trial study). In this example, (e.g., associated with a clinical trial study) the computer system stores data in data storage 16 (… Figure 1A Transmit information specifying the group count at a planned time, at a periodic time, intermittently, continuously, etc.

[0055] Database module 60 may also store output data 60c generated by neural network 22. As an example, output data 60c may include near real-time predictions of enrollment for a specific clinical trial study and / or for a specific location within a specific clinical trial study. For instance, output data 60c may be used to generate location predictions, which are then aggregated into study predictions.

[0056] Furthermore, database module 60 may store processing rules 60d (e.g., rules specifying the type of historical performance metrics to be retrieved for a site and / or other historical data to be retrieved, including, for example, historical data associated with the same disease domain as the clinical trial study targeted to generate near-real-time predictions), which specify how the neural network 22 can be used to process the data in database module 60 to generate near-real-time enrollment predictions and / or which data to retrieve from one or more data stores. (See also: Regarding...) Figure 6 The historical data discussed also includes data from designated competitive field trials.

[0057] In another example, processing rule 60d may include one or more rules for implementing, instructing, tuning, or prompting, and operating neural network 22 to produce output data 60c. For example, one or more rules may specify that training data 60a is provided to neural network 22 for training or prompting (e.g., enabling neural network 22 to generate enrollment predictions, identify trends and / or correlations between the content of clinical trial input data and the topics therein, and generate new outputs based on these identified trends and / or correlations). As another example, one or more rules may specify that input data 60b is provided to neural network 22 (e.g., to generate output data 60c representing enrollment predictions). As another example, one or more rules may specify that the generated output data 60c is presented to a user and / or stored for future retrieval and / or processing (e.g., using database module 60). As another example, one or more rules may specify one or more tools that facilitate specific actions performed by neural network 22. For example, tools may specify certain actions or operations that neural network 22 can perform to generate enrollment predictions, retrieve data, and generate content based on the retrieved data, etc. This article provides a more detailed description of the example data processing techniques.

[0058] As described above, the data processing system 11 also includes a communication module 62. The communication module 62 allows data to be transmitted to and from the data processing system 11. For example, the communication module 62 can be communicatively connected to a network, enabling it to communicate with client device 14 ( Figure 1A The system transmits data and receives data from the client device 14. Information received from the client device 14 can be processed (e.g., using processing module 64) and stored (e.g., using database module 60).

[0059] As described above, the data processing system 11 also includes a processing module 64. The processing module 64 processes data stored in or otherwise accessible by the data processing system 11. For example, the processing module 64 may be used to perform one or more of the operations described herein (e.g., operations associated with the neural network 22).

[0060] In some implementations, software applications can be used to facilitate the tasks described herein. As an example, the application can be installed on the data processing system 11 and / or the client device 14. Furthermore, the user can interact with the application to input data and / or commands to the data processing system 11, and to view the data generated by the data processing system 11.

[0061] refer to Figure 3 The diagram illustrates a calibration environment 70. Calibration environment 70 includes a calibrator 74 for calibrating or fine-tuning neural networks (e.g., recurrent neural networks). Calibration environment 70 also includes a data source 72 and a data storage 104. In operation, data source 72 sends or transfers a modeling dataset 106 to a preprocessor 78. Modeling dataset 106 includes experimental-level and test-site-level data. For example, data source 72 transfers modeling dataset 106 to preprocessor 78, which organizes the data in the correct sequence and transforms it to accurately reflect the calendar month. Maintaining the sequential order of the data is crucial in the context of data preprocessing because current neural networks are designed to retain the internal state of the memory that processes the input sequence, thereby capturing temporal dependencies and patterns over time, as discussed in further detail below.

[0062] The preprocessor 78 outputs the preprocessed data to the segmenter 80, which splits the preprocessed data into a test dataset 106a and a training dataset 106b, as will be described herein. The calibrator 74 includes a tuner 76 for determining the values ​​(optimal values) of the hyperparameters of the neural network. In this example, the tuner 76 includes an encoder 82. The segmenter 80 transmits the training dataset 106b to the encoder 82. The encoder 82 performs various types of encoding on the training dataset 106b, such as one-hot encoding. One-hot encoding is a method used to convert categorical variables into binary format. The main purpose of one-hot encoding is to ensure that categorical data can be effectively used in machine learning models. The encoder 82 transmits the encoded training dataset to the input engine 84.

[0063] In this example, input engine 84 acts as the input layer in the neural network, receiving information that will be used to fine-tune the network. Input engine 84 feeds the encoded training dataset to hidden layer engine 100, which begins the forward pass through the RNN as described herein. At each step, the RNN takes two inputs: the current input received from input engine 84 and the hidden state from the previous time step. The RNN computes these inputs to produce a new hidden state generated by hidden state engine 104. The new hidden state is then passed to the next time step (e.g., received by input engine 84), and the process continues. Once a transition is reached during training, hidden layer engine 100 outputs the proposed tuned model to validation tester 102, which uses test dataset 106a to test the effectiveness of the tuned model. If the tuned model has sufficient accuracy, validation tester 102 outputs a calibrated or tuned model 108 to data storage 104. Additional details of the calibration environment are provided below.

[0064] Given the sequential nature of the input data (where each time period is influenced by the time periods preceding it), RNNs are ideally suited for this task. RNNs can effectively learn from the input history to predict future trends by remembering important information from past time periods while incorporating new data.

[0065] In practice, this means that if a study is falling behind its target enrollment trajectory, an RNN model can identify this trend early, allowing stakeholders to take swift corrective action. Similarly, if a study is progressing faster than expected, resources can be reallocated to ensure optimal efficiency. Thus, using deep learning models such as RNNs provides a powerful tool for transforming raw enrollment data into actionable insights.

[0066] In the field of accurately predicting research enrollment at various stages, complex, multidimensional, and sequential datasets require sufficiently sophisticated deep learning models to discern underlying patterns and interdependencies. Recurrent Neural Networks (RNNs) are designed to work with the inherent sequential data in clinical trial enrollment data. While traditional machine learning (ML) models treat all input data as independent, RNNs can identify patterns across time due to their “recursive” nature. The “recursive” part of an RNN refers to its ability to remember or retain previous outputs in its internal memory while processing the current input. This ability allows RNN units to connect information from earlier steps to later steps as a sequence, making it ideal for processing large datasets with a temporal element.

[0067] Furthermore, RNNs can simplify feature engineering for time-series data. Feature engineering is the process of transforming raw data into features that better represent the underlying problem, thereby improving model performance. In the case of time-series data, traditionally, lagged versions of variables, trends, seasonal indicators, and other time components are manually generated to provide sufficient context to the model. Leveraging the inherent ability of RNNs to process sequential data, RNNs can encode information from previous data points, eliminating the need for feature engineering.

[0068] Traditional RNN models have significant limitations in handling the vanishing gradient problem, which can be interpreted as the influence of distant past events becoming increasingly less apparent until it disappears completely. This limits the ability of traditional RNN models to process long sequences and retain information from earlier stages of the sequence. GRU addresses some of the challenges encountered by traditional RNNs (known as the "vanishing gradient problem"). This means that RNNs may struggle to connect information from earlier times in sequences with later elements. By utilizing the gating mechanism in GRU, which controls the amount of internal state information that will be retained or forgotten for the next new element, the model is better able to learn long-term dependencies.

[0069] The training data consisted of a mixture of raw trial enrollment data and derived features processed using the Enrollment Modeling (EM) data pipeline. This pipeline generated two main metric streams: study site data and study site-month data. The study site data focused on descriptive details of each study site, including site capacity, past performance, and attributes influencing patient enrollment. The study site-month data focused on monthly snapshots, detailing enrollment rates and illustrating the dynamic evolution of sites over time.

[0070] In this example, the database (e.g., Figure 3 The EDC 72 in the database stores data from clinical trials that span decades, conducted across hundreds of countries, covering hundreds of indicators and encompassing many therapeutic areas.

[0071] Preprocessing the input data (e.g., by preprocessor 78) involves organizing the data in the correct sequence and transforming it to accurately reflect the calendar months. Maintaining the sequential order of the data is crucial in the context of data preprocessing because RNNs are designed to preserve the internal state, or "memory," of processing the input sequence, thereby capturing temporal dependencies and patterns (as previously described). By preserving the sequential nature of calendar months in the data, RNNs can effectively learn from past months to generate accurate predictions of future months. The primary data granularity working together is at the research-site level, and the techniques described in this paper precisely isolate research-site-, research-specific, and country-specific features through the use of unique combinations of IDs (e.g., database IDs, research IDs, and cluster IDs (unique healthcare facility IDs)).

[0072] Each research site has a rich set of features attached to it. This information includes both time-varying and constant features. Time-varying or monthly-varying features contain data that changes monthly and provide insights through aggregation, such as the total number of patients recruited in each month during the study enrollment period within a country, the active research sites during that period, and the average number of patients per site at the national level. On the other hand, examples of constant features include the study's indications, its treatment area, phase, and associated sponsors, as well as country-specific data such as population and density.

[0073] The preprocessed data is further split into training and test sets to ensure that the model (e.g., a neural network) can generalize to new, unseen data. Cross-validation on the training set is used for hyperparameter tuning and feature selection. The most common type is k-fold cross-validation, where the initial samples are randomly divided into k equal-sized subsamples. Of these k subsamples, a single subsample is retained as validation data for testing the model, and the remaining k-1 subsamples are used as training data. In this data splitting process, each study is treated as a single entity during the sampling period. This approach ensures that the same study is not split across both the training and test sets. In the context of RNNs, maintaining the temporal order of the data is crucial. The temporal series cross-validation used adheres to this temporal order, allowing the model to learn meaningful temporal patterns during training, which can then be tested against unseen future data.

[0074] Furthermore, for the training set, transformations tailored to the properties of each feature type (including categorical features, time-varying numerical features, and constant numerical features) are performed. This method ensures that the RNN can efficiently learn patterns from the data. For the same feature (e.g., categorical features), one-hot encoding is used for its transformation. Other categorical features are initially converted to their corresponding index formats (e.g., to generate integer-encoded categorical features). Embeddings are generated, for example, by merging these integer-encoded categorical features into different embedding layers before being passed to the RNN. The embedding layers essentially function as lookup tables maintaining dense vectors for each category. That is, in order to merge features into the embedding layers, a feature (if it has not already been encoded) is encoded as a categorical value, and then the encoded value is fed as input to the embedding layers; this allows the model to learn a vector representation of the feature, thereby capturing the relationships between different values ​​of the feature within the embedding space during training. Utilizing embedding layers to encode categorical features offers several advantages, such as reduced dimensionality compared to one-hot encoding, and the ability to capture semantic relationships between different categories. Numerical features (e.g., population density) are z-normalized to aid model training. In contrast, time-varying numerical features (such as the number of patients enrolled and the number of activation sites) are preserved in their original form, thus leveraging the inherent ability of RNNs to handle time-series data.

[0075] After initializing the embedding layer, the RNN requires input to maintain a consistent length across batches. Since sequences have varying lengths, shorter sequences are typically padded by adding zeros to the ends to match the desired sequence length. Conversely, overly long sequences are truncated to ensure they conform to the specified length.

[0076] Before starting the forward pass through a two-layer RNN, an initial hidden state is set up. This initial context for the RNN is created by passing the embeddings and numerical features through a series of fully connected linear layers. The forward pass through the RNN begins with the first "K" elements of the sequence, where 'K' represents the initial known segment of the data sequence. In this particular context, 'K' reflects the number of months since the study began its enrollment processing. At each time step, the RNN unit takes two inputs: the current input and the hidden state from the previous time step. The RNN computes these inputs to produce a new hidden state. The new hidden state is then passed to the next time step. This process is repeated in a loop starting at element K+1 and continuing until the maximum predicted length. This process allows the RNN to efficiently process the sequence and preserve information from previous time steps.

[0077] After processing, activation functions are applied to the raw output values ​​of the RNN. When focusing on predicting continuous non-negative values, the ReLU (Rectified Linear Unit) activation function is used at the output layer. ReLU effectively sets negative values ​​to zero and allows positive values ​​to remain unchanged. The inclusion of ReLU introduces non-linearity, which helps the model capture complex patterns while maintaining computational efficiency.

[0078] During training, various techniques are employed to prevent overfitting. One such technique is dropout, which randomly sets the proportion of input units to 0 at each update during training time, which helps prevent overfitting. In addition to preventing overfitting (i.e., the model learning too much from the noise in the training data and not generalizing well to unseen data), noise is added directly to the embeddings and other inputs to the model.

[0079] To facilitate faster convergence during training, a "teacher forcing" technique is employed. Instead of consistently using the model's own previous predictions as input to the current time step, actual ground truth values ​​are occasionally inserted. This approach tends to accelerate model convergence, resulting in a more stable and accurate model. Training processing is performed in iterations equal to the batch size (which constitutes one epoch—i.e., the entire training dataset is passed through a single complete pass of the learning algorithm). This processing is applied to the cross-validation (CV) set (a subset of the training data) to ensure the model does not overfit and generalizes well to unseen data.

[0080] Once activations are generated, the RNN's output is matched to the actual values ​​using a loss function. After calculating the loss, the model's parameters are updated. Backpropagation calculates the gradient of the loss relative to the model's parameters, which is then used for updates.

[0081] After each period, a validation test set is used to evaluate the model's performance. This involves processing the validation data with the current model parameters, making predictions under different choices of K (1, 3, 8, etc.), where the predicted values ​​are compared with the actual values ​​at the study-site and study levels. This involves calculating the absolute difference between the predicted and actual monthly inclusion values ​​for each month.

[0082] To optimize model performance, hyperparameter tuning using a cross-validation set is performed. The tuning process utilizes a grid search to systematically explore the range of hyperparameter values. This exhaustive search ensures a more thorough exploration, leading to the optimal hyperparameter configuration.

[0083] Hyperparameters include the learning rate, dropout rate, and teacher coercion rate, as well as the dimensions of the embedding and linear layers, the number of layers in the RNN, the L2 decay rate (a regularization method that prevents overfitting by penalizing large weights), and the level of noise added to the input. The dimensions of the embedding and linear layers determine the complexity and ability of the model to capture information. The number of layers in an RNN can affect its ability to capture complex patterns in the data. Adding noise to the input is a regularization technique that can improve generalization by making the model less sensitive to small changes in the input.

[0084] After selecting the optimal hyperparameters based on the cross-validation set performance, the model is then retrained on the full training set, including the cross-validation set. This process ensures that the model is tuned and ready to be evaluated on unseen data.

[0085] Once the optimal hyperparameters are fine-tuned, the entire dataset is used for training. This approach ensures that the model has access to the maximum amount of available information, thus leveraging every bit of potential insight provided by the data.

[0086] Models (e.g., neural networks) provide monthly enrollment predictions specific to individual research sites. Post-processing can aggregate research-level enrollment data across calendar months. This granularity allows stakeholders to gain insights not only at a macro level but also to understand the dynamics at individual research sites.

[0087] Figure 4 An example of training a neural network is illustrated. For instance, training set 120 is input with various types of input data (e.g., including numerical and categorical features). Training set 120 may include clinical trial study data 120a describing clinical trial studies (including, for example, attributes of the clinical trial study and the disease domain, numerical and categorical features, etc.). Clinical trial study data 120 is mapped to labels 120b via a mapping structure. The mapping structure includes pointers or other types of structures that associate one data item with another. Label 120b specifies the enrollment count of the clinical trial study represented in clinical trial study data 120a.

[0088] Training set 120 can include any suitable number of training datasets. Individual training examples in training set 120 can include contextual data specifying the name and other attributes of the clinical trial study. Contextual data may also include information about the data structure, including, for example, which field stores or otherwise indicates the disease domain of the clinical trial study. The specification of this field assists the neural network in requesting and retrieving historical data associated with that disease domain.

[0089] Neural network 124 can be trained by neural network trainer 122 to generate real-time ingress predictions. In some implementations, neural network trainer 122 can compute a numerical representation (e.g., in vector form) of the training data used for machine training. In some implementations, neural network training techniques include feature extraction to build different neurons within the neural network. One or more features can transform the more likely ingress target to be achieved. For example, a particular feature can correspond to the achievement of a particular ingress target, such that the strength of the current feature leads to the determination of a particular metric (e.g., an indicator of the ingress target being achieved) for the corresponding event (e.g., an event indicating a start date, a completion date, and an ingress target).

[0090] In some implementations, a single machine learning model (e.g., a neural network) can be trained. In other implementations, multiple models can be trained based on a training set of 120. Once trained, the trained neural network is able to receive and process requests. For example, given the input data for a clinical trial study, the trained neural network can predict enrollment.

[0091] To fine-tune the neural network, a small, high-quality, and diverse training dataset is used. This strategy prevents overfitting while ensuring the model performs well across different scenarios. The training dataset includes various ingress counts to provide balanced training data. Using the prepared dataset (as described above), the neural network is fine-tuned via supervised learning (where inputs include numerical values, categorical values, etc.), and labels indicate the ingress count. For example, numerical values ​​include historical data such as historical performance metrics, competitive field trials, and real-time data. Categorical values ​​include country (e.g., "KOR", "CHN"), indication (e.g., "684" following ICD9 code), stage (e.g., "Phase III", "Phase IV", "Phase II", "Phase I"), gender (e.g., "female", "male"), etc.

[0092] refer to Figure 5The system workflow 130 of the data processing system described herein is illustrated. In operation, the EDC system 134 transmits data to the data processing system specifying the sites participating in a particular trial and their enrollment to date. Using this information, the data processing system updates 136 the participating sites and their enrollment to date. Based on these updates, the user selects 132 studies to track and predict enrollment data. The user can add / edit 138 study and site attributes as needed, such as study enrollment objectives, site activation and completion dates. After the user has edited or modified the site attributes, the user requests 140 forecasts, for example, to predict enrollment up to a specific target date. A neural network model generates 142 forecasts for each participating site and / or for each selected site. The data processing system performs post-processing 144 and also aggregates the data to make it viewable at the study level and / or country level. The data processing system generates one or more graphical user interfaces for the user to view 145 aggregated study enrollment forecasts and country details. Based on the visualizations in the graphical user interface, the user decides 146 whether any intervention is needed.

[0093] refer to Figure 6 The example input and output are shown. In this example, input 150 includes research data (including real-time data) and historical data. The research data includes research attributes and location attributes, as well as actual enrollment performance. Input 150 is fed into neural network 152, which produces predicted output data 154. In this example, predicted output data 154 is broken down by month and also across various locations. Predicted output data 154 also includes aggregations for viewing predicted enrollment at the national and research-specific levels.

[0094] refer to Figure 7A This illustrates a graphical user interface 160 that allows users to add and / or edit locations and location attributes as needed. In this example, the graphical user interface 160 includes a table with columns 160a, 160b, 160c, and 160d. The table also includes rows 160e, 160f, and 160g. Column 160a specifies the information type, such as, for example, target inclusion. Column 160b specifies the source of this information type. Column 160c specifies the status of this information type. Column 160d specifies the various actions that can be taken for this information type. Rows 160e-160g specify the various information types. The actions specified by column 160d include controls 160h and 160i. Selecting control 160h allows the user to edit the target, such as target inclusion. Selecting control 160i causes the data processing system of this paper to update the prediction of inclusion up to the target date.

[0095] refer to Figure 7BThe diagram illustrates a graphical user interface 170 that allows users to modify various attributes of a specific study. The graphical user interface includes control 170a, which allows users to modify specific indications for the study. Graphical user interface 170 includes control 170b, which allows users to specify the sex of patients to be included in the study used for prediction. Control 170c allows users to specify the age range to be used in the prediction. Control 170d allows users to specify the number of study arms to be used in the prediction. Graphical user interface 170 also includes a section 170e for specifying target enrollment for a specific study.

[0096] refer to Figure 7C The diagram illustrates a graphical user interface 180 that allows users to modify various venue-level attributes. In this example, the graphical user interface 180 includes controls 180a-180c that allow users to specify venue activation dates. The graphical user interface 180 also includes controls 180d-180f that allow users to specify venue enrollment end dates.

[0097] refer to Figure 8A The diagram illustrates a graphical user interface 190 with aggregated study enrollment forecasts. The graphical user interface 190 includes a visual representation 190a specifying predicted enrollments up to various dates, a visual representation 190b specifying target enrollments, and a visual representation 190c specifying actual enrollments.

[0098] refer to Figure 8B The graphical user interface 192 shows an aggregated view of predicted enrollments for a specific study. In this example, the aggregated view is at the national level.

[0099] refer to Figure 8C The diagram illustrates a graphical user interface 194 that provides a more granular view of predictions for various locations. In this example, the graphical user interface 194 includes rows 194a-194c, each specifying a particular location. The graphical user interface 194 also includes a column 194d specifying the actual inclusions for the location to date and a column 194e specifying the predictions for the location.

[0100] refer to Figure 9This paper illustrates the processing 200 used to develop and validate neural networks for input prediction as described herein. In this example, the EDC system 202 accesses experimental-level data 204 and test-site-level data. The data processing system described herein uses the experimental-level data 204 to prepare the modeling dataset 206. The preparation of the modeling dataset is described herein. As described herein, the data processing system splits the modeling dataset 208 into training and test sets. The training data is stored in data storage 212. The test data is stored in data storage 210. The data processing system defines 214 the model architecture and fits the model. The data processing system evaluates 216 the model accuracy. The data processing system refines 218 the model by exploring different feature sets, architectures, and hyperparameter values. The data processing system determines 220 whether the model achieves satisfactory accuracy or whether there are additional variables to explore. When the model does not achieve satisfactory accuracy or there are additional variables to explore, the data processing system returns to defining and fitting the model. However, when the data processing system determines that the model has achieved satisfactory accuracy, or that there are no additional variables to explore, the data processing system determines the final model. The data processing system then uses test data stored in data storage 210 to test the final model and evaluates and reports the model accuracy against the test data.

[0101] refer to Figure 10 Visualizations of model (e.g., neural network) performance on a test set with over 2,000 historical studies are shown.226 For each study, the aggregated predicted enrollment count on the last-patient-in day of the study is compared to the actual enrollment count of the study, and the absolute error (AE) is calculated. This exercise is performed assuming known enrollments for different months (k) (from 1 to 60). The AEs calculated at the same k are averaged across studies to obtain the mean absolute error (MAE). Each MAE value corresponds to a point in the figure below, with lower values ​​indicating better model performance.

[0102] Additionally, the model performance (with several baselines) is compared to several baselines. The solid black line (e.g., the lowest line) represents the performance of the deep learning model, the shaded line (e.g., the top line) represents the baseline for predicting future enrollment using historical enrollment rates within a country, and the middle line (the line with white circles) represents the line for predicting future enrollment using the average observed enrollment rate of current studies up to month k. The deep learning model (e.g., the neural network described in this paper) outperforms both baseline methods.

[0103] refer to Figure 11The diagram illustrates a process 230 for generating a target pattern for use with a neural network to produce near real-time predictions of clinical trial enrollment. In operation, the data processing system causes (231) to draw one or more user interfaces having one or more input sections for inputting data for a specified clinical trial. The data processing system receives (232) input data corresponding to data input into one or more sections of the user interface, wherein the received input data specifies a given clinical trial. Based on the received input data, the data processing system generates (233) a target pattern for the given clinical trial, wherein the target pattern specifies the target enrollment and completion dates for the given clinical trial. In some examples, the target enrollment and completion dates are input into a graphical user interface. In other examples, the data processing system uses input of data for the specified clinical trial (e.g., the name of the clinical trial) to look up or retrieve information about the clinical trial (e.g., target enrollment, start date, completion date, site name, etc.) from a data store.

[0104] The data processing system accesses (234) historical and enrollment data for a given clinical trial from one or more hardware storage devices, where the historical data comes from one or more studies conducted in the disease area associated with the clinical trial, and where the enrollment data specifies the actual enrollments in the clinical trial. In this example, the data processing system requests CTMS to generate predictions for the disease area of ​​the targeted clinical trial study, and then requests historical data (e.g., historical enrollment data) for clinical trial studies in the same or similar disease area. The data processing system also retrieves from CTMS the actual enrollment data for the targeted clinical trial study.

[0105] The data processing system inputs (235) the target pattern, enrollment data, and historical data into the neural network. The data processing system receives (236) prediction data from the neural network, which specifies one or more near-real-time predictions for enrollment in the clinical trial. The data processing system outputs (237) one or more visualizations of the received prediction data to one or more user interfaces.

[0106] Figure 12 An example computing system according to an implementation of this disclosure is depicted. System 240 can be used for any of the operations described with respect to the various implementations discussed herein. System 240 may include one or more processors 242, memory 246, one or more storage devices 250, and one or more I / O devices 254 controllable via one or more input / output (I / O) interfaces 252. The various components 242, 246, 250, 252, or 254 may be interconnected via at least one system bus 248, which enables data transfer between various modules and components of system 240.

[0107] One or more processors 242 may be configured to process instructions for execution within system 240. One or more processors 242 may include one or more single-threaded processors, one or more multi-threaded processors, or both. One or more processors 242 may be configured to process instructions stored in memory 246 or on storage device 250. One or more processors 242 may include one or more hardware-based processors, each including one or more cores. One or more processors 242 may include one or more general-purpose processors, one or more dedicated processors, or both.

[0108] Memory 246 may store information within system 240. In some implementations, memory 246 includes one or more computer-readable media. Memory 246 may include any number of volatile memory cells, any number of non-volatile memory cells, or both. Memory 246 may include read-only memory, random access memory, or both. In some examples, memory 246 may be used as active or physical memory by one or more software modules.

[0109] One or more storage devices 250 may be configured to provide (e.g., persistent) mass storage to system 240. In some implementations, one or more storage devices 250 may include one or more computer-readable media. For example, one or more storage devices 250 may include floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices. One or more storage devices 250 may include read-only memory, random access memory, or both. One or more storage devices 250 may include one or more of internal hard disk drives, external hard disk drives, and removable drives.

[0110] One or both of memory 246 and (one or more) storage devices 250 may include one or more computer-readable storage media (CRSM), including, for example, non-transitory computer-readable media. CRSM may include one or more of electronic storage media, magnetic storage media, optical storage media, magneto-optical storage media, quantum storage media, mechanical computer storage media, etc. CRSM can provide storage for computer-readable instructions describing data structures, processes, applications, programs, other modules, or other data used for the operation of system 240. In some implementations, CRSM may include data storage that provides storage for computer-readable instructions or other information in a non-transitory format. CRSM may be incorporated into system 240 or may be external to system 240. CRSM may include read-only memory, random access memory, or both. One or more CRSMs suitable for tangibly embodying computer program instructions and data may include any type of non-volatile memory, including but not limited to: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. In some examples, (one or more) processors 242 and memory 246 may be supplemented by or incorporated into one or more application-specific integrated circuits (ASICs).

[0111] System 240 may include one or more I / O devices 254. One or more I / O devices 254 may include one or more input devices, such as a keyboard, mouse, pen, game controller, touch input device, audio input device (e.g., microphone), gesture input device, haptic input device, image or video capture device (e.g., camera), or other devices. In some examples, one or more I / O devices 254 may also include one or more output devices, such as a display, one or more LEDs, audio output devices (e.g., speakers), printers, haptic output devices, etc. One or more I / O devices 254 may be physically incorporated into one or more computing devices of system 240, or may be external to one or more computing devices of system 240.

[0112] System 240 may include one or more I / O interfaces 252 to enable components or modules of system 240 to control, interact with, or otherwise communicate with (one or more) I / O devices 254. The I / O interfaces 252 may enable information to be transmitted within or outside system 240 or between components of system 240 via serial communication, parallel communication, or other types of communication. For example, the I / O interfaces 252 may conform to a version of the RS-232 standard for serial ports or a version of the IEEE 1284 standard for parallel ports. As another example, the I / O interfaces 252 may be configured to provide connectivity via a Universal Serial Bus (USB) or Ethernet. In some examples, the I / O interfaces 252 may be configured to provide a serial connection conforming to a version of the IEEE 1394 standard.

[0113] One or more I / O interfaces 252 may also include one or more network interfaces that enable communication between computing devices in system 240 or between system 240 and other network-connected computing systems. One or more network interfaces may include one or more network interface controllers (NICs) or other types of transceiver devices configured to send and receive communications on one or more networks using any network protocol.

[0114] The computing devices of system 240 can communicate with each other or with other computing devices using one or more networks. Such networks can include public networks such as the Internet, private networks such as institutional or personal intranets, or any combination of private and public networks. Networks can include any type of wired or wireless network, including but not limited to local area networks (LANs), wide area networks (WANs), wireless local area networks (WWANs), wireless LANs (WLANs), and mobile communication networks (e.g., 3G, 4G, Edge, etc.). In some implementations, communication between computing devices can be encrypted or otherwise protected. For example, communication can employ one or more public or private encryption keys, passwords, digital certificates, or other credentials supported by security protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols.

[0115] System 240 may include any number of computing devices of any type. One or more computing devices may include, but are not limited to: personal computers, smartphones, tablet computers, wearable computers, implantable computers, mobile gaming devices, e-book readers, automotive computers, desktop computers, laptop computers, notebook computers, game consoles, home entertainment devices, network computers, server computers, mainframe computers, distributed computing devices (e.g., cloud computing devices), microcomputers, system-on-a-chip (SoC), and system-in-package (SiP). Although the examples herein may describe one or more computing devices as one or more physical devices, implementations are not limited thereto. In some examples, a computing device may include one or more virtual computing environments, hypervisors, emulations, or virtual machines executing on one or more physical computing devices. In some examples, two or more computing devices may include clusters, clouds, farms, or other groups of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, shared networking capabilities, or other aspects.

[0116] This specification uses the term "configured to" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed on it, which, in operation, causes the system to perform that operation or action. For a computer program configured to perform a particular operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0117] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in one or more of these combinations. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium) for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more of these combinations. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0118] The term data processing device (also referred to herein as a data processing system) means data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or multiple computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0119] Computer programs (which may also be referred to or described as programs, software, software applications, apps, modules, software modules, scripts, or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and computer programs can be deployed in any form (including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment). A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file holding other programs or data (e.g., stored in one or more scripts within a markup language document), stored in a single file dedicated to the program in question, or stored in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code sections). Computer programs can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected via data communication networks.

[0120] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple data collections, which can be organized and accessed differently.

[0121] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same or multiple computers.

[0122] The processing and logic flow described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flow can also be performed by dedicated logic circuitry (e.g., FPGA or ASIC), or by a combination of dedicated logic circuitry and one or more programmable computers.

[0123] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory (ROM) or random access memory (RAM), or both. The basic components of a computer are the CPU for making or executing instructions and one or more memory devices for storing instructions and data. The CPU and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks), or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few).

[0124] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0125] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) via which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0126] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0127] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0128] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the specification within the context of separate embodiments may also be implemented in combination in a single embodiment. On the other hand, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0129] Similarly, although operations are depicted in a specific order in the drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequence, or requiring all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0130] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions.

[0131] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. An apparatus for facilitating software development to use a neural network in generating near real-time predictions of enrollment in clinical studies, said apparatus comprising: One or more processors; as well as One or more hardware storage devices for storing instructions, said instructions being executable by said one or more processors to perform one or more first operations, said one or more first operations including: Receive one or more instructions in a development environment, said one or more instructions being executable by said one or more processors to perform one or more second operations, said one or more second operations including: This enables the creation of one or more user interfaces with one or more input sections for inputting data from a specified clinical trial. Receive input data from one or more user interfaces corresponding to data input into one or more portions of the user interfaces, wherein the received input data specifies a given clinical trial; Receive the target pattern for the given clinical trial from one or more user interfaces; Access to historical and enrollment data for the given clinical trial from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease area associated with the clinical trial, and wherein the enrollment data specifies the actual enrollment in the clinical trial; The target pattern, the input data received for the given clinical trial, the enrollment data, and the historical data are input into the neural network; Receive predictive data from the neural network specifying one or more near real-time predictions for enrollment in the clinical trial; and Output one or more visualizations of the received prediction data to one or more user interfaces.

2. The device according to claim 1, wherein, The clinical trial is associated with one or more sites, wherein the accessed enrollment data is for a given site within the one or more sites, and wherein the prediction data specifies one or more near-real-time predictions of enrollment for the clinical trial at the given site within the one or more sites.

3. The device according to claim 1, wherein, The target pattern specifies the enrollment target, activation date, and / or completion date.

4. The device according to claim 1, wherein, The neural network in question is a recurrent neural network.

5. The device according to claim 1, wherein, The one or more visualizations provide a research-level first view or a national-level second view.

6. A method for generating a target pattern, implemented by a data processing system, said target pattern being used in conjunction with a neural network to generate a near real-time prediction of enrollment in a clinical study, said method comprising: This enables the creation of one or more user interfaces with one or more input sections for inputting data from a specified clinical trial. Receive input data from one or more user interfaces corresponding to data input into one or more portions of the user interfaces, wherein the received input data specifies a given clinical trial; Based on the received input data, the data processing system generates a target pattern for the given clinical trial, wherein the target pattern specifies the target enrollment and completion dates for the given clinical trial; Access to historical and enrollment data for the given clinical trial from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease area associated with the clinical trial, and wherein the enrollment data specifies the actual enrollment in the clinical trial; The target pattern, the input data, and the historical data are input into the neural network; Receive predictive data from the neural network specifying one or more near real-time predictions for enrollment in the clinical trial; and Output one or more visualizations of the received prediction data to one or more user interfaces.

7. The method according to claim 6, wherein, The clinical trial is associated with one or more sites, wherein the accessed enrollment data is for a given site within the one or more sites, and wherein the prediction data specifies one or more near-real-time predictions of enrollment for the clinical trial at the given site within the one or more sites.

8. The method according to claim 6, wherein, The target pattern specifies the enrollment target, activation date, and / or completion date.

9. The method according to claim 6, further comprising: The neural network is calibrated using the following methods: Accessing clinical trial data from hardware storage devices; The clinical trial data was divided into test data and training data; For the training data Identify one or more features in the training data; Encode one or more of the features; Transmit at least a portion of one or more encoded features to one or more hidden layers of the neural network; Based on the output of one or more hidden layers, determine one or more values ​​of one or more hyperparameters of the neural network; and The transmission and determination are repeated until the neural network output satisfies one or more values ​​of one or more thresholds.

10. The method of claim 6, further comprising: The neural network is trained using supervised learning techniques based on training data to predict one or more near real-time predictions, wherein the neural network is configured to receive the target pattern, input data specifying the given clinical trial, the enrollment data, and the historical data as inputs, and to process the inputs to generate outputs specifying one or more near real-time predictions for enrollment in the given clinical trial, wherein the neural network includes a plurality of artificial neurons connected by edges and aggregated into a plurality of neural network layers including at least an input layer and an output layer, wherein each edge is configured to transmit a signal from one artificial neuron to another, and wherein the output of each artificial neuron is calculated based on the input of that artificial neuron according to a plurality of weights.

11. The method of claim 10, further comprising: The data processing system processes new data using the plurality of artificial neurons in a trained neural network based on the values ​​of the plurality of weights to identify one or more near-real-time predictions for enrollment in the given clinical trial, wherein the artificial neurons in the input layer are configured to receive the new data as input, and the artificial neurons in the output layer are configured to generate new outputs identifying near-real-time predictions for enrollment, and wherein the new data includes the target pattern and the received input data.

12. The method of claim 6, further comprising: Training the neural network to predict the intake includes: Receive training data, wherein one or more portions of the training data are associated with a label specifying an in-group count; Define a first squared error function, which quantifies the prediction error of the neural network based on the training data; and The weights of the neural network are set to minimize the first prediction error.

13. The method according to claim 6, wherein, The neural network in question is a recurrent neural network.

14. The method according to claim 6, wherein, The one or more visualizations provide a research-level first view or a national-level second view.

15. The method of claim 6, further comprising: Determine an inclusion group for one or more near-real-time predictions by specifying an amount less than a threshold; In response to the determination, functional data is generated, which, upon receipt by the clinical trial management system, causes the clinical trial management system to perform one or more operations. At least one of these operations is allocating memory and processing resources for generating one or more communications and transmitting them over a network to one or more client devices, each client device associated with the location of the clinical trial study. The communications also inform the user of the client device that has received the communications that the enrollment is less than a threshold and prompt the user to perform one or more actions. The functional data is transmitted to the clinical trial management system.

16. The method according to claim 15, wherein, The functional data mentioned is an alarm.

17. One or more machine-readable hardware storage devices for generating a target pattern, said target pattern being used in conjunction with a neural network to generate near real-time predictions indicative of enrollment in a clinical study, said one or more machine-readable hardware storage devices storing instructions executable by one or more processors to perform operations including: This enables the creation of one or more user interfaces with one or more input sections for inputting data from a specified clinical trial. Receive input data from one or more user interfaces corresponding to data input into one or more portions of the user interfaces, wherein the received input data specifies a given clinical trial; Based on the received input data, the data processing system generates a target pattern for the given clinical trial, wherein the target pattern specifies the target enrollment and completion dates for the given clinical trial; Access to historical and enrollment data for the given clinical trial from one or more hardware storage devices, wherein the historical data comes from one or more studies conducted in a disease area associated with the clinical trial, and wherein the enrollment data specifies the actual enrollment in the clinical trial; The target pattern, the input data, and the historical data are input into the neural network; Receive predictive data from the neural network specifying one or more near real-time predictions for enrollment in the clinical trial; and Output one or more visualizations of the received prediction data to one or more user interfaces.

18. One or more machine-readable hardware storage devices according to claim 17, wherein, The clinical trial is associated with one or more sites, wherein the accessed enrollment data is for a given site within the one or more sites, and wherein the prediction data specifies one or more near-real-time predictions of enrollment for the clinical trial at the given site within the one or more sites.

19. One or more machine-readable hardware storage devices according to claim 17, wherein, The target pattern specifies the enrollment target, activation date, and / or completion date.

20. One or more machine-readable hardware storage devices according to claim 17, wherein, The operation also includes: The neural network is calibrated using the following methods: Accessing clinical trial data from hardware storage devices; The clinical trial data was divided into test data and training data; For the training data Identify one or more features in the training data; Encode one or more of the features; Transmit at least a portion of one or more encoded features to one or more hidden layers of the neural network; Based on the output of one or more hidden layers, determine one or more values ​​of one or more hyperparameters of the neural network; and The transmission and determination are repeated until the neural network output satisfies one or more values ​​of one or more thresholds.

21. A computer program product storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 6-16.

Images