Artificial intelligence-driven clinical trial protocol data acquisition enhancement system and method

Abstract

To provide a method and system for artificial intelligence driven clinical trial protocol data acquisition and augmentation.SOLUTION: The method includes receiving a user request in a natural language via a user interface, wherein the user request is combined with context data to generate a prompt for a large language model, inputting the prompt into the model to generate a model response including a database query written in a database query language and metadata, retrieving clinical trial protocol data from a first database based on the database query, generating an API request based on the database query and / or metadata, and executing an API call using the generated API request to retrieve clinical trial metrics, and generating a response to the user request based on the retrieved protocol data and the metrics.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates generally to artificial intelligence driven clinical trial protocol data acquisition and augmentation. [Background technology]

[0002] Clinical research (clinical trials) often have diverse requirements, and each clinical trial has its own unique characteristics and requirements. A research protocol is a key document that defines the content of a clinical trial and its requirements. A research protocol is a text document that includes information about the overall picture of the study, its objectives, target patients, procedures, etc. Traditionally, the process of creating research protocols required medical professionals, such as doctors working for pharmaceutical companies, to create these protocols as formal documents. This process was inefficient, and these professionals often resisted using technology to assist in creating research protocols. Summary of the Invention [Problem to be solved by the invention]

[0003] Traditional clinical trial design software (such as proprietary software from clinical trial management system vendors or open-source solutions like OpenClinica and REDCap) focuses on data collection and management, without actively supporting protocol design. These tools typically require significant human input and do not leverage artificial intelligence (AI) to dynamically suggest improvements or predict outcomes based on historical data. Additionally, they generally lack the ability to provide real-time metrics or predictive analytics.

[0004] Some traditional systems integrate AI to optimize specific aspects of trials, such as patient recruitment or site selection. Examples include IBM Watson for Clinical Trial Matching and Antidote's Match, which focus on using AI to match patients with trials. However, while these tools leverage AI to some extent, they do not provide comprehensive assistance with trial protocol design itself. Their focus is narrower, often limited to improving recruitment and operational aspects after the protocol is finalized.

[0005] Traditional protocol repositories and analytical tools (e.g., clinicaltrials.gov) provide databases of registered clinical trial protocols. While some analytical tools provide insights based on this data, they do not guide the protocol design process. Furthermore, these repositories are static and primarily for reference purposes, and do not offer interactive tools for creating or modifying protocols based on predictive modeling. [Means for solving the problem]

[0006] Disclosed embodiments provide advanced tools for clinical trial design and management, including techniques that leverage generative artificial intelligence (AI) to analyze past clinical trial protocols and related data. This technology assists in the creation of efficient and effective clinical trial protocols by suggesting optimized procedures, estimating key trial metrics, and improving the decision-making process in clinical trial management.

[0007] The disclosed embodiments are relevant to medical informatics in that they are useful beyond clinical trial applications in other areas of medical data management, such as patient treatment planning, hospital resource allocation, and public health research.

[0008] Disclosed embodiments relate to educational tools. In academic and training settings, this technology can be used to simulate clinical trial design scenarios, helping students and new researchers understand the complexities of trial design and how various factors affect trial outcomes.

[0009] In disclosed embodiments, a model (e.g., a generative artificial intelligence (AI) model) specific to a clinical research indicator is developed and fine-tuned to enhance its functionality as an expert system for clinical trial protocol generation. This involves training the model to understand and interpret nuances in clinical data and trial results, giving it the ability to generate and optimize trial protocols based on learned patterns and predictive analytics. The fine-tuning process ensures that the model provides accurate, context-sensitive recommendations that reflect the complexity and specific requirements of the clinical trial.

[0010] The disclosed embodiments provide an AI-based agent that serves as the primary interface between the end user and the system. The agent is configured to route user queries to the appropriate system and perform processing and response. While the agent's primary interaction occurs with the model for immediate adjustments to the trial protocol based on user input, the agent is also configured to interface with other clinical trial data systems. For example, if a user proposes new inclusion criteria, this information is processed by the model and the clinical trial protocol is updated. The same information is communicated to a cost estimation service, and an updated budget analysis is displayed to the user. This multifaceted approach enables the agent to provide comprehensive services, improving user experience and operational efficiency.

[0011] In one aspect, the disclosed embodiments provide a method, system, and computer-readable medium for artificial intelligence-driven clinical trial protocol data acquisition and augmentation. The method includes receiving a natural language user request via a user interface. The method further includes combining the user request with contextual data to generate a prompt for a large-scale language model. The method further includes inputting the prompt into the model to generate a model response including a database query written in a database query language and metadata. The method further includes acquiring clinical trial protocol data from at least a first database based, at least in part, on the database query. The method further includes generating one or more application programming interface (API) requests based on some or all of the acquired protocol data and metadata. The method further includes executing one or more API calls using the generated API requests to acquire one or more clinical trial metrics. The method further includes generating a response to the user request based on some or all of the acquired protocol data and metrics. The method further includes outputting the response to the user interface.

[0012] Implementations can include one or more of the following features, either alone or in combination.

[0013] When combining the user request with the context data, the context data can include a schema of the first database. The context data can include domain knowledge definitions, including one or more of the following: an electronic data capture query definition and a patient burden index definition. The context data can include an output format definition. The context data can include persona instructions for the model.

[0014] When inputting prompts to the model, the database query language may be SQL. The first database may store historical clinical trial protocol data, and when retrieving the clinical trial protocol data, the first database may be accessed via an API call to a publicly available uniform resource locator (URL). Retrieving the clinical trial protocol data further includes retrieving additional clinical trial protocol data from a second database and correlating the clinical trial protocol data retrieved from the first database with the clinical trial protocol data retrieved from the second database using corresponding National Clinical Trial (NCT) numbers. When executing the one or more API calls using the generated API request, the one or more API calls are made to one or more of a screening failure prediction tool, a budget calculation tool, and a patient burden index calculation tool.

[0015] The method can include parsing the model response to extract a database query, executing the database query against at least a first database, and replacing at least a portion of the metadata with the retrieved protocol data to generate an extended model response. The method can include parsing the extended model response to extract metadata and comparing variables of the extracted metadata with variables of the API.

[0016] Inputting prompts to the model to generate a model response including a database query and metadata can use a zero-shot learning training process. The method can further include iteratively refining the prompts to improve model performance. Inputting prompts to the model to generate a model response including a database query and metadata can use a first-shot learning training process. The method can further include inputting a small number of manually labeled examples to train the model. The method can further include fine-tuning the model using a continuously updated curated dataset. The method can further include, after output, receiving a user rating of the response via a user interface and training the model based at least in part on the user rating. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 illustrates an artificial intelligence driven clinical trial protocol data acquisition and augmentation system in accordance with disclosed embodiments. [Figure 2] FIG. 1 illustrates the extraction and correlation of clinical trial data between a first publicly available database and a second privately available database. [Figure 3A] FIG. 1 is a diagram showing a response obtained when a system prompt is generated from a user request and input into a large-scale language model (LLM). [Figure 3B] FIG. 1 is a diagram showing a response obtained when a system prompt is generated from a user request and input into a large-scale language model (LLM). [Figure 4] FIG. 1 illustrates a data model schema and corresponding database tables. [Figure 5] FIG. 1 illustrates a verification and training system in which pairs of user requests and database queries verified by users are stored in a database and used for training. [Figure 6]FIG. 1 illustrates an example method for artificial intelligence driven clinical trial protocol data acquisition and augmentation, according to disclosed embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.

[0019] Disclosed embodiments provide an AI-driven protocol assistant that leverages natural language processing and machine learning (i.e., artificial intelligence) to automate the generation and optimization of clinical trial protocols, significantly reducing manual work and errors. Disclosed features include real-time optimization capabilities that adapt protocols based on new data and evolving study goals. In embodiments, the system uses advanced data mining techniques to analyze historical clinical trial data, thereby improving the predictive accuracy of new protocols.

[0020] Disclosed embodiments utilize predictive analytics to estimate key trial metrics and enable proactive adjustments. For example, a strategic recommendation engine can provide targeted advice tailored to specific trial goals, such as increasing diversity or reducing costs. Additionally, disclosed embodiments provide a user-friendly interface, making advanced trial design accessible to a broader research workforce. These capabilities address inefficiencies and limitations in the current trial design process by enabling more informed, dynamic, and data-driven decision-making.

[0021] The disclosed systems and methods address several particular technical problems.

[0022] One technical problem that the disclosed embodiments solve is the lack of dynamic optimization. Conventional tools typically do not dynamically suggest protocol adjustments based on real-time data input or changing research environments. In contrast, the approach described herein provides continuous optimization suggestions as new data becomes available or as user-defined parameters are modified (e.g., to increase diversity or reduce cost).

[0023] Another technical problem solved by the disclosed embodiments is the underutilization of historical data in conventional approaches. Traditional clinical trial design methods may not fully utilize the vast amount of data available from past studies. In contrast, the disclosed approach leverages historical protocols and results to support better study design decisions and improve the predictive accuracy and relevance of protocols.

[0024] Another technical problem that the disclosed embodiments solve is the lack of predictive capabilities for trial metrics. Traditional tools typically do not predict trial outcomes or metrics such as patient burden index, query rates, or enrollment success rates. In the disclosed embodiments, an AI-driven assistant uses historical data and statistical models to predict these metrics, allowing researchers to proactively modify protocols to resolve potential issues before the trial begins.

[0025] Another technical problem that the disclosed embodiments solve is the limitations of decision support for improving user-defined priorities. Specifically, traditional approaches lack support tools that provide recommendations tailored to specific goals, such as improving participant diversity or effectively managing clinical trial costs. The disclosed embodiments use AI-driven tools to suggest specific trial steps that help achieve clinical trial site, country, and strategic goals.

[0026] Among other benefits, the disclosed embodiments improve the efficiency of clinical trial protocol generation by automating protocol generation and optimization. The features disclosed herein significantly reduce the time required to create and modify clinical trial documentation, allowing research teams to allocate more resources to other important aspects of trial preparation and execution. Furthermore, the real-time optimization capabilities disclosed herein enable protocols to be rapidly adjusted in response to new data or changing trial goals. This capability allows clinical trial designs to rapidly adapt to new information, significantly accelerating the iterative process of trial design, which traditionally requires manual modifications.

[0027] Embodiments of the present disclosure essentially provide a strategic recommendation engine, offering targeted suggestions for improving specific aspects of clinical trials (e.g., increasing participant diversity or reducing operational costs). These suggestions are based on deep data analysis and predictive modeling, providing more advanced and nuanced insights than traditional tools. Furthermore, by integrating predictive analytics, the disclosed functionality not only predicts clinical trial outcomes but also provides indicators that foresee potential challenges in patient recruitment, retention, and other key areas. This proactive approach improves trial planning and management, minimizing risk and improving overall trial success.

[0028] The disclosed embodiments achieve an additional benefit by leveraging machine learning to analyze historical data, ensuring that the protocols generated are not only based on current best practices, but also reflect lessons learned from past successes and failures. This historical insight improves the accuracy of protocol recommendations and reduces costly errors and oversights.

[0029] FIG. 1 illustrates an artificial intelligence (AI)-driven clinical trial protocol data acquisition and enhancement system according to a disclosed embodiment. Clinical trial protocol data (e.g., historical clinical trial protocol data) is used as the basis for generating clinical trial protocols (or "study protocols") for future clinical trials. A study protocol is a text document that includes information about the overall study design, objectives, patient population, procedures, etc. For example, a study protocol may include the background and rationale, objectives (e.g., primary, secondary, exploratory), type (e.g., randomized, double-blind, placebo-controlled), number of subjects, duration and phase, inclusion criteria, assessments and procedures (e.g., details of medical tests and examinations, schedule of events and visits, etc.), subject treatment (e.g., investigational drug, dose, administration method), drug efficacy and safety monitoring procedures, adverse event and serious adverse event definitions and procedures, data management and statistical methods, quality control and quality assurance, ethical considerations, publication and data sharing policies, references to relevant scientific literature, and various forms (e.g., informed consent, questionnaires, surveys, etc.).

[0030] In the disclosed system, a user accesses a user interface 110 and enters a natural language user request to obtain clinical trial protocol data. For example, a user request might be, "Create a protocol for a Phase 2 oncology study." The user request can also be configured to obtain other types of information from a variety of external sources. For example, a user might request estimates for various clinical trial metrics, such as diversity score, patent burden index (PBI), study cost estimate (budget), etc.

[0031] In an embodiment, the user interface 110 can be implemented as an agent 120. The agent 120 is a software and / or hardware subsystem configured to engage in natural language communication with a user and coordinate the acquisition of clinical trial data according to the methods disclosed herein. As described in more detail below, the agent 120 is capable of intelligently coordinating the interaction between user input, the historical clinical trial data database 140, and external application programming interfaces (APIs). The primary role of the agent 120 is to ensure that the generated clinical trial protocols are comprehensive, accurate, and up-to-date, leveraging internal and external data sources as needed.

[0032] In a hardware-software-based system, an "agent" (e.g., agent 120) refers to a conceptual model that encapsulates specific functionality of the overall software system. Identifying and describing agents is a technique for abstracting and partitioning specific tasks or functions, making the system easier to understand and manage. However, it is important to note that this is merely a model or framework for understanding the system's behavior. In reality, the specific functionality assigned to an agent may be performed by other parts of the system. This situation arises from the interconnected nature of software systems, where different modules or components collaborate to perform tasks. Thus, while agents are useful for visualizing and describing the functionality of the disclosed system, actual execution may involve various pieces of software other than the "agent."

[0033] A user request is received by agent 120, as shown by the line labeled "1" in the diagram. As described in more detail below, agent 120 combines the user request with contextual data (e.g., see the discussion of Figures 3A-3B) to generate a prompt for artificial intelligence-based model 130 (e.g., a large-scale language model LLM). Agent 120 inputs this generated prompt into model 130 (line labeled "2" in the diagram), which generates a model response including a database query written in a database query language (e.g., Structured Query Language SQL) and associated metadata. Agent 120 uses the database query to retrieve clinical trial protocol data from historical database 140 (shown by line "3").

[0034] In the disclosed embodiment, the agent 120 is pre-configured with knowledge of all executable external APIs 160 (APIs to which API calls can be sent). Each API 160 is documented with its endpoints, required payload structure (e.g., variables to pass), and expected response format. Based on the database query and / or metadata generated by the model 130, the agent 120 generates API requests and executes the API calls using the generated API requests (line "4") to retrieve API data such as clinical trial metrics. In the embodiment, the API calls are made to one or more of the APIs 160, such as screening failure prediction, budget calculation, and patient burden index calculation.

[0035] In a disclosed software system, the term "external API" is typically used to enable communication or interaction with another software system or service. It provides a set of rules and protocols that dictate how software systems should interact with each other, allowing for the seamless exchange of data and functionality. In some cases, an external API may be part of another system that resides on the same server, a different server, or a cloud-based platform. It may be developed for internal use by the same organization or provided by a third-party software system. However, the term "external" does not necessarily mean that an external API is located remotely or part of a third-party system. It may simply refer to an API that exists outside of the primary software system in question, operates independently, and can be leveraged to enhance or extend the functionality of the core system. Even if an API is part of the same overall system or developed by the same organization, it may be called an "external API" if it is designed to operate independently or in conjunction with other systems.

[0036] Agent 120 generates a response to the user's request based on the obtained protocol data and metrics and outputs the response to user interface 110. The response may include a natural language response to the user (denoted by line "5") and API data (line "6") as a result of the API call. In an embodiment, these outputs may be combined into a single response to the user (e.g., a single natural language response).

[0037] In the illustrated example, the response reads, "We have created a protocol based on the phases and indications you provided." The created protocol may be in text format following the response, or in a separate file in document or spreadsheet format, or both. The file may be sent to the user or made available for download. In some cases, the response may include a link to access the created protocol.

[0038] Figure 2 illustrates clinical trial data extraction and correlation between a first public database and a second proprietary database. In an embodiment, the first database stores historical clinical trial protocol data and is accessible via API calls to publicly available unique resource locators (URLs). The primary source of historical data is clinicaltrials.gov, a public website that maintains an extensive record of clinical trial protocols.

[0039] As described above, agent 120 uses the generated database query to retrieve clinical trial protocol data from database 140 (see FIG. 1 ). In some cases, if a specific NCT number is specified in the user's request, the agent may use a query that includes the specific NCT number. In such cases, the system generates API request 210, which is used to make an API call to API 220 of public database 230 published on clinicaltrials.gov. API 220 allows for the retrieval and expansion of structured data based on various parameters, such as study phase, condition, intervention, and outcome. This is shown in the illustrated example database response 240.

[0040] In some embodiments, proprietary data is integrated into this extraction process to enrich the dataset. In such cases, the system may access a proprietary database 250, such as clinical trial data compiled over many years by a company involved in establishing and managing clinical trials (e.g., the applicant, Medidata Solutions, Inc.). Queries for such access may use all or part of the response 240 from the first public database 230. In some cases, only the NCT is used as a query parameter. Results 260 obtained from the second proprietary database 250 may include data not stored in the public database 230 (e.g., trial budgets and patient burden indexes).

[0041] The information in both databases can be correlated, for example, based on NCT number, to generate a more complete response. In embodiments, the correlated response formed from the query of the first public database 230 and the second proprietary database 250 can be combined with additional data (such as the clinical trial metrics described above) obtained via API calls performed by the agent 120.

[0042] In an embodiment, clinical trial protocol database 270 (such as database 140 shown in FIG. 1) is built based on linked data from public database 230 and proprietary database 250, as well as data from other sources. Historical database 270 is built as a comprehensive database to house a vast collection of digitized historical clinical trial protocols and facilitate data harmonization and ingestion into systems designed to support future applications. Database 270 can also integrate various operational metrics (e.g., query volume, enrollment rate, and other relevant performance indicators) obtained from Medidata's extensive proprietary datasets. This foundational database 270 provides the raw material needed to train and optimize AI models, enabling them to learn from accurate and rich historical data.

[0043] Figures 3A and 3B show the process of generating system prompts from user requests, the responses obtained when the system prompts are input into a model (e.g., a large-scale language model (LLM)), and extending the model responses using API calls.

[0044] System prompts provide detailed input instructions for the LLM to read, interpret, and generate the desired output. As described above, to generate prompts, agent 120 combines a user request (e.g., a natural language request) with context data to input into model 130 (see FIG. 1). This context data may include various components that assist model 130 in processing the user request. For example, a schema for a database (e.g., database 140). The schema provides a structured framework or blueprint that describes how data in database 140 is organized and accessed, and assists model 130 in constructing database queries (e.g., SQL queries).

[0045] Context data may also include domain knowledge definitions. These definitions may include various elements, such as EDC query definitions that specify how to query data stored in an EDC system (e.g., MedidataRave® EDC). They may also include patient burden index (PBI) definitions that explain how to interpret PBI data.

[0046] Another possible component of context data is an output format definition, which specifies the desired structure and format of the processed data. For example, a valid SQL dialect (e.g., ANSI SQL) might be specified to be compatible with a particular type of database software. The output format definition might also define any relevant metadata to be included in the response from the model (e.g., inclusion criteria or trial phase provided).

[0047] Context data may also include persona instructions for the model. These instructions establish the role the model should play as part of this solution, such as the tone and perspective it should take when formulating a response to a user request. For example, a model might adopt the persona of a clinical assistant who assists non-technical users (e.g., physicians) in creating study protocols.

[0048] In the example of Figure 3A, the user request is "Please provide a list of inclusion criteria for the Phase 2 study." In response, a system prompt is generated that includes a persona instruction ("You are a clinical assistant..."), an output format definition ("SQL query in ANSI SQL dialect"), a database schema ("db_schema"), and a metadata section ("...extract the following metadata from the user request..."), which includes tags and instructions to the model. The system prompt is input to the model, and a model response is generated. In the example, the model response includes the database query ("SELECT inclusion_criterion FROM eligibility...") and metadata.

[0049] In the example of Figure 3B, the user request is "Please provide a list of common activities in an oncology study over a 12-month period." This generates a system prompt, which includes a persona instruction ("You are a clinical assistant..."), an output format definition ("SQL query in ANSI SQL dialect"), a database schema ("db_schema"), and a metadata section ("...extract the following metadata from the user request..."), which includes tags and instructions to the model. The system prompt is input to the model, which generates a model response. The model response, in an example, includes the database query ("SELECT activities_name FROM activities...") and metadata.

[0050] The model response is used to retrieve clinical trial protocol data from database 140 (see FIG. 1) based at least in part on the database query. In the example of FIG. 3A, the response from model 130 (in this case, the LLM response) includes an SQL query and metadata sections that correspond to the metadata sections of the user request. Specific tags specified by the user in the user request are "marked" by inserting a message between the relevant tags. In this example, these are inclusion_criteria and exclusion_criteria. A "NULL" is inserted between metadata tags for which no metadata was retrieved from the user request. Examples include activities, condition, and study_duration. Additionally, a value of "2" is inserted between the trial_phase tag based on the user request.

[0051] In the example shown in Figure 3B, the model response contains an SQL query and metadata sections corresponding to the metadata sections in the user request. Specific tags requested by the user in the user request are "marked" by inserting a message between the relevant tags (in this example, the activities tags). The indicator "NULL" is inserted between metadata tags for which no metadata was retrieved from the user request, such as inclusion_criteria, exclusion_criteria, and trial_phase. Based on the user request, the value "12" is inserted between the study_duration tags and the term "oncology" is inserted between the condition tags.

[0052] In some embodiments, retrieving clinical trial protocol data from a database may include parsing the model responses to extract database queries and executing the database queries against the database, and may also replace at least a portion of the metadata with the retrieved protocol data to generate an augmented model response.

[0053] In the example in Figure 3A, the marker "[Requested by user]" between the inclusion_criteria tags is replaced with "Age is 18 years old." Also, the placeholder "NULL" between the condition tags is replaced with "oncology." In the example in Figure 3B, the marker "[Requested by user]" between the activities tags is replaced with "Blood draw / X-ray / DNA test."

[0054] The system generates one or more application programming interface (API) requests based on the retrieved protocol data and / or metadata. In an example, this includes parsing the enhanced model response to extract metadata and comparing variables in the extracted metadata with variables in the API. This may be followed by using the generated API requests to execute one or more API calls to retrieve clinical trial metrics, such as screening failure predictions, patient burden indexes, estimated costs, etc.

[0055] In the example of Figure 3A, if metadata is present between the condition tag and the trial_phase tag, it is sufficient to call the Cost Estimation API. In other cases, invoking the Cost Estimation API requires including metadata between the study_duration tags, in which case this API is not called in the example shown (although other APIs may be called). In the example of Figure 3B, if metadata is present between the activities tags, it may be sufficient to call the Patient Burden Index API.

[0056] Figure 4 illustrates a data model schema and corresponding database tables. In an embodiment, the database schema includes tables for storing protocol information such as study phases, conditions, interventions, outcomes, demographic data, and other relevant attributes. This schema may be designed based on the CDISC Unified Study Design Model (USDM) standard, which ensures consistency and compatibility with industry standards. Additionally, as discussed above, the data model may be extended with proprietary extensions to fit specific requirements.

[0057] In an embodiment, the database is normalized to minimize redundancy and ensure data integrity. This process reduces duplication and dependencies by organizing data into tables and defining relationships between tables. To optimize query performance, indexes are implemented to enable fast retrieval of data during the protocol generation process. Indexes are created for key attributes such as protocol identifiers (e.g., NCT numbers), study phases, conditions, and interventions. This ensures efficient access to the most frequently queried data. Additionally, query optimization techniques are applied to further improve performance, enabling the system to handle large volumes of data processing with minimal latency.

[0058] Figure 5 shows a diagram of a validation and training system in which user-validated user request / database query pairs are stored in a database and used for training. Such training is performed during system use to improve the system's accuracy. In addition to training performed before use, the output data undergoes a human review process in which domain experts check the protocols generated based on the system's responses against established standards and requirements and provide feedback on the accuracy and relevance of the content.

[0059] The system receives user requests 510, which are natural language statements providing queries and / or instructions for providing results to the system, as described in the description of Figure 1. Query generation 520 of the LLM model receives the user request and processes it to generate a database query (e.g., in the form of SQL code). The database query is received by database query execution 530, which executes the SQL code against database 540 (i.e., retrieves data from the database based on the database query). LLM model 520 returns a response. This response is sent to response output 550, such as a user interface, for output to the user.

[0060] Once the system is operational, end users have the ability to rate the responses and outputs. Positive user ratings indicate high accuracy and relevance, while areas for improvement are identified through low ratings. As explained in more detail below, positively rated examples are stored in a training database and used to continuously improve the system's performance. This iterative feedback loop allows the model to evolve and improve over time, maintaining high accuracy and reliability in capturing data that is used to generate clinical trial protocols.

[0061] Responses from the LLM model 520 are sent to user validation 560 to generate training data. This can be done, for example, through user input at a user interface. The user reviews each response and rates its accuracy, for example, by providing a positive or negative indication or score. The responses are sent to training data generation 570, and responses that receive a positive indication from the user are paired with a corresponding user request and stored in training database 580. The user request / response pairs (e.g., user request / SQL code pairs) can be used for future model training. Thus, in an embodiment, after the responses are output, user ratings of the responses can be received via a user interface, and the model can be trained based at least in part on the user ratings.

[0062] To train the model 520, multiple approaches can be employed in various combinations.

[0063] In zero-shot learning, model 520 leverages pre-trained knowledge to generate output without the need for specific examples or a clear definition of "good results." System prompts can be iteratively refined to provide clearer or more comprehensive information to improve model performance. For example, additional scheme or formatting information can be included as context information for the system prompt. Thus, in an embodiment, the process of inputting a prompt (e.g., system prompt 320 in FIG. 3A) into a model to generate a model response, including a database query (and metadata), can use a zero-shot learning training process, iteratively refining the prompt to improve model performance.

[0064] In short-shot learning, the model 520 is adapted to a particular task and its performance improved by training it using a small number of examples. Such examples can be obtained from a training database 580 and / or pre-defined examples. Thus, in an embodiment, the process of inputting prompts to the model to generate model responses including database queries (and metadata) can use a short-shot learning training process, which may include inputting a small number of manually labeled examples to train the model.

[0065] In fine-tuning training, the model 520 is trained on a user-validated, curated dataset (e.g., the training database 580 described above). The fine-tuning process allows the model to refine its understanding and produce more accurate and relevant outputs based on the evolving dataset. The frequency of fine-tuning an LLM depends on the availability of curated datasets.

[0066] FIG. 6 illustrates a method 600 for artificial intelligence-driven clinical trial protocol data acquisition and augmentation, according to a disclosed embodiment. The method 600 includes receiving (610) a natural language user request via a user interface and combining the user request with contextual data to generate (620) a prompt for a large-scale language model. The method further includes inputting the prompt into a model to generate (630) a model response including a database query in a database query language and metadata, and retrieving (640) clinical trial protocol data from at least a first database based, at least in part, on the database query. The method further includes generating (650) one or more application programming interface (API) requests based, in part or in whole, on the retrieved protocol data and metadata. The method further includes executing (660) one or more API calls using the generated API requests to retrieve one or more clinical trial metrics. The method further includes generating (670) a response to the user request based, at least in part, on the retrieved protocol data and metrics, and outputting (680) the response to the user interface.

[0067] Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Likewise, aspects of the present invention may be embodied in hardware, software, or a combination of both. Aspects of the present invention may be embodied as a computer program product recorded on a computer-readable medium having computer-readable program code recorded thereon.

[0068] The computer-readable medium may be a computer-readable storage medium, which may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0069] The computer program code in embodiments of the present invention can be written in any suitable programming language and / or scripting language. The program code can be executed on a single computer or on multiple computers. The computer can include a processing unit in communication with a computer-usable medium, the computer-usable medium including a set of instructions, and the processing unit can include a machine learning algorithm designed and / or trained to execute the set of instructions.

[0070] The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. receiving a user request in natural language via a user interface; a prompt generation step that combines the user request with the context data to generate a prompt for the large-scale language model; a prompt input step of inputting the prompt into the large-scale language model to generate a model response including a database query written in a database query language and metadata; obtaining clinical trial protocol data from a first database based at least in part on the database query; a request generation step of generating one or more API (Application Programming Interface) requests based on some or all of the acquired clinical trial protocol data and metadata; executing one or more API calls using the generated API request to obtain one or more clinical trial metrics; generating a response to the user request based at least in part on the acquired protocol data and the clinical trial metrics; a response output step of outputting the response to the user interface; Including, A method for acquiring and augmenting clinical trial protocol data using artificial intelligence.

2. The method of claim 1 , wherein in the prompt generating step, the context data includes a schema of a first database.

3. The method of claim 1 , wherein in the prompt generating step, the context data includes a domain knowledge definition, including at least one of an electronic data collection query definition and a patient burden index definition.

4. The method of claim 1 , wherein in the prompt generating step, the context data includes an output format definition.

5. The method of claim 1 , wherein in the prompt generating step, the contextual data includes a persona designation for the model.

6. The method of claim 1 , wherein in the step of inputting the prompt into the model, the database query language is SQL.

7. 2. The method of claim 1, wherein the first database stores past clinical trial protocol data, and in the step of obtaining the clinical trial protocol data, the first database is accessed via an API call to a publicly available uniform resource locator (URL).

8. The protocol data acquisition step includes: obtaining additional clinical trial protocol data from a second proprietary database; linking the clinical trial protocol data obtained from the first database with the clinical trial protocol data obtained from the second database using corresponding National Clinical Trial (NCT) numbers; 8. The method of claim 7, comprising:

9. The method of claim 1 , wherein in the executing step, the API call is made to at least one of a screening failure prediction tool, a budget calculation tool, and a patient burden index calculation tool.

10. an extraction step of analyzing the model response to extract the database query; the protocol data obtaining step includes executing a database query on the first database; replacing at least a portion of the metadata with the retrieved protocol data to generate an extended model response; The method of claim 1 further comprising:

11. analyzing the augmented model response to extract the metadata; In the request generation step, a step of comparing the extracted metadata variables with API variables; The method of claim 10 further comprising:

12. 10. The method of claim 1, wherein the prompting step uses a zero-shot learning training process.

13. The method of claim 12 , further comprising iteratively refining the prompts to improve performance of the model.

14. 10. The method of claim 1, wherein the prompting step uses a few-shot learning training process.

15. The method of claim 14 , further comprising inputting a small number of manually labeled examples to train the model.

16. The method of claim 1 , further comprising fine-tuning the model using a continuously updated curated dataset.

17. after the outputting step, receiving a user rating of the response via the user interface; and training the model based at least in part on the user ratings.

18. receiving a user request in natural language via a user interface; a prompt generation step that combines the user request with the context data to generate a prompt for the large-scale language model; an input step of inputting the prompt into the large-scale language model to generate a model response including a database query written in a database query language and metadata; obtaining clinical trial protocol data from at least one first database based at least in part on the database query; a request generation step of generating one or more API (Application Programming Interface) requests based on some or all of the acquired clinical trial protocol data and metadata; executing one or more API calls using the generated API request to obtain one or more clinical trial metrics; generating a response to the user request based at least in part on the acquired protocol data and the clinical trial metrics; a response output step of outputting the response to the user interface; 1. An artificial intelligence driven clinical trial protocol data acquisition augmentation system comprising a computer having one or more processors in communication with a memory storing instructions for executing the steps of:

19. 1. A non-transitory computer-readable medium storing instructions that are executable by one or more processors of a computer to cause the one or more processors to perform an artificial intelligence (AI)-driven clinical trial protocol data acquisition and augmentation method, the instructions comprising: The method comprises: a prompt generation step that combines the user request with the context data to generate a prompt for the large-scale language model; an input step of inputting the prompt into the large-scale language model to generate a model response including a database query written in a database query language and metadata; obtaining clinical trial protocol data from at least one first database based at least in part on the database query; a request generation step of generating one or more API (Application Programming Interface) requests based on some or all of the acquired clinical trial protocol data and metadata; executing one or more API calls using the generated API request to obtain one or more clinical trial metrics; generating a response to the user request based at least in part on the acquired protocol data and the clinical trial metrics; a response output step of outputting the response to the user interface; 1. A computer-readable medium comprising:

Images