Ensemble ai systems and methods for automated real-time clinical evidence synthesis, generation, analysis, and discovery

Abstract

A method for automated real-time processing of clinical evidence, implemented by a computing system includes receiving a user input that contains a clinical question; generating parameters for a Population, Intervention, Comparison, Outcome (PICO) table using a large language model (LLM) relevant to the clinical question; retrieving clinical research studies from multiple databases based on the parameters as approved by a user electronic device; screening these studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolving discrepancies between the Al systems via a researcher audit interface; and generating a list of clinical research studies based on the screenings.

Description

ENSEM BLE Al SYSTEMS AN D METHODS FOR AUTOMATED REAL-TI MECLIN ICAL EVIDENCE SYNTHESIS, GEN ERATION, ANALYSIS, ANDDISCOVERYFIELD OF THE INVENTION

[0001] The field of the invention is healthcare technology and, in particular, a system and method for enhancing clinical evidence synthesis, generation, analysis and discovery.BACKGROUND OF THE INVENTION

[0002] The following description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.

[0003] In clinical research, addressing specific clinical questions often necessitates the use of large datasets. Traditionally, these datasets are reviewed through manual methods that are both time-consuming and prone to human error. Automated systems have been introduced to streamline this process. However, these systems often lack robustness and can fail to integrate complex data points effectively, leading to incomplete or biased analyses.

[0004] Furthermore, new studies and data are continuously published. This poses a challenge for existing systems to stay current without constant manual intervention. As a result, evidence synthesis projects may quickly become outdated, leading to a noticeable degree of research waste. Existing systems are also limited in their ability to assess the quality of research studies, often overlooking critical factors such as fraud, validity of results, and publication bias, which can significantly impact clinical decisions and patient outcomes.

[0005] Thus, there is still a need for a system that allows for real-time, and improved processing of clinical research across diverse medical databases and to mitigate some of the errors inherent in current clinical evidence processing systems. Methods and systems that provide improved reliability, adaptability, and comprehensiveness in analyzing and processing clinical data are needed.BRIEF DESCRIPTION OF DRAWINGS

[0006] Various objects, features, aspects, and advantages of the inventive subject matter will become more apparent from the following detailed description of preferred embodiments, along with the accompanying drawing figures in which like numerals represent like components.

[0007] FIG. 1 is a block diagram of a system architecture for a clinical evidence system in accordance with an example of the present specification.

[0008] FIG. 2 is a flow diagram for an operational process of the clinical evidence system of FIG. 1.

[0009] FIG. 3 is a flow diagram for additional stages of the operational process of clinical evidence system of FIG. 2.

[0010] FIG. 4 to FIG. 11 are screenshots of a client application in accordance with an example of the present specification.DETAILED DESCRIPTION OF THE INVENTION

[0011] This detailed description provides an explanation of the embodiments of the present specification. The present specification encompasses a variety of systems, methods, and non- transitory computer-readable media.

[0012] A method for automated real-time processing of clinical evidence, implemented by a computing system includes: receiving a user input that contains a clinical question; generating parameters for a Population, Intervention, Comparison, Outcome (PICO) table using a large language model (LLM) relevant to the clinical question; retrieving clinical research studies from multiple databases based on the parameters as approved by a user electronic device; screening these studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolving discrepancies between the Al systems via a researcher audit interface; and generating a list of clinical research studies based on the screening. Advantageously, this method and system enhances the review of clinical evidence using artificial intelligence techniques.

[0013] All publications herein are incorporated by reference to the same extent as if each individual publication or patent application were specifically and individually indicated to be incorporated by reference. Where a definition or use of a term in an incorporated reference is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.

[0014] In some embodiments, the numbers expressing quantities of features used to describe and claim certain embodiments of the invention are to be understood as being modified in some instances by the term "about." Accordingly, in some embodiments, the numerical parameters set forth in the written description and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed considering the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable. The numerical values presented in some embodiments of the invention may contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.

[0015] As used in the description herein and throughout the claims that follow, the meaning of "a," "an," and "the" includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise.

[0016] The recitation of ranges of values herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range. Unless otherwise indicated herein, each individual value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of examples, or exemplary language (e.g. "such as") provided with respect to certain embodiments herein is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.

[0017] Groupings of alternative elements or embodiments of the invention disclosed herein are not to be construed as limitations. Each group member can be referred to and claimed individually or in any combination with other members of the group or other elements found herein. One or more members of a group can be included in, or deleted from, a group for reasons of convenience and / or patentability. When any such inclusion or deletion occurs, the specification is herein deemed to contain the group as modified thus fulfilling the written description of all Markush groups used in the appended claims.

[0018] A computing system may include one or more processors, memory, and storage devices configured with software and / or firmware to implement the specified functionalities. The hardware is designed to support the computational demands of ensemble Al methods including but not limited to Bayesian analysis, decision tree algorithms, and machine learning techniques.

[0019] The system may process clinical data through a series of computational steps, including data collection, cleaning, normalization, and analysis. Machine learning algorithms implemented on the system can analyze large datasets to identify patterns, make predictions, or generate recommendations based on the trained models.

[0020] Training of machine learning models involves feeding large datasets into the system, where the data is used to gradually adjust the model's parameters until it achieves the desired level of accuracy or performance. This training process can be performed using a variety of algorithms, including supervised, unsupervised, or reinforcement learning techniques, depending on the nature of the problem being addressed. Fine-tuning these algorithms can also be used for adapting pre-trained models to specific tasks or datasets.

[0021] The system includes a user interface that allows users to interact with various functionalities, providing inputs, configuring settings, and receiving outputs. This interface can be web-based, mobile, or desktop applications, designed to facilitate user interaction with the underlying processes and to display the results of the analysis in an understandable and actionable manner.

[0022] The system can be implemented on cloud-based infrastructure, allowing scalable computing resources and storage capacity to accommodate the needs of large-scale applications. Networking technologies enable the system to access distributed data sources, integrate with other systems, and provide services to remote users over the Internet.

[0023] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0024] FIG. 1 depicts a system architecture diagram for system 100, according to an example of the present specification. The system 100 includes server 101, which hosts several functional modules 102. These modules include a clinical studies search engine 112, a data analyticsmodule 114, a data extraction module 116, a user interaction module 118, an evidence synthesis module 120, a bias detection and correction module 121, a validation and verification module 124, and an interoperability module 126. Each module plays a role in the processing and analysis of clinical data, as discussed in greater detail below.

[0025] Server 101 also incorporates one or more persistent data storage 104, shown as full text data storage 104-1, extracted data storage (unstructured) 104-2, and extracted data storage (structured) 104-3, which respectively manage full-text documents, unstructured extracted data, and structured extracted data.

[0026] Connected via network 106, server 101 interacts with multiple Large Language Model Systems (LLMs), designated as 110-1, 110-2, and 110-3. Network 106 facilitates the transfer and handling of data, supporting the system's capability to manage and transfer clinical information.

[0027] Still with reference to FIG. 1, the clinical studies search engine module 112 of system 100 is designed to automate and optimize the retrieval of clinical research studies from multiple medical and scientific databases. According to one example, the databases under search include the OpenAlex™ database, which includes over 250 million peer-reviewed publications.

[0028] The data analytics module 114 processes and analyzes the extracted data from clinical studies to identify keywords and concepts.

[0029] The data extraction module 116 automates the retrieval of specific data points from both structured and unstructured formats within the stored clinical studies.

[0030] The user interaction module 118 serves as the interface between the system 100 and the user electronic devices 108 shown in FIG. 1. Module 118 collects user inputs, displays data outputs, and provides tools for user interaction with the system 100, such as querying, reviewing results, and customizing data views. This module enhances user experience and engagement, making the system accessible and adaptable to various user needs.

[0031] The evidence synthesis module 120 integrates and compiles data from multiple sources to generate comprehensive reports and insights. It synthesizes findings from individual studies into coherent summaries, supporting systematic reviews and meta-analyses. This module translates complex data into actionable clinical evidence.

[0032] The bias detection and correction module 122 identifies and mitigates potential biases within the data or the analysis processes. This includes biases from study design, publication bias, and data collection. By correcting these biases, the module ensures that the clinical evidence produced is improved in both accurate and reliable.

[0033] The validation and verification module 124 ensures the accuracy and reliability of the outputs of the system 100. Module 124 conducts checks and balances, such as cross-verifying extracted data with original sources and running consistency tests across different data sets. This module is used for maintaining the integrity of the clinical evidence processing system.

[0034] The interoperability module 126 facilitates the ability of the system 100 to communicate and function with other IT systems, databases, and software. It ensures that data formats, protocols, and interfaces are standardized or adaptable, allowing data exchange and integration across diverse health information systems.

[0035] FIG. 2 is a flow diagram that illustrates a method of clinical evidence processing in accordance with an example of the present specification. The process begins at 200. At 202, the user interaction module 118 receives an input comprising a clinical question from the user electronic device 108. At 204, the clinical studies search engine 110 generates parameters for a PICO (Population, Intervention, Comparison, Outcome) table relevant to the clinical question. At 206, the user interaction module 118 presents these parameters to the user for confirmation or changes. At 210, once confirmed, the server 101 generates selection criteria for the search. At 212, the system 100 searches a data repository for abstract clinical research studies that match the selection criteria. This may involve accessing the full text data storage 104-1 or external databases. At 214, a first level screening of the identified research studies is performed to assess their relevance. At 216, the accuracy of the first level screening results is verified. At 218, a second level screening of full-text clinical research studies is performed. At 220, a further validation process is conducted. At 222, a list of the clinical research studies responsive to the clinical question is stored in the extracted data storage (structured) 104-3. The process concludes at 226.

[0036] The first and second level screenings mentioned in steps 214 and 218 are conducted using an ensemble Al system, designed to enhance the precision of the screening process. The word 'ensemble' refers to the use of multiple independent Al entities, shown as LLMs 110 in FIG. 1, to perform parallel assessments of the research studies' titles and abstracts at Level 1, and full texts at Level 2. According to one example, one of the LLMs 110 can be fine-tuned for specific tasks while another uses reinforcement learning techniques, allowing them to learn from their distinct mistakes and avoid homogeneity in error generation. This dual Al approach mirrors a human screening processes but with enhanced efficiency and the ability to handle vast datasets. Discrepancies between the two Ais' conclusions can be audited by researchers, or afurther Al entity, providing an additional layer of validation and ensuring that only the most relevant and accurate studies are included in the list. This process results in a comprehensive, transparent, and explainable list of clinical research studies, which adheres to the principles of open and accessible scientific inquiry by including all relevant studies published globally, irrespective of language. This methodology leverages the capabilities of advanced LLMs to refine and enhance the search and validation stages of clinical evidence synthesis.

[0037] FIG. 3 is a flow diagram that illustrates extended clinical evidence processing, in accordance with an example of the present specification. FIG. 3 introduces advanced processing and analysis features to further enhance decision-making and data integrity. The initial steps 200 through 222 in FIG. 3 mirror those in FIG. 2, beginning with receiving a clinical question and concluding with storing a list of responsive clinical research studies. At 300, the process extends by enhancing decision-making using ensemble or multiple Al systems to improve the accuracy and reliability of the system. At 302, the artificial intelligence processes are optimized through fine-tuning and reinforcement learning techniques. This stage ensures the Al systems remain effective over time. At 304, relevant data points are extracted from the identified research studies, a process facilitated by the data extraction module 116. At 306, the extracted data undergo an assessment for instances of fraud, outliers, misclassification, and publication bias. This step maintains the integrity of the clinical evidence and is conducted by the bias detection and correction module 122. At 310, 312, and 314, a feasibility assessment is conducted. This includes assessing clinical homogeneity 310, statistical homogeneity 312, and evaluating network connectivity 314. These assessments ensure that the studies are comparable, and the data are integrated. At 316, the process involves analyzing indirect drug study relationships to uncover hidden correlations and insights, which could be used for new clinical hypothesis generation. At 318, errors in the original manuscripts or the process itself are identified and addressed, helping to refine the overall quality and accuracy of the synthesized evidence. At 320, feedback is provided to the large language model systems 110 to support continuous learning and improvement. This feedback loop makes the Al components more responsive and accurate in their outputs. The process concludes at 322, having incorporated a range of analytical techniques and Al-enhanced capabilities to provide a synthesis of clinical evidence.

[0038] FIG. 4 is a screenshot 400 of a user interface for use in accordance with the system of FIG. 1, in accordance with an example of the present specification. This user interface is designed to facilitate user interaction and manage various aspects of the automated real-timeclinical evidence synthesis process. Along the left side pane of the dashboard, several navigational buttons are presented: 'Dashboard' 410, 'Al Chat' 412, 'Librarian Validation' 414, 'Reviewer Validation' 416, 'Research' 418, and 'Users' 420. In this instance, the 'Al Chat' button 412 is highlighted as active, indicating that the Al Chat feature is currently being displayed on the main screen. On the main screen, a dialogue bubble 402 displays a system output requesting the user to provide a clinical question. This feature initiates the data processing workflow. Below the dialogue bubble, a text entry box 404 is shown where the user inputs the clinical question using their electronic device. This input acts as the trigger for the system to begin its automated functions, starting with the generation of a PICO table relevant to the inputted clinical question. Once the PICO table is generated, it is displayed within another dialogue bubble 406. This visualization allows the user to review the generated parameters and ensure they align with the specifics of the clinical query they are investigating. Next, to finalize the acceptance of the PICO table parameters, an 'Approve' button 408 is positioned near the PICO table display. This button allows the user to confirm the accuracy and relevance of the PICO table, thereby advancing the process to the subsequent stages of data retrieval and analysis.

[0039] FIG. 5 illustrates a subsequent stage of the user interface dashboard from FIG. 4, following the approval of the PICO table parameters. This screenshot 500 captures the continued interaction process between the user and the system as it advances through the clinical data processing workflow. After the 'Approve' button 408 is selected confirming the PICO table, the dashboard updates to display a new system output at 502. This output requests further details from the user, specifically asking for information about the inclusion and exclusion criteria that should apply to the clinical studies search. These criteria are used for refining the search results to ensure that they are relevant and tailored to the specific clinical question. Below the system output, a text entry box 504 is presented where the user can input the requested details about the inclusion and exclusion criteria.

[0040] FIG. 6 illustrates a subsequent stage of the user interface dashboard from FIG. 5, following the user's entry of inclusion and exclusion criteria as shown in FIG. 5. This screenshot 600 captures how the system organizes and displays the specified criteria to the user for review and confirmation. In this view, the inclusion and exclusion criteria are presented in a structured tabular format, which aids in clarity and ease of verification. There are two main tables presented: 'Inclusion Criteria' 602 and 'Exclusion Criteria' 604. Each table lists the criteria for review by the user, categorizing what attributes or conditions should be considered ordisregarded in the retrieval and analysis of clinical studies. The inclusion criteria 602 outline the parameters that must be met for studies to be considered relevant. These might include specific population characteristics, intervention types, outcome measures, and other relevant factors that align with the clinical question and research objectives. Conversely, the exclusion criteria 604 detail the parameters that disqualify studies from being included in the analysis. These could encompass certain study designs, date ranges, geographic locations, and other factors that are deemed irrelevant or that could bias the research outcomes.

[0041] FIG. 7 illustrates a subsequent stage of the user interface dashboard from FIG. 6, continuing from the confirmation of inclusion and exclusion criteria as shown in FIG. 6. This screenshot 700 demonstrates the interactive process as the system prepares to execute the search based on the approved criteria. After the user confirms the inclusion and exclusion criteria by clicking an 'Approve' button 702, the system generates and displays a proposed search strategy in dialogue box 704. This search strategy outlines the approach the system will take to locate and retrieve relevant clinical studies, incorporating the detailed criteria previously set by the user. Additionally, the dashboard presents two actionable buttons that offer different paths for proceeding with the search strategy: 'Continue without validation' 706 and 'Validate with expert' 708. The 'Continue without validation' button 706 allows the user to proceed directly with the search without further review, suitable for scenarios where time constraints are a priority or when the user is confident in the accuracy of the input criteria and the systemgenerated search strategy. The 'Validate with expert' button 708, conversely, provides an option to have the search strategy reviewed by a domain expert. This validation step is crucial for ensuring the strategy's comprehensiveness and alignment with best research practices, potentially increasing the reliability of the search outcomes by incorporating expert feedback.

[0042] FIG. 8 displays a specific interface from the dashboard associated with the librarian validation phase, continuing the flow from the option to 'Validate with expert' as depicted in FIG. 7. This screenshot 800 is designed to facilitate expert review and validation of the search strategy and associated data, ensuring a rigorous search methodology. Screenshot 800 is divided into two main columns: the first column 802 presents a detailed listing of various attributes and values. These attributes include study characteristics such as study type, date, lead sponsor, sponsor type, and details regarding the study's design like allocation, assignment, primary purpose, masking details, and observational design elements. Each row in this column provides specific data points for an expert reviewer to assess the relevance in relation to the searchstrategy. The second column 804, adjacent to the first, prompts for a "supporting quote". This column includes a text field for each row where the expert can enter citations or comments that justify the inclusion or exclusion of the study based on the data presented in the first column. This feature allows the validating expert to provide direct feedback and supporting evidence, to substantiate the decision-making process and ensure the accuracy of the search strategy.

[0043] FIG. 9 shows the Level 1 validation screen of the dashboard, part of the validation process described earlier. Screenshot 900 is designed for the initial review of studies retrieved from the search strategy. Screenshot 900 consists of two main areas. Area 902 lists the articles that have been identified as potentially relevant, providing information such as the title, abstract and a validation checkmark. This list allows validators to quickly assess each article's relevance. Area 904 is an interface where validators can exclude studies from further consideration. It allows studies that do not meet the inclusion criteria to be selected and a reason for exclusion may be entered, viewed or edited. This feature ensures that decisions are documented, maintaining a record of the validation process.

[0044] FIG. 10 illustrates the Level 2 validation screen of the dashboard, which facilitates the full-text screening of studies. This screen is an extension of the validation process and builds on the preliminary assessments made in Level 1, as shown in FIG. 9. Similar to the previous validation screen, FIG. 10 is divided into two main areas. Area 1002 displays a list of articles that have passed the initial Level 1 screening and are now subject to more detailed full-text reviews. This list includes the same type of essential information as before— titles, abstract, and validation check— to aid validators in their in-depth assessment of each study's content and relevance. Area 1004 provides the interface for excluding studies based on the full-text review. Validators can select articles that, upon closer examination of the full text, fail to substantiate their initial relevance. The interface requires validators to specify reasons for each exclusion, for documentation purposes.

[0045] FIG. 11 displays a dashboard summarizing the "Methods" used in a systematic literature review process according to an example of the present specification, detailing how a study selection and search strategy were implemented. In the top left of the dashboard, box 1102 presents the clinical question for the literature review process. Below the clinical question, box 1104 provides an explanation of the systematic literature review process. Further down, box 1106 describes the screening process, which is divided into two stages. The first stage involves screening titles and abstracts (Level 1) to identify studies that potentially meet the inclusioncriteria. The second stage (Level 2) involves a more detailed review of the full texts of these selected studies. As noted above, during both stages, an Al system evaluates the records against the inclusion and exclusion criteria.. Following this Al assessment, a human expert reviews and validates the Al's decisions, ensuring accuracy and correcting any discrepancies (using a review interface not shown in the drawings). On the top right of the dashboard, box 1108 displays the inclusion and exclusion criteria organized into two separate tables. Below the criteria tables, box 1110 outlines the search strategy employed.

[0046] With the list of validated clinical research studies, the system 100 is equipped to conduct in-depth analysis and generate actionable clinical insights. By applying advanced algorithmic processes, the system can validate the integrity of the studies through various analytical techniques, including the use of Benford's law for detecting fraudulent research, Bayesian outlier analysis for identifying statistical anomalies, and assessments of publication bias. Furthermore, the system evaluates the clinical, metadata, and statistical homogeneity of the studies to ensure consistency and relevancy across the dataset. The inclusion of network connectivity assessments aids in understanding the interrelationships and dependencies within the data. Additionally, meta-regression adjustments can be employed to account for effects of identified heterogeneities on clinical outcomes. These processes not only refine the validity and reliability of the information but also enable the extraction of insights, contributing to advanced, evidence-based clinical insights and discoveries.

[0047] One general aspect includes a method including the steps of: receiving a user input that contains a clinical question; generating parameters for a Population, Intervention, Comparison, Outcome (PICO) table using a large language model (LLM) relevant to the clinical question; retrieving clinical research studies from multiple databases based on the parameters as approved by a user electronic device; screening these studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolving discrepancies between the Al systems via a researcher audit interface; and generating a list of clinical research studies based on the screenings.

[0048] Implementations may include one or more of the following features: retrieving full texts of clinical research studies for a second level of screening using the two Al systems; presenting the researcher audit interface designed for a medical librarian, wherein the researcher audit interface displays the search strategy and results, enabling the medical librarian to validate the accuracy and appropriateness of the search strategy and results, and wherein the validation bythe medical librarian informs reinforcement learning processes for the Al systems; applying an algorithmic process to validate the list of clinical research studies, comprising: checking for fraudulent research using Benford's law, Bayesian outlier analysis, extreme misclassification outlier analysis, and publication bias assessment; applying an algorithmic process to conduct feasibility assessments, comprising the assessment of clinical homogeneity, study metadata homogeneity, statistical homogeneity, and network connectivity; applying an algorithmic process to perform meta-regression adjustments to account for effects of identified heterogeneities on outcomes, employing Bayesian analysis.

[0049] A system for automated real-time processing of clinical evidence is disclosed including a memory storing instructions and a processor configured to execute the instructions to receive an input from a user electronic device comprising a clinical question; generate parameters for a Population, Intervention, Comparison, Outcome (PICO) table relevant to the clinical question using a large language model (LLM); retrieve clinical research studies from multiple databases based on the parameters; screen the retrieved studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolve discrepancies between the Al systems through a researcher audit interface; generate a list of clinical research studies based on the screenings; and display the list on a display of the user electronic device.

[0050] A non-transitory computer-readable storage medium is disclosed. The storage medium stores instructions that, when executed by at least one processor, cause the at least one processor to receive an input from a user electronic device comprising a clinical question; generate parameters for a Population, Intervention, Comparison, Outcome (PICO) table relevant to the clinical question using a large language model (LLM); retrieve clinical research studies from multiple databases based on the parameters; screen the retrieved studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolve discrepancies between the Al systems through a researcher audit interface; generate a list of clinical research studies based on the screenings; and display the list on a display of the user electronic device.

[0051] While the invention has been described with reference to the specific embodiments, it will be understood by those skilled in the art that various changes may be made without departing from the scope of the present specification. Furthermore, the scope of the present specification is not intended to be limited to the specific embodiments described herein.Additionally, the range of embodiments described herein is not intended to limit the scope of the present specification. Rather, the invention encompasses all modifications and variations within the scope of the present specification.

Claims

CLAIMS1. A method for automated real-time processing of clinical evidence, implemented by a computing system, comprising: receiving an input from a user electronic device comprising a clinical question; generating parameters for a Population, Intervention, Comparison, Outcome (PICO) table relevant to the clinical question using a large language model (LLM); retrieving clinical research studies from multiple databases based on the parameters; screening the retrieved studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolving discrepancies between the Al systems through a researcher audit interface; generating a list of clinical research studies based on the screenings; and displaying the list on a display of the user electronic device.

2. The method of claim 1, further comprising: retrieving full texts of clinical research studies for a second level of screening using the two Al systems.

3. The method of claim 1, further comprising: presenting the researcher audit interface designed for a medical librarian, wherein the researcher audit interface displays the search strategy and results, enabling the medical librarian to validate the accuracy and appropriateness of the search strategy and results, and wherein the validation by the medical librarian informs reinforcement learning processes for the Al systems.

4. The method of claim 1, further comprising: applying an algorithmic process to validate the list of clinical research studies, comprising: checking for fraudulent research using Benford's law, Bayesian outlier analysis, extreme misclassification outlier analysis, and publication bias assessment.

5. The method of claim 1, further comprising: applying an algorithmic process to conduct feasibility assessments, comprising the assessment of clinical homogeneity, study metadata homogeneity, statistical homogeneity, and network connectivity.

6. The method of claim 1, further comprising: applying an algorithmic process to perform meta-regression adjustments to account for effects of identified heterogeneities on outcomes, employing Bayesian analysis.

7. A system for automated real-time processing of clinical evidence, comprising: a memory storing instructions; and a processor configured to execute the instructions to: receive an input from a user electronic device comprising a clinical question; generate parameters for a Population, Intervention, Comparison, Outcome (PICO) table relevant to the clinical question using a large language model (LLM); retrieve clinical research studies from multiple databases based on the parameters; screen the retrieved studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolve discrepancies between the Al systems through a researcher audit interface; generate a list of clinical research studies based on the screenings; and display the list on a display of the user electronic device.

8. At least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: receive an input from a user electronic device comprising a clinical question; generate parameters for a Population, Intervention, Comparison, Outcome (PICO) table relevant to the clinical question using a large language model (LLM); retrieve clinical research studies from multiple databases based on the parameters;screen the retrieved studies using two independent artificial intelligence (Al) systems that analyze titles and abstracts or executive summaries; resolve discrepancies between the Al systems through a researcher audit interface; generate a list of clinical research studies based on the screenings; and display the list on a display of the user electronic device.

Images