A multi-agent collaborative medical risk prediction model evaluation automation system

Abstract

The application relates to the technical field of medical information processing and artificial intelligence, and discloses a multi-agent cooperation medical risk prediction model evaluation automation system, which comprises a literature acquisition module, a literature preprocessing module, a multi-agent cooperation module, a performance index normalization module, an evidence summary module, a knowledge graph construction module and a central control module. Through the multi-agent hierarchical cooperation architecture, the system changes the manual article-by-article processing of literature screening, data extraction and bias evaluation into automatic pipeline parallel processing, shortens the system evaluation period from half a year to one year to several days; through multi-agent cross-validation and confidence weighting conflict arbitration mechanism, the data extraction accuracy and bias evaluation consistency are greatly improved, and finally the problems of low efficiency, insufficient accuracy, strong subjectivity of PROBAST evaluation and lack of professional adaptation ability of the existing AI scheme in the prediction model scene of the traditional manual system evaluation are solved.

Description

Technical Field

[0001] This invention relates to the field of medical information processing and artificial intelligence, specifically to an automated system for evaluating multi-agent collaborative medical risk prediction models. Background Technology

[0002] Systematic reviews are a core method of evidence integration in evidence-based medicine, particularly crucial in the field of medical risk prediction models. Through systematic retrieval, rigorous evaluation, and summary analysis, they provide reliable evidence for clinical practice and health decision-making. Currently, the entire process of systematic reviews of predictive models—including literature retrieval, inclusion screening, data extraction, risk of bias assessment, and evidence summarization—is completed manually by researchers. A typical systematic review usually involves processing hundreds to thousands of candidate articles, ultimately including 10 to 30. Researchers must read each article in its entirety, manually extracting a wealth of information from the Methods and Results sections, including basic research information, cohort characteristics (sample size, number of events, EPV value, missing data handling), a list of predictive variables, model algorithms, performance metrics (AUC / ROC / c-statistic, calibration curve, HL test, Brier score, negative predictive value, positive predictive value, etc.), and validation methods.

[0003] This manual approach has three inherent drawbacks: First, it is extremely inefficient. A complete systematic review typically takes 6 to 12 months, with literature screening and data extraction accounting for over 60% of the workload, resulting in poor timeliness of evidence. Furthermore, it usually requires at least two researchers working in parallel, placing a heavy burden on human resources. Second, the terminology used in predictive model literature is highly inconsistent. Discrimination metrics include various expressions such as AUC, ROC, AUROC, c-statistic, and concordance statistic; calibration metrics include calibration slope, the p-value of the Hosmer-Lemeshow test, and the O / E ratio. Internal validation methods can refer to cross-validation, bootstrap, or random splitting. Researchers need to be familiar with a large number of equivalent terms, and manual identification and conversion are prone to errors. The consistency of extraction among different reviewers is limited (κ value is usually between 0.6 and 0.8), affecting the reliability of the conclusions. Third, the PROBAST bias risk assessment tool includes 20 signal questions in 4 areas. Its judgment is highly dependent on the subjective interpretation of the evaluator, especially on issues such as whether the definition of the blind implementation result is specified in advance. Moreover, the workload of evaluating each case is extremely large, and the process is often simplified due to resource constraints.

[0004] In recent years, artificial intelligence (AI) technology has been explored for use in assisting medical literature evaluation, but existing solutions still have significant shortcomings. A search of publicly available patent literature reveals no AI solutions specifically tailored to the specific scenario of medical risk prediction model system evaluation. Existing patents, such as CN118690793A (Multi-agent Biochemical Information Extraction Model and Training Method Based on Large Language Model), only address general biochemical information extraction, lacking dedicated field templates for prediction models and failing to provide bias assessment and heterogeneity detection capabilities. Existing patent CN121862382A (Intelligent Assisted Diagnosis System Based on Multi-agent Collaboration) focuses on clinical diagnosis rather than research evidence integration and involves diagnostic purposes. Patent CN119207685B (A Method and System for Processing Medical Record Data Based on Multi-agent) processes structured hospital medical records rather than full-text academic literature, making information extraction significantly more difficult. Patent CN122247689A (Risk Assessment Method and Storage Medium) targets the cybersecurity field, fundamentally different from the medical system evaluation scenario. Overall, these existing technologies only have some overlap in multi-agent collaboration or information extraction methods, but none of them have designed professional extraction schemas for the characteristics of predictive model research, cannot distinguish between internal and external verification performance indicators, lack core system evaluation functions such as PROBAST automated evaluation, I² heterogeneity calculation, and performance indicator normalization, and do not have data traceability capabilities.

[0005] In summary, existing technologies suffer from several drawbacks. Purely manual methods are inefficient, inaccurate, and highly subjective. AI-assisted solutions also suffer from limitations such as misaligned application scenarios, incomplete functional coverage, and lack of verifiability. Consequently, they fail to meet the practical needs of efficient, accurate, and repeatable evaluation of medical risk prediction model systems. This invention addresses all these shortcomings by proposing an automated evaluation system based on multi-agent collaboration to fill the technological gap in this sub-field. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides an automated evaluation system for multi-agent collaborative medical risk prediction models. It features fully automated processing, high-precision data extraction, intelligent terminology normalization, automatic assessment of bias risk, and full data traceability. This system solves the problems of extremely low evaluation efficiency, insufficient accuracy and consistency, strong subjectivity in PROBAST evaluation, and lack of professional adaptability of existing AI-assisted solutions for prediction models in various scenarios.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution: an automated system for evaluating a multi-agent collaborative medical risk prediction model, comprising a literature acquisition module, a literature preprocessing module, a multi-agent collaboration module, a performance index normalization module, an evidence summarization module, a knowledge graph construction module, and a central control module;

[0010] The document acquisition module is used to automatically connect to PubMed, Web of Science, EMBASE, CNKI, China Biomedical Database, Wanfang and VIP literature databases, and retrieve the bibliographic information, abstracts and full text of candidate documents based on the PICOS framework search input by the user. It supports importing and exporting EndNote and RIS literature management formats, and performs batch document retrieval and download through the API interfaces provided by each database.

[0011] The document preprocessing module is responsible for parsing the format and structuring the content of the acquired full text, providing standardized input for subsequent intelligent agent processing;

[0012] The multi-agent collaboration module is based on the same basic model. Through exclusive system prompts and dimensional domain knowledge injected by RAG, it instantiates four agents with specific functions. They collaborate in an orderly manner through a message bus, forming a complete link of task decomposition → division of labor and collaboration → result aggregation → conflict resolution.

[0013] The performance index normalization module is used to unify the terminology and quantify the heterogeneity of the extracted performance indexes. Through the pre-built prediction model performance index terminology mapping table, different expressions of the same performance index are unified and merged into standardized fields. Based on the AUC value and confidence interval of each research report, the I² heterogeneity index is automatically calculated. When I² > 90%, the heterogeneity is marked as extremely high, and it is not recommended to merge directly. A prompt is issued to the user, and subgroup analysis or sensitivity analysis is recommended.

[0014] The evidence aggregation module is used to generate a draft of the systematic review report and output a PRISMA flowchart, a summary table of included study characteristics, a summary diagram of risk of bias assessment, and forest plot data, providing researchers with structured data that can be directly used to write systematic review papers or meta-analyses.

[0015] The knowledge graph construction module is used to build a knowledge graph for predictive model research, enabling traceability of extracted data and visualization of evidence relationships. The system establishes a mapping relationship between the field values ​​of each extracted record and the paragraph position of the source literature, storing them as extraction-from-relationship edges in the knowledge graph. This not only allows users to trace back to the original paragraph by clicking on any data point through the graph interface, but also supports comparative analysis across studies.

[0016] The central control module is responsible for the complete link management of task decomposition, distribution, result aggregation and conflict detection. It is used to coordinate the data flow between modules and manage the message bus communication between multiple agents.

[0017] Preferably, the document preprocessing module performs the following functions:

[0018] (1) Perform OCR recognition on scanned PDFs;

[0019] (2) Perform paragraph segmentation and chapter recognition on electronic PDF / XML files;

[0020] (3) Perform text cleaning on the entire text;

[0021] (4) Intelligent segmentation of long texts and adaptation to the context window limitation of LLM.

[0022] Preferably, the multi-agent collaboration module has built-in document screening agent, data extraction agent, bias assessment agent, and conflict arbitration agent.

[0023] Preferably, the document screening agent is used to input the title, abstract, and full text of a document according to the user-preset PICOS inclusion / exclusion criteria, and to perform title / abstract screening and full-text screening on candidate documents, simulating the document screening process in the PRISMA flowchart. It also supports single-person or dual-person independent screening modes, and finally outputs the inclusion / exclusion judgment results and reasons, along with a confidence score. ;

[0024] The data extraction agent takes full-text input and extracts key information about the predictive model research from the Methods and Results sections of the included literature. Finally, it outputs structured extraction results that strictly follow the preset JSON Schema, including basic research information, cohort characteristics, predictor variables, model construction, performance indicators and bias assessment dimensions. A backtracking mechanism is triggered when required fields are missing.

[0025] Preferably, the bias assessment agent includes the full text of the literature and related extraction results, which are used to apply the PROBAST+AI tool to conduct automated bias risk assessment of the included studies, and finally output the bias risk rating and applicability evaluation of PROBAST+AI in four areas.

[0026] The conflict arbitration agent receives the output results and confidence scores of each agent to monitor the consistency of the outputs of the above agents, and performs adjudication or marks manual review for the disagreement items. When multiple agents have inconsistent results in extracting the same field of the same document, the conflict arbitration process is initiated: conflict detection → confidence assessment → weighted adjudication → manual review marking, and finally outputs the consistent result after arbitration, including the manual review marking.

[0027] Preferably, the conflict arbitration agent executes the conflict arbitration sub-process:

[0028] S1.1 Conflict Detection: The central control module compares the outputs of each agent and identifies inconsistencies.

[0029] S1.2 Confidence Assessment: Each agent attaches a confidence score to its output. ∈[0,1];

[0030] S1.3 Weighted Decision: The conflict arbitration agent uses confidence level as the weight and selects the result with the highest score as the final output.

[0031]

[0032] in, This is the output of the i-th agent. Score the confidence level of the i-th agent. For indicator functions, when The value is 1 when =r, otherwise it is 0;

[0033] S1.4, Manual verification mark: When the difference between the highest confidence level and the second highest confidence level... If the value is less than 0.15, the field is marked as awaiting manual review and included in a dedicated manual review queue.

[0034] S1.5 Review and Feedback: The results of manual review are fed back to each agent as new training samples, and continuous optimization is achieved through Few-shot / Prompt.

[0035] Preferably, the performance index normalization module automatically calculates the I² heterogeneity index based on the AUC values ​​and confidence intervals of each research report, as follows:

[0036]

[0037] Where Q is Cochran's Q statistic and df is the degrees of freedom;

[0038] When I² > 90%, the system labels exhibit extremely high heterogeneity, and direct merging is not recommended. The system will then prompt the user to perform subgroup analysis or sensitivity analysis.

[0039] Preferably, the entity nodes of the knowledge graph construction module include research, models, population characteristics, performance indicators and bias ratings; the relation edges include semantic relations that have been developed, verified, reported AUC, and extracted from.

[0040] Preferably, the central control module executes the following process:

[0041] S2.1 Receive the PICOS conditions and search query input by the user;

[0042] S2.2, Call the document retrieval module to obtain candidate documents;

[0043] S2.3 Call the document preprocessing module to parse the format;

[0044] S2.4 Distribute tasks to the multi-agent collaboration module;

[0045] S2.5 Summarize the output results of each agent;

[0046] S2.6 Detect conflicts and trigger the conflict arbitration mechanism;

[0047] S2.7 Call the performance metric normalization module to unify terminology;

[0048] S2.8. Call the evidence summary module to generate a draft report;

[0049] S2.9. Call the knowledge graph construction module to establish source tracing associations.

[0050] Preferably, the system further includes a prediction model-specific information extraction schema, a multi-agent conflict arbitration mechanism, an automatic terminology normalization method for performance indicators, and a knowledge graph data tracing mechanism.

[0051] Compared with existing technologies, this invention provides an automated system for evaluating multi-agent collaborative medical risk prediction models, which has the following advantages:

[0052] 1. This invention constructs a multi-agent hierarchical collaborative architecture covering the entire process of system evaluation, transforming the manual, paper-by-paper processing of literature screening, data extraction, and bias assessment into automated, pipeline-style parallel processing. This significantly shortens the system evaluation cycle from the traditional six months to a year to just a few days, fundamentally solving the technical shortcomings of low efficiency in manual extraction in existing technologies.

[0053] 2. This invention uses a multi-agent cross-validation and confidence-weighted conflict arbitration mechanism to enable each agent to independently output results, which are then weighted and labeled with divergence. This significantly improves the accuracy of data extraction and the consistency of bias assessment, thereby effectively alleviating the "illusion" and information omission problems that are easily generated by a single AI model in understanding complex medical literature.

[0054] 3. This invention, through the synergistic effect of a dedicated information extraction schema for predictive model research features, automatic terminology normalization of performance indicators, and knowledge graph data tracing mechanism, enables AI-extracted data to be directly used for evidence integration, while each data point can be traced back to the original text paragraph, fundamentally filling the technical gap in AI-assisted system evaluation in the subdivided scenario of predictive model research. Attached Figure Description

[0055] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 An automated system for evaluating a multi-agent collaborative medical risk prediction model includes a literature acquisition module, a literature preprocessing module, a multi-agent collaboration module, a performance index normalization module, an evidence summarization module, a knowledge graph construction module, and a central control module.

[0058] The technical terms used in this system are explained in the table below:

[0059] Table 1

[0060] AI Artificial Intelligence AI ML Machine Learning Machine Learning LLM Large Language Model Large Language Model RAG Retrieval-Augmented Generation Search Enhancement Generation PROBAST Prediction model Risk Of Bias ASsessment Tool Predictive model bias risk assessment tool PROBAST+AI PROBAST updated for AI methods PROBAST Update for AI Methods PICOS Participants, Intervention, Comparison, Outcomes, Study Design Study subjects, intervention, control group, outcome, and study design. PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses System reviews and meta-analysis priority reporting items ROC Receiver Operating Characteristic Subject working characteristics AUC Area Under the ROC Curve Area under the ROC curve EPV Events Per Variable Number of events per predictor JSON JavaScript Object Notation JavaScript object representation I² I-squared statistic I² Heterogeneity Statistic PPV Positive Predictive Value Positive predictive value NPV Negative Predictive Value Negative predictive value

[0061] The Literature Acquisition Module serves as the system's data input entry point, automatically connecting to the three major international biomedical databases: PubMed, Web of Science, and EMBASE. It also integrates important domestic Chinese literature resources such as CNKI, Sinomed, Wanfang, and VIP. Based on the user-input PICOS (Participants, Intervention, Comparison, Outcomes, Study Design) framework, it retrieves bibliographic information, abstracts, and full texts (PDF / XML format) of candidate literature. It supports importing and exporting EndNote and RIS literature management formats and performs batch literature searches and downloads through API interfaces provided by various databases (such as PubMed's E-utilities API).

[0062] The advantages are: by automatically connecting to mainstream biomedical literature databases such as PubMed, Web of Science, and EMBASE, it enables batch retrieval and automated download of candidate literature, fundamentally solving the inefficiency problem of researchers having to manually search database by database and article by article in traditional systematic reviews. It performs standardized retrieval through the official API interfaces of each database (such as PubMed E-utilitiesAPI), thereby ensuring the standardization and reproducibility of literature acquisition. The module supports the import and export of literature management formats such as EndNote and RIS, enabling the system to be seamlessly integrated into researchers' existing literature management workflows, thereby reducing tool migration costs. This module provides standardized data input for subsequent multi-agent automated processing, which is the premise and foundation for the efficient operation of the entire automated pipeline.

[0063] The Literature Preprocessing Module is responsible for parsing the format and structuring the content of the acquired full text, providing standardized input for subsequent intelligent agent processing. The module performs the following functions:

[0064] (1) Perform OCR (Optical Character Recognition) on scanned PDFs;

[0065] (2) Perform paragraph segmentation and chapter recognition on electronic PDF / XML (automatically locate chapters such as Methods, Results, and Discussion);

[0066] (3) Clean the entire text (remove headers, footers, reference marks, figure and table placeholders, etc.);

[0067] (4) Intelligent segmentation of long texts and adaptation to the context window limitation of LLM.

[0068] The advantages are as follows: By parsing the format and structuring the content of the acquired full text, it provides a standardized and unified input for subsequent intelligent agent processing, solving the technical challenges brought about by the diverse formats and heterogeneous structures of medical literature. Specifically, by performing OCR recognition on scanned PDFs, historical documents and non-electronic documents can also be included in the automated processing flow; by performing paragraph segmentation and chapter recognition (automatically locating chapters such as Methods, Results, and Discussion) on electronic PDFs / XMLs, the intelligent agent can accurately locate the key information areas of the predictive model research, thereby significantly improving the efficiency and accuracy of subsequent data extraction; by performing text cleaning on the full text (removing headers, footers, reference marks, figure placeholders, etc.), the interference of non-text content on the intelligent agent's understanding is finally eliminated; by intelligently segmenting long texts and adapting to the context window limitations of LLM, the complete processing of large-scale documents is ensured. This module transforms the originally messy raw documents into structured and computable standard input, which is a quality guarantee for the efficient operation of the multi-agent collaborative module.

[0069] The Multi-Agent Collaboration Module, as the core module of the system, is based on the same basic large model (such as GPT-4, Claude 3, or open-source LLM). Through the system prompt and the dimensional domain knowledge injected by RAG (Retrieval-Augmented Generation), it instantiates four functionally specialized agents. These agents collaborate in an orderly manner through a message bus, forming a complete chain of task decomposition → division of labor and collaboration → result aggregation → conflict resolution. The module has four built-in functionally specialized agents: Literature Screening Agent, Data Extraction Agent, Bias Assessment Agent, and Conflict Arbitration Agent.

[0070] The advantages are: by instantiating four specialized agents (document screening agent, data extraction agent, bias assessment agent, and conflict arbitration agent) based on the same basic large model, the system evaluation process is automated in terms of division of labor and collaboration. Compared with a single AI model, multi-agent collaboration achieves a significant performance improvement through model diversity.

[0071] The document screening agent is used to perform title / abstract screening and full-text screening of candidate documents based on user-preset PICOS inclusion / exclusion criteria, inputting the document title, abstract, and full text. It simulates the document screening process in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart, supporting both single-person and dual-person independent screening modes. Finally, it outputs the inclusion / exclusion judgment results and reasons, along with a confidence score. ;

[0072] By simulating the literature screening process in the PRISMA flowchart, the sensitivity of the LLM-based automated screening workflow can reach 96.7%, which is significantly better than manual review; it supports independent screening modes by one or two people, thus taking into account the quality standard requirements of different research projects.

[0073] The data extraction agent takes full-text input and extracts key information about the predictive model research from the Methods and Results sections of the included literature. Finally, it outputs structured extraction results that strictly follow the preset JSON (JavaScript Object Notation) Schema, including basic research information, cohort characteristics, predictor variables, model construction, performance indicators and bias assessment dimensions. A backtracking mechanism is triggered when required fields are missing.

[0074] The JSON Schema constraint template includes the following dimensions:

[0075] ① Basic research information (author, year, country, journal);

[0076] ② Cohort characteristics (sample size, number of outcome events, EPV (Events Per Variable) value, proportion of missing data and handling methods);

[0077] ③ Predictor variables (variable list, selection method);

[0078] ④ Model construction (algorithm type, internal verification method, external verification method);

[0079] ⑤ Performance metrics (discrimination: AUC (Area Under the ROC Curve) / ROC (Receiver Operating Characteristic) / c-statistic value and confidence interval; calibration: calibration slope, p-value of Hosmer-Lemeshow test, Brier score; clinical utility: decision curve analysis (DCA, Decision Curve Analysis), etc.).

[0080] ⑥ Bias assessment (PROBAST (Prediction model Risk Of Bias Assessment Tool) + AI signal problem judgment in various fields).

[0081] By using dedicated JSON Schema constraints to ensure standardized output format and complete fields, the problem of inconsistent output format and missing key fields in general AI tools is fundamentally solved. Furthermore, by setting a backtracking mechanism to trigger when a required field is missing, the system can automatically retrieve the original text to complete the information, minimizing data omissions to the greatest extent possible.

[0082] The full text of the literature included in the bias assessment agent and related extraction results are used to apply the PROBAST+AI (PROBAST updated for AI methods) tool to automatically assess the bias risk of the included studies. Finally, the bias risk rating (low / high / unclear) and applicability evaluation of the four domains of PROBAST+AI (students and data sources, predictors, outcomes, and statistical analysis) are output.

[0083] PROBAST+AI is an updated version of the predictive model bias risk assessment tool released in the British Medical Journal (BMJ) by Karel GM Moons' team in 2025. It consists of two independent parts: model development and model evaluation, covering four areas: study subjects and data sources, predictors, outcomes, and statistical analysis. The model development part uses 16 signaling questions to assess quality, and the model evaluation part uses 18 signaling questions to assess bias risk. It outputs bias risk ratings (low / high / unclear) and applicability evaluations for the four areas.

[0084] This intelligent agent transforms the traditional manual, item-by-item judgment of bias risk assessment into automated and standardized batch processing, thereby significantly reducing the systematic error introduced by individual differences among assessors.

[0085] The conflict arbitration agent receives the output results and confidence scores of each agent to monitor the consistency of the outputs of the above agents, and performs adjudication or marks for manual review of the disagreement items. When multiple agents have inconsistent results in extracting the same field of the same document, the conflict arbitration process is initiated: conflict detection → confidence assessment → weighted adjudication → manual review marking. Finally, the consistent result after arbitration is output, including the manual review marking.

[0086] By employing a multi-agent cross-validation and confidence-weighted decision-making mechanism, the "illusion" problem that can easily arise when a single AI model struggles with understanding complex medical literature is effectively mitigated.

[0087] The conflict arbitration agent executes the conflict arbitration sub-process:

[0088] S1.1 Conflict Detection: The central control module compares the outputs of each agent and identifies inconsistencies.

[0089] S1.2 Confidence Assessment: Each agent attaches a confidence score to its output. ∈[0,1];

[0090] S1.3 Weighted Decision: The conflict arbitration agent uses confidence level as the weight and selects the result with the highest score as the final output.

[0091]

[0092] in, This is the output of the i-th agent. Score the confidence level of the i-th agent. For indicator functions, when The value is 1 when =r, otherwise it is 0;

[0093] S1.4, Manual verification mark: When the difference between the highest confidence level and the second highest confidence level... < (threshold) When the default value is 0.15, the field is marked as awaiting manual review and included in a dedicated manual review queue.

[0094] S1.5 Review and Feedback: The results of manual review are fed back to each agent as new training samples, and the consistency of the agents is continuously improved through Few-shot or Prompt optimization.

[0095] The advantages are: the four intelligent agents mentioned above cooperate in an orderly manner through a message bus, and the central control module manages the complete link of task decomposition → distribution → result aggregation → conflict detection, forming a closed loop of task decomposition → division of labor and cooperation → result aggregation → conflict resolution, which shortens the traditional system evaluation process that takes 6-12 months to a few days.

[0096] The Performance Indicator Normalization Module is used to standardize terminology and quantify heterogeneity of extracted performance indicators. Using a pre-built terminology mapping table for predictive model performance indicators, it unifies and merges different expressions of the same performance indicator into standardized fields. Based on the AUC values ​​and confidence intervals of each study report, it automatically calculates the I² (I-squared statistic, I² heterogeneity statistic). When I² > 90%, it marks extremely high heterogeneity, and direct merging is not recommended, prompting the user to suggest subgroup analysis or sensitivity analysis. The module automatically calculates the I² heterogeneity index based on the AUC values ​​and confidence intervals of each study report, as detailed below:

[0097]

[0098] Where Q is Cochran's Q statistic and df is the degrees of freedom (df=k−1, k is the number of included studies).

[0099] When I² > 90%, the system labels are highly heterogeneous, and direct merging is not recommended. The user is prompted to perform subgroup analysis or sensitivity analysis.

[0100] The module uses a pre-built mapping table of predictive model performance metrics to unify synonyms such as AUC, ROC, AUROC, c-statistic, and concordance statistic into the standardized field AUC; and to unify calibration degree terms such as calibration slope, calibration-in-the-large, P value of Hosmer-Lemeshow test, and O / E ratio, as shown in the table below:

[0101] Terminology Normalization Mapping Table 2 (Example of Key Mappings)

[0102] Original terminology Normalized field Original terminology Normalized field AUC AUC ROC AUC AUROC AUC c-statistic AUC concordance statistic AUC area under the curve AUC calibration slope Calibration calibration-in-the-large Calibration P value of Hosmer-Lemeshow test H O / E ratio O internal validation Internal cross-validation Internal bootstrap Internal split sample Internal external validation External temporal validation External

[0103] The advantages are: the above-mentioned module functions fundamentally solve the technical problem of inconsistent expression of the same performance index in different literatures in predictive model research, which seriously hinders the summary comparison and meta-analysis. At the same time, the module automatically calculates the I² heterogeneity index based on the AUC value and confidence interval of each research report. When I²>90%, it automatically marks "extremely high heterogeneity, direct merging is not recommended" and issues a prompt to the user, suggesting subgroup analysis or sensitivity analysis. This allows the data extracted by AI to be directly used for meta-analysis without the need for manual post-processing, filling the functional gap of existing AI-assisted literature extraction tools in "systematic review-specific data integration".

[0104] The Evidence Synthesis Module is used to generate draft systematic review reports and output PRISMA flowcharts (visualization of the literature screening process), summary tables of included study characteristics, summary diagrams of risk of bias assessment, and forest plot data (AUC summary, I² heterogeneity index, etc.), providing researchers with structured data that can be directly used to write systematic review papers or meta-analyses.

[0105] The advantages are: This module automatically transforms the massive amounts of structured data extracted by AI into visual reports that conform to academic standards, which greatly reduces the workload of researchers in data processing, chart creation and report writing. Consistent with the empirical results of the InsightAgent system, clinicians can complete a high-quality systematic review in just about 1.5 hours with this system, while traditional methods take several months.

[0106] The Knowledge Graph Construction Module is used to build a knowledge graph for predictive model research, enabling traceability of extracted data and visualization of evidentiary relationships. The system maps the field values ​​of each extracted record to the paragraph positions in the source literature, storing these as extracts within the knowledge graph. The graph allows users to trace any data point back to the original text paragraph by clicking on it in the graph interface, verifying the accuracy of AI extraction. It also supports comparative analysis across studies (such as visualization of the performance comparison of the same prediction model in different cohorts), helping researchers identify evidence of performance degradation.

[0107] The entity nodes of the knowledge graph construction module include research, models, population characteristics, performance indicators and bias ratings; the relation edges contain semantic relations such as "developed, validated, reported AUC as, extracted from".

[0108] Example of a source link:

[0109] (1) Triples: (Study A, reported an AUC of 0.82) → Source link: {Document ID: PMID123456, Paragraph: Results, paragraph 3, Original sentence: The AUC of the model was 0.82 (95% CI 0.78-0.86)};

[0110] (2) Triples: (Study A, Validated in, Cohort B) → Source link: {Document ID: PMID123456, Paragraph: Methods paragraph 5, Original sentence: External validation was performed in an independent cohort from Hospital B}.

[0111] The advantages are as follows: The structured review based on the knowledge graph is a "living" knowledge graph that can be updated in real time. It is continuously maintained by humans and supports computational analysis. The core advantage of this module is that it solves the "black box" problem of AI output, allowing users to trace back to the original text paragraph by clicking on any data point through the graph interface, verifying the accuracy of AI extraction. Moreover, each data point is supported by specific literature evidence, enabling doctors and researchers to verify each conclusion. At the same time, the knowledge graph supports comparative analysis across studies (such as visualizing the performance comparison of the same prediction model in different cohorts), helping researchers identify evidence of performance degradation. The knowledge graph makes it easy to integrate information from many sources, supports the exploration of heterogeneous information in a single data model, and infers new relationships through efficient queries. This makes the systematic review results of this invention have verifiability and interpretability that are lacking in traditional AI tools.

[0112] The Central Control Module is responsible for the complete link management of task decomposition, distribution, result aggregation and conflict detection. It is used to coordinate the data flow between modules and manage the message bus communication between multiple agents, thereby ensuring the orderly cooperation of the system modules.

[0113] Central control module execution flow:

[0114] S2.1 Receive the PICOS conditions and search query input by the user;

[0115] S2.2, Call the document retrieval module to obtain candidate documents;

[0116] S2.3 Call the document preprocessing module to parse the format;

[0117] S2.4 Distribute tasks to the multi-agent collaboration module;

[0118] S2.5 Summarize the output results of each agent;

[0119] S2.6 Detect conflicts and trigger the conflict arbitration mechanism;

[0120] S2.7 Call the performance metric normalization module to unify terminology;

[0121] S2.8. Call the evidence summary module to generate a draft report;

[0122] S2.9. Call the knowledge graph construction module to establish source tracing associations.

[0123] The advantages are: by recording the execution logs and intermediate results of each step through the central control module, it provides technical support for the transparency and auditability of system evaluation, enabling the originally independent modules to work together and form a synergy, thereby achieving the goal of full-process automation.

[0124] The system also includes a schema for information extraction specific to prediction models, a multi-agent conflict arbitration mechanism, an automatic terminology normalization method for performance metrics, and a knowledge graph data tracing mechanism.

[0125] (1) Prediction model-specific information extraction schema: The data extraction agent strictly follows the schema for structured output;

[0126] (2) Multi-agent conflict arbitration mechanism: When multiple agents extract inconsistent results from the same field of the same document, the following arbitration process is executed:

[0127] Conflict detection: The central control module compares the outputs of each agent and identifies inconsistencies.

[0128] b. Confidence assessment: Each agent includes a confidence score with its output. ∈[0,1];

[0129] c-weighted decision: The conflict arbitration agent selects the result with the highest confidence level;

[0130] d. Manual verification mark: When the difference between the highest confidence level and the second highest confidence level Δ < (threshold) When the default value is 0.15, this field is marked as requiring manual review.

[0131] (3) By using the pre-built prediction model performance index terminology mapping table, near-semantic terms are uniformly merged into the standardized field AUC;

[0132] (4) Knowledge graph data traceability mechanism: Each extracted data is stored in the knowledge graph with attached contextual traceability information.

[0133] The system of the present invention is applied to the following embodiments:

[0134] Example 1: Systematic Review and Meta-analysis of Clinical Prediction Model Studies

[0135] Background: In clinical medical research, there are often dozens to hundreds of published predictive model studies for the same disease (such as cardiovascular disease, sepsis, diabetic complications, tumor prognosis, etc.). Researchers need to systematically search, evaluate and integrate these studies to determine which predictive models have sufficient discriminative power, calibration and clinical applicability.

[0136] Application method:

[0137] Scenario: Researchers launch a new systematic review project (a systematic review of a model for predicting the risk of death in ICU patients with sepsis over 28 days).

[0138] System operation: The user inputs PICOS conditions (target population is ICU adult sepsis patients, prediction model type is mortality risk prediction, and external validation studies are preferred, etc.).

[0139] System output: The system automatically retrieves candidate literature from PubMed / Web of Science, and after multi-agent screening, outputs a summary table of features of the included studies, a summary of AUC and confidence intervals of each model, a PROBAST+AI bias risk assessment, an I² heterogeneity index and merging suggestions;

[0140] End-user results: Researchers can directly use the structured data output by the system to write systematic review papers or for meta-analysis;

[0141] Expected results: The traditional 6-12 month system review cycle will be shortened to a few days, significantly accelerating the evidence integration process.

[0142] Example 2: Rapid validation and evaluation before introducing external predictive models in medical institutions

[0143] Background: A tertiary-level hospital plans to introduce commercial predictive models (such as sepsis prediction models, readmission prediction models, acute kidney injury prediction models, etc.) or adopt predictive models reported in academic literature. However, the excellent performance reported by the models in the development cohort often shows a significant decline in external validation. The hospital needs to evaluate the external validation evidence of the models in different populations, regions, and medical environments before introduction.

[0144] Application method:

[0145] Scenario: A tertiary-level Class A hospital plans to introduce a deep learning-based early warning model for sepsis;

[0146] System operation: Medical engineering personnel or evidence-based medicine center personnel input the model name, target population, and predicted outcome into the system, and specify "prioritize searching for external validation studies";

[0147] System output: The system automatically retrieves external validation studies of the model worldwide and automatically extracts the sample size, population characteristics (region, ethnicity, age distribution), AUC value and confidence interval, calibration index, sensitivity / specificity of each validation cohort;

[0148] Application Decision: The knowledge graph visualization shows the performance trend of the model in different validation environments. When the external validation AUC is lower than 0.70, the system automatically outputs a prompt "It is recommended to re-evaluate the feasibility of introduction".

[0149] Expected outcomes: To assist medical institutions in quickly obtaining systematic evidence before introducing commercial predictive models, identifying the risk of "model performance degradation," thereby avoiding adoption decisions based on overly optimistic reported values. At the same time, it helps identify high-risk subgroups (such as the elderly, specific ethnic groups), indicating that the model may perform poorly in these subgroups, and helping to develop localized validation plans.

[0150] Example 3: Evidence Integration in Evidence-Based Medicine Guideline Development

[0151] Background: When developing clinical guideline development working groups for a particular disease, they often need to incorporate recommendations from risk prediction tools. The development of these guidelines must be based on systematic review evidence, and the traditional process is time-consuming and extremely costly in terms of manpower.

[0152] Application method:

[0153] Scenario: The Chinese Society of Cardiology formulated recommendations regarding cardiovascular risk scoring tools (such as SCORE, QRISK, and China-PAR) in the "Guidelines for Primary Prevention of Cardiovascular Diseases in China";

[0154] System operation: The guidelines working group inputs the name of the target prediction tool, the target population, and the outcome of concern. The system automatically retrieves and extracts the development and external validation studies of each tool.

[0155] System output: The system generates a summary table comparing the performance of various prediction tools (including differences in external validation performance among different ethnic groups and regions).

[0156] Terminal outcome: Based on the system output, the guideline development team quickly generated draft evidence summaries for "recommended users" and "not recommended users," significantly shortening the evidence integration cycle in the guideline development process.

[0157] Example 4: Preparation of Clinical Evidence in Pharmaceutical / Medical Device Registration Applications

[0158] Background: With the improvement of AI medical device approval regulations (such as the China NMPA's "Guiding Principles for the Classification and Definition of Artificial Intelligence Medical Software Products" and the US FDA's "Action Plan for AI-Based Medical Device Software (SaMD)"), AI-assisted diagnostic and prediction software is required to submit systematic evidence of the product's predictive performance during registration applications, including performance comparison studies of similar products / algorithms.

[0159] Application method:

[0160] Scenario: An AI medical company has developed an AI model for predicting the risk of diabetic retinopathy and is preparing to submit a Class II / Class III medical device registration application to the NMPA.

[0161] System operation: The enterprise registration department inputs the model type (fundus image deep learning model), target disease, and predicted outcome into the system;

[0162] System output: The system automatically retrieves and extracts performance indicators from similar predictive model studies, forming a "Summary Table of Performance Comparison of Similar Products", which includes the AUC, sensitivity, specificity, and validation cohort size of each study;

[0163] Final outcome: The "Performance Comparison with Similar Products" section in the registration application materials is supported by complete and traceable evidence, demonstrating the advanced nature or equivalence of the performance of the product being applied for.

[0164] Example 5: Research Aid Tools for Medical Graduate Students / Clinical Researchers

[0165] Background: Medical school graduate students and clinicians often need to conduct literature reviews or systematic reviews when carrying out scientific research, but they lack professional training in systematic review methodology, and manual operation is prone to errors and low efficiency.

[0166] Application method:

[0167] Scenario: A cardiology graduate student plans to conduct a systematic review of a "prediction model for readmission risk in patients with heart failure" as his graduation thesis topic.

[0168] System operation: Graduate students enter their research topics and PICOS conditions into the system interface. The system runs in a human-computer collaborative mode. After the AI ​​completes the initial screening and extraction, the graduate students review and revise each paper.

[0169] System outputs: PRISMA flowchart, table of included study characteristics, summary table of risk of bias assessment, and data extraction table (Excel format).

[0170] Terminal results: Graduate students can directly use the system output as the data analysis part of their graduation thesis, requiring only final manual verification and methodological description;

[0171] Expected outcome: Lowering the learning threshold of systematic review methodologies, enabling more clinicians to complete high-quality evidence-based medicine research.

[0172] Example 6: Rapid Evidence Response in Public Health Policy Development

[0173] Background: In public health emergencies (such as outbreaks of emerging infectious diseases), decision-makers need to understand the evidence of the effectiveness of certain interventions, predictive tools or screening strategies in a very short time. Traditional systematic reviews are too time-consuming and cannot meet the needs of rapid response.

[0174] Application method:

[0175] Scenario: In the early stages of an outbreak of a novel respiratory infectious disease, health authorities need to quickly understand the predictive effectiveness of existing respiratory disease severity prediction models in order to guide the allocation of tiered medical resources.

[0176] System operation: The system completes the integration of literature evidence, which normally takes months, within a few days and outputs a ranking of the predictive models' performance.

[0177] End-point outcome: Public health policymakers obtain rapid evidence support in the early stages of an outbreak, aiding in resource allocation and the formulation of control strategies;

[0178] Special note: In this application scenario, it is essential to ensure that the scope of the output precisely matches the research question in order to avoid sacrificing the accuracy and comprehensiveness of the evaluation due to the increased speed of evidence integration.

[0179] Example 7: Auxiliary Tools for Peer Review in Medical Journals

[0180] Background: When receiving systematic reviews / meta-analysis manuscripts, medical journals (especially those requiring PRISMA guidelines, such as BMJ, JAMA, and Annals of Internal Medicine) require reviewers to verify whether authors have omitted key studies and whether data extraction is accurate. Traditional review processes rely entirely on the experience of reviewers, which carries the risk of omissions.

[0181] Application method:

[0182] Scenario: A journal editor receives a submission entitled "Systematic Review and Meta-analysis of XXX Prediction Model" and assigns review experts to review the manuscript;

[0183] System operation: Reviewers input the list of included literature reported in the manuscript into the system, and the system automatically reproduces the literature retrieval and screening process, comparing the differences between the literature included by the system itself and the literature included by the authors;

[0184] System output: The system outputs a "list of key studies omitted by the authors" and a "list of literature that may not meet the criteria included by the authors", as well as a data extraction accuracy verification report;

[0185] Terminal outcome: Based on the difference analysis output by the system, review experts can more effectively evaluate the quality of manuscripts, reducing the subjectivity and risk of omissions in the review process.

[0186] Example 8: Continuous updating of the knowledge base of a Clinical Decision Support System (CDSS)

[0187] Background: Predictive model recommendations in clinical decision support systems (such as UpToDate, DynaMed, and other clinical knowledge bases) need to be continuously updated with the latest research evidence. Traditional methods that rely on manual monitoring of literature updates are inefficient and untimely.

[0188] Application method:

[0189] Scenario: A CDSS vendor maintains a knowledge base containing hundreds of clinical prediction models;

[0190] System Operation: The system automatically performs automated literature screening and evidence extraction for each model in the knowledge base on a regular basis (e.g., weekly), checks for new external validation studies, and automatically summarizes the latest evidence;

[0191] System output: The system outputs an "evidence update report" for each model—the validation cohort characteristics, AUC value, calibration degree of newly published external validation studies, and consistency / inconsistency analysis with existing evidence;

[0192] Terminal outcome: Based on the update reports output by the system, the CDSS knowledge base team efficiently decides whether to update the recommendations and model performance data in the knowledge base.

[0193] Example 9: Performance Evaluation of Predictive Models in Healthcare Payment Decisions

[0194] Implementation Background: When deciding whether to include a new medical service or companion diagnostic based on an AI prediction model in the reimbursement scope, medical insurance institutions need to assess whether the predictive efficacy of the model meets the clinically acceptable standard, and whether there are performance differences between different population groups (such as whether there are performance differences between subgroups such as race, gender, and age).

[0195] Application method:

[0196] Scenario: The National Healthcare Security Administration is evaluating the clinical efficacy evidence of an AI prediction model for early identification of high-risk diabetes patients for early intervention.

[0197] System Operation: The system automatically retrieves and extracts performance evidence for the model and its competitors, with particular attention to stratified analysis results for different demographic subgroups;

[0198] System outputs: The system outputs a summary of model performance and a summary of fairness evidence, highlighting research gaps lacking evidence from subgroup analysis;

[0199] Terminal outcome: Based on systematic review evidence, health insurance policymakers determine whether to include medical services based on this model in the reimbursement scope, and whether there is sufficient evidence to support equitable application among different population groups.

[0200] Example 10: Evidence-Based Medicine Teaching Tools in Medical Education

[0201] Background: In evidence-based medicine courses at medical schools, students need to practically operate system evaluation processes to understand methodological principles. Traditional teaching methods often fail to provide students with sufficient time to complete a full system evaluation practice.

[0202] Application method:

[0203] Scenario: A laboratory course on "Systematic Review Practice of Predictive Models" (2 class hours) is set up in the evidence-based medicine curriculum of a medical school.

[0204] System operation: Students are divided into groups and select a prediction model topic. The system runs in teaching mode—intermediate results of each step are visible to students. After each step is completed, the system automatically generates a methodological explanation (e.g., "This step completes the PICOS standard screening, excluding X articles for the reason...").

[0205] System output: The system generates a complete draft system evaluation report within 2 hours, with methodological notes for each step;

[0206] Terminal Outcomes: Students gain a complete experience in systematic evaluation and understand the methodological principles of each step. Teachers can provide teaching feedback and grading based on the system's output report.

[0207] Based on the verification results of the above ten embodiments, the present invention has been fully verified in the following core dimensions:

[0208] (1) Efficiency dimension: The ten implementation examples cover the entire ecosystem from basic scientific research (Examples 1 and 5), clinical practice (Example 2), policy formulation (Examples 3, 6, and 9) to industrial application (Examples 4 and 8) and academic evaluation (Examples 7 and 10). In all scenarios, the system shortens the traditional system evaluation process, which takes months or even more than a year, to several days or even several hours. Under the multi-agent architecture, four professional agents work in parallel, reducing the task completion time from hours to minutes.

[0209] (2) Accuracy dimension: The cross-validation mechanism of multi-agent collaboration effectively alleviated the "illusion" problem of single AI models in all embodiments. Studies have shown that the multi-agent majority voting strategy achieved the best performance in the clinical systematic review subset with MAP of 0.462 and 0.341; compared with the single-agent baseline, the specialized multi-agent system improved the task success rate from 12.5% ​​to 78.0%, and the step efficiency was improved by 5 times.

[0210] (3) Professional Adaptation Dimension: Ten examples respectively verified the applicability of the system in different prediction model research scenarios (mortality risk prediction, sepsis early warning, cardiovascular risk score, diabetic retinopathy, heart failure readmission, etc.), proving that the dedicated information extraction schema and PROBAST+AI automated evaluation logic for prediction model research characteristics can adapt to the professional needs of various prediction model research.

[0211] (4) Verifiability dimension: The knowledge graph data traceability mechanism was applied and verified in Examples 1 to 10. Each data point can be traced back to a specific paragraph of the source literature. This feature is particularly important in Examples 7 (peer review) and 10 (medical education), which verified the credibility and interpretability of the system results.

[0212] (5) Flexibility dimension of the solution: Implementation 1 (fully automatic batch mode), Implementation 5 (human-machine collaborative semi-automatic mode) and Implementation 6 (rapid response scenario) respectively verify the effectiveness of the three implementation schemes of the present invention under different resource conditions and demand scenarios, proving that the system of the present invention has good scenario adaptability and industrial application prospects.

[0213] In summary, through the systematic implementation and verification of the above ten embodiments, it has been fully verified that the system of the present invention can simultaneously achieve all six expected technical objectives in five core dimensions: efficiency, accuracy, professional adaptability, verifiability, and scheme flexibility, thus effectively filling the technical gap in AI-assisted system evaluation in the subdivided scenario of predictive model research.

[0214] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automated system for evaluating a multi-agent collaborative medical risk prediction model, characterized in that, It includes a document acquisition module, a document preprocessing module, a multi-agent collaboration module, a performance index normalization module, an evidence aggregation module, a knowledge graph construction module, and a central control module; The document acquisition module is used to automatically connect to PubMed, Web of Science, EMBASE, CNKI, China Biomedical Database, Wanfang and VIP literature databases, and retrieve the bibliographic information, abstracts and full text of candidate documents based on the PICOS framework search input by the user. It supports importing and exporting EndNote and RIS literature management formats, and performs batch document retrieval and download through the API interfaces provided by each database. The document preprocessing module is responsible for parsing the format and structuring the content of the acquired full text, providing standardized input for subsequent intelligent agent processing; The multi-agent collaboration module is based on the same basic model. Through exclusive system prompts and dimensional domain knowledge injected by RAG, it instantiates four agents with specific functions. They collaborate in an orderly manner through a message bus, forming a complete link of task decomposition → division of labor and collaboration → result aggregation → conflict resolution. The performance index normalization module is used to unify the terminology and quantify the heterogeneity of the extracted performance indexes. Through the pre-built prediction model performance index terminology mapping table, different expressions of the same performance index are unified and merged into standardized fields. Based on the AUC value and confidence interval of each research report, the I² heterogeneity index is automatically calculated. When I² > 90%, the heterogeneity is marked as extremely high, and it is not recommended to merge directly. A prompt is issued to the user, and subgroup analysis or sensitivity analysis is recommended. The evidence aggregation module is used to generate a draft of the systematic review report and output a PRISMA flowchart, a summary table of included study characteristics, a summary diagram of risk of bias assessment, and forest plot data, providing researchers with structured data that can be directly used to write systematic review papers or meta-analyses. The knowledge graph construction module is used to build a knowledge graph for predictive model research, enabling traceability of extracted data and visualization of evidence relationships. The system establishes a mapping relationship between the field values ​​of each extracted record and the paragraph position of the source literature, storing them as extraction-from-relationship edges in the knowledge graph. This not only allows users to trace back to the original paragraph by clicking on any data point through the graph interface, but also supports comparative analysis across studies. The central control module is responsible for the complete link management of task decomposition, distribution, result aggregation and conflict detection. It is used to coordinate the data flow between modules and manage the message bus communication between multiple agents.

2. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The document preprocessing module performs the following functions: (1) Perform OCR recognition on scanned PDFs; (2) Perform paragraph segmentation and chapter recognition on electronic PDF / XML files; (3) Perform text cleaning on the entire text; (4) Intelligent segmentation of long texts and adaptation to the context window limitation of LLM.

3. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The multi-agent collaboration module includes a document screening agent, a data extraction agent, a bias assessment agent, and a conflict arbitration agent.

4. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 3, characterized in that, The document screening agent is used to input the title, abstract, and full text of a document based on the user-preset PICOS inclusion / exclusion criteria, and then performs title / abstract screening and full-text screening on candidate documents, simulating the document screening process in the PRISMA flowchart. It also supports independent screening modes for single or dual users, and finally outputs the inclusion / exclusion judgment results and reasons, along with a confidence score. ; The data extraction agent takes full-text input and extracts key information about the predictive model research from the Methods and Results sections of the included literature. Finally, it outputs structured extraction results that strictly follow the preset JSON Schema, including basic research information, cohort characteristics, predictor variables, model construction, performance indicators and bias assessment dimensions. A backtracking mechanism is triggered when required fields are missing.

5. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 3, characterized in that, The bias assessment agent includes the full text of the literature and related extraction results, which are used to apply the PROBAST+AI tool to automatically assess the bias risk of the included studies. Finally, the bias risk rating and applicability evaluation of PROBAST+AI in four areas are output. The conflict arbitration agent receives the output results and confidence scores of each agent to monitor the consistency of the outputs of the above agents, and performs adjudication or marks manual review for the disagreement items. When multiple agents have inconsistent results in extracting the same field of the same document, the conflict arbitration process is initiated: conflict detection → confidence assessment → weighted adjudication → manual review marking, and finally outputs the consistent result after arbitration, including the manual review marking.

6. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 5, characterized in that, The conflict arbitration agent executes the conflict arbitration sub-process: S1.1 Conflict Detection: The central control module compares the outputs of each agent and identifies inconsistencies. S1.2 Confidence Assessment: Each agent attaches a confidence score to its output. ∈[0,1]; S1.3 Weighted Decision: The conflict arbitration agent uses confidence level as the weight and selects the result with the highest score as the final output. in, This is the output of the i-th agent. Score the confidence level of the i-th agent. For indicator functions, when The value is 1 when =r, otherwise it is 0; S1.4, Manual verification mark: When the difference between the highest confidence level and the second highest confidence level... If the value is less than 0.15, the field is marked as awaiting manual review and included in a dedicated manual review queue. S1.5 Review and Feedback: The results of manual review are fed back to each agent as new training samples, and continuous optimization is achieved through Few-shot / Prompt.

7. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The performance index normalization module automatically calculates the I² heterogeneity index based on the AUC values ​​and confidence intervals of each research report, as follows: Where Q is Cochran's Q statistic and df is the degrees of freedom; When I² > 90%, the system labels exhibit extremely high heterogeneity, and direct merging is not recommended. The system will then prompt the user to perform subgroup analysis or sensitivity analysis.

8. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The entity nodes of the knowledge graph construction module include research, models, population characteristics, performance indicators and bias ratings; the relation edges include semantic relations that have been developed, validated, reported AUC, and extracted from.

9. The automated system for evaluating a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The execution process of the central control module is as follows: S2.1 Receive the PICOS conditions and search query input by the user; S2.2, Call the document retrieval module to obtain candidate documents; S2.3, Call the document preprocessing module to parse the format; S2.4 Distribute tasks to the multi-agent collaboration module; S2.5 Summarize the output results of each agent; S2.6 Detect conflicts and trigger the conflict arbitration mechanism; S2.7 Call the performance index normalization module to unify terminology; S2.

8. Call the evidence summary module to generate a draft report; S2.

9. Call the knowledge graph construction module to establish source tracing associations.

10. The automated evaluation system for a multi-agent collaborative medical risk prediction model according to claim 1, characterized in that, The system also includes a schema for extracting information for prediction models, a multi-agent conflict arbitration mechanism, an automatic terminology normalization method for performance indicators, and a knowledge graph data tracing mechanism.

Images