Clinical scoring system based on large language model multi-agent and self-evolution method thereof

Abstract

The invention relates to the field of medical artificial intelligence, in particular to a clinical scoring system combining large language model multi-agents and machine learning and a self-evolution method of the clinical scoring system, and provides the clinical scoring system based on the large language model multi-agents and the self-evolution method of the clinical scoring system. The objective of the invention is to overcome the defects of data missing sensitivity, insufficient interpretability, weak dynamic updating capability and the like of an existing clinical scoring system. According to the system, through a multi-agent collaboration architecture, the natural language interaction capability of a large language model (LLM) and the precise prediction capability of a machine learning model are deeply fused, and a self-evolution mechanism is introduced to realize dynamic optimization, so that the reliability, flexibility and transparency of clinical decisions are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence, and in particular relates to a clinical scoring system and a self-evolution method thereof that combines a large language model multi-agent with machine learning. Background Art

[0002] Currently, clinical scoring systems are primarily divided into two categories: traditional expert consensus-based scoring systems and machine learning-based scoring systems. While traditional scoring systems, such as APACHE and SOFA, are simple and easy to use, their predictive accuracy is often limited by manually set weights and thresholds. With the advancement of machine learning technology, machine learning-based clinical scoring systems are gradually emerging. These systems, by learning from large amounts of clinical data, can provide more accurate predictions.

[0003] Meanwhile, the application of large language models (LLMs) in the medical field has also made significant progress in recent years. Through post-training methods or retrieval-augmented generation (RAG) technology, LLMs can acquire medical knowledge and provide a natural language interface, greatly improving the user experience of medical information acquisition and decision support.

[0004] However, current clinical scoring systems based on machine learning have the following obvious shortcomings:

[0005] -Limited flexibility and inability to effectively handle missing data. Although there are some methods to handle missing values, model performance will degrade significantly when the proportion of missing values ​​is large.

[0006] High barriers to use and development. Building machine learning models still requires advanced data science skills and knowledge, making it difficult for clinicians to master, which limits their widespread application.

[0007] -The model has poor interpretability. Understanding the model's decision-making process requires professional knowledge and is difficult to explain in natural language, which reduces clinicians' trust in the model.

[0008] Lack of dynamic self-update and real-time evolution capabilities. Once a model is established, it is difficult to adapt to changes in the distribution of clinical data. For example, when a new therapy leads to improved patient prognosis, the previously established prediction model may not be able to adapt to this change.

[0009] Current large-scale medical language models also have limitations. While they capture medical knowledge through post-training or RAG techniques, their essence is to predict the next token based on context using an autoregressive model. This approach relies heavily on the probability distribution between tokens in the original data, lacking true thinking and reasoning capabilities. It can only perform targeted restoration based on compressed parameter knowledge, is prone to "hallucinations," and cannot guarantee the accuracy and timeliness of the generated content.

[0010] Current general-purpose intelligent agent systems rely heavily on the model's own knowledge and code level. Because current generative AI models struggle to ensure output accuracy, they are less applicable to clinical scoring scenarios and struggle to meet the reliability requirements of medical scoring systems. Furthermore, the medical process requires high model interpretability, making it difficult for general-purpose intelligent agents to leverage existing tools to meet the high reliability, flexibility, and model interpretability requirements of medical scenarios. Summary of the Invention

[0011] The problem to be solved by this application is how to integrate large language models with machine learning technology end-to-end to build an intelligent clinical scoring system that has both natural language interaction capabilities and can accurately process clinical data. Specifically, this application aims to solve the following problems: 1. How to improve the flexibility and robustness of the clinical scoring system in the face of missing data; 2. How to lower the threshold for use and construction of the clinical scoring system to make it more friendly to clinicians; 3. How to improve the interpretability of the model to make the decision-making process more transparent; 4. How to achieve dynamic self-update and real-time evolution of the system so that it can adapt to changes in the distribution of clinical data. The purpose of the present invention is to provide a clinical scoring system based on a large language model and multiple agents, including:

[0012] User interaction layer, providing a natural language interface to receive user instructions;

[0013] The multi-agent layer includes data preprocessing agents, model building agents, model evaluation agents, and model interpretation agents, which collaborate to complete feature verification, model generation, performance evaluation, and result interpretation;

[0014] The machine learning model layer integrates multiple machine learning algorithms to generate personalized prediction models;

[0015] Data layer, storing clinical data and model parameters;

[0016] The self-evolution module optimizes prompt words, model parameters and tool combinations through log analysis to achieve dynamic updates.

[0017] In the system of the present invention, the self-evolution module includes:

[0018] Log file analyzer to extract error patterns and performance bottlenecks from general logs, event logs, and trace logs;

[0019] Prompt word optimizer, which optimizes natural language templates based on historical interaction records;

[0020] Model optimizer, which tunes hyperparameters via grid search and retrains the model.

[0021] In the system described herein, the multi-agent layer collaborates via magnetic single-group chat, employing the AutoGen framework for task allocation and process control. For example, after completing a performance evaluation, the model evaluation agent automatically sends the results to the model interpretation agent, triggering the SHAP value analysis process.

[0022] In the system described in the present invention, the model interpretation agent uses SHAP (SHapley AdditiveexPlanations) value technology to generate a feature contribution visualization report and output it in Markdown format.

[0023] The system of the present invention is characterized in that the dynamic feature detection module includes:

[0024] Feature input agent, verifies the validity of user input features;

[0025] Model generation agent, based on effective feature selection algorithm and generate personalized model;

[0026] Report the agent and generate an evaluation report including AUC, F1 score, and SHAP explanation.

[0027] The system described in the present invention is characterized in that the system integrates common machine learning models such as gradient boosting tree and XGBoost and data pre-processing technologies such as synthetic minority class oversampling technology.

[0028] The system of the present invention is characterized in that the user interaction layer supports triggering the following operations through natural language instructions:

[0029] Specify the feature generation model;

[0030] Input patient data to obtain prediction results and interpretation;

[0031] Query model performance metrics and feature importance.

[0032] The present invention also provides a self-evolution method for a clinical scoring system based on a large language model and multiple agents, characterized by comprising:

[0033] Analyze log files for errors and performance metrics;

[0034] Optimize the prompt word template to improve interaction stability;

[0035] Dynamically adjust machine learning model parameters and tool calling order;

[0036] Automatically collect new clinical data and update the model.

[0037] The method described in the present invention is characterized in that the self-evolution method defines a reward function through reinforcement learning to optimize the intelligent agent collaboration strategy.

[0038] The present invention provides a computer-readable storage medium, characterized in that it stores an execution program of a clinical scoring system based on a large language model and multiple agents.

[0039] The key innovations of this application are: 1. A multi-agent collaborative architecture is designed, which consists of a data preprocessing agent, a model building agent, a model evaluation agent, and a model interpretation agent. Each agent has a clear division of labor and works together; 2. An agent self-evolution mechanism based on self-reflection is introduced. When the system performance is poor or encounters difficulties, the evolution process can be automatically triggered, and the code, prompt words and tool combinations are optimized by analyzing the log context information; 3. A deep integration of large language models and machine learning models is achieved. The large language model is responsible for natural language interaction and knowledge extraction, and the machine learning model is responsible for accurate data modeling and prediction; 4. A set of dynamic data update and model adjustment mechanisms are developed, so that the system can automatically update and optimize according to the latest clinical data. 5. A dynamic feature detection and personalized model generation system is implemented, which can dynamically generate and evaluate prediction models based on user-specified features; 6. A patient personalized result interpretation system is developed, which can load the optimal model to make predictions for individual patients and provide detailed explanations and visual reports using SHAP values. This paper proposes a large language model (LLM)-based multi-agent clinical scoring system and its self-evolutionary approach, aiming to address the shortcomings of existing clinical scoring systems, such as sensitivity to missing data, insufficient interpretability, and weak dynamic update capabilities. Through a multi-agent collaborative architecture, the system deeply integrates the natural language interaction capabilities of a large language model (LLM) with the precise predictive capabilities of machine learning models. It also introduces a self-evolutionary mechanism for dynamic optimization, significantly improving the reliability, flexibility, and transparency of clinical decision-making.

[0040] This invention realizes the end-to-end fusion of natural language interaction and data modeling for the first time, solving the practical pain points in clinical decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 : System architecture diagram (showing the four-layer structure and module interaction);

[0042] Figure 2 : Flowchart of the self-evolution mechanism (log analysis → optimization → execution closed loop);

[0043] Figure 3 : Flowchart of dynamic feature detection and model generation;

[0044] Figure 4 : Flowchart of the patient results interpretation system. DETAILED DESCRIPTION

[0045] The present invention is described in detail below through examples. It is necessary to point out that the following examples are not to be construed as limiting the scope of protection of the present invention. If a person skilled in the art makes some non-essential improvements and adjustments to the present invention based on the above-mentioned invention content, they still fall within the scope of the present invention.

[0046] The clinical scoring system based on large language model multi-agent and its self-evolution method proposed in this application has an overall architecture divided into four layers: user interaction layer, multi-agent layer, machine learning model layer and data layer.

[0047] 1 Overall system architecture

[0048] The system consists of three core modules:

[0049] 1.1 Intelligent Agent Self-Evolution System

[0050] The system consists of three main components: a log file analyzer, a prompt word optimizer, and a model optimizer. The log file analyzer is responsible for reading and analyzing three types of files generated by the system: general logs, event logs, and trace logs, extracting key information for system optimization. The prompt word optimizer focuses on optimizing the prompt word templates used in the system, effectively improving system stability and interaction quality. The model optimizer is primarily responsible for automatically adjusting model parameters in data analysis tools by analyzing performance indicators to improve model performance.

[0051] The system leverages Microsoft's AutoGen framework to enable multi-agent collaboration, automatically analyzing errors, warnings, and performance metrics in log files and improving the existing system accordingly. This includes automatically acquiring the latest log files and extracting key information, analyzing error patterns and performance bottlenecks in the logs, generating optimization suggestions and improvement plans, and automatically executing system optimization operations, forming a complete self-evolutionary closed loop. This includes a complete iterative process from log collection to error pattern recognition to parameter optimization to model retraining to performance verification.

[0052] 1.2 Dynamic Feature Detection and Personalized Model Generation System

[0053] The system consists of four specialized agents, forming an efficient collaborative chain. The Feature Input Agent is responsible for receiving and processing feature requirements entered by users through a natural language interface. The Feature Analysis Agent is responsible for loading raw data and analyzing whether the user-entered features exist in the dataset, ensuring the validity of model building. The Model Generation Agent applies machine learning algorithms based on verified valid features to generate personalized predictive models. The Reporting Agent is responsible for generating detailed model reports based on model results or generating error reports if features are invalid.

[0054] The system is comprehensive in functionality, including dynamically reading feature lists from raw data files, rigorously checking whether user-entered features exist in the original data, generating personalized prediction models based on valid features, providing interpretable analysis of model predictions using SHAP values, and generating detailed model reports or error reports. The entire system utilizes a magnetic single-group chat format to organize agent collaboration, strictly adhering to the multi-agent collaboration model of the AutoGen framework to ensure seamless and efficient collaboration among agents.

[0055] 1.3 Patient Personalized Result Interpretation System

[0056] The system also consists of four specialized agents that form a complete analysis chain. The Data Input Agent is responsible for receiving patient data and validating its format and integrity to ensure that it meets model input requirements. The Prediction Agent is responsible for loading the system's top-performing prediction model and making accurate predictions based on the input patient data. The Interpretation Agent uses SHAP value technology to conduct in-depth analysis of the prediction results, revealing the contribution of each feature to the prediction. The Report Agent is responsible for integrating the analysis results and generating a structured Markdown-formatted analysis report for easy understanding and use by medical staff.

[0057] The system offers comprehensive functionality, including automatic loading of the optimal model from a model directory for prediction, providing detailed explanations of individual patient predictions using SHAP value technology, generating comprehensive reports containing data summaries, prediction results, SHAP interpretations, and medical explanations, and saving reports in Markdown format in a designated directory for easy access. The system also utilizes magnetic single-group chat to organize agent collaboration, enabling an end-to-end patient analysis process from data input to report generation.

[0058] 2. Agent Self-Evolution Mechanism

[0059] The system introduces a self-reflective agent self-evolution mechanism, implemented through a log file analysis agent. This component automatically reads and analyzes three types of log files generated by the system: general logs, event logs, and trace logs. It identifies errors, warnings, and performance bottlenecks, and extracts key information for system optimization.

[0060] The self-evolution process includes four key steps: machine learning code optimization, agent prompt word improvement, tool combination call optimization, and automatic updating of training data. In the machine learning code optimization step, the model optimizer component automatically adjusts model parameters and optimizes feature engineering methods by analyzing model performance indicators to improve prediction accuracy. It also uses model performance and operational efficiency as part of the reward function to constrain and improve the original machine learning code. In the agent prompt word improvement step, the prompt word optimization agent optimizes the prompt word template based on historical interaction records to improve the stability and quality of system responses. In the tool combination call optimization step, the system dynamically adjusts the tool call order and parameter settings based on log analysis results to improve workflow efficiency. In the automatic updating of training data, the system can collect new clinical data and retrain the model so that the model can adapt to dynamic changes in clinical data distribution.

[0061] 3 Dynamic Feature Detection and Personalized Model Generation

[0062] This system implements dynamic feature detection and personalized model generation functions. The main process includes five steps: feature input and verification, data preprocessing, model training and selection, model interpretation and report generation.

[0063] During the feature input and verification phase, users enter features of interest through a natural language interface. The feature analysis agent loads the raw data and checks whether the features exist. The system then determines whether there are sufficient valid features to build a predictive model. During data preprocessing, the system uses a median strategy to fill missing values, scientifically partitions the dataset into training, validation, and test sets, uses a normalizer to standardize features, and addresses class imbalance through synthetic minority oversampling.

[0064] In the model training and selection phase, the system trains various types of models, including logistic regression, random forest, gradient boosting tree, and XGBoost, uses the grid search method for hyperparameter optimization, and comprehensively evaluates model performance based on multiple indicators such as area under the curve, accuracy, and F1 score. Ultimately, the model with the best performance is selected as the prediction model. In the model interpretation phase, the system uses SHAP value technology to interpret the model prediction results, generate feature importance rankings and visualization charts, and create SHAP dependency graphs to intuitively display the relationship between features and prediction results. In the report generation phase, the system generates a detailed report containing model performance, feature importance, and visual explanations, and saves it in Markdown format for easy reading and sharing by medical staff.

[0065] 4. Patient personalized result interpretation system

[0066] This system realizes the individual patient analysis function. The main process includes four steps: model loading, patient data processing, prediction and interpretation, and report generation.

[0067] During the model loading phase, the Model Analyzer class automatically loads the best-performing prediction model from the model directory and retrieves the model's feature list and feature importance information to prepare for subsequent analysis. During the patient data processing phase, the Patient Data Processor class verifies that the patient data contains all necessary features and preprocesses the patient data to align its format with the model training data, ensuring accurate predictions.

[0068] During the prediction and interpretation phase, the system uses the loaded optimal model to predict patient data, calculates the SHAP value to explain the model's prediction for that patient, and generates a feature contribution visualization chart to intuitively demonstrate the degree and direction of each feature's influence on the prediction result. During the report generation phase, the report generator class creates a structured report that comprehensively includes a patient data summary, prediction results, SHAP interpretation, and a medically informed clinical explanation. The report is then saved in a designated directory for easy access and reference by physicians.

[0069] 5. User Interaction Methods

[0070] The system provides a natural language interactive interface, allowing users to perform various functions through conversation. Users can enter features of interest, and the system will automatically generate a personalized prediction model based on these features. Users can also enter patient data to obtain prediction results and detailed visual explanations. Users can query model performance and feature importance information at any time to understand the basis for the model's predictions. Users can also obtain clinical explanations and treatment recommendations based on medical knowledge to assist in medical decision-making. This natural language interaction method greatly reduces the threshold for using the system, allowing clinicians to fully utilize artificial intelligence technology to assist in clinical decision-making without having to master complex data science knowledge.

[0071] Beneficial effects

[0072] This application uses a deep fusion architecture of large language models and machine learning, as well as a self-reflective agent self-evolution mechanism, to achieve the following technical effects:

[0073] 1. Improved flexibility and robustness of the clinical scoring system:

[0074] Dynamic feature detection mechanism allows the system to build models based on available features, effectively dealing with data missing issues

[0075] Multi-model fusion strategy ensures reliable predictions under different data conditions

[0076] 2. Lowering the threshold for using and building clinical scoring systems:

[0077] Natural language interface eliminates the need for clinicians to master complex data science knowledge

[0078] Automated feature validation and model generation streamlines model building

[0079] 3. Improved model interpretability:

[0080] SHAP value explanation mechanism transforms complex machine learning decisions into intuitive visualizations

[0081] Structured Markdown reports make forecast results and interpretations easy to understand

[0082] Feature importance ranking and dependency graphs show the impact of each feature on the prediction

[0083] 4. Realize the dynamic self-update and real-time evolution of the system:

[0084] Log file analysis agents can automatically analyze system logs and identify improvement opportunities

[0085] The prompt word optimization agent and model optimization agent can optimize the original system based on the analysis results

[0086] Dynamic model generation mechanism enables the system to adapt to different feature combinations and data distributions

[0087] 5. Alleviates the “hallucination” problem of large language models:

[0088] Machine learning models based on real data provide accurate prediction results, and realistic structured reports based on SHAP value explanations ensure the transparency and reliability of model decisions.

[0089] The present invention uses a collaborative mechanism between LLM and machine learning. LLM is responsible for parsing natural language instructions, and the machine learning model is modeled based on dynamically verified features. The two achieve functional complementarity through the division of labor among intelligent agents. The traditional view is that the "black box" nature of LLM conflicts with the interpretability requirements of medical scenarios. However, the present invention combines SHAP value interpretation with text reports generated by LLM to prove that the two can work together to meet clinical credibility requirements. The self-evolution-driven optimization of the present invention automatically identifies performance bottlenecks through log analysis, triggering model parameter adjustments and prompt word iterations, rather than relying on manual rules.

[0090] The present invention achieves "parallel" collaboration through a multi-agent architecture. For example, the model interpretation agent simultaneously calls SHAP values ​​and LLM to generate clinical recommendations, solving the following technical difficulties, such as real-time mapping of the text output of LLM with the structured input of the machine learning model; and designing a verification mechanism to prevent the "hallucination" of LLM from contaminating the machine learning model input.

Claims

1. A clinical scoring system based on a large language model and multiple agents, characterized by: include: User interaction layer, providing a natural language interface to receive user instructions; The multi-agent layer includes data preprocessing agents, model building agents, model evaluation agents, and model interpretation agents, which collaborate to complete feature verification, model generation, performance evaluation, and result interpretation; The machine learning model layer integrates multiple machine learning algorithms to generate personalized prediction models; Data layer, storing clinical data and model parameters; The self-evolution module optimizes prompt words, model parameters and tool combinations through log analysis to achieve dynamic updates.

2. The system according to claim 1, wherein: The self-evolution module includes: Log file analyzer to extract error patterns and performance bottlenecks from general logs, event logs, and trace logs; Prompt word optimizer, which optimizes natural language templates based on historical interaction records; Model optimizer, which tunes hyperparameters via grid search and retrains the model.

3. The system according to claim 1, wherein: The multi-agent layer collaborates through magnetic single-group chat and uses the AutoGen framework to achieve task allocation and process control.

4. The system according to claim 1, wherein: The model explanation agent uses SHAP value technology to generate a feature contribution visualization report and outputs it in Markdown format.

5. The system according to claim 1, wherein: The dynamic feature detection module includes: Feature input agent, verifies the validity of user input features; Model generation agent, based on effective feature selection algorithm and generate personalized model; Report the agent and generate an evaluation report including AUC, F1 score, and SHAP explanation.

6. The system according to claim 1, wherein: The training includes various types of models including logistic regression, random forest, gradient boosting tree and XGBoost.

7. The system according to claim 1, wherein: The user interaction layer supports triggering the following operations through natural language commands: Specify the feature generation model; Input patient data to obtain prediction results and interpretation; Query model performance metrics and feature importance.

8. A self-evolution method based on the system of claim 1, characterized in that: include: Analyze log files for errors and performance metrics; Optimize the prompt word template to improve interaction stability; Dynamically adjust machine learning model parameters and tool calling order; Automatically collect new clinical data and update the model.

9. The method according to claim 8, characterized in that The self-evolution method defines the reward function through reinforcement learning and optimizes the agent collaboration strategy.

10. A computer-readable storage medium, characterized in that The device stores an execution program of the system according to any one of claims 1 to 8.

Images