Medical language model evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a medical language model evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of medical health, and the method is operated in an isolated containerization environment, and achieves the safety compliance of the whole evaluation process; the performance evaluation of the medical ability is carried out by calling OpenCompass, so that the upgrading from'knowledge evaluation 'to'clinical ability evaluation' is realized; through safety assessment based on the preset clinical safety and ethical rule base, automatic clinical safety and ethical risk quantification is realized; and the target evaluation configuration file is automatically obtained based on the target evaluation template, and the comprehensive evaluation report is finally generated, so that the operation threshold of the user is reduced, and the automation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical health, and in particular to a medical language model evaluation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] The rapid development of LLM (Large Language Model) has shown great potential in professional fields such as medicine, finance and education. Comprehensive, objective and reproducible evaluation of LLM is an important cornerstone for AI (Artificial Intelligence) safety and development. The evaluation of LLM mainly relies on open-source evaluation frameworks. OpenCompass ("Compass") as the current mainstream tool provides a standardized model evaluation process: 1. Rich model and dataset support: It supports hundreds of datasets (such as MMLU (Massive Multitask Language Understanding), C-Eval, GSM8K (Grade School Math 8K, Basic School Math 8K)) and multiple model access methods (such as HuggingFace local model, API (Application Programming Interface) model).

[0003] 2. Configuration-driven evaluation: Users define the model, dataset and evaluation method to be evaluated by writing a Python configuration file (Config).

[0004] 3. Standardized execution process: The evaluation is divided into two stages of "inference" and "evaluation". The inference stage allows the model to run the dataset in batches and generate answers; the evaluation stage scores the answers according to standard answers or specific rules (such as "LLM-as-Judge", i.e. LLM as a judge).

[0005] 4. Result summary: Finally, a table containing various indicators (such as accuracy Accuracy, F1 score F1-score) is generated for horizontal comparison between models.

[0006] In the medical scenario, researchers attempt to use the existing OpenCompass scheme for model evaluation, which usually includes the following two ways: Method 1, using general medical benchmarks: configure OpenCompass to run public medical knowledge question answering sets such as MedQA, PubMedQA, MedMCQA, etc. These data sets mainly test the model's medical knowledge memory ability in the form of multiple choice questions.

[0007] Method 2, human-assisted evaluation: AI engineers use OpenCompass to run the model, export the model's raw answers (such as diagnosis recommendations for a simulated case), and then organize a team of expert doctors to manually score these answers.

[0008] However, when directly applying general evaluation frameworks such as OpenCompass to the medical field, which is a high-risk and highly regulated special field, there are problems such as evaluation dimensions not matching medical core capabilities, clinical safety and ethical risks, data security and compliance risks, high operation threshold, and low automation efficiency, making it impossible to meet the evaluation needs of the medical field. SUMMARY

[0009] The purpose of the present application is to provide a medical language model evaluation method, device, electronic equipment and storage medium, to at least solve one of the above problems.

[0010] In a first aspect, the present application provides a medical language model evaluation method applied in an isolated containerized environment; the medical language model evaluation method comprises: parsing the received evaluation request to obtain a to-be-evaluated model identifier and a target evaluation template, the target evaluation template being one of a plurality of preset medical capability evaluation templates; determining a target data set identifier and a target evaluation configuration file according to the target evaluation template; calling OpenCompass to perform performance evaluation of the medical capability of a target medical language model corresponding to the to-be-evaluated model identifier according to the target evaluation configuration file and a target data set corresponding to the target data set identifier, to obtain a performance evaluation result and raw output content of the target medical language model; calling a preset clinical safety and ethics rule library to perform safety evaluation on the raw output content, to obtain a safety evaluation result; generating a comprehensive evaluation report according to the performance evaluation result and the safety evaluation result.

[0011] In an optional implementation, according to the target evaluation template, the target data set identifier and the target evaluation configuration file are determined, comprising: According to the mapping relationship between the preset evaluation template and the data set, and the configuration file, the target data set identifier and the target evaluation configuration file corresponding to the target evaluation template are determined; wherein the mapping relationship between the evaluation template and the data set, and the configuration file includes a group of data sets and their evaluation configuration files mapped by each medical ability evaluation template.

[0012] In an optional implementation, according to the target evaluation configuration file and the target data set corresponding to the target data set identifier, the OpenCompass is called to perform performance evaluation of the medical ability of the target medical language model corresponding to the to-be-evaluated model identifier, to obtain a performance evaluation result and original output content of the target medical language model, including: According to the to-be-evaluated model identifier and the target data set identifier, the target medical language model and the target data set are pulled; According to the target evaluation configuration file and the target data set, the run.py executor of OpenCompass is called to perform batch inference on the target medical language model, to obtain a performance evaluation result and original output content; wherein the performance evaluation result includes an accuracy rate and scores of a plurality of preset medical evaluation dimensions corresponding to the target data set.

[0013] In an optional implementation, the preset clinical safety and ethics rule library is called to perform safety evaluation on the original output content, to obtain a safety evaluation result, including: The clinical safety and ethics rule library is called to perform cross-validation and pattern matching on the original output content, to obtain a safety evaluation result; wherein the clinical safety and ethics rule library includes one or more of a contraindication and drug interaction database, a clinical guideline rule library, a symptom "critical value" rule library, an ethics bottom line data set, and a special disease gold standard data set, and the safety evaluation result includes a number of high-risk errors and a clinical safety score.

[0014] In an optional implementation, according to the performance evaluation result and the safety evaluation result, a comprehensive evaluation report is generated, including: The performance evaluation result and the safety evaluation result are summarized and visualized to obtain a comprehensive evaluation report, and the comprehensive evaluation report includes a comprehensive conclusion, a medical ability radar chart, and a high-risk error list, and the medical ability radar chart is used to display evaluation scores of a plurality of preset medical evaluation dimensions.

[0015] In an optional implementation, the plurality of preset medical evaluation dimensions include a plurality of medical knowledge question and answer, medical reasoning, medical safety, ethics, medical language understanding, and medical language generation.

[0016] In an optional implementation, the medical language model evaluation method further includes: In the process of processing the evaluation request, a complete operation audit log is recorded.

[0017] In a second aspect, the present application provides a medical language model evaluation device applied to an isolated containerized environment; the medical language model evaluation device comprises: A request analysis module is configured to parse a to-be-evaluated model identifier and a target evaluation template from the received evaluation request, the target evaluation template being one of a plurality of preset medical capability evaluation templates; A template matching module is configured to determine a target data set identifier and a target evaluation configuration file according to the target evaluation template; A performance evaluation module is configured to call OpenCompass to perform performance evaluation of a medical capability of a target medical language model corresponding to the to-be-evaluated model identifier according to the target evaluation configuration file and a target data set corresponding to the target data set identifier, and obtain a performance evaluation result and original output content of the target medical language model; A safety evaluation module is configured to call a preset clinical safety and ethics rule library to perform safety evaluation on the original output content, and obtain a safety evaluation result; A report generation module is configured to generate a comprehensive evaluation report according to the performance evaluation result and the safety evaluation result.

[0018] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, and the processor executing the computer program to implement the medical language model evaluation method of any one of the preceding embodiments.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, and the computer program being executed by a processor to perform the medical language model evaluation method of any one of the preceding embodiments.

[0020] The medical language model evaluation method, apparatus, electronic device, and storage medium provided by this invention are applied in an isolated containerized environment. The method includes: parsing a received evaluation request to obtain an identifier for the model to be evaluated and a target evaluation template, where the target evaluation template is one of several preset medical capability evaluation templates; determining a target dataset identifier and a target evaluation configuration file based on the target evaluation template; calling OpenCompass to perform a performance evaluation of the target medical language model corresponding to the identifier for the model to be evaluated, based on the target dataset corresponding to the target dataset and the target evaluation configuration file and the identifier, to obtain the performance evaluation result and the original output content of the target medical language model; calling a preset clinical safety and ethics rule base to perform a safety evaluation of the original output content, to obtain a safety evaluation result; and generating a comprehensive evaluation report based on the performance evaluation result and the safety evaluation result. Running in an isolated containerized environment ensures security and compliance throughout the entire evaluation process; by calling OpenCompass for performance evaluation of medical capabilities, it upgrades from "knowledge evaluation" to "clinical capability evaluation"; through security evaluation based on a pre-set clinical safety and ethics rule base, it achieves automated quantification of clinical safety and ethical risks; and by automatically obtaining target evaluation configuration files based on target evaluation templates, it ultimately generates a comprehensive evaluation report, reducing the user's operational threshold and improving automation efficiency. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a medical language model evaluation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an orchestration management system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a medical language model evaluation device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the present application will be described clearly and completely below in connection with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0024] It is found through research that when OpenCompass is directly applied to the medical scene, the following serious defects exist: 1. Serious disconnection between evaluation dimensions and medical core capabilities: The core of the medical scene is "clinical decision-making" rather than "knowledge contest". MedQA and other multiple-choice benchmarks can only evaluate the model "knows what" (such as the typical symptoms of a certain disease), but cannot evaluate its "how to make decisions". It cannot measure the key capabilities of the model in the real clinical workflow, such as the logic of differential diagnosis, the ability of multi-round inquiry, the reasoning ability of interpreting complex test reports, and the comprehensive ability of developing treatment plans.

[0025] 2. Evaluation gap of clinical safety and ethical risks: The output of the general framework is the "accuracy" and other performance indicators, which is far from enough in medicine. The existing technology completely lacks automated "safety" and "ethics" quantitative evaluation. For example, does the model suggest a "contraindicated" drug? Does it give a diagnosis and treatment suggestion with inducement or irresponsibility in the simulated consultation? These fatal defects are completely ignored in the existing scheme.

[0026] 3. Huge hidden danger of data security and compliance: If medical evaluation is to be truly valuable, it must use "disease-specific datasets" or "de-identified RWD (Real-World Data)". These data are extremely sensitive patient PHI (Protected Health Information), which are strictly protected by security regulations. OpenCompass, as an open source tool, does not consider running in a private, isolated "Safe Haven" environment in its standard process design, lacks necessary access control, data non-deposition, operation audit, and other compliance mechanisms, and has a serious risk of data leakage.

[0027] 4. High operation threshold and low automation efficiency: The dimensions of medical evaluation (such as safety, compliance, professionalism) are much more complex than general evaluation. If you want to use OpenCompass to evaluate the "diabetes management ability" of a model, you need an AI engineer to manually filter ten relevant data sets and write a complex Python Config file for each data set. This process not only requires deep programming skills, but also requires medical knowledge. Medical professionals (doctors, researchers) cannot operate independently, resulting in long evaluation cycles, high labor costs, and inability to support rapid model iteration.

[0028] Based on this, the embodiment of the application provides a medical language model evaluation method and device, electronic equipment and storage medium, which adopts a model evaluation scheme designed for the medical field, automatic and safe, which can overcome the above-mentioned defects of the prior art in the application of OpenCompass in the medical field.

[0029] The embodiment of the application aims to achieve the following specific objectives: 1. Realize the safety and compliance of the whole evaluation process: provide a safe and controllable execution environment, encapsulate OpenCompass and its evaluation process as a whole, and ensure that it can run in the private intranet ("safe room") of a hospital or research institution. All evaluation data (especially sensitive disease data) do not leave the safety boundary in the whole life cycle (loading, reasoning, evaluation), and audit logs are provided to meet the strict requirements of medical data compliance.

[0030] 2. Upgrade from "knowledge evaluation" to "clinical ability evaluation": get rid of the dependence on MedQA and other shallow multiple-choice questions. The embodiment of the application aims to build an automated workflow that can call OpenCompass to perform more complex "multi-round dynamic evaluation sets" that simulate real clinical paths, thereby evaluating the comprehensive decision-making ability of the model in the "diagnosis-control target-treatment-health management" whole process.

[0031] 3. Establish an automated clinical safety and ethical risk quantification mechanism: the core goal of the application is to quantify the "safety" of the model. The scheme will automatically identify and quantify high-risk errors in the model output, such as "violation of diagnosis and treatment guidelines", "recommendation of contraindicated drugs", "ignorance of critical values", etc., and generate a "clinical safety score (Clinical Safety Score)" independent of accuracy.

[0032] 4. Lower the threshold for medical experts to use, achieve end-to-end automation: eliminate complex Python configuration. The embodiment of the application provides a "templated" configuration interface for medical tasks. Medical professionals only need to select an "evaluation template" (such as "hypertension evaluation" and "diabetes evaluation") and the model to be evaluated, and all OpenCompass configuration files can be automatically generated. One key to start, execute, summarize, and generate customized reports for medical dimensions.

[0033] In order to facilitate the understanding of the present embodiment, first, a medical language model evaluation method disclosed by the present embodiment is introduced in detail.

[0034] The embodiment of the application provides a medical language model evaluation method, which is applied to an isolated containerized environment, and the method can be executed by an electronic device with data processing capability. The method is an Opencompass open source tool-based model evaluation automation scheme in a medical scenario, which can be used for evaluation of AI base (special disease large model). Referring to the flowchart of a medical language model evaluation method shown in the figure, the method mainly includes the following steps S110 to S150: Figure 1 Step S110, the model to be evaluated and the target evaluation template are parsed from the received evaluation request. The target evaluation template is one of the preset multiple medical capability evaluation templates.

[0035] In the present embodiment, multiple medical capability evaluation templates are preset, such as "diagnostic capability evaluation", "control target capability evaluation", "ethical compliance evaluation", "patient communication capability evaluation", etc. Users such as doctors can select the model to be evaluated and the target evaluation template on the user interface, and then click the evaluation button to trigger the evaluation request. The model to be evaluated is the unique identifier of the medical language model (hereinafter referred to as the target medical language model) that needs to be evaluated. The medical language model is an LLM in the medical field.

[0036] Step S120, according to the target evaluation template, determine the target data set identifier and the target evaluation configuration file.

[0037] In the present embodiment, in order to realize "zero code" evaluation, multiple medical capability evaluation templates are preset, and the user only needs to select the required medical capability evaluation template to automatically generate the required evaluation configuration file. The target data set corresponding to the target evaluation template can be one or more data sets.

[0038] ​In a possible implementation, a mapping relationship between the evaluation template and the data set and the configuration file is preset, and the mapping relationship between the evaluation template and the data set and the configuration file includes a group of data sets mapped by each medical capability evaluation template and an evaluation configuration file of the group of data sets. Based on this, the target data set identifier corresponding to the target evaluation template and the target evaluation configuration file can be determined according to the preset mapping relationship between the evaluation template and the data set and the configuration file. In this way, the target data set identifier corresponding to the target evaluation template and the target evaluation configuration file can be directly found, which is convenient and fast.

[0039] In another possible implementation, each medical capability evaluation template maps a group of data sets, and each data set is provided with an evaluation configuration file thereof. Based on this, the target data set identifier of the group of data sets mapped by the target evaluation template can be determined first, and then the target evaluation configuration file can be generated by aggregating the evaluation configuration files of each data set in the group of data sets. In this way, when the medical capability evaluation template is updated (for example, added, modified, etc.), only the mapping relationship between the medical capability evaluation template and the data set needs to be updated, and the evaluation configuration file corresponding to the medical capability evaluation template does not need to be updated at the same time, so that the maintenance operation can be reduced, and the consistency of data can be ensured.

[0040] In step S130, the OpenCompass is called to perform performance evaluation of the target medical language model corresponding to the to-be-evaluated model identifier according to the target data set corresponding to the target evaluation configuration file and the target data set identifier, to obtain a performance evaluation result and original output content of the target medical language model.

[0041] In this embodiment, the OpenCompass tool chain, the evaluation data set, and all to-be-evaluated models are encapsulated in a sealed containerized environment, which can be deployed behind the internal network firewall of a medical institution. All evaluation tasks are executed in a sealed computing instance, so that data is not landed and not leaked.

[0042] In some possible embodiments, the above step S130 can include: pulling the target medical language model and the target data set according to the to-be-evaluated model identifier and the target data set identifier; performing batch inference on the target medical language model according to the target evaluation configuration file and the target data set by calling the run.py executor of the OpenCompass, to obtain a performance evaluation result and original output content; and wherein the performance evaluation result includes an accuracy rate and scores of a plurality of preset medical evaluation dimensions corresponding to the target data set.

[0043] In this embodiment, OpenCompass can be called to perform a complex "multi-round dynamic evaluation set" that simulates a real clinical path, so as to evaluate the comprehensive decision-making ability of the model in the whole process of "diagnosis-control target-treatment-health management". Based on this, the above target data set can include a clinical scenario containing patient basic information, initial performance, and timeline dynamic evolution, which includes at least three evaluation stages: information collection and diagnosis stage, examination and treatment stage, follow-up management and adjustment stage. When batch reasoning is performed, for each data set in the target data set, the current scene information can be input into the target medical language model for each evaluation stage, the decision response of the target medical language model is received, the next stage scene information is dynamically generated based on the decision response and the preset clinical logic, including patient state change, examination result, treatment response and complication, and multi-round dynamic interaction is realized; then the clinical scene is controlled to evolve in the order of timeline and decision branch, and the complete output of the target medical language model is recorded at each decision point, including diagnosis reasoning, examination selection, treatment plan, patient communication content and follow-up plan, so as to realize continuous evaluation. Then the accuracy rate can be evaluated, and multi-dimensional ability quantitative evaluation can be performed on each data set; wherein the accuracy rate can be evaluated based on the consistency of the model output result and the preset standard result; the multi-dimensional ability quantitative evaluation can include automatic scoring of the decision of the target medical language model at each stage based on a plurality of pre-defined medical evaluation dimensions, to obtain the score of each data set in each medical evaluation dimension.

[0044] Further, when performing multi-round dynamic interaction, a decision time limit can be set to simulate the time pressure of the real clinical environment. The scene evolution path can be dynamically adjusted according to the quality of the model decision, including: when the model decision conforms to the best practice, evolving towards the healing direction; when the model decision has defects, generating corresponding complication or disease deterioration scenarios; when the model requires additional information, providing an information acquisition mechanism consistent with clinical practice. In this way, the comprehensive decision-making and coping ability of the model under pressure is fully evaluated.

[0045] Step S140, calling a preset clinical safety and ethics rule library to perform safety evaluation on the original output content to obtain a safety evaluation result.

[0046] The clinical safety and ethics rule library is preset in the embodiment, and the original output content generated by the model can be automatically evaluated for safety to realize safety evaluation. In some possible embodiments, the step S140 can include: calling the clinical safety and ethics rule library to cross-verify and pattern-match the original output content to obtain a safety evaluation result; wherein the clinical safety and ethics rule library includes one or more of a contraindication and drug interaction database, a clinical guideline rule library, a symptom "critical value" rule library, an ethics bottom line data set, and a special disease gold standard data set, and the safety evaluation result includes a number of high-risk errors and a clinical safety score. The special disease gold standard data set refers to a reference data set that is considered the most accurate and reliable in the research, diagnosis or treatment of a specific disease.

[0047] The cross-verification refers to a process of mutual comparison and verification of the same information from multiple independent sources or dimensions. In the embodiment, the cross-verification can include multi-rule source verification, context consistency check, expert knowledge alignment, and time dimension verification. The multi-rule source verification can be a comparison of the same clinical advice output by the model with a drug interaction library, a contraindication database, a diagnosis and treatment guideline standard, an ethics specification text, and other multiple independent rule sources. The context consistency check can be a comparison of whether the advice given by the model at different stages of the same case is self-consistent (for example, whether there is a logical conflict between the previous diagnosis and the subsequent treatment plan). The expert knowledge alignment can be a cross-verification of the model output with multiple authoritative clinical guidelines (such as the American Heart Association Guidelines and the Chinese Diabetes Prevention Guidelines). The time dimension verification can be a check of whether the advice is consistent with the development law of the disease course (for example, whether the acute phase treatment and the recovery period management are reasonably connected). The pattern matching refers to the use of a corresponding matching pattern when matching with the rules. Different rules can adapt to different matching patterns.

[0048] The cross-verification provides a wide range of verification to ensure that the advice is reasonable and reliable in multiple dimensions. The pattern matching provides a deep screening to accurately identify high-risk errors. The combination of the two forms a three-dimensional safety filter net. This double mechanism ensures that both common risk patterns can be efficiently captured and complex clinical decisions can be rigorously reviewed in multiple dimensions.

[0049] In step S150, a comprehensive evaluation report is generated according to the performance evaluation result and the safety evaluation result.

[0050] The performance evaluation result and the safety evaluation result can be summarized and visualized to obtain a comprehensive evaluation report, which can include a comprehensive conclusion, a medical ability radar chart and a high-risk error list, etc. The medical ability radar chart is used to display the evaluation scores of multiple preset medical evaluation dimensions. The comprehensive conclusion can include a text conclusion output by a large language model, evaluation scores of medical evaluation dimensions and a comprehensive score, etc. The evaluation score of the medical evaluation dimension can be an average score (e.g. 100 points) of all data sets in the corresponding medical evaluation dimension. The comprehensive score can be obtained by weighting and summing the evaluation scores of each medical evaluation dimension based on preset medical evaluation dimension weights.

[0051] Optionally, the multiple preset medical evaluation dimensions can include multiple of medical knowledge question answering, medical reasoning, medical safety, ethics, medical language understanding and medical language generation. By integrating these dimensions, it can not only determine whether the model's answer is correct, but also evaluate whether it is safe and usable in real clinical scenarios. By generating a medical ability radar chart, a precise direction can be provided for model optimization. For example, a certain model may have a high score in knowledge question answering, but perform poorly in the safety dimension, indicating that it needs to strengthen its risk identification ability to avoid the situation of "correct knowledge but dangerous application".

[0052] Further, the medical language model evaluation method can further include recording complete operation audit logs during processing of the evaluation request. This detailed recording of the execution process from request parsing, template matching to each evaluation step makes the evaluation no longer a "black box", enhancing the credibility and reproducibility of the results. This comprehensive audit mechanism transforms the one-time evaluation activity into an auditable, analyzable and iterative standardized process, significantly improving the rigor and long-term value of the evaluation system.

[0053] The medical language model evaluation method provided by the embodiments of the present application runs in an isolated containerized environment, achieving safety compliance of the entire evaluation process; by calling OpenCompass for performance evaluation of medical ability, it realizes the upgrade from "knowledge evaluation" to "clinical ability evaluation"; by safety evaluation based on a preset clinical safety and ethics rule library, it realizes automatic quantification of clinical safety and ethics risks; based on the target evaluation template, the target evaluation configuration file is automatically obtained, and finally the comprehensive evaluation report is generated, reducing the operation threshold of the user and improving the automation efficiency.

[0054] The medical language model evaluation method can be implemented by an automated and safe orchestration management system constructed on the upper layer of OpenCompass, which is dedicated to the medical scene. As shown in Figure 2 The orchestration management system includes the following core modules: 1. Trusted Secure Space & Data Management Module 201: This module is the security foundation of the solution. It encapsulates the OpenCompass toolchain, evaluation datasets, models to be evaluated, and all modules of the embodiments of the invention in an isolated containerized environment. This environment is deployed behind the medical institution's internal network firewall, and all evaluation tasks are executed in isolated computing instances, ensuring that data is not landed and not leaked, and recording complete operation audit logs.

[0055] 2. Medical Capability Evaluation Template & Configuration Auto-generation Module 202: This module is an interactive portal for medical professionals, enabling "zero-code" evaluation. It preinstalls a rich set of medical capability evaluation templates (such as "diagnostic capability evaluation", "control target capability evaluation", "ethical compliance evaluation", "patient communication capability evaluation"). Each template is mapped to a carefully selected set of OpenCompass datasets (including public medical sets and private disease sets) and their complex evaluation configuration files in the background. Users only need to select the template on the interface, and this module will automatically generate all the required Python Config files for OpenCompass.

[0056] 3. Clinical Safety & Ethical Rules Database Module 203: This module is the core of the safety evaluation implementation. It is an "fact-checking" database independent of OpenCompass, containing: Contraindications and drug interaction database: such as (aspirin -> gastric ulcer patient -> contraindication); Clinical guideline rule base: such as (mild infection -> should not use -> high-level antibiotics); Symptom "critical value" rule base: such as (chest pain -> possible heart attack); Ethical bottom line dataset: such as ("Can I take medicine by myself?" -> "Please consult a doctor"); Disease-specific gold standard dataset, manually annotated by medical experts.

[0057] 4. Automated Orchestration & OpenCompass Invocation Engine 204: This module is the execution center of the system. It receives the templated request from the Medical Capability Evaluation Template & Configuration Auto-generation Module 202, automatically pulls the model and data from the secure environment of the Trusted Secure Space & Data Management Module 201, and invokes the OpenCompass run.py executor to start batch inference. After OpenCompass completes the "evaluation" step (generates accuracy), this engine will immediately start the next step.

[0058] 5. Clinical safety and risk quantification analysis module 205: This module is triggered after the OpenCompass evaluation is completed. It extracts the raw output of the model (e.g., treatment plan text) and calls the clinical safety and ethics rule base of the clinical safety and ethics rule base module 203 for cross-validation and pattern matching. It automatically parses the model answer, quantifies the number of "high-risk errors" (e.g., the number of contraindication violations, the number of red flag symptom omissions), and generates a "clinical safety score" according to the severity.

[0059] 6. Medical dimension customization report generation module 206: This module is responsible for summarizing the "performance score" of OpenCompass (e.g., MedQA accuracy) and the "clinical safety score" of the clinical safety and risk quantification analysis module 205. It automatically generates a comprehensive report that is easy for medical decision makers to understand, including a "medical capability radar chart" (showing knowledge, reasoning, safety, ethics, etc. dimensions) and a detailed "critical error list", providing key decision-making basis for clinical application of the model.

[0060] Corresponding to the medical language model evaluation method described above, the embodiment of the present application also provides a medical language model evaluation device, which is applied to an isolated containerized environment. Referring to Figure 3 The structure diagram of a medical language model evaluation device is shown in the figure. The device comprises: The request analysis module 301 is used to parse the received evaluation request to obtain the to-be-evaluated model identifier and the target evaluation template, and the target evaluation template is one of the preset plurality of medical capability evaluation templates; The template matching module 302 is used to determine the target data set identifier and the target evaluation configuration file according to the target evaluation template; The performance evaluation module 303 is used to call OpenCompass to perform performance evaluation of the medical capability of the target medical language model corresponding to the to-be-evaluated model identifier according to the target evaluation configuration file and the target data set corresponding to the target data set identifier, to obtain the performance evaluation result and the raw output content of the target medical language model; The safety evaluation module 304 is used to call the preset clinical safety and ethics rule base to perform safety evaluation on the raw output content to obtain the safety evaluation result; The report generation module 305 is used to generate a comprehensive evaluation report according to the performance evaluation result and the safety evaluation result.

[0061] The medical language model evaluation device provided by the embodiment of the present application runs in an isolated containerized environment, realizes safe compliance of the whole evaluation process, realizes upgrading from 'knowledge evaluation' to 'clinical ability evaluation' by calling OpenCompass to perform performance evaluation of medical ability, realizes automatic quantitative evaluation of clinical safety and ethics risk based on a preset clinical safety and ethics rule library, automatically obtains a target evaluation configuration file based on a target evaluation template, and finally generates a comprehensive evaluation report, thereby reducing the operation threshold of the user and improving the automation efficiency.

[0062] Further, the template matching module 302 is specifically configured to: determine a target data set identifier and a target evaluation configuration file corresponding to a target evaluation template according to a mapping relationship between the preset evaluation template and the data set and the configuration file, wherein the mapping relationship between the evaluation template and the data set and the configuration file includes a group of data sets and their evaluation configuration files mapped by each medical ability evaluation template.

[0063] Further, the performance evaluation module 303 is specifically configured to: pull a target medical language model and a target data set according to the to-be-evaluated model identifier and the target data set identifier; call a run.py executor of OpenCompass to perform batch inference on the target medical language model according to the target evaluation configuration file and the target data set, and obtain a performance evaluation result and original output content; wherein the performance evaluation result includes an accuracy rate and scores of a plurality of preset medical evaluation dimensions corresponding to the target data set.

[0064] Further, the safety evaluation module 304 is specifically configured to: call a clinical safety and ethics rule library to cross-verify and pattern-match the original output content, and obtain a safety evaluation result; wherein the clinical safety and ethics rule library includes one or more of a contraindication and drug interaction database, a clinical guideline rule library, a symptom 'critical value' rule library, an ethics bottom line data set, and a special disease gold standard data set, and the safety evaluation result includes a number of high-risk errors and a clinical safety score.

[0065] Further, the report generation module 305 is specifically configured to: summarize and visually process the performance evaluation result and the safety evaluation result, and obtain a comprehensive evaluation report, wherein the comprehensive evaluation report includes a comprehensive conclusion, a medical ability radar chart, and a high-risk error list, and the medical ability radar chart is used to display evaluation scores of a plurality of preset medical evaluation dimensions.

[0066] Further, the plurality of preset medical evaluation dimensions include a plurality of medical knowledge question and answer, medical reasoning, medical safety, ethics, medical language understanding, and medical language generation.

[0067] Further, the medical language model evaluation device further includes: The log recording module is configured to record a complete operation audit log during processing of the evaluation request.

[0068] The medical language model evaluation device provided by the embodiment has the same implementation principle and technical effects as the medical language model evaluation method, and for brevity, the part of the medical language model evaluation device embodiment not mentioned can refer to the corresponding content in the medical language model evaluation method.

[0069] As shown in Figure 4 The electronic device 400 provided by the embodiment of the present application includes a processor 401, a memory 402 and a bus. The memory 402 stores a computer program that can run on the processor 401. When the electronic device 400 runs, the processor 401 communicates with the memory 402 through the bus. The processor 401 executes the computer program to implement the medical language model evaluation method described above.

[0070] Specifically, the memory 402 and the processor 401 can be general memory and processor, which are not specifically limited here.

[0071] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is run by a processor to execute the medical language model evaluation method in the foregoing method embodiment. The computer readable storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0072] The term "and / or" in the present document is only used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent three cases of existence of A alone, existence of A and B simultaneously, and existence of B alone. In addition, the term "at least one" in the present document means any one of a plurality of or any combination of at least two of a plurality of, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0073] In all examples shown and described herein, any specific value should be interpreted as merely exemplary and not as a limitation, and thus other examples of the example embodiments can have different values.

[0074] The computer program product of the present application can be a computer program embodied on a storage medium. Such storage medium can be a floppy disk, a hard disk, a CD-ROM, a RAM, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory. Such computer program product can also be a download, which can be stored on or transferred from a server and downloaded via a data network.

[0075] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. The described device embodiments are merely schematic, for example, the division of the modules is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0076] The module explained as a separate component can or can not be physically separate, and the component shown as a module can or can not be a physical module, that is, can be located in one place, or can be distributed to a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0077] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can be physically present alone, or two or more modules can be integrated into one module.

[0078] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating medical language models, characterized in that, Applied to isolated, containerized environments; the medical language model evaluation method includes: The identification of the model to be evaluated and the target evaluation template are obtained from the received evaluation request. The target evaluation template is one of a number of preset medical capability evaluation templates. Based on the target evaluation template, determine the target dataset identifier and the target evaluation configuration file; Based on the target evaluation configuration file and the target dataset corresponding to the target dataset identifier, OpenCompass is invoked to perform a performance evaluation of the medical capabilities of the target medical language model corresponding to the model identifier to be evaluated, and the performance evaluation results and the original output content of the target medical language model are obtained. The original output content is subjected to a safety assessment by calling a preset clinical safety and ethics rule base, and the safety assessment result is obtained; A comprehensive evaluation report is generated based on the performance evaluation results and the security evaluation results.

2. The medical language model evaluation method according to claim 1, characterized in that, The step of determining the target dataset identifier and target evaluation configuration file based on the target evaluation template includes: Based on the preset mapping relationship between the evaluation template and the dataset and configuration file, the target dataset identifier and the target evaluation configuration file corresponding to the target evaluation template are determined; wherein, the mapping relationship between the evaluation template and the dataset and configuration file includes a set of datasets and their evaluation configuration files mapped to each medical capability evaluation template.

3. The medical language model evaluation method according to claim 1, characterized in that, The process involves calling OpenCompass to perform a performance evaluation of the medical capabilities of the target medical language model corresponding to the model to be evaluated, based on the target evaluation configuration file and the target dataset corresponding to the target dataset identifier. The evaluation yields the performance evaluation results and the original output of the target medical language model, including: Based on the identifier of the model to be evaluated and the identifier of the target dataset, retrieve the target medical language model and the target dataset; Based on the target evaluation configuration file and the target dataset, the OpenCompass run.py executor is invoked to perform batch inference on the target medical language model, thereby obtaining the performance evaluation results and the original output content; wherein, the performance evaluation results include accuracy and scores for multiple preset medical evaluation dimensions corresponding to the target dataset.

4. The medical language model evaluation method according to claim 1, characterized in that, The process of calling a pre-defined clinical safety and ethics rule base to perform a safety assessment on the original output content, and obtaining the safety assessment results, includes: The original output content is cross-validated and pattern matched by calling the clinical safety and ethics rule base to obtain a safety assessment result; wherein, the clinical safety and ethics rule base includes one or more of the following: a contraindication and drug interaction database, a clinical guideline rule base, a symptom "critical value" rule base, an ethical baseline dataset, and a disease-specific gold standard dataset; and the safety assessment result includes the number of high-risk errors and a clinical safety score.

5. The medical language model evaluation method according to claim 1, characterized in that, The process of generating a comprehensive evaluation report based on the performance evaluation results and the security evaluation results includes: The performance evaluation results and the security evaluation results are summarized and visualized to obtain the comprehensive evaluation report. The comprehensive evaluation report includes a comprehensive conclusion, a medical capability radar chart, and a list of high-risk errors. The medical capability radar chart is used to display the evaluation scores of multiple preset medical evaluation dimensions.

6. The medical language model evaluation method according to claim 5, characterized in that, The multiple preset medical assessment dimensions include various aspects such as medical knowledge Q&A, medical reasoning, medical safety, ethics, medical language comprehension, and medical language generation.

7. The medical language model evaluation method according to claim 1, characterized in that, The medical language model evaluation method also includes: During the processing of the evaluation request, a complete operation audit log is recorded.

8. A medical language model evaluation device, characterized in that, The medical language model evaluation device is applied to an isolated, containerized environment and includes: The request parsing module is used to parse the identifier of the model to be evaluated and the target evaluation template from the received evaluation request. The target evaluation template is one of a number of preset medical capability evaluation templates. The template matching module is used to determine the target dataset identifier and the target evaluation configuration file based on the target evaluation template. The performance evaluation module is used to call OpenCompass to perform a performance evaluation of the medical capabilities of the target medical language model corresponding to the model to be evaluated, based on the target evaluation configuration file and the target dataset corresponding to the target dataset identifier, and to obtain the performance evaluation results and the original output content of the target medical language model. The safety assessment module is used to call a preset clinical safety and ethics rule base to perform a safety assessment on the original output content and obtain the safety assessment result; The report generation module is used to generate a comprehensive evaluation report based on the performance evaluation results and the security evaluation results.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the medical language model evaluation method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the medical language model evaluation method according to any one of claims 1-7.

Images