Medical large model evaluation system

Abstract

The present disclosure relates to a medical large model evaluation system, comprising: a task identification module configured to identify a target task type and a target task difficulty of an input task; a dynamic evaluation module configured to determine at least one target evaluation index matched with the task from a preset evaluation index library according to the target task type and the target task difficulty; and a multi-model collaboration module configured to evaluate the capability of a plurality of preset target medical large models in processing the task based on the target evaluation index to obtain a target evaluation score. The present disclosure can objectively and efficiently evaluate the performance of a medical large model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a large medical model evaluation system. Background Art

[0002] With the recent rapid development of medical artificial intelligence (AI) technology, the application of big medical models in diagnosis, treatment recommendations, medical education, and other fields has gradually increased. However, how to scientifically, objectively, and efficiently evaluate the performance of big medical models has become a major bottleneck restricting their further development. Summary of the Invention

[0003] In order to solve the above technical problems, the present disclosure provides a medical large model evaluation system.

[0004] According to one aspect of the present disclosure, a medical large model evaluation system is provided, the system comprising:

[0005] A task identification module, used to identify the target task type and target task difficulty of the input task;

[0006] a dynamic evaluation module, configured to determine at least one target evaluation indicator that matches the task from a preset evaluation indicator library based on the target task type and the target task difficulty; and

[0007] The multi-model collaboration module is used to evaluate the ability of multiple preset target medical large models to handle the task based on the target evaluation indicators to obtain target evaluation scores.

[0008] According to another aspect of the present disclosure, a method for evaluating a large medical model is provided, the method comprising:

[0009] Identify the target task type and target task difficulty of the input task;

[0010] Determining at least one target evaluation indicator that matches the task from a preset evaluation indicator library according to the target task type and the target task difficulty; and

[0011] Based on the target evaluation indicators, the capabilities of multiple preset target medical large models to handle the tasks are evaluated to obtain target evaluation scores.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0013] processor;

[0014] a memory for storing instructions executable by the processor;

[0015] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned medical large model evaluation method.

[0016] The present disclosure also provides a computer-readable storage medium storing a computer program for executing the above-mentioned medical large model evaluation method.

[0017] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0018] The medical big model evaluation system provided by the embodiment of the present disclosure includes: a task identification module, which is used to identify the target task type and target task difficulty of an input task; a dynamic evaluation module, which is used to determine at least one target evaluation indicator that matches the task from a preset evaluation indicator library based on the target task type and the target task difficulty; and a multi-model collaboration module, which is used to evaluate the ability of multiple preset target medical big models to handle the task based on the target evaluation indicator to obtain a target evaluation score.

[0019] Through modular design including task identification module, dynamic evaluation module, multi-model collaboration module, sampling module and arena evaluation module, this solution can effectively conduct comprehensive evaluation of large medical models in multi-task scenarios, significantly improve the scientificity, objectivity and adaptability of the evaluation, and provide an efficient evaluation tool for the development and iteration of large medical models. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is a schematic diagram of the framework of the medical large model evaluation system according to an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a method for performing adversarial evaluation by an arena evaluation module according to an embodiment of the present disclosure;

[0024] Figure 3 This is a flowchart of the medical large model evaluation method according to an embodiment of the present disclosure;

[0025] Figure 4This is a schematic diagram of the structure of the electronic device described in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0027] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0028] How to scientifically, objectively, and efficiently evaluate the performance of large medical models is crucial for their application and development in diagnosis, treatment recommendations, medical education, and other fields. To address this issue, the present disclosure provides a large medical model evaluation system.

[0029] This disclosure first considers some relevant evaluation methods, referring to the following situations.

[0030] Existing large-scale medical model evaluation methods have limitations: Traditional natural language processing (NLP) evaluation metrics (such as BLEU and ROUGE) primarily focus on character matching or shallow semantic clues, making them ill-suited to the complex tasks in medical scenarios. This is particularly true in open-ended evaluations, where standardized answers are often relied upon, often relying on individual physicians' diagnoses as reference answers, leading to highly subjective evaluation results.

[0031] Directly using large models as referees is biased: In recent years, the method of using large models to score generated results has become increasingly popular. However, in medical scenarios, this method often has biases and cannot guarantee complete objectivity and reliability. In addition, the complexity and professionalism of medical tasks require more scientific evaluation methods.

[0032] The multi-task nature of medical scenarios increases the difficulty of evaluation: Tasks in medical scenarios are diverse, encompassing objective questions, disease diagnosis, treatment plan development, and other subjective clinical environments. These tasks require distinct evaluation methods, making a single approach difficult to fully cover. For example, objective questions can be directly assessed through accuracy, while treatment plans require a comprehensive assessment across multiple dimensions, including accuracy, rationality, and feasibility.

[0033] Lack of effective performance monitoring during model iteration: With the rapid development of large medical models, iterative model training requires continuous performance monitoring. However, existing methods lack dynamic testing benchmarks and scientific comparative evaluation mechanisms, making it difficult to quantify and compare the effects of model improvements.

[0034] Efficient utilization of computing resources is a key issue: In medical scenarios, data annotation is costly and computing resources are limited. Existing evaluation methods often require comprehensive evaluation of large-scale data, resulting in wasted resources. How to select representative datasets through intelligent sampling to ensure scientific evaluation while reducing computing resource consumption has become a pressing issue.

[0035] Based on the above situation, in view of the multi-task characteristics of medical scenarios, the medical large model evaluation system provided by the embodiment of the present disclosure can dynamically adapt to different task types and coordinate multi-model collaboration through modular design, which can better improve the limitations of relevant evaluation methods; and it also achieves more efficient model performance evaluation through sampling test benchmark data and arena mode, thereby comprehensively improving the reliability, objectivity and practicality of medical large models in actual applications.

[0036] Example 1:

[0037] Figure 1 An architectural diagram of a medical large model evaluation system provided in an embodiment of the present disclosure is provided. The medical large model evaluation system includes at least: a task identification module 110, a dynamic evaluation module 120 and a multi-model collaboration module 130; and may further include: a sampling module 140 and an arena evaluation module 150.

[0038] In this embodiment, the task identification module 110 is used to identify the target task type and target task difficulty of an input task.

[0039] In this embodiment, the task identification module 110 intelligently analyzes the input task to determine its type and difficulty. Using the identified target task type and difficulty, it dynamically selects the most appropriate medical model and evaluation indicators, thereby ensuring the accuracy and scientific nature of the evaluation.

[0040] The dynamic evaluation module 120 is configured to determine at least one target evaluation indicator that matches the task from a preset evaluation indicator library according to the target task type and target task difficulty.

[0041] The dynamic evaluation module 120 in this embodiment calls a preset evaluation index library based on the output of the task identification module 110 and flexibly selects appropriate target evaluation indicators. The dynamic evaluation module 120 supports dynamic adaptation in multi-task scenarios and can select single or multiple target evaluation indicators from the evaluation index library based on the target task type and target task difficulty of the task. For example, in the case where the task is a subjective question with no standard answer, the dynamic evaluation module 120 can combine multiple dimensional evaluation indicators such as semantic matching, logical consistency and professional scoring to conduct a comprehensive evaluation, thereby avoiding the singleness and limitations of traditional evaluation methods.

[0042] The multi-model collaboration module 130 is used to evaluate the ability of multiple preset target medical large models to process tasks based on target evaluation indicators to obtain target evaluation scores.

[0043] In this embodiment, the multi-model collaboration module 130 coordinates multiple specialized medical models, or even more powerful ones, to form an "expert group" to jointly evaluate complex tasks. This multi-model collaboration module 130 can simulate multidisciplinary consultations in real-world medical scenarios and objectively score open-ended responses from multiple perspectives. By integrating the evaluation results of multiple target medical models, the multi-model collaboration module 130 can significantly improve the professionalism and reliability of the evaluation.

[0044] In the following embodiments, each module in the medical large model evaluation system, such as the task identification module 110, the dynamic evaluation module 120 and the multi-model collaboration module 130, is described in detail.

[0045] In medical scenarios, the diversity of task types puts forward different requirements for model evaluation. Traditional single evaluation methods are difficult to adapt to these complex tasks, which can easily lead to deviations or inaccuracies in the evaluation results. Therefore, this embodiment provides a task identification module 110, whose core goal is to perform intelligent analysis of the input tasks through the central model, determine the type and difficulty of the task, and dynamically select the most suitable evaluation method based on the determined target task type and target task difficulty, so as to ensure the accuracy and scientificity of the evaluation; the selected evaluation method may include: medical large model and evaluation indicators.

[0046] In this embodiment, the method for the task identification module 110 to identify the target task type and target task difficulty of the input task may include:

[0047] First, the task information of the input task is obtained; the task information includes: input question, standard answer and output answer of the model to be evaluated.

[0048] In a specific embodiment, the model to be evaluated performs intent recognition on the input question in the task and obtains an output answer. The input question in the task is pre-labeled with the corresponding standard answer. The model to be evaluated can be a model obtained by fine-tuning a general model using preset intent recognition data. Fine-tuning can enhance the accuracy of the model to be evaluated in intent recognition.

[0049] Secondly, the target task type of the task is identified based on the task information.

[0050] This specific embodiment can identify the type of task based on the above input question, standard answer, and output answer to determine the target task type. For example:

[0051] Objective tasks (such as multiple-choice questions) usually contain clear options and question descriptions. Based on this, the target task type can be determined to be an objective task based on the question description of the input question and the fact that both the standard answer and the output answer are options.

[0052] Open-ended question-answering tasks usually ask questions in natural language, and the answers have a high degree of freedom. Based on this, the target task type can be determined to be an open-ended question-answering task based on the fact that the input question is text content in natural language, and that both the standard answer and the output answer are text content that express certain semantics but have a high degree of freedom.

[0053] Treatment planning tasks usually include patient medical history information and complex medical background descriptions. Based on this, the target task type can be determined to be a treatment planning task, as the input question contains key content such as patient medical history information and complex medical background descriptions, and both the standard answer and the output answer are text content expressing the treatment plan.

[0054] Then, the semantic complexity score and context dependency score of the task are calculated; and the target task difficulty of the task is determined based on the semantic complexity score and context dependency score.

[0055] In a specific embodiment, semantic complexity analysis and context dependency analysis are required. Semantic complexity analysis can be understood as calculating the semantic complexity of the task text, such as the sparsity of sentence embeddings and keyword density, to obtain a semantic complexity score to assess the difficulty of understanding the task.

[0056] For example: simple multiple-choice questions usually have lower semantic complexity; diagnosis and treatment planning tasks have higher semantic complexity because they involve multidisciplinary knowledge.

[0057] Context dependency analysis can be understood as analyzing the degree of task dependence on context information and the context dependency score to judge the difficulty of the task.

[0058] For example, open-ended question-answering tasks are usually more context-dependent; multiple-choice tasks are less context-dependent.

[0059] After obtaining the semantic complexity score and context dependency score of the task, the target task difficulty of the task can be determined by referring to the following formula:

[0060] D=w s *S c +w c *C d

[0061] Among them, S c represents the semantic complexity score, C d represents the context dependency score, w s and w c is the corresponding weight.

[0062] According to the above embodiment, the output of the task identification module 110 may at least include: the target task type, which is used to clarify whether the task belongs to an objective task, an open-ended question-and-answer task, or a treatment plan task; the target task difficulty, which is used to quantify the complexity of the task and provide a basis for the selection of subsequent evaluation methods.

[0063] The task identification module 110 provided in the above embodiment intelligently analyzes input tasks and dynamically selects the most appropriate assessment method based on the target task type (e.g., objective question, disease diagnosis, medication recommendation, treatment plan, etc.) and task difficulty. This module ensures precise adaptation of the assessment method by determining task difficulty and classifying task types, avoiding the limitations of traditional, single-minded assessment methods.

[0064] In this embodiment, the core objective of dynamic evaluation module 120 is to flexibly invoke a pre-defined evaluation metric library based on the output of task identification module 110 and dynamically select the target evaluation metric that best suits the current task. This dynamic evaluation module 120 is adaptable to multi-task scenarios and comprehensively evaluates the output of the model being evaluated using multi-dimensional indicators, ensuring the scientific and comprehensive nature of the evaluation results.

[0065] The dynamic evaluation module 120 determines at least one target evaluation indicator that matches the task from a preset evaluation indicator library based on the target task type and target task difficulty. The implementation process may include:

[0066] According to the preset matching relationship between the task type and the evaluation indicator, candidate evaluation indicators that match the target task type are determined from the preset evaluation indicator library; according to the difficulty of the target task, at least one target evaluation indicator that matches the task is selected from the candidate evaluation indicators.

[0067] In a specific embodiment, the dynamic evaluation module 120 relies on a preset evaluation index library, which includes a variety of evaluation indexes applicable to different task types. The evaluation indexes in the evaluation index library can refer to the following examples.

[0068] Character-based metrics are used to indicate the degree of character matching between the standard answer and the output answer of the model being evaluated. These metrics include accuracy, recall, BLUE, Rouge, and F1 scores.

[0069] Vector space-based indicators are used to represent the distance between the standard answer and the output answer of the model to be evaluated in the vector space. More specifically, it is the distance between the text corresponding to the standard answer and the output answer in the vector space; it includes: text similarity, BertScore, etc.

[0070] The automatic evaluation indicator based on the big model refers to the use of the medical big model as an automatic evaluator, conducting multi-dimensional automatic evaluation through prompt words, and returning the evaluation results in the required format; among them, the multi-agent mode is a special form of this indicator, which uses multiple medical big models to debate or vote to improve the objectivity and accuracy of the evaluation.

[0071] An indicator based on the arena model, which is used to transform the direct scoring model into a preference evaluation model, and obtain the relative ranking of a group of models to be evaluated by calculating the ELO score.

[0072] In this embodiment, candidate evaluation indicators that match the target task type may be determined from the evaluation indicator library according to a preset matching relationship between the task type and the evaluation indicator.

[0073] The matching relationship between task types and evaluation indicators is shown in the example provided in Table 1.

[0074] Table 1: Matching relationship between task types and evaluation metrics

[0075]

[0076] Referring to Table 1 above, the task indicators for objective task matching generally include: accuracy, recall, and F1 score. These evaluation indicators are used to measure the degree of match between the output answer of the model to be evaluated and the standard answer.

[0077] The task indicators for open-ended question-answering task matching generally include: semantic matching, logical consistency, and fluency; the above evaluation indicators generally use a single large medical model for multi-dimensional automated evaluation.

[0078] The task indicators for matching medical professional question-answering tasks generally include: professionalism, rationality, consistency, and feasibility; the above evaluation indicators can be used to comprehensively evaluate the output of the task through multi-model collaborative scoring.

[0079] According to the matching relationship between the above task types and evaluation indicators, multiple evaluation indicators matching the target task type can be used as candidate evaluation indicators; and then, at least one target evaluation indicator matching the task is selected from the candidate evaluation indicators in combination with the difficulty of the target task.

[0080] For example, the higher the difficulty of the target task, the more target evaluation indicators are selected from the candidate evaluation indicators. Alternatively, different evaluation indicators are preset with different reference values. The higher the difficulty of the target task, the target evaluation indicators with the higher total reference value are selected from the candidate evaluation indicators.

[0081] Based on the above embodiments, the following possible scenarios are provided here.

[0082] For objective tasks, such as multiple-choice questions in the physician qualification examination, which have objective standardized answers, "accuracy" can be selected as the target evaluation indicator to calculate the degree of match between the output answer of the model to be evaluated and the standard answer.

[0083] For open-ended question-answering tasks, such as medical consultation, the dynamic evaluation module 120 can be called to conduct a comprehensive evaluation combining multi-dimensional indicators such as semantic matching, logical consistency, and professional scoring.

[0084] For diagnosis and treatment plan tasks, such as drug recommendations and diagnosis and treatment plan tasks, the multi-model collaboration module 130 can be called to form an "expert group" to jointly score the answers and conduct a comprehensive evaluation based on multi-dimensional indicators such as accuracy, rationality and feasibility.

[0085] The dynamic evaluation module 120 provided in the above embodiment uses a pre-defined evaluation index library based on task identification results (i.e., target task type and difficulty) to flexibly select evaluation indicators. For example, for objectivity issues, "accuracy" is used as the target evaluation indicator. For more complex tasks such as medical diagnosis and treatment planning, a large model is used to conduct an automated comprehensive evaluation of multiple metrics (such as accuracy, rationality, and feasibility). This dynamic evaluation module 120 supports dynamic adaptation in multi-task scenarios, significantly improving the flexibility and scientific nature of evaluation.

[0086] In this embodiment, the core goal of the multi-model collaboration module 130 is to coordinate multiple target medical models (including specialized models or models with enhanced performance) to form an "expert team" to jointly evaluate complex tasks. This multi-model collaboration module 130 simulates multidisciplinary consultations in real-world medical scenarios, objectively evaluating the output of the model being evaluated from multiple perspectives to ensure the professionalism and reliability of the evaluation results. This multi-model collaboration mechanism effectively improves the comprehensiveness and accuracy of evaluations, making it particularly suitable for complex tasks such as open-ended question-answering and diagnosis and treatment planning.

[0087] In this embodiment, the multi-model collaboration module 130 may include at least a model determination unit and a joint evaluation unit. Through the above units, the ability of multiple preset target medical large models to process tasks is evaluated based on target evaluation indicators to obtain target evaluation scores.

[0088] Among them, the model determination unit is used to determine multiple target medical large models that match the task in a pre-built model pool according to the target task type; wherein the different medical large models included in the model pool correspond to different task types.

[0089] The model determination unit in the multi-model collaboration module 130 coordinates multiple specialized or high-performance target medical models through the central model to form an "expert group" to jointly evaluate the task. The specific implementation of this step is as follows:

[0090] Build a model pool containing multiple large medical models, each of which has strong performance in a specific field or task type. For example:

[0091] Medical Model A: The character is a general practitioner, who is good at the initial diagnosis of common diseases;

[0092] Medical Model B: The character is a specialist doctor who is good at diagnosing diseases in a specific medical specialty;

[0093] Medical Model C: The character is a medical expert who is good at diagnosing difficult and complicated medical diseases.

[0094] Then, based on the output of the task identification module 110, the task is assigned to the most suitable target medical model. For example:

[0095] For open-ended question answering tasks, we can just call the medical model A;

[0096] For the preliminary diagnosis task, medical large model A and medical large model B are called first;

[0097] For the medication recommendation task, medical model B and medical model C are called first.

[0098] The joint evaluation unit is used to jointly evaluate the ability of multiple target medical large models to process tasks based on target evaluation indicators to obtain target evaluation scores.

[0099] In this embodiment, the joint evaluation unit is specifically configured to:

[0100] Based on the target evaluation indicators, the ability of multiple target medical large models to handle tasks is evaluated respectively to obtain multiple first evaluation scores, and a target evaluation score is determined from the multiple first evaluation scores based on a voting mechanism;

[0101] and / or,

[0102] Based on the target evaluation index, the ability of any target medical big model to perform the task is evaluated to obtain a second evaluation score, and the second evaluation score is corrected multiple times through other target medical big models to obtain a target evaluation score; among which, other target medical big models are other models among multiple target medical big models except any target medical big model.

[0103] Specifically, the voting mechanism requires each target medical model to give an independent evaluation result, namely the first evaluation score; and then selects the first evaluation score with the largest weight as the final target evaluation score by counting.

[0104] The debate mechanism begins with one of the target medical models performing an initial evaluation, resulting in a second evaluation score. The other target medical models then have the right to modify the second evaluation score. The other target medical models then check the rationality of the second evaluation score and make corrections based on the results. This process repeats until the number of iterations reaches a preset threshold or all target medical models agree. The latest second evaluation score, after multiple iterations and corrections, is used as the final target evaluation score.

[0105] The composite mechanism model independently scores, combining the characteristics of the voting and debate mechanisms described above. Target Medical Macro Models A and B vote to evaluate the candidate, generating a candidate score. Target Medical Macro Model C then modifies the candidate score. In this model, only Target Medical Macro Model C has the authority to modify the score.

[0106] Through the above voting mechanism and debate mechanism and their review mechanism, the multi-model collaboration module 130 can improve the efficiency and effectiveness of multi-model collaboration.

[0107] To improve the efficiency and effectiveness of multi-model collaboration, the multi-model collaboration module 130 in this embodiment can provide an optimization mechanism for multi-model collaboration. The optimization mechanism for multi-model collaboration mainly includes three aspects: model selection optimization, scoring standard optimization, and collaboration efficiency optimization.

[0108] The model selection optimization is used to optimize the process of selecting the target medical model. Accordingly, the model determination unit can also be used to:

[0109] According to the target task type, multiple candidate medical big models that match the task are determined in the pre-built model pool; according to the difficulty of the target task, at least one target medical big model that matches the task is selected from the multiple candidate medical big models.

[0110] Specifically, for simple tasks whose target task difficulty is lower than the preset difficulty value, only one or two target medical big models can be selected from multiple candidate medical big models for evaluation, thereby reducing the consumption of computing resources by reducing the number of models.

[0111] For complex tasks where the target task difficulty is not lower than the preset difficulty value, more target medical big models can be selected from multiple candidate medical big models for joint evaluation to ensure the comprehensiveness of the evaluation results.

[0112] Model selection optimization dynamically selects the target medical model that best suits the current task by analyzing task difficulty and model capabilities, meeting the needs of computing resources and comprehensive evaluation.

[0113] In this embodiment, the scoring standard optimization is used to optimize the evaluation standard. The scoring standard can be continuously optimized by analyzing the evaluation results of historical tasks. Accordingly, the multi-model collaboration module 130 also includes: a scoring standard optimization unit; the scoring standard optimization unit is used to:

[0114] Adjusting target evaluation indicators and their weights according to at least one of the target task type, target task difficulty, and model performance of the target medical large model;

[0115] Based on the adjusted target evaluation indicators and their weights, the ability of multiple target medical large models to handle tasks is evaluated separately to obtain multiple third evaluation scores;

[0116] The preset scoring threshold is adjusted according to the distribution of the third evaluation scores.

[0117] This embodiment of optimizing the scoring criteria can improve the accuracy of evaluation by adjusting target evaluation indicators and their weights and optimizing scoring thresholds.

[0118] In this embodiment, collaboration efficiency optimization is used to improve the efficiency of multi-model collaboration through parallel computing and task allocation optimization. Specifically, for independent scoring tasks, parallel computing is used to reduce scoring time; for collaborative scoring tasks, a task allocation optimization algorithm is used to ensure the balance of scoring tasks.

[0119] According to the above embodiment, the output of the multi-model collaboration module 130 may include at least:

[0120] The evaluation results of a single target medical big model, that is, the independent evaluation indicators and first evaluation score of each target medical big model; the comprehensive evaluation results of multiple target medical big models, that is, the final target evaluation score generated by the fusion of scoring results; the evaluation report can include a detailed analysis of the scoring results and the advantages and disadvantages of the output of each target medical big model, providing a reference for model optimization.

[0121] The multi-model collaboration module 130 provided in the above embodiment coordinates multiple target medical large models to form an "expert group" to jointly evaluate complex tasks. It can simulate multidisciplinary consultations in real medical scenarios, objectively score open-ended answers from multiple perspectives, and ensure the professionalism and reliability of the evaluation results.

[0122] In some embodiments, the large model evaluation system may further include: a sampling module 140 and an arena evaluation module 150; wherein,

[0123] The sampling module 140 is used to sample the labeled data in the medical scene to obtain test benchmark data.

[0124] The arena evaluation module 150 is used to perform adversarial evaluation on the model performance of two to-be-evaluated models based on the test benchmark data.

[0125] In this embodiment, the sampling module 140 samples the labeled data in the medical scene to obtain the test benchmark data, which may include the following steps (1)-(4).

[0126] (1) Obtaining labeled data with labeled information in medical scenarios; wherein the labeled information includes: task category and department category.

[0127] Specifically, we need to obtain real annotated data in medical scenarios. The annotated data carries clear annotation information, including but not limited to: department categories (such as internal medicine, surgery, pediatrics, etc.) and task categories (such as multiple-choice questions, open-ended questions, diagnosis and treatment plans, etc.).

[0128] (2) Select the first data from the labeled data based on semantic embedding and clustering algorithms.

[0129] Specifically, the semantic distribution of the labeled data is analyzed by using semantic embedding and clustering algorithms, and the first data is selected from the labeled data according to the semantic distribution, thereby ensuring the diversity of the sampled data.

[0130] (3) Select the second data from the labeled data according to the task category and department category.

[0131] Specifically, the second data is selected from the labeled data through distribution analysis of task categories and department categories, so as to ensure the coverage of the sampled data.

[0132] (4) The first data and the second data are determined as test reference data.

[0133] According to the data selection method of the above steps (2) and (3), representative first data and second data can be selected according to the diversity and coverage of the labeled data, and the first data and second data can be determined as test benchmark data.

[0134] The test benchmark data determined by the above method specifically includes, for example, 50 physician qualification knowledge data selected from the annotated data of internal medicine, 30 open-ended question and answer data sampled from the surgical data, and 20 diagnosis and treatment plan data sampled from the pediatric data.

[0135] Furthermore, this embodiment can dynamically update the test benchmark data as the medical model evolves and task requirements change. For example, as the performance of the medical model improves, the difficulty of the test benchmark data can be increased; as new tasks are introduced, test benchmark data related to the new tasks can be added.

[0136] This embodiment uses intelligent sampling to select representative small-scale data from the labeled data in medical scenarios as test benchmark data. The labeled data contains clear department categories and task categories. By analyzing the diversity and coverage of the data, the representativeness and scientific nature of the selected test benchmark data are ensured, thereby ensuring the quality of the evaluation while reducing the consumption of computing resources.

[0137] In this embodiment, the core goal of the arena evaluation module 150 is to introduce an adversarial evaluation mechanism to allow the model to perform performance evaluation in a pairwise comparison arena mode based on the same test benchmark data, and to quantify the model's capabilities in combination with the ELO (Elorating system) scoring system.

[0138] Reference Figure 2 The arena evaluation module 150 performs adversarial evaluation on the model performance of the two models to be evaluated based on the test benchmark data, and the following steps A1-A4 can be referred to.

[0139] A1: For the two models to be evaluated, multiple rounds of competitive evaluation operations are performed based on the test benchmark data. The competitive evaluation operations include steps A2 and A3.

[0140] In a specific embodiment, a dynamic matching algorithm can be used to select models with similar performance for competition. The selected models are the models to be evaluated. In the initial stage of the competitive evaluation, or during the first N competitive evaluation cycles, a base model can be selected to compete against an improved model to assess the effectiveness of the model improvements. In subsequent stages, or after N competitive evaluation cycles, model versions with similar performance are selected for competition to ensure a fair and scientific evaluation.

[0141] A2: Evaluate the model performance of each model to be evaluated based on the test benchmark data to obtain the current evaluation score.

[0142] In each round of competitive evaluation, the two models to be evaluated generate their own answers to the same test benchmark data, and evaluate the answers using evaluation indicators to obtain the corresponding current evaluation scores. Finally, the winner is determined based on the current evaluation scores of the two models to be evaluated.

[0143] Through multiple rounds of competitive evaluation, the performance of the two models under evaluation on different tasks is comprehensively assessed. For example, in the first round of competitive evaluation, the performance of the two models under evaluation is evaluated on the multiple-choice task; in the second round of competitive evaluation, the performance of the two models under evaluation is evaluated on the open-ended question-answering task; and in the third round of competitive evaluation, the performance of the two models under evaluation is evaluated on the diagnosis and treatment planning task.

[0144] It can be understood that in each round of competitive evaluation operation, each model to be evaluated will obtain its own current evaluation score.

[0145] A3, updating the current ELO score of each model to be evaluated according to the two current evaluation scores; wherein, when the competitive evaluation operation is performed for the first time, the current ELO score of the model to be evaluated is a preset initial ELO score.

[0146] In this embodiment, the arena evaluation module 150 uses the ELO scoring system to quantify the model's capabilities. The ELO scoring system is a scoring method widely used in arena evaluation that can dynamically adjust the model's score to reflect the model's relative strength.

[0147] In one specific implementation, an initial ELO score is set in advance for each model to be evaluated, usually 1500 points, such as: Model A: 1500 points, Model B: 1500 points; Model A and Model B are two models to be evaluated.

[0148] In each round of competitive evaluation, taking any of the models to be evaluated as an example, the current ELO score of the model to be evaluated is updated based on the win or loss results of the two models to be evaluated, using the following formula:

[0149] R′=R+K*(SE)

[0150] Where R is the current ELO score of the model to be evaluated, R′ is the updated ELO score of the model to be evaluated; K is the score update coefficient, which is usually 32; S is the actual game result, with the winner getting 1 point, the loser getting 0 points, and the draw getting 0.5 points; E is the expected win rate of the model to be evaluated.

[0151] A4. After completing multiple rounds of competitive evaluation operations, the latest ELO score of each model to be evaluated is determined as its respective target ELO score; the target ELO score is used to represent the model performance of the model to be evaluated.

[0152] In this embodiment, the competitive evaluation operation is performed multiple times to achieve multiple rounds of competition, and the ELO score of the model to be evaluated is continuously updated. After completing a preset number of rounds of competitive evaluation operations, the target ELO score of each model to be evaluated is finally determined. For example, after 10 rounds of competitive evaluation operations, the ELO score of model A is updated to 1600 points, and the ELO score of model B is updated to 1550 points.

[0153] According to the above embodiment, the arena evaluation module 150 can provide clear feedback for model optimization by performing adversarial evaluation on the model performance of two models to be evaluated. The specific implementation is as follows:

[0154] 1) Win / Loss Statistics. The results of each round of competitive evaluation are tallied for the two models being evaluated, and their performance on different tasks is analyzed. For example, Model A has an 80% win rate on the multiple-choice task, while Model B has a 60% win rate on the open-ended question-answering task.

[0155] 2) Score Trend Analysis: Analyze the improvement effects of the two models under evaluation by analyzing the changing trends of the ELO scores. For example, if the ELO score of Model A increases from 1500 to 1600, it indicates a significant improvement in model performance; while the ELO score of Model B decreases from 1500 to 1450, it indicates that the model performance needs further optimization.

[0156] 3) Task Performance Analysis. Based on the evaluation results of each task category, analyze the strengths and weaknesses of the two models to be evaluated. For example, Model A performs well on the multiple-choice task but poorly on the treatment planning task; Model B performs well on the open-ended question-answering task but has problems with logical consistency.

[0157] 4) Optimization Recommendations: Based on the evaluation results, provide specific recommendations for model optimization. For example, enhance the logical reasoning ability of Model A in the diagnosis and treatment planning task; improve the semantic matching ability of Model B in the open-ended question answering task.

[0158] According to the above embodiment, the output of the arena assessment module 150 may include at least:

[0159] The ELO score, i.e. the final ELO score of each model to be evaluated, reflects the overall strength of the model to be evaluated. The win-loss statistics, i.e. the results of each round of competition, reflect the performance of the model to be evaluated on different tasks. The evaluation report can include score trend analysis, task performance analysis, and optimization suggestions, providing clear feedback for the iteration of the model to be evaluated.

[0160] In the above embodiment, the sampling module 140 selects annotated data from medical scenarios to identify representative, small-scale data as the Arena test benchmark data. The Arena evaluation module 150 uses a pairwise comparison method to perform adversarial scoring on the test benchmark data for the models to be evaluated. Incorporating the ELO scoring system, the model's performance is quantified, ensuring the efficiency and scientific nature of the evaluation and comprehensively quantifying the effects of model improvement. The Arena evaluation mechanism not only improves the scientific nature and objectivity of the evaluation, but also significantly reduces computing resource consumption while providing clear feedback for model iteration.

[0161] In summary, the medical large model evaluation system provided by the embodiment of the present disclosure, through a modular design such as the task identification module 110, the dynamic evaluation module 120, the multi-model collaboration module 130, the sampling module 140 and the arena evaluation module 150, can effectively conduct a comprehensive evaluation of the medical large model in multi-task scenarios, significantly improve the scientificity, objectivity and adaptability of the evaluation, and provide an efficient evaluation tool for the development and iteration of the medical large model.

[0162] The task identification module 110 comprehensively improves data quality by screening out high-quality data through a comprehensive evaluation of multi-dimensional indicators. This not only provides a solid data foundation for model fine-tuning, but also effectively reduces the negative impact of low-quality data on model training, significantly reduces the computational complexity of subsequent processing, and improves data processing efficiency.

[0163] Dynamic Evaluation Module 120 flexibly utilizes multi-dimensional evaluation metrics (such as semantic matching, logical consistency, and professionalism) based on the target task type and difficulty, enabling accurate assessment of model outputs. This module, through dynamic weight adjustment and a comprehensive scoring mechanism, ensures scientific and comprehensive evaluation results, providing clear guidance for model optimization and significantly improving model performance in complex tasks.

[0164] The Multi-Model Collaboration Module 130 utilizes an "expert group" mechanism to coordinate multiple target medical models for joint evaluation of complex tasks. This module comprehensively analyzes model outputs from multiple perspectives, ensuring the professionalism and reliability of evaluation results. Furthermore, dynamic task allocation and score fusion mechanisms improve evaluation efficiency, making it particularly suitable for highly specialized scenarios such as healthcare.

[0165] The sampling module 140 efficiently selects representative small-scale test benchmark data from massive data through semantic diversity analysis and coverage analysis, significantly reducing the consumption of computing resources; this mechanism can dynamically update the test benchmark data to adapt to different task requirements and ensure the comprehensiveness and pertinence of the evaluation results.

[0166] Arena Evaluation Module 150 selects representative models for comparison and comprehensively evaluates the effectiveness of model improvements. Using a competitive evaluation mechanism and an ELO scoring system, Arena Evaluation Module 150 quantifies the relative capabilities of models. Through multiple rounds of competition and analysis of scoring trends, the module significantly enhances the objectivity and scientific nature of the evaluation, providing a reliable basis for model iteration.

[0167] Therefore, this paper, through a modular design, organically combines quality assessment, dynamic evaluation, multi-model collaboration, arena evaluation, and intelligent sampling mechanisms to form an efficient, accurate, and professional medical large-scale model evaluation system. This medical large-scale model evaluation system not only improves the performance of models in professional tasks but also significantly reduces the cost of data processing and evaluation, providing strong support for the application of models in highly specialized fields such as medicine.

[0168] Example 2:

[0169] Corresponding to the above-mentioned medical large model evaluation system, such as Figure 3 As shown, this embodiment provides a large medical model evaluation method, comprising the following steps:

[0170] S102, identifying the target task type and target task difficulty of the input task;

[0171] S104, determining at least one target evaluation indicator that matches the task from a preset evaluation indicator library based on the target task type and target task difficulty; and

[0172] S106, based on the target evaluation indicators, evaluate the ability of the preset multiple target medical large models to handle tasks and obtain the target evaluation scores.

[0173] In one embodiment, the medical large model evaluation method may further include the following steps:

[0174] Sampling labeled data in medical scenarios to obtain test benchmark data;

[0175] An adversarial evaluation is performed on the model performance of the two models to be evaluated based on the test benchmark data.

[0176] The method provided in this embodiment has the same implementation principle and technical effects as those of the aforementioned system embodiment. For the sake of brief description, for matters not mentioned in the method embodiment, reference can be made to the corresponding content in the aforementioned system embodiment.

[0177] Example 3:

[0178] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0179] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0180] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the medical large model evaluation method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0181] In one example, the electronic device 400 may further include an input device 403 and an output device 404 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0182] In addition, the input device 403 may also include, for example, a keyboard, a mouse, and the like.

[0183] The output device 404 can output various information to the outside, including determined distance information, direction information, etc. The output device 404 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0184] Of course, to simplify, Figure 4Only some of the components related to the present disclosure in the electronic device 400 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 400 may further include any other appropriate components according to specific application scenarios.

[0185] Example 4:

[0186] Furthermore, this embodiment also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned medical large model evaluation method.

[0187] The embodiments of the present disclosure provide a computer program product of a medical large model evaluation method, device, electronic device, and medium, including a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0188] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0189] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A large medical model evaluation system, characterized in that: The system comprises: A task identification module, used to identify the target task type and target task difficulty of the input task; a dynamic evaluation module, configured to determine at least one target evaluation indicator that matches the task from a preset evaluation indicator library based on the target task type and the target task difficulty; and A multi-model collaboration module is used to evaluate the ability of multiple preset target medical large models to handle the task based on the target evaluation index to obtain a target evaluation score; The multi-model collaboration module includes: a model determination unit, configured to determine, based on the target task type, a plurality of target medical macro models that match the task in a pre-built model pool; wherein different medical macro models included in the model pool correspond to different task types; a joint evaluation unit, configured to jointly evaluate the capabilities of the plurality of target large medical models in processing the task based on the target evaluation index to obtain a target evaluation score; The joint evaluation unit is further configured to: Based on the target evaluation index, the capabilities of the plurality of target large medical models to process the task are evaluated respectively to obtain a plurality of first evaluation scores, and a target evaluation score is determined from the plurality of first evaluation scores based on a voting mechanism; and / or, Based on the target evaluation index, the ability of any target medical macro model to perform the task is evaluated to obtain a second evaluation score, and the second evaluation score is corrected multiple times through other target medical macro models to obtain a target evaluation score; wherein, the other target medical macro models are other models among the multiple target medical macro models except for any target medical macro model.

2. The system according to claim 1, wherein: The task identification module is further configured to: Obtaining task information of an input task; wherein the task information includes: an input question, a standard answer, and an output answer of the model to be evaluated; identifying a target task type of the task according to the task information; calculating a semantic complexity score and a context dependency score for the task; A target task difficulty of the task is determined based on the semantic complexity score and the context dependency score.

3. The system according to claim 1, wherein: The dynamic evaluation module is further configured to: Determine, from a preset evaluation indicator library, a candidate evaluation indicator that matches the target task type based on a preset matching relationship between the task type and the evaluation indicator; At least one target evaluation indicator that matches the task is selected from the candidate evaluation indicators according to the difficulty of the target task.

4. The system according to claim 1, wherein: The model determination unit is further configured to: According to the target task type, determining a plurality of candidate medical large models matching the task in a pre-built model pool; At least one target medical large model matching the task is selected from a plurality of candidate medical large models according to the difficulty of the target task.

5. The system according to claim 1, wherein: The multi-model collaboration module further includes a scoring standard optimization unit; the scoring standard optimization unit is configured to: Adjusting the target evaluation indicators and their weights according to at least one of the target task type, the target task difficulty, and the model performance of the target medical macromodel; Based on the adjusted target evaluation indicators and their weights, respectively evaluating the capabilities of the plurality of target large medical models in processing the tasks to obtain a plurality of third evaluation scores; The preset scoring threshold is adjusted according to the distribution of the third evaluation scores.

6. The system according to claim 1, wherein: The system further comprises: a sampling module and an arena assessment module; wherein, The sampling module is used to sample the labeled data in the medical scene to obtain test benchmark data; The arena evaluation module is used to perform adversarial evaluation on the model performance of two to-be-evaluated models based on the test benchmark data.

7. The system according to claim 6, characterized in that The sampling module is also used for: Acquire annotated data carrying annotated information in a medical scenario; wherein the annotated information includes: task category and department category; Selecting first data from the labeled data according to a semantic embedding and clustering algorithm; selecting second data from the labeled data according to the task category and the department category; The first data and the second data are determined as the test benchmark data; wherein the first data and the second data are representative data selected from the labeled data.

8. The system according to claim 6, wherein: The Arena Assessment Module is also used to: For the two models to be evaluated, multiple rounds of competitive evaluation operations are performed based on the test benchmark data, and the competitive evaluation operations include: Evaluate the model performance of each of the models to be evaluated based on the test benchmark data to obtain a current evaluation score; and The current ELO score of each of the models to be evaluated is updated according to the two current evaluation scores; wherein, when the competitive evaluation operation is performed for the first time, the current ELO score of the model to be evaluated is a preset initial ELO score; When multiple rounds of the competitive evaluation operation are completed, the latest ELO score of each of the models to be evaluated is determined as its respective target ELO score; the target ELO score is used to represent the model performance of the model to be evaluated.

Images