Medical large model evaluation system

Abstract

The invention relates to a medical large model evaluation system, and the system comprises a task recognition module which is used for recognizing the target task type and target task difficulty of an input task; the dynamic evaluation module is used for determining at least one target evaluation index matched with the task from a preset evaluation index library according to the target task type and the target task difficulty; and the multi-model cooperation module is used for evaluating the task processing capability of a plurality of preset target medical large models based on the target evaluation index to obtain a target evaluation score. According to the invention, the performance of the medical large model can be objectively and efficiently evaluated.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to a medical large model evaluation system. Background Art

[0002] Recently, with the rapid development of medical artificial intelligence technology, the application of medical large models in the fields of diagnosis, treatment recommendations, medical education, etc. has gradually increased. However, how to scientifically, objectively, and efficiently evaluate the performance of medical large models has become an important bottleneck restricting their further development. Summary of the Invention

[0003] To solve the above technical problems, the present disclosure provides a medical large model evaluation system.

[0004] According to one aspect of the present disclosure, there is provided a medical large model evaluation system, the system comprising: A task recognition module, configured to recognize the target task type and target task difficulty of the input task; A dynamic evaluation module, configured to determine at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and the target task difficulty; and, A multi-model collaboration module, configured to evaluate the capabilities of a preset plurality of target medical large models to process the task based on the target evaluation index, and obtain a target evaluation score.

[0005] According to another aspect of the present disclosure, there is also provided a medical large model evaluation method, the method comprising: Recognizing the target task type and target task difficulty of the input task; Determining at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and the target task difficulty; and, Evaluating the capabilities of a preset plurality of target medical large models to process the task based on the target evaluation index, and obtaining a target evaluation score.

[0006] According to another aspect of the present disclosure, there is also provided an electronic device, the electronic device comprising: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above medical large model evaluation method.

[0007] The present disclosure also provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being used to execute the above medical large model evaluation method.

[0008] The technical solution provided by the embodiments of the present disclosure has the following advantages compared with the prior art: The medical large model evaluation system provided by the embodiments of the present disclosure includes: a task recognition module, configured to recognize the target task type and the target task difficulty of the input task; a dynamic evaluation module, configured to determine at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and the target task difficulty; and a multi-model collaboration module, configured to evaluate the capabilities of a preset plurality of target medical large models to process the task based on the target evaluation index, and obtain a target evaluation score.

[0009] Through the modular design of the task recognition module, the dynamic evaluation module, the multi-model collaboration module, the sampling module and the arena evaluation module, this solution can effectively and comprehensively evaluate medical large models for multi-task scenarios, significantly improving the scientificity, objectivity and adaptability of the evaluation, and providing an efficient evaluation tool for the development and iteration of medical large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] Figure 1 It is a schematic framework diagram of the medical large model evaluation system according to the embodiments of the present disclosure; Figure 2 It is a flowchart of the arena evaluation module executing the adversarial evaluation method according to the embodiments of the present disclosure; Figure 3 It is a flowchart of the medical large model evaluation method according to the embodiments of the present disclosure; Figure 4 It is a schematic structural diagram of the electronic device according to the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] In order to better understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0014] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0015] How to scientifically, objectively, and efficiently evaluate the performance of medical large models is crucial for the application and development of medical large models in fields such as diagnosis, treatment recommendations, and medical education. To address this issue, embodiments of the present disclosure provide a medical large model evaluation system.

[0016] The present disclosure first considers some related evaluation methods and refers to the following several situations.

[0017] Existing medical large model evaluation methods have limitations: Traditional natural language processing (NLP) evaluation metrics (such as BLEU, ROUGE, etc.) mainly target character matching or shallow semantic clues and are difficult to meet the requirements of complex tasks in medical scenarios. Especially in the evaluation of open-ended answers, there is a lack of standardized answers, and usually only a doctor's diagnosis can be relied on as the reference answer, resulting in strong subjectivity of the evaluation results.

[0018] Directly using a large model as a referee has biases: In recent years, the method of using a large model to score generated results has gradually become popular. However, in medical scenarios, this method often has biases and cannot guarantee complete objectivity and reliability. Moreover, the complexity and professionalism of medical tasks require more scientific evaluation methods.

[0019] The multi-task characteristics of medical scenarios increase the evaluation difficulty: The task types in medical scenarios are diverse, including objective questions, disease diagnosis, subjective clinical environments such as treatment plan formulation, etc. These tasks have different requirements for evaluation methods, and a single evaluation method is difficult to comprehensively cover. For example, objective questions can be directly evaluated by "correct rate", while treatment plans need to be comprehensively evaluated from multiple dimensions such as accuracy, rationality, and feasibility.

[0020] There is a lack of effective performance monitoring means during the model iteration process: With the rapid development of medical large models, the iterative training of the model requires continuous monitoring of its performance. However, existing methods lack dynamic test benchmarks and scientific comparative evaluation mechanisms, making it difficult to quantify and compare the improvement effects of the models.

[0021] The efficient utilization of computing resources has become a key issue: In medical scenarios, the cost of data annotation is high and computing resources are limited. Existing evaluation methods usually require a comprehensive evaluation of large-scale data, resulting in resource waste. How to select a representative data set through intelligent sampling to ensure the scientific nature of the evaluation and reduce the consumption of computing resources has become an urgent problem to be solved.

[0022] Based on the above situation, in view of the multi-task characteristics of the medical scenario, the medical large model evaluation system provided by the embodiments of the present disclosure can dynamically adapt to different task types and coordinate the cooperation of multiple models through modular design, which can better improve the limitations of related evaluation methods; and, it also realizes a more efficient evaluation of the model performance through sampling test benchmark data and the arena mode, so as to comprehensively improve the reliability, objectivity and practicability of the medical large model in practical applications.

[0023] Embodiment 1: Figure 1 FIG. is a schematic structural diagram of a medical large model evaluation system provided by an embodiment of the present disclosure. The medical large model evaluation system at least includes: a task recognition module 110, a dynamic evaluation module 120, and a multi-model cooperation module 130; and, it may further include: a sampling module 140 and an arena evaluation module 150.

[0024] In this embodiment, the task recognition module 110 is used to identify the target task type and target task difficulty of the input task.

[0025] The task recognition module 110 in this embodiment judges the type and difficulty of the task through intelligent analysis of the input task. By using the identified target task type and target task difficulty, the most suitable medical large model and evaluation index can be dynamically selected, thereby ensuring the accuracy and scientificity of the evaluation.

[0026] The dynamic evaluation module 120 is used to determine at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and target task difficulty.

[0027] The dynamic evaluation module 120 in this embodiment calls the preset evaluation index library based on the output of the task recognition module 110 and flexibly selects appropriate target evaluation indexes. The dynamic evaluation module 120 supports dynamic adaptation in multi-task scenarios and can select single or multiple target evaluation indexes from the evaluation index library according to the target task type and target task difficulty of the task. For example, in the case where the task is a subjective question without a standard answer, the dynamic evaluation module 120 can comprehensively evaluate by combining multi-dimensional evaluation indexes such as semantic matching, logical consistency, and professional scoring, so as to avoid the singleness and limitations of traditional evaluation methods.

[0028] The multi-model cooperation module 130 is used to evaluate the ability of a preset plurality of target medical large models to process tasks based on the target evaluation index, and obtain a target evaluation score.

[0029] The multi-model collaboration module 130 in this embodiment is used to coordinate multiple professional medical large models or medical large models with stronger performance to form an "expert group" for joint evaluation of complex tasks. The multi-model collaboration module 130 can simulate multi-disciplinary consultations in real medical scenarios and objectively score open-ended answers from multiple perspectives. By integrating the evaluation results of multiple target medical large models, the multi-model collaboration module 130 can significantly improve the professionalism and reliability of the evaluation.

[0030] In the following embodiments, each module in the medical large model evaluation system, such as the task recognition module 110, the dynamic evaluation module 120, and the multi-model collaboration module 130, will be described in detail.

[0031] In a medical scenario, the diversity of task types poses different requirements for model evaluation. Traditional single evaluation methods are difficult to adapt to these complex tasks and are prone to deviations or inaccuracies in evaluation results. Therefore, this embodiment provides a task recognition module 110, whose core objective is to intelligently analyze the input task through a central model, judge the type and difficulty of the task, and dynamically select the most suitable evaluation method according to the judged target task type and target task difficulty, so as to ensure the accuracy and scientific nature of the evaluation; the selected evaluation methods may include: medical large models and evaluation metrics.

[0032] In this embodiment, the method by which the task recognition module 110 identifies the target task type and target task difficulty of the input task may include: First, obtain the task information of the input task; among them, the task information includes: the input question, the standard answer, and the output answer of the model to be evaluated.

[0033] In a specific embodiment, the input question in the task is subjected to intent recognition by the model to be evaluated to obtain an output answer. Among them, the input question in the above task is pre-marked with a corresponding standard answer. The model to be evaluated may be a model obtained by fine-tuning a general model using preset intent recognition data. Through fine-tuning, the accuracy of the model to be evaluated in intent recognition can be enhanced.

[0034] Second, identify the target task type of the task according to the task information.

[0035] This specific embodiment can identify the type of the task based on the above input question, standard answer, and output answer, and determine the target task type. For example: Objective tasks (such as multiple-choice questions) usually contain clear options and question descriptions; based on this, according to the question description of the input question, and the standard answer and output answer are both options, it can be determined that the target task type is an objective task.

[0036] Open-ended question tasks usually pose questions in natural language form, and the answers have a relatively high degree of freedom. Based on this, it is possible to determine that the target task type is the open-ended question task type according to the input question being text content in natural language form, and both the standard answer and the output answer being text content that expresses a certain semantics but has a relatively high degree of freedom.

[0037] Diagnosis and treatment plan tasks usually contain patient medical record information and complex medical background descriptions. Based on this, it is possible to determine that the target task type is the diagnosis and treatment plan task type according to the input question containing key content such as patient medical record information and complex medical background descriptions, and both the standard answer and the output answer being text content that expresses the diagnosis and treatment plan.

[0038] Then, calculate the semantic complexity score and context dependence score of the task; and then determine the target task difficulty of the task based on the semantic complexity score and context dependence score.

[0039] In a specific embodiment, semantic complexity analysis and context dependence analysis need to be performed. Among them, semantic complexity analysis can be understood as obtaining a semantic complexity score by calculating the semantic complexity of the text of the task, such as the sparsity of sentence embeddings, keyword density, etc., so as to evaluate the understanding difficulty of the task.

[0040] For example: Simple multiple-choice questions usually have a low semantic complexity; diagnosis and treatment plan tasks have a high semantic complexity due to involving multidisciplinary knowledge.

[0041] Context dependence analysis can be understood as judging the task difficulty by analyzing the degree of dependence of the task on context information and obtaining a context dependence score.

[0042] For example: Open-ended question tasks usually have a high context dependence; multiple-choice question tasks have a low context dependence.

[0043] After obtaining the semantic complexity score and context dependence score of the task, the target task difficulty of the task can be determined with reference to the following formula: D = w s *S c + w c *C d Wherein, S c represents the semantic complexity score, C d represents the context dependence score, and w s and w c are the corresponding weights.

[0044] According to the above embodiments, the output of the task recognition module 110 may at least include: a target task type, which is used to clarify the type of the task, such as an objective task, an open-ended question task, or a diagnosis and treatment plan task; and a target task difficulty, which is used to quantify the complexity of the task and provide a basis for the selection of subsequent evaluation methods.

[0045] The task recognition module 110 provided in the above embodiments can intelligently analyze the input task and dynamically select the most suitable evaluation method according to the target task type (such as objective questions, disease diagnosis, drug recommendation, diagnosis and treatment plan, etc.) and the target task difficulty. Through task difficulty judgment and type classification, this module ensures the precise adaptation of the evaluation method and avoids the limitation of the single traditional evaluation method.

[0046] In this embodiment, the core objective of the dynamic evaluation module 120 is to flexibly call a preset evaluation index library according to the output content of the task recognition module 110 and dynamically select the target evaluation index most suitable for the current task from it. The dynamic evaluation module 120 can adapt to multi-task scenarios, comprehensively evaluate the output of the model to be evaluated through multi-dimensional indexes, and ensure the scientificity and comprehensiveness of the evaluation results.

[0047] The dynamic evaluation module 120 determines at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and the target task difficulty. The implementation process may include: Determine candidate evaluation indexes matching the target task type from the preset evaluation index library according to the preset matching relationship between the task type and the evaluation indexes; select at least one target evaluation index matching the task from the candidate evaluation indexes according to the target task difficulty.

[0048] In a specific embodiment, the dynamic evaluation module 120 relies on a preset evaluation index library, which includes various evaluation indexes applicable to different task types. The evaluation indexes in the evaluation index library can refer to the following examples.

[0049] Character-based indexes, which are used to represent the character matching degree between the standard answer and the output answer of the model to be evaluated; they include: accuracy, recall, BLUE, Rouge, and F1 score, etc.

[0050] Vector space-based indexes, which are used to represent the distance between the standard answer and the output answer of the model to be evaluated in the vector space. More specifically, it is the distance between the text corresponding to the standard answer and the output answer in the vector space; they include: text similarity, BertScore, etc.

[0051] Automatic evaluation metrics based on large models, which refer to using a medical large model as an automatic evaluator, conducting multi-dimensional automatic evaluation through prompts, and returning the evaluation results in the required format; among them, the multi-agent mode is a special form of this metric, where multiple medical large models are used for debate or voting to improve the objectivity and accuracy of the evaluation.

[0052] Metrics based on the arena mode, which are used to change the direct scoring mode into a preference evaluation model, and obtain the relative rankings of a group of models to be evaluated through the calculation of ELO scores.

[0053] In this embodiment, according to the preset matching relationship between the task type and the evaluation metric, candidate evaluation metrics matching the target task type can be determined from the above-mentioned evaluation metric library.

[0054] Among them, the matching relationship between the task type and the evaluation metric refers to the examples provided in Table 1.

[0055] Table 1: Matching relationship between task type and evaluation metric

[0056] Referring to Table 1 above, the task metrics matching objective tasks generally include: Accuracy, Recall, F1 score, etc.; the output answers of the model to be evaluated and the standard answers are measured by the above evaluation metrics for the matching degree.

[0057] The task metrics matching open-ended question-and-answer tasks generally include: semantic matching, logical consistency, fluency, etc.; the above evaluation metrics generally use a single medical large model for multi-dimensional automatic evaluation.

[0058] The task metrics matching medical professional question-and-answer tasks generally include: professionalism, rationality, consistency, feasibility, etc.; the above evaluation metrics can comprehensively evaluate the output of the task through the method of multi-model collaborative scoring.

[0059] According to the matching relationship between the above task type and the evaluation metric, multiple evaluation metrics matching the target task type can be used as candidate evaluation metrics; then, at least one target evaluation metric matching the task is selected from the candidate evaluation metrics in combination with the target task difficulty.

[0060] Exemplarily, the higher the target task difficulty, the more target evaluation metrics are selected from the candidate evaluation metrics. Or, different evaluation metrics have different reference values, and the higher the target task difficulty, the higher the total reference value of the target evaluation metrics selected from the candidate evaluation metrics.

[0061] Based on the above embodiments, the following several possible scenarios are provided here.

[0062] For objective tasks, such as multiple-choice questions in the medical licensing examination that have objectively standardized answers, "accuracy rate" can be selected as the target evaluation indicator to calculate the matching degree between the output answer of the model to be evaluated and the standard answer.

[0063] For open-ended question-and-answer tasks, such as medical consultation questions, the dynamic evaluation module 120 can be invoked to conduct a comprehensive evaluation by combining multi-dimensional indicators such as semantic matching, logical consistency, and professionalism scoring.

[0064] For medical treatment plan tasks, such as drug recommendation and medical treatment plan tasks, the multi-model collaboration module 130 can be invoked to form an "expert group" to jointly score the answers and conduct a comprehensive evaluation from multi-dimensional indicators such as accuracy, rationality, and feasibility.

[0065] The dynamic evaluation module 120 provided in the above embodiments is based on the task recognition results (i.e., the target task type and target task difficulty), invokes the preset evaluation index library, and flexibly selects evaluation indicators. For example, for objective questions, "accuracy rate" is used as the target evaluation indicator; for tasks such as medical diagnosis and medical treatment plan with higher complexity, a large model is invoked to conduct an automated comprehensive evaluation with multi-dimensional indicators (such as accuracy, rationality, and feasibility). The dynamic evaluation module 120 supports dynamic adaptation in multi-task scenarios and can significantly improve the flexibility and scientific nature of evaluation.

[0066] In this embodiment, the core goal of the multi-model collaboration module 130 is to form an "expert group" to conduct a joint evaluation of complex tasks by coordinating multiple target medical large models (including professional models or models with stronger performance). The multi-model collaboration module 130 simulates multi-disciplinary consultations in real medical scenarios, objectively evaluates the output results of the model to be evaluated from multiple perspectives, and ensures the professionalism and reliability of the evaluation results. Through the multi-model collaboration mechanism, the comprehensiveness and accuracy of evaluation can be effectively improved, especially for complex tasks such as open-ended question-and-answer and medical treatment plan.

[0067] In this embodiment, the multi-model collaboration module 130 can at least include a model determination unit and a joint evaluation unit. Through the above units, based on the target evaluation indicator, the capabilities of multiple preset target medical large models to process tasks are evaluated to obtain the target evaluation score.

[0068] Among them, for the model determination unit, it is used to determine multiple target medical large models that match the task in the pre-constructed model pool according to the target task type; among them, different medical large models included in the model pool correspond to different task types.

[0069] The model determination unit in the multi-model collaboration module 130 coordinates multiple professional or high-performance target medical large models through the central model to form an "expert group" for joint evaluation of tasks. The specific implementation of this step is as follows: Construct a model pool containing multiple medical large models, and each medical large model has strong performance in a specific field or task type. For example: Medical large model A: Its role is a general practitioner, and it is good at the preliminary diagnosis of common diseases; Medical large model B: Its role is a specialist doctor, and it is good at the diagnosis of diseases in specific medical specialties; Medical large model C: Its role is a medical expert, and it is good at the diagnosis of difficult and miscellaneous medical diseases.

[0070] Then, according to the output content of the task recognition module 110, the task is assigned to the most suitable target medical large model. For example: For an open-ended question-and-answer task, just call medical large model A; For a preliminary diagnosis task, give priority to calling medical large model A and medical large model B; For a medication recommendation task, give priority to calling medical large model B and medical large model C.

[0071] For the joint evaluation unit, it is used to jointly evaluate the task processing capabilities of multiple target medical large models based on the target evaluation indicators to obtain the target evaluation score.

[0072] In this embodiment, the joint evaluation unit is specifically used for: Based on the target evaluation indicators, evaluate the task processing capabilities of multiple target medical large models respectively to obtain multiple first evaluation scores, and determine the target evaluation score from the multiple first evaluation scores based on the voting mechanism; And / or, Based on the target evaluation indicators, evaluate the task ability of any one target medical large model to obtain a second evaluation score, and use other target medical large models to correct the second evaluation score multiple times to obtain the target evaluation score; where the other target medical large models are the other models among the multiple target medical large models except any one target medical large model.

[0073] Specifically, the voting mechanism requires each target medical large model to give an independent evaluation result, that is, the first evaluation score; then select the first evaluation score with the largest weight as the final target evaluation score by counting.

[0074] The debate mechanism initializes the evaluation score by one of the target medical large models to obtain a second evaluation score; other target medical large models have the right to correct the second evaluation score. Then, other target medical large models check whether the second evaluation score is reasonable and correct the second evaluation score according to the result of the reasonableness check. Repeat the above process. When the number of iterations reaches the preset number threshold or all target medical large models have no objections, the latest second evaluation score after multiple iterations and corrections is used as the final target evaluation score.

[0075] The composite mechanism model independently scores, combines the characteristics of the above two modes of the voting mechanism and the debate mechanism, and conducts a voting-based evaluation by target medical large model A and target medical large model B to obtain candidate evaluation scores; then target medical large model C corrects the candidate evaluation scores. In this mode, only target medical large model C has the right to modify the evaluation score.

[0076] Through the above voting mechanism, debate mechanism and their review mechanism, the multi-model collaboration module 130 can improve the efficiency and effect of multi-model collaboration.

[0077] In order to improve the efficiency and effect of multi-model collaboration, the multi-model collaboration module 130 in this embodiment can provide an optimization mechanism for multi-model collaboration. The optimization mechanism for multi-model collaboration mainly includes three aspects: model selection optimization, scoring standard optimization, and collaboration efficiency optimization.

[0078] Among them, the model selection optimization is used to optimize the process of selecting target medical large models. Correspondingly, the model determination unit can also be used for: Determine multiple candidate medical large models matching the task in the pre-constructed model pool according to the target task type; select at least one target medical large model matching the task from multiple candidate medical large models according to the target task difficulty.

[0079] Specifically, for simple tasks where the target task difficulty is lower than the preset difficulty value, one or two target medical large models can be selected and called from multiple candidate medical large models for evaluation, and the consumption of computing resources can be reduced by reducing the number of models.

[0080] For complex tasks where the target task difficulty is not lower than the preset difficulty value, more target medical large models can be selected and called from multiple candidate medical large models for joint evaluation to ensure the comprehensiveness of the evaluation results.

[0081] The model selection optimization dynamically selects the most suitable target medical large model for the current task by analyzing the task difficulty and model capabilities, meeting the requirements of computing resources and evaluation comprehensiveness.

[0082] In this embodiment, the scoring criterion optimization is used to optimize the evaluation criterion, which can be achieved by analyzing the evaluation results of historical tasks and continuously optimizing the scoring criterion. Correspondingly, the multi-model collaboration module 130 further includes: a scoring criterion optimization unit; the scoring criterion optimization unit is used for: Adjust the target evaluation index and its weight according to at least one of the target task type, target task difficulty, and model performance of the target medical large model; Based on the adjusted target evaluation index and its weight, evaluate the capabilities of multiple target medical large models to process tasks respectively, and obtain multiple third evaluation scores; Adjust the preset scoring threshold according to the distribution of the third evaluation scores.

[0083] The embodiment of the scoring criterion optimization can improve the evaluation accuracy by adjusting the target evaluation index and its weight and optimizing the scoring threshold.

[0084] In this embodiment, the collaboration efficiency optimization is used to improve the efficiency of multi-model collaboration through parallel computing and task allocation optimization. Specifically, for independent scoring tasks, a parallel computing method is adopted to reduce the scoring time; for collaborative scoring tasks, a task allocation optimization algorithm is adopted to ensure the balance of scoring tasks.

[0085] According to the above embodiments, the output of the multi-model collaboration module 130 can at least include: The evaluation results of individual target medical large models, that is, the independent evaluation indexes and first evaluation scores of each target medical large model; the comprehensive evaluation results of multiple target medical large models, that is, the final target evaluation scores generated by fusing the scoring results; the evaluation report, which can include a detailed analysis of the scoring results and the advantages and disadvantages of the outputs of each target medical large model, providing a reference for model optimization.

[0086] The multi-model collaboration module 130 provided in the above embodiment coordinates multiple target medical large models to form an "expert group" to jointly evaluate complex tasks, can simulate multi-disciplinary consultations in real medical scenarios, objectively score open-ended answers from multiple perspectives, and ensure the professionalism and reliability of the evaluation results.

[0087] In some embodiments, the large model evaluation system may further include: a sampling module 140 and an arena evaluation module 150; wherein, The sampling module 140 is used to sample the labeled data in the medical scenario to obtain test benchmark data.

[0088] The arena evaluation module 150 is used to conduct adversarial evaluation of the model performances of two models to be evaluated based on the test benchmark data.

[0089] In this embodiment, the sampling module 140 samples the labeled data in the medical scenario to obtain the test benchmark data, which may include the following steps (1)-(4).

[0090] (1) Obtain the labeled data carrying labeled information in the medical scenario; wherein, the labeled information includes: task category, department category.

[0091] Specifically, obtain the real labeled data in the medical scenario. The labeled data carries clear labeled information, including but not limited to: department category (such as internal medicine, surgery, pediatrics, etc.) and task category (such as multiple-choice questions, open-ended questions, treatment plans, etc.).

[0092] (2) Select the first data from the labeled data according to the semantic embedding and clustering algorithm.

[0093] Specifically, analyze the semantic distribution of the labeled data through the semantic embedding and clustering algorithm, and select the first data from the labeled data according to the semantic distribution, so as to ensure the diversity of the sampled data.

[0094] (3) Select the second data from the labeled data according to the task category and department category.

[0095] Specifically, select the second data from the labeled data through the distribution analysis of the task category and department category, so as to ensure the coverage rate of the sampled data.

[0096] (4) Determine the first data and the second data as the test benchmark data.

[0097] According to the data selection methods in steps (2) and (3) above, representative first data and second data can be selected according to the diversity and coverage rate of the labeled data, and the first data and the second data are determined as the test benchmark data.

[0098] Specific examples of the test benchmark data determined in the above manner include, for example, selecting 50 medical practitioner qualification knowledge data from the labeled data of internal medicine, sampling 30 open-ended question data from the surgical data, and sampling 20 treatment plan data from the pediatric data.

[0099] In addition, in this embodiment, the test benchmark data can also be dynamically updated as the medical large model iterates and the task requirements change. For example: after the performance of the medical large model is improved, increase the difficulty of the test benchmark data; after a new task is introduced, increase the test benchmark data related to the new task.

[0100] In this embodiment, through intelligent sampling, representative small-scale data is selected from the labeled data in the medical scenario as the test benchmark data. The labeled data includes clear department categories and task categories. By analyzing the diversity and coverage of the data, the representativeness and scientific nature of the selected test benchmark data are ensured, thereby reducing the consumption of computing resources while ensuring the evaluation quality.

[0101] In this embodiment, the core objective of the arena evaluation module 150 is to introduce an adversarial evaluation mechanism, enabling the model to perform performance evaluation in a pairwise comparison arena mode based on the same test benchmark data, and quantifying the model's capabilities in combination with the ELO (Elo rating system) scoring system.

[0102] Refer to Figure 2 , the arena evaluation module 150 performs adversarial evaluation on the model performances of two models to be evaluated based on the test benchmark data, which can refer to the following steps A1 - A4.

[0103] A1. For two models to be evaluated, perform multiple rounds of arena evaluation operations based on the test benchmark data. The arena evaluation operations include steps A2 and A3.

[0104] In a specific embodiment, models with similar performances can be selected for confrontation through a dynamic matching algorithm, and the selected models are the models to be evaluated. Among them, in the initial stage of the adversarial evaluation, or in the stage of performing the arena evaluation operations in the first N times, a basic model and an improved model can be selected for confrontation to evaluate the improvement effect of the model. In the subsequent stage, or in the stage of performing the arena evaluation operations after N times, model versions with similar performances are selected for confrontation to ensure the fairness and scientific nature of the evaluation.

[0105] A2. Evaluate the model performances of each model to be evaluated based on the test benchmark data respectively to obtain the current evaluation scores.

[0106] In each round of arena evaluation operation, the two models to be evaluated generate their respective answers for the same test benchmark data, and the answers are evaluated through evaluation metrics to obtain the corresponding current evaluation scores. Finally, the winner and loser are determined based on the respective current evaluation scores of the two models to be evaluated.

[0107] Through multiple rounds of arena evaluation operations, comprehensively evaluate the performances of the two models to be evaluated on different tasks. For example: in the first round of arena evaluation operation, evaluate the performances of the two models to be evaluated on the multiple-choice question task; in the second round of arena evaluation operation, evaluate the performances of the two models to be evaluated on the open-ended question task; in the third round of arena evaluation operation, evaluate the performances of the two models to be evaluated on the diagnosis and treatment plan task.

[0108] It can be understood that in each round of competitive evaluation operation, each model to be evaluated will obtain its respective current evaluation score.

[0109] A3. Update the current ELO scores of each model to be evaluated respectively according to the two current evaluation scores; among them, when the competitive evaluation operation is executed for the first time, the current ELO score of the model to be evaluated is the preset initial ELO score.

[0110] In this embodiment, the arena evaluation module 150 uses the ELO scoring system to quantify the capabilities of the models. The ELO scoring system is a scoring method widely used in arena evaluations, which can dynamically adjust the scores of the models and reflect the relative strengths of the models.

[0111] In a specific implementation manner, an initial ELO score is preset for each model to be evaluated, usually 1500 points. For example: Model A: 1500 points, Model B: 1500 points; Model A and Model B are two models to be evaluated.

[0112] In the confrontation of each round of competitive evaluation operation, taking any one of the models to be evaluated as an example, referring to the following formula, update the current ELO score of the model to be evaluated according to the win-loss results of the two models to be evaluated: R′ = R + K * (S - E) Wherein, R is the current ELO score of the model to be evaluated, R′ is the updated ELO score of the model to be evaluated; K is the score update coefficient, usually taking a value of 32; S is the actual game result, the winner gets ① point, the loser gets 0 points, and a draw gets 0.5 points; E is the expected winning rate of the model to be evaluated.

[0113] A4. When multiple rounds of competitive evaluation operations are completed, determine the current latest ELO scores of each model to be evaluated as their respective target ELO scores; the target ELO scores are used to represent the model performance of the models to be evaluated.

[0114] In this embodiment, multiple rounds of competitive evaluation operations are executed to achieve multiple rounds of confrontation, and the ELO scores of the models to be evaluated are continuously updated. When the preset number of rounds of competitive evaluation operations is completed, the target ELO scores of each model to be evaluated are finally determined. For example, after 10 rounds of competitive evaluation operations, the ELO score of Model A is updated to 1600 points, and the ELO score of Model B is updated to 1550 points.

[0115] According to the above embodiments, the arena evaluation module 150 can provide clear feedback for model optimization by conducting adversarial evaluations on the model performances of two models to be evaluated. The specific implementation is as follows: 1) Win-loss statistics. Count the win-loss results of the two models to be evaluated in each round of competitive evaluation operations, and analyze the performance of the two models to be evaluated on different tasks. For example: The winning rate of Model A in the multiple-choice question task is 80%, and the winning rate of Model B in the open-ended question task is 60%.

[0116] 2) Scoring trend analysis. Analyze the improvement effects of the two models to be evaluated through the changing trends of ELO scores. For example: The ELO score of Model A increases from 1500 to 1600, indicating a significant improvement in model performance; the ELO score of Model B decreases from 1500 to 1450, indicating that the model performance needs further optimization.

[0117] 3) Task performance analysis. Analyze the advantages and disadvantages of the two models to be evaluated according to the evaluation results of task categories. For example: Model A performs excellently in the multiple-choice question task but poorly in the diagnosis and treatment plan task; Model B performs well in the open-ended question task but has problems with logical consistency.

[0118] 4) Optimization suggestions. Provide specific suggestions for model optimization based on the evaluation results. For example: Enhance the logical reasoning ability of Model A in the diagnosis and treatment plan task; improve the semantic matching ability of Model B in the open-ended question task.

[0119] According to the above embodiments, the output of the arena evaluation module 150 can at least include: The ELO score, that is, the final ELO score of each model to be evaluated, reflecting the comprehensive strength of the model to be evaluated; the win-loss statistics, that is, the win-loss results of each round of confrontation, reflecting the performance of the model to be evaluated on different tasks; the evaluation report, which can include scoring trend analysis, task performance analysis and optimization suggestions, providing clear feedback for the iteration of the model to be evaluated.

[0120] In the above embodiments, the sampling module 140 selects labeled data in the medical scenario to determine a representative small-scale data as the test benchmark data for the arena. The arena evaluation module 150 conducts adversarial scoring on the models to be evaluated on the test benchmark data through pairwise comparison, and combines the ELO scoring system to quantify the performance of the models, which can ensure the efficiency and scientific nature of the evaluation and comprehensively quantify the improvement effect of the models. Through the arena evaluation mechanism, not only can the scientific nature and objectivity of the evaluation be improved, but also the consumption of computing resources can be significantly reduced, and at the same time, clear feedback can be provided for model iteration.

[0121] In summary, the medical large model evaluation system provided by the embodiments of the present disclosure, through the modular design of the task recognition module 110, the dynamic evaluation module 120, the multi-model collaboration module 130, the sampling module 140, and the arena evaluation module 150, can effectively conduct a comprehensive evaluation of the medical large model for multi-task scenarios, significantly improving the scientificity, objectivity, and adaptability of the evaluation, and providing an efficient evaluation tool for the development and iteration of the medical large model.

[0122] Among them, the task recognition module 110 can comprehensively improve the data quality. Through the comprehensive evaluation of multi-dimensional indicators, high-quality data can be screened out. This not only provides a solid data foundation for model fine-tuning, but also effectively reduces the negative impact of low-quality data on model training, significantly reducing the computational workload of subsequent processing and improving the data processing efficiency.

[0123] The dynamic evaluation module 120 can flexibly call multi-dimensional evaluation indicators (such as semantic matching, logical consistency, and professionalism scoring) according to the target task type and target task difficulty to achieve accurate evaluation of the model output. By dynamically adjusting the weights and comprehensive scoring mechanism, this module ensures the scientificity and comprehensiveness of the evaluation results, provides a clear direction for model optimization, and significantly improves the performance of the model in complex tasks.

[0124] The multi-model collaboration module 130 coordinates multiple target medical large models to jointly evaluate complex tasks by introducing the "expert group" mechanism. This module can comprehensively analyze the model output from multiple perspectives to ensure the professionalism and reliability of the evaluation results. At the same time, through the dynamic task allocation and score fusion mechanism, the evaluation efficiency is improved, especially suitable for high-professional scenarios such as medical care.

[0125] The sampling module 140 efficiently screens out representative small-scale test benchmark data from massive data through semantic diversity analysis and coverage analysis, significantly reducing the consumption of computing resources; this mechanism can dynamically update the test benchmark data to adapt to different task requirements, ensuring the comprehensiveness and pertinence of the evaluation results.

[0126] The arena evaluation module 150 can select representative models to be evaluated for comparison to comprehensively evaluate the improvement effect of the models. The arena evaluation module 150 quantifies the relative capabilities of the models through the adversarial evaluation mechanism and the ELO scoring system; through multiple rounds of confrontation and scoring trend analysis, the objectivity and scientificity of the evaluation are significantly improved, providing a reliable basis for model iteration.

[0127] Therefore, through modular design, the present disclosure organically combines quality assessment, dynamic assessment, multi-model collaboration, arena assessment, and intelligent sampling mechanism to form an efficient, accurate, and professional medical large model evaluation system. This medical large model evaluation system not only improves the performance of the model in professional tasks but also significantly reduces the costs of data processing and evaluation, providing strong support for the application of the model in highly professional fields such as medicine.

[0128] Embodiment 2: Corresponding to the above-mentioned medical large model evaluation system, as Figure 3 shown, this embodiment provides a medical large model evaluation method, including the following steps: S102, identifying the target task type and target task difficulty of the input task; S104, determining at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and target task difficulty; and, S106, evaluating the capabilities of a preset plurality of target medical large models to process tasks based on the target evaluation index to obtain a target evaluation score.

[0129] In one embodiment, the medical large model evaluation method may further include the following steps: Sampling the labeled data in the medical scenario to obtain test benchmark data; Performing adversarial evaluation on the model performances of two models to be evaluated based on the test benchmark data.

[0130] For the method provided in this embodiment, its implementation principle and the resulting technical effects are the same as those of the foregoing system embodiment. For a brief description, for parts not mentioned in the method embodiment, reference may be made to the corresponding content in the foregoing system embodiment.

[0131] Embodiment 3: Figure 4 The following is a schematic structural diagram of an electronic device provided for an embodiment of the present disclosure. As Figure 4 shown, the electronic device 400 includes one or more processors 401 and a memory 402.

[0132] The processor 401 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0133] The memory 402 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 401 may run the program instructions to implement the medical large model evaluation method of the embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage media.

[0134] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0135] In addition, the input device 403 may further include, for example, a keyboard, a mouse, and so on.

[0136] The output device 404 may output various information to the outside, including the determined distance information, direction information, etc. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0137] Of course, for simplicity, Figure 4 only some of the components related to the present disclosure in the electronic device 400 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 400 may further include any other appropriate components.

[0138] Embodiment 4: Furthermore, this embodiment further provides a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is used to execute the above-mentioned medical large model evaluation method.

[0139] The computer program product of a medical large model evaluation method, device, electronic device and medium provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the methods described in the previous method embodiments. For specific implementation, reference can be made to the method embodiments, which will not be elaborated here.

[0140] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0141] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical large model evaluation system, characterized in that, The system includes: A task recognition module for recognizing the target task type and target task difficulty of the input task; A dynamic evaluation module for determining at least one target evaluation index matching the task from a preset evaluation index library according to the target task type and the target task difficulty; and A multi-model collaboration module for evaluating the capabilities of a preset plurality of target medical large models in processing the task based on the target evaluation index to obtain a target evaluation score.

2. The system according to claim 1, characterized in that The task recognition module is further configured to: Obtain the task information of the input task; wherein, the task information includes: the input question, the standard answer, and the output answer of the model to be evaluated; Recognize the target task type of the task according to the task information; Calculate the semantic complexity score and context dependence score of the task; Determine the target task difficulty of the task based on the semantic complexity score and the context dependence score.

3. The system according to claim 1, wherein The dynamic evaluation module is further configured to: Determine candidate evaluation indexes matching the target task type from a preset evaluation index library according to a preset matching relationship between the task type and the evaluation indexes; Select at least one target evaluation index matching the task from the candidate evaluation indexes according to the target task difficulty.

4. The system according to claim 1, wherein The multi-model collaboration module includes: A model determination unit for determining a plurality of target medical large models matching the task in a pre-constructed model pool according to the target task type; wherein, different medical large models included in the model pool correspond to different task types; A joint evaluation unit for jointly evaluating the capabilities of the plurality of target medical large models in processing the task based on the target evaluation index to obtain a target evaluation score.

5. The system according to claim 4, characterized in that, The model determination unit is further configured to: Determine a plurality of candidate medical large models matching the task in a pre-constructed model pool according to the target task type; Select at least one of the target medical large models matching the task from the plurality of candidate medical large models according to the target task difficulty.

6. The system according to claim 4, wherein The joint evaluation unit is further configured to: Evaluate the capabilities of the plurality of target medical large models in processing the task respectively based on the target evaluation index to obtain a plurality of first evaluation scores, and determine the target evaluation score from the plurality of first evaluation scores based on a voting mechanism; And / or Evaluate the capability of any one of the target medical large models in processing the task based on the target evaluation index to obtain a second evaluation score, and correct the second evaluation score multiple times through other target medical large models to obtain the target evaluation score; wherein, the other target medical large models are other models among the plurality of target medical large models except the any one target medical large model.

7. The system according to claim 4, characterized in that, The multi-model collaboration module further includes: a scoring criterion optimization unit; the scoring criterion optimization unit is configured to: Adjust the target evaluation index and its weight according to at least one of the target task type, the target task difficulty, and the model performance of the target medical large model. Based on the adjusted target evaluation metrics and their weights, the capabilities of multiple target medical large models in processing the task are respectively evaluated to obtain multiple third evaluation scores; According to the distribution of the third evaluation scores, the preset scoring threshold is adjusted.

8. The system according to claim 1, wherein The system further includes: a sampling module and an arena evaluation module; wherein, The sampling module is used to sample the labeled data in the medical scenario to obtain test benchmark data; The arena evaluation module is used to perform adversarial evaluation on the model performances of two models to be evaluated based on the test benchmark data.

9. The system according to claim 8, wherein, The sampling module is further used for: Obtaining labeled data carrying labeling information in the medical scenario; wherein, the labeling information includes: task category, department category; Selecting first data from the labeled data according to semantic embedding and clustering algorithms; Selecting second data from the labeled data according to the task category and the department category; Determining the first data and the second data as the test benchmark data.

10. The system according to claim 8, wherein The arena evaluation module is further used for: For the two models to be evaluated, performing multiple rounds of arena evaluation operations based on the test benchmark data, and the arena evaluation operations include: Evaluating the model performances of the models to be evaluated respectively based on the test benchmark data to obtain current evaluation scores; and Updating the current ELO scores of the models to be evaluated respectively according to the two current evaluation scores; wherein, when the arena evaluation operation is performed for the first time, the current ELO score of the model to be evaluated is the preset initial ELO score; When multiple rounds of the arena evaluation operations are completed, respectively determining the current latest ELO scores of the models to be evaluated as their respective target ELO scores; the target ELO scores are used to represent the model performances of the models to be evaluated.

Images