Evaluation optimization method for medical big language model evaluation questions

Abstract

The invention discloses an evaluation optimization method for medical big language model evaluation questions, and the method comprises the steps: firstly evaluating the medical fact consistency, logic self-consistency and expression definition of the evaluation questions, and obtaining a medical fact consistency score, a logic self-consistency score and an expression definition score; and then calculating a comprehensive quality score of the evaluation question based on the medical fact consistency score, the logic self-consistency score and the expression definition score, comparing the comprehensive quality score with a threshold value, if the comprehensive quality score is greater than or equal to the threshold value, determining that the quality is qualified, otherwise, optimizing the evaluation question, and evaluating again until an expected requirement is met. The method can effectively solve the problem that a generative model is easy to generate illusion, and ensures the medical accuracy of evaluation questions.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to an evaluation and optimization method for medical large language model assessment questions. Background Technology

[0002] In recent years, medical large-scale language models have developed rapidly. To ensure their performance, they need to be evaluated before formal use, including assessments of medical knowledge mastery, clinical reasoning ability, and safety. The quality and scale of the evaluation set directly determine the reliability of the evaluation results. Currently, the generation of evaluation sets for medical large-scale language models mainly relies on manual construction by medical experts. These experts manually extract key information, write questions (Prompts), and formulate standard answers (Ground Truths) by consulting electronic medical records (EMRs), medical guidelines, or literature. Specifically, the process begins with manual selection of materials from public datasets such as MIMIC-III and PubMedQA, or from anonymized hospital data. Then, simple scripts are used to remove garbled characters or irrelevant characters for data cleaning. Subsequently, medical experts write question-answer pairs based on the materials, perform manual annotation, and finally format the pairs, saving them as JSON or CSV for evaluation. Currently, some technical solutions attempt to use large models to assist in generating evaluation sets, that is, to use general large models, such as ChatGPT, to directly generate questions based on a piece of text. However, manual review is still required to ensure the accuracy of medical facts.

[0003] It is evident that existing methods heavily rely on manual labor. The time-consuming and labor-intensive process of manually selecting and writing questions makes it difficult to construct large-scale evaluation sets covering multiple diseases and departments in a short period. Therefore, they cannot keep up with the demands of frequent model iterations, exhibiting insufficient automation and scalability. Furthermore, existing technologies lack automated structuring and standardization mechanisms. Medical data is highly heterogeneous, and the lack of automated processing and unified standards makes it difficult to form highly consistent datasets. This results in inconsistent semantic expression styles and difficulty distributions in the generated evaluation samples, affecting the fairness of the evaluation results. In existing processes, the generated content often lacks systematic quality control methods, easily introducing "toxic samples" containing medical factual errors or logical contradictions, severely impacting the accuracy of the evaluation and making it difficult to guarantee data quality. Moreover, the generation and evaluation of existing evaluation sets are disconnected, failing to utilize referee models to score and provide feedback on the generated samples. This prevents low-quality samples from being automatically corrected, hindering the self-improvement of dataset quality. Overall, existing evaluation set generation methods have fragmented preparation processes, with tools at each stage being disjointed. They lack a unified system to manage the entire lifecycle from raw data acquisition to final dataset output, making end-to-end management difficult and limiting their ability to be continuously updated. Summary of the Invention

[0004] To address some or all of the problems in existing technologies and in order to generate a high-quality evaluation set for medical large language models, this invention provides an evaluation optimization method for medical large language model evaluation questions, comprising: The medical fact consistency, logical consistency, and clarity of expression of the assessment questions are evaluated to obtain a medical fact consistency score. Logical consistency score and the score for clarity of expression ; Based on the medical fact consistency score, logical consistency score, and clarity of expression score, the overall quality score of the assessment questions is calculated. The overall quality score is compared with a threshold. If it is greater than or equal to the threshold, the quality is considered acceptable. Otherwise, the assessment questions are optimized and re-evaluated until the expected requirements are met.

[0005] Furthermore, the quality of the assessment questions is evaluated using a referee model trained on subjective question quality evaluation data annotated by medical experts, wherein the subjective question quality evaluation data includes question samples, expert quality scores, and expert scoring reasons.

[0006] Furthermore, the training objectives of the referee model include: The quality score prediction task takes the question content and raw medical data as input and outputs the score results for each quality dimension. Error type classification task, which is used to identify the types of errors present in the questions, including medical factual errors, logical contradictions, missing information, and ambiguity of expression; The modification suggestion generation task learns to generate structured modification suggestion text based on expert reasoning annotations.

[0007] Furthermore, the medical factual consistency score is used to measure whether the sample of assessment questions conflicts with original medical data or authoritative guidelines, and it is calculated according to the following formula: , in, This represents the i-th medical fact in the sample of assessment questions. This indicates the corresponding facts in Yuan Shu's data source, and This is a consistency indicator function.

[0008] Furthermore, the logical consistency score is used to evaluate whether the conditions given in the question stem of the assessment question are sufficient to derive the reference answer and whether there is an internal logical contradiction. It is obtained by simulating the reasoning process through the referee model.

[0009] Furthermore, the clarity score is used to assess whether the language of the assessment question is fluent and whether there is any ambiguity or risk of multiple interpretations. It is calculated by combining the semantic consistency and ambiguity detection results through the adjudication model.

[0010] Furthermore, the overall quality score of the assessment questions is calculated based on the following formula: , in, , , Preset weights.

[0011] Furthermore, optimizing the assessment questions includes: The test question samples are subjected to error diagnosis and structured analysis. Based on the error type and misalignment location results, the corresponding correction strategy is selected to modify the test question samples, and the question content is regenerated based on the original medical data.

[0012] Furthermore, the correction strategy includes: If the error is located in a certain option but there is no problem in the question stem itself, then the content of the question stem remains unchanged, and only the problematic option is regenerated; If the information in the question stem is insufficient to support the current answer, then the question stem should be supplemented without changing the core meaning of the question. If the conditions in the question stem are reasonable, but the current answer cannot be derived from the information in the question stem, and the question stem cannot be modified, then medical reasoning will be performed again based on the conditions in the question stem, and the content of the corresponding options will be adjusted accordingly. If the question is unclear or ambiguous, the question stem or options should be rewritten with semantic equivalence.

[0013] Furthermore, the evaluation optimization method also includes: If the number of corrections exceeds a preset value and the assessment question still does not meet the expected requirements, then the assessment question is discarded.

[0014] This invention provides an evaluation and optimization method for medical large language model assessment questions. It evaluates and optimizes the generated assessment questions in real time, effectively addressing the hallucination problem inherent in generative models and ensuring the medical accuracy of the assessment set. The feedback-based iterative mechanism allows the system to continuously correct errors, resulting in increasingly higher quality samples as the runtime increases. Attached Figure Description

[0015] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0016] Figure 1 This is a flowchart illustrating a method for generating an automatically optimized medical large language model evaluation set according to an embodiment of the present invention. Figure 2 This diagram illustrates a flowchart of a medical data standardization processing method according to an embodiment of the present invention. Figure 3 This diagram illustrates a version management and release method for a medical large language model evaluation set according to an embodiment of the present invention. Figure 4 The diagram shows a structural schematic of a system for generating an automatically optimized medical large language model evaluation set according to an embodiment of the present invention. Detailed Implementation

[0017] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific quantities, materials, and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0018] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0019] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0020] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor.

[0021] To achieve automated production of high-quality, large-scale medical evaluation data, this invention provides a fully automated method and system for preparing medical evaluation datasets. The system includes a closed-loop process of large-scale model generation, quality control of the judge model, and feedback iteration. Specifically, it utilizes a generative large-scale model to batch-construct initial evaluation samples based on intermediate data. Then, a finely tuned judge model is introduced to score the generated samples across multiple dimensions, including factual accuracy, logical coherence, and clarity. Finally, a negative feedback loop is constructed, using judge opinions to guide the generation model in correcting low-scoring samples, ultimately achieving end-to-end automated preparation and distribution.

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0023] Figure 1 This diagram illustrates a flowchart of a method for generating an automatically optimized medical large language model evaluation set according to an embodiment of the present invention. Figure 1 As shown, a method for generating an automatically optimized evaluation set for a large medical language model includes: First, in step 101, a sample of assessment questions is generated. The sample of assessment questions is generated based on real medical data, and includes a question stem, options, a standard answer, and explanations. In one embodiment of the invention, the sample of assessment questions is generated through a large-scale model-based assessment sample generation unit. This unit utilizes the generation capabilities of a large language model to batch-construct assessment questions based on standardized data. In another embodiment of the invention, standardized data is injected into pre-built prompt word templates to guide the large-scale model to generate a complete sample containing a "question stem," "options," "standard answer," and "explanation." The prompt word templates include multiple sets for different question types, such as single-choice, multiple-selection, case analysis, and question-and-answer. In another embodiment of the invention, to ensure the generated questions are rich in sentence structure and examination perspectives, and to avoid repetitive patterns, random seeds and diversity parameters can be introduced during the generation process. In one embodiment of the present invention, the standardized data is obtained by standardizing real medical data, wherein the real medical data includes electronic medical record system (EMR) data, medical literature database data, clinical guideline database data, and publicly available medical knowledge graphs, etc., wherein, for example, paper medical record scans or image-format medical reports can be processed by an optical character recognition (OCR) module. Figure 2 This diagram illustrates a flowchart of a medical data standardization processing method according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, the medical data standardization processing method includes: First, in step 201, data anonymization is performed. Sensitive information in the medical data is identified and masked. In one embodiment of the present invention, sensitive information such as patient name, ID, and address is automatically identified and masked using Named Entity Recognition (NER) technology. Next, in step 202, the mapping is standardized. Named entity recognition (NER) technology is used to automatically identify information such as diseases, drugs, and examinations, and heterogeneous terms from different sources are standardized and mapped to standard medical terms such as SNOMED CT and ICD10 to eliminate ambiguity. Finally, in step 203, structuring is performed. Using natural language processing technology, unstructured medical records, discharge summaries, and other text are converted into structured JSON objects. Specifically, the standardized medical data is structurally reorganized to obtain structured JSON objects. For example, "The patient had a fever yesterday, with a temperature of 39 degrees Celsius" is transformed into {"symptom": "fever", "value": "39℃", "time": "yesterday"}. After standardization, the structured medical data is injected into the prompt word template, which automatically generates sample assessment questions. Next, in step 102, quality assessment. The quality of the test question sample is assessed to address the correctness issue of automatically generated content. In one embodiment of the invention, the quality of the test question sample is assessed using a referee model. This referee model is trained on a large amount of subjective question quality evaluation data annotated by human medical experts. The training data includes at least question samples, expert quality scores, and expert scoring reasons. The question samples include, but are not limited to, different types of medical subjective questions such as case analysis questions, diagnostic reasoning questions, treatment decision-making questions, and guideline comprehension questions. Each question includes a stem, candidate answers or reference answers, and a corresponding original medical data source, such as medical record excerpts or guideline excerpts. The expert quality scores are obtained by cross-scoring of question quality by at least two medical experts. The scoring dimensions cover consistency of medical facts, logical consistency, and clarity of expression. Each dimension uses discrete rank or continuous numerical scoring and is normalized to a score of 0 to 1. The expert scoring reasons refer to the clear reasons given by experts for low or medium-scoring questions, such as: the option conflicts with the patient's drug allergy history. The scoring reasons are used as supervisory signals to train the judging model's reason generation and error localization capabilities. In one embodiment of the invention, the judging model employs a fine-tuning or instruction-based fine-tuning method based on a large language model. Its training objectives include a quality score prediction task, an error type classification task, and a modification suggestion generation task. The input to the quality score prediction task is the question content and original medical data, and the output is the score results for each quality dimension. The error type classification task is used to identify the error types present in the question, such as medical factual errors, logical contradictions, missing information, and ambiguous expressions. The modification suggestion generation task learns to generate structured modification suggestion text based on expert reason annotations. To enhance the model's generalization ability to different question types, in one embodiment of the invention, different types of subjective question prompt templates are introduced during training to simulate evaluation instructions in real-world usage scenarios. As mentioned above, in one embodiment of the invention, the question quality includes a medical factual consistency score, a logical consistency score, and a clarity score. The medical factual consistency score is used to measure whether the questions and answers in the assessment sample conflict with original medical data or authoritative guidelines. In one embodiment of the invention, the consistency confidence level is obtained by using the adjudication model to judge the medical entities, attributes, and relationships involved in the questions. , in, This represents the i-th medical fact in the sample of assessment questions. This indicates the corresponding facts in Yuan Shu's data source, and This is a consistency indicator function.

[0024] The logical consistency score is used to evaluate whether the conditions given in the question stem of the evaluation question sample are sufficient to derive the reference answer and whether there are internal logical contradictions. In one embodiment of the present invention, the reasoning process is simulated through the referee model to determine the derivability from the question stem to the answer, and its score is defined as follows: , Its value is estimated by the referee model through condition generation or discrimination.

[0025] The clarity of expression score is used to assess whether the language of the test question samples is fluent and whether there is any ambiguity or risk of multiple interpretations. In one embodiment of the present invention, the clarity of expression score is calculated by combining the semantic consistency and ambiguity detection results through the referee model. It should be understood that, in some other embodiments of the present invention, an ensemble learning strategy may also be used to evaluate the test sample, that is, to introduce multiple models with different architectures, such as BERT, GPT-4, Claude, etc., to form a "jury" and decide whether the sample is qualified through a voting mechanism, so as to further reduce the false positive rate; Next, in step 103, threshold determination. Based on the medical factual consistency score, logical consistency score, and clarity of expression score, the overall quality score of the assessment question sample is calculated: , in, , , The weights are preset. The overall quality score is compared with a threshold. If it is greater than or equal to the threshold, it indicates that the corresponding evaluation question sample meets the expected requirements and is stored in the candidate pool. If the overall quality score is lower than the threshold, the corresponding evaluation question sample is marked as "to be optimized". In one embodiment of the present invention, a "human-in-the-loop" mode can be further introduced, in which marginal samples with low confidence of the adjudication model are automatically distributed to human experts for final adjudication, and the adjudication results are used to fine-tune the adjudication model. Next, in step 104, sample optimization. The evaluation question samples marked "to be optimized" are corrected and optimized. In one embodiment of the invention, the referee model is used to perform error diagnosis and structured analysis on the evaluation question samples marked "to be optimized". Specifically, the referee model performs alignment analysis on the question content, reference answer, and original medical data source to identify quality defects in the questions and output structured error diagnosis information. The structured diagnosis information includes at least error type, error location, and modification suggestion text to guide the subsequent question correction process. The error type describes the nature of the question quality defect, such as inconsistencies between the generated content and the original medical records or medical guidelines, medical conflicts, or logical contradictions where the question stem conditions cannot support the current answer's reasoning conclusion. The error location identifies the specific location where the error occurs, such as a statement in the question stem, a candidate option, or the reasoning relationship between the answer and the question stem. The modification suggestion text refers to the modification suggestion text for the corresponding error, given in natural language, explaining the cause of the error and suggesting feasible correction directions, for example: Option C's description of the drug conflicts with the patient's penicillin allergy history in the question stem. Through the structured parsing method described above, the system can clearly distinguish whether a problem stems from a conflict in medical facts, insufficient logical reasoning, or unclear expression of the question, thus providing the generation unit with a directly executable optimization basis. After the error diagnosis information is sent to the evaluation sample generation unit, the evaluation sample generation unit selects an appropriate correction strategy to modify the question based on the error type and error location result, and regenerates the question content based on the original medical data. In one embodiment of the present invention, the correction strategy includes an option correction strategy, a question stem supplementation strategy, an answer re-derivation strategy, and an expression rewriting strategy. When the error is located in a certain option and the question stem itself is not problematic, the generation unit keeps the question stem content unchanged and only regenerates the problematic option. For example, if the question stem clearly states that the patient has a history of penicillin allergy, and if a certain option recommends the use of penicillin-type drugs, the generation unit, after parsing the medical conflict, will regenerate an alternative treatment plan without relevant contraindications based on the same disease background and replace the original option with the new option content. When the adjudication model determines that the information provided by the question stem is insufficient to support the current answer, i.e., the logical consistency is insufficient, the generation unit supplements the question stem without changing the core semantics of the question. For example, in a diagnostic reasoning question, if only the patient's chief symptoms are described and multiple possible diagnoses cannot be distinguished, the generation unit can supplement key laboratory test results or imaging findings based on the original case data, thus ensuring that the question stem conditions uniquely support the correct answer. When the question stem conditions are reasonable but the current answer cannot be derived from the question stem information, and the question stem cannot be modified, the generation unit re-performs medical reasoning based on the question stem conditions, selects a more reasonable answer, and simultaneously adjusts the corresponding option content.For example, if the question stem describes a patient with typical signs of heart failure, and the original answer points to an infectious disease, the generation unit will re-derive the diagnosis and correct the answer to a heart failure-related diagnosis consistent with the question stem's conditions. When the judging model identifies unclear wording or semantic ambiguity in the question, the generation unit rewrites the question stem or options semantically. For example, vague statements such as "recently elevated abnormal indicators" are rewritten as medical descriptions that clearly define specific indicators and their magnitudes, thereby improving the clarity of the question. It should be understood that these correction strategies can be implemented individually or in combination as needed.

[0026] After the correction is completed, the quality of the optimized test question samples is evaluated again. If they are qualified, they are stored in the candidate pool; otherwise, they are optimized again. In one embodiment of the present invention, in order to improve efficiency, if the number of corrections exceeds a preset value and the test question samples still do not meet the expected requirements, the test question samples are discarded. That is, before entering step 104, it is first checked whether the number of iterations exceeds the upper limit. If it exceeds the upper limit, the process proceeds to step 105 and the samples are discarded; if it does not exceed the upper limit, the process proceeds to step 104. Finally, in step 106, the evaluation set is packaged and published. When the number of evaluation question samples in the candidate pool reaches a preset value, or the generation time reaches a preset value, the dataset is automatically packaged and generated. Figure 3 This diagram illustrates a version management and release method for a medical large language model evaluation set according to an embodiment of the present invention. Figure 3 As shown, the version management and release of the medical large language model evaluation set includes: First, in step 301, consistency and distribution detection is performed. A consistency distribution detection is conducted on the evaluation question samples in the candidate pool. If the detection passes, proceed to step 302; otherwise, continue waiting for the generation of new evaluation question samples. Next, in step 302, version tagging. Each release requires version tagging of the generated evaluation set (e.g., v1.0, v1.1), recording the generation time, data source range, and pass rate statistics for version management purposes; Next, in step 303, a statistical report is generated. Based on version information and consistency and distribution detection results, the generated statistics include recording information such as the departmental distribution and difficulty distribution of the evaluation dataset; Finally, in step 304, the format is encapsulated and published. The evaluation dataset is exported with one click to a format supported by the evaluation framework, such as HuggingFace Dataset, JSON, Parquet, etc., achieving end-to-end delivery, published to the evaluation platform, and archived for historical versions.

[0027] Thus, the automatic generation of the medical large language model evaluation set has been completed. It integrates the entire process of data access, generation, evaluation, iteration and release, and realizes large-scale preparation with no or very little human intervention.

[0028] Based on the generation method described above Figure 4 This diagram illustrates the structure of a system for generating an automatically optimized medical large language model evaluation set, according to an embodiment of the present invention. Figure 4 As shown, a system for generating an automatically optimized medical large language model evaluation set includes a data processing module 401, a generation module 402, an evaluation module 403, an optimization module 404, and a publishing module 405. The data processing module 401 addresses the issues of diverse data sources and complex structures by de-identifying, standardizing, and structuring data from different sources. The generation module 402 generates evaluation question samples based on structured medical data and regenerates the evaluation question samples based on the modification suggestions from the optimization module 404. The evaluation module 403 assesses the quality of the generated evaluation question samples. The optimization module 404 provides modification suggestions based on the evaluation results from the evaluation module. The publishing module 405 is responsible for the final data encapsulation and publishing.

[0029] The generation method and system in this invention adhere to unified standards throughout the entire process, from data cleaning at the source to the final dataset release. This eliminates subjective biases and formatting issues caused by manual operation, enabling the generation of tens of thousands of high-quality evaluation questions within hours at a cost that is only a fraction of that of manual preparation. This meets the urgent need for frequent model iterations. The introduction of the referee model is equivalent to equipping each question with a "24-hour online review expert," effectively solving the problem of hallucination that generative models are prone to, ensuring the medical accuracy of the evaluation set, and significantly improving data quality and consistency. The feedback-based iterative mechanism can continuously correct errors; as the running time increases, the quality of the generated samples will become increasingly higher, rather than stagnating like in traditional methods.

[0030] Experiments on the MedBench Chinese medical large-scale model evaluation platform, using actual medical datasets, have verified that the generation method described above can stably generate medical evaluation datasets that meet the expected structure and quality requirements. The automatic verification and judging mechanism can effectively identify and correct low-quality samples, validating the feasibility of this technical solution in practical applications. Furthermore, the structured question-answer pairs generated by the aforementioned method can not only be used for evaluation but also as high-quality instruction tuning data to improve the training effect of large-scale medical models. The generation method can also be extended to other vertical fields such as law and finance, where the accuracy of facts is extremely important, to construct corresponding professional evaluation sets.

[0031] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. An evaluation and optimization method for medical large language model assessment questions, characterized in that, include: The medical fact consistency, logical consistency, and clarity of expression of the assessment questions are evaluated to obtain a medical fact consistency score. Logical consistency score and the score for clarity of expression ; Based on the medical fact consistency score, logical consistency score, and clarity of expression score, the overall quality score of the assessment questions is calculated. The overall quality score is compared with a threshold. If it is greater than or equal to the threshold, the quality is considered acceptable. Otherwise, the assessment questions are optimized and re-evaluated until the expected requirements are met.

2. The evaluation and optimization method as described in claim 1, characterized in that, The quality of the assessment questions is evaluated using a referee model trained on subjective question quality evaluation data annotated by medical experts. This subjective question quality evaluation data includes question samples, expert quality scores, and expert scoring reasons.

3. The evaluation and optimization method as described in claim 2, characterized in that, The training objectives of the referee model include: The quality score prediction task takes the question content and raw medical data as input and outputs the score results for each quality dimension. Error type classification task, which is used to identify the types of errors present in the questions, including medical factual errors, logical contradictions, missing information, and ambiguity of expression; The task of generating modification suggestions learns and generates structured modification suggestion text based on expert annotations.

4. The evaluation and optimization method as described in claim 1, characterized in that, The medical fact consistency score is used to measure whether the sample of assessment questions conflicts with original medical data or authoritative guidelines, and it is calculated according to the following formula: ， in, This represents the i-th medical fact in the sample of assessment questions. This indicates the corresponding facts in Yuan Shu's data source, and This is a consistency indicator function.

5. The evaluation and optimization method as described in claim 2, characterized in that, The logical consistency score is used to evaluate whether the conditions given in the question stem of the assessment question are sufficient to derive the reference answer and whether there is an internal logical contradiction. It is obtained by simulating the reasoning process through the judge model.

6. The evaluation and optimization method as described in claim 2, characterized in that, The clarity score is used to assess whether the language of the assessment question is fluent and whether there is any ambiguity or risk of multiple interpretations. It is calculated by combining the semantic consistency and ambiguity detection results through the adjudication model.

7. The evaluation and optimization method as described in claim 1, characterized in that, The overall quality score of the assessment questions is calculated based on the following formula: ， in, , , Preset weights.

8. The evaluation and optimization method as described in claim 1, characterized in that, Optimizing the assessment questions includes: The test question samples are subjected to error diagnosis and structured analysis. Based on the error type and misalignment location results, the corresponding correction strategy is selected to modify the test question samples, and the question content is regenerated based on the original medical data.

9. The evaluation and optimization method as described in claim 8, characterized in that, The correction strategy includes: If the error is located in a certain option but there is no problem in the question stem itself, then the content of the question stem remains unchanged, and only the problematic option is regenerated; If the information in the question stem is insufficient to support the current answer, then the question stem should be supplemented without changing the core meaning of the question. If the conditions in the question stem are reasonable, but the current answer cannot be derived from the information in the question stem, and the question stem cannot be modified, then medical reasoning will be performed again based on the conditions in the question stem, and the content of the corresponding options will be adjusted accordingly. If the question is unclear or ambiguous, the question stem or options should be rewritten with semantic equivalence.

10. The evaluation and optimization method as described in claim 1, characterized in that, Also includes: If the number of corrections exceeds a preset value and the assessment question still does not meet the expected requirements, then the assessment question is discarded.

Images