Automatic generation method of semantic consistent evaluation questions, electronic equipment and medium

Abstract

The invention discloses a semantic consistent evaluation question automatic generation method, an electronic device and a medium, and the method comprises the steps: firstly generating candidate questions based on the function of a to-be-evaluated large language model and the technical field to which the to-be-evaluated large language model belongs, then carrying out the semantic consistency detection of the candidate questions, and removing the candidate questions which do not pass the detection. Through semantic consistency detection, generated questions with logic errors can be automatically identified and eliminated, and the stability of evaluation data is ensured.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a method, electronic device and medium for automatically generating semantically consistent evaluation questions. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly being used in the medical field, such as for assisted diagnosis, medical record generation, and medical consultation. To ensure the safety and effectiveness of these models in practical applications, scientific and objective evaluation is crucial.

[0003] Existing technologies typically employ fixed benchmark datasets to evaluate medical language models. Specifically, this involves first constructing a static medical assessment dataset, which may be compiled by medical experts or collected from past medical exam questions, containing various question types such as multiple choice, multiple selection, and case analysis questions. This dataset usually remains unchanged after publication; some improvement schemes introduce a small-scale manual update mechanism, where experts manually replace some questions every few months, or new questions are collected through crowdsourcing. At the start of the evaluation, the medical language model under test is connected to the evaluation system. The system sends questions from the question bank sequentially or randomly to the model. Finally, the model's answers are collected and compared with standard answers to generate a scoring report. It can be seen that in existing technologies, researchers input a fixed question bank into the model, record the model's output, and calculate metrics such as accuracy, recall, or F1 score.

[0004] Existing medical language model evaluation question banks exhibit static and fixed characteristics. Because evaluation questions remain unchanged over extended periods, model developers may unintentionally or intentionally mix evaluation data into the training corpus, causing the model to memorize specific questions rather than mastering medical logic—a phenomenon known as overfitting. This causes evaluation results to gradually deviate from the model's true reasoning and generalization abilities, failing to objectively reflect the model's true performance during continuous iteration. Secondly, the updating of evaluation questions relies primarily on manual adjustments, lacking a fixed-cycle automated replacement mechanism. Manual question generation and review are inefficient and costly, making it difficult to automatically generate, replace, and release new questions at a predetermined frequency, such as weekly or monthly. This results in an unstable question bank update rhythm, failing to meet the needs of high-frequency evaluation and continuous monitoring during model development. Furthermore, existing technologies lack semantic consistency control during question replacement, making it difficult to maintain a stable difficulty distribution. When adding or replacing questions, they are often based on simple manual selection or random sampling, without systematic semantic transformation or difficulty constraints. This can easily lead to large fluctuations in question difficulty and inconsistent distribution of tested skills across different assessment periods, severely impacting the horizontal comparability of assessment results (i.e., comparisons between different models) and the vertical continuity (i.e., comparisons between different versions of the same model). Furthermore, existing technologies lack a dynamic question generation mechanism based on semantic transformation and question bank extraction. They do not fully utilize automated methods in natural language processing, such as semantic rewriting, question rewriting, and context transformation. This makes it difficult to generate a large number of structurally diverse but identical assessment questions while ensuring consistency in medical semantics, such as diagnostic conclusions and treatment principles, thus hindering the large-scale construction of dynamic question banks. Simultaneously, existing processes primarily rely on manual review, lacking the introduction of dedicated quality control models to automatically detect the medical correctness, logical consistency, and clarity of expression in newly generated questions. This can easily lead to the omission of potential errors or low-quality questions, affecting the authority of the assessment. Existing technologies also lack a prior model-based verification mechanism for assessment questions. Difficulty consistency cannot be quantitatively guaranteed, and the assessment of question difficulty relies heavily on expert experience, without introducing a stable prior model to "test" and verify new questions. The inability to use quantitative methods to ensure that new and old questions fall within the same ability range leads to uncontrollable fluctuations in assessment scores as the questions change. Summary of the Invention

[0005] To address some or all of the problems in the prior art, the first aspect of this invention provides a method for automatically generating semantically consistent evaluation questions, comprising: Based on the functionality and technical field of the large language model to be evaluated, candidate questions are generated; and Perform semantic consistency checks on the candidate questions and remove those that fail the checks.

[0006] Furthermore, based on the seed question bank, candidate questions are generated through semantic transformation.

[0007] Furthermore, the semantic transformation includes: context reconstruction, reverse questioning and negation changes, terminology synonym substitution, and numerical fine-tuning.

[0008] Furthermore, the semantic consistency detection includes: performing semantic consistency detection on the generated candidate questions and the corresponding seed questions in the seed question bank to determine whether the candidate questions will cause the standard answer to change; if so, the detection fails.

[0009] Furthermore, the automatic generation method also includes: Before performing semantic consistency detection, the candidate questions are validated according to rules to check whether the candidate question format is standardized, whether the options are complete, and whether there are any sensitive words.

[0010] Furthermore, the automatic generation method also includes: Before performing semantic consistency detection, the candidate questions are evaluated for readability, and generated results that are obscure or ambiguous are eliminated.

[0011] Based on the automatic generation method described above, the present invention also provides an electronic device for automatically generating semantically consistent assessment questions, which includes a memory and a processor, wherein the memory is configured to store a computer program that executes the automatic generation method described above when the processor is running.

[0012] The present invention also provides a computer-readable storage medium for automatically generating semantically consistent assessment questions, which stores a computer program that executes the automatic generation method as described above when running on a processor.

[0013] This invention provides an automatic generation method for semantically consistent evaluation questions. Firstly, it automatically generates evaluation questions with diverse structures based on semantic transformation, which helps to comprehensively evaluate the model's generalization ability and robustness. Simultaneously, through automated semantic consistency detection, it can intercept low-quality data in real time, ensuring the reliability of the evaluation data. Attached Figure Description

[0014] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0015] Figure 1 This diagram illustrates a flowchart of an automatic update method for an evaluation question bank for a large language model, according to an embodiment of the present invention. Figure 2This diagram illustrates a flowchart of an automatic generation method for evaluation questions of a large language model according to an embodiment of the present invention. Figure 3 This diagram illustrates a method for automatically generating evaluation questions for a large language model, according to yet another embodiment of the present invention. Figure 4 This diagram illustrates a flowchart of an automatic generation method for evaluation questions of a large language model, according to another embodiment of the present invention. Figure 5 This diagram illustrates the structure of an automatic update system for an evaluation question bank of a large language model, according to an embodiment of the present invention. Figure 6 A schematic diagram illustrating a sliding window update mechanism according to an embodiment of the present invention is shown. Detailed Implementation

[0016] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0017] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0018] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0019] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure, for example, by hardware programming a programmable device such as a field-programmable gate array (FPGA). When a module is implemented using firmware, the module's function can be written in the form of program code into a read-only memory of the device, such as an EPROM or EEPROM, and the corresponding function of the module can be implemented when the program code is executed by a processor.

[0020] To address the technical problems in existing large language model evaluation technologies, such as static question banks leading to overfitting or memorizing questions, lack of automated update mechanisms resulting in high maintenance costs, lack of semantic consistency and difficulty control during question updates leading to incomparable evaluation results, and lack of automated quality control and prior model verification mechanisms making it difficult to guarantee question quality and data reliability, this invention provides a method and system capable of periodically and automatically generating, verifying, and publishing high-quality evaluation questions. This ensures that while preventing model cheating through question manipulation, it maintains long-term stability in evaluation difficulty and the skills tested, thereby objectively reflecting the true changes in the capabilities of the medical large language model during continuous iteration.

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0022] Figure 1 This diagram illustrates a flowchart of an automatic update method for an evaluation question bank of a large language model, according to an embodiment of the present invention. Figure 1 As shown, an automatic update method for an evaluation item bank for large language models includes: First, in step 101, new questions are generated. New questions are generated specifically for the type and function of the large language model to be evaluated. In some embodiments of this invention, the generated new questions can be further verified, eliminating those that fail verification and streamlining the verified questions as candidates; and Finally, in step 102, the question bank is updated periodically. The evaluation question bank of the medical large language model is updated according to a fixed update cycle. Using a sliding window strategy, each update replaces a specified proportion of the questions in the evaluation question bank with new questions, such as... Figure 6As shown. The fixed update cycle and specified percentage are determined based on actual needs, such as the type of the large language model and its domain. For technical fields with rapid updates and iterations, the fixed update cycle should be relatively short, and the specified percentage can be correspondingly larger. Conversely, for large language models with a smaller target audience and higher specialization, the fixed update cycle can be relatively longer, and the percentage of each update can be reduced. For example, for a large medical language model, the fixed update cycle could be one week, and the specified percentage could be 20%.

[0023] In one embodiment of the present invention, after each update, the updated question bank needs to be version-numbered, and the source and difficulty coefficient of the seed questions corresponding to that version need to be recorded to ensure the traceability of the evaluation results. After the model under test is connected, questions are distributed from the latest version of the question bank, answers are collected, and an analysis report is generated. The analysis report includes scores and the ability change curves of the model at different time points. At the same time, the question bank obtained by the timed updates can also be used as data augmentation corpus for the training stage of the large language model to help the model improve its robustness to different expression methods.

[0024] Figure 2 This diagram illustrates a flowchart of an automatic generation method for evaluation questions of a large language model, according to an embodiment of the present invention. Figure 2 As shown, an automatic generation method for evaluation items in large language models includes: First, in step 201, candidate questions are generated. New candidate questions are generated specifically for the type and function of the large language model to be evaluated. In one embodiment of the invention, a rule-based template filling technique is used, that is, multiple question-answer templates are predefined, and questions are generated by randomly filling in data that conforms to the logic of the professional domain. For example, for a large medical language model used for disease diagnosis, multiple clinical medical record templates can be predefined, and symptom and sign data that conform to medical logic can be randomly input to generate questions. In another embodiment of the invention, candidate questions are generated based on a seed question bank through semantic transformation, thereby automatically generating evaluation questions with diverse structures while maintaining the semantics and answers. The seed question bank includes high-quality benchmark seed questions that have undergone rigorous manual review, and each seed question not only includes a stem, options, and a standard answer, but also a structured metadata tag. The metadata tag includes information such as multi-level classification and difficulty level. For example, for a medical large language model, the metadata tags for seed questions include: primary medical departments, such as cardiology and gastroenterology; secondary disease classifications, such as hypertension and coronary heart disease; assessment skills, such as diagnostic reasoning, medication advice, and contraindication judgment; difficulty coefficients; and corresponding medical knowledge graph entity links. In addition, real data, such as de-identified electronic medical records (EMRs), can be introduced and converted into seed questions through automated cleaning and annotation tools. These seed questions serve as the root source for dynamically generating candidate questions, ensuring the accuracy of assessing core professional knowledge. In one embodiment of the invention, generating candidate questions through semantic transformation refers to automatically generating new questions that are semantically equivalent but have different expressions based on seed questions. In another embodiment of the invention, candidate questions are generated by combining prompt engineering with a large language model, or by entity replacement strategies based on the corresponding professional knowledge graph. Specifically, the generation of candidate questions includes context reconstruction, reverse questioning and negation changes, terminological synonym substitution, and numerical fine-tuning. The context reconstruction mentioned above refers to keeping the core features and answers unchanged while changing only some non-core features. For example, for a medical language model, the core attributes and diagnosis of the patient can be kept unchanged, while the non-core attributes of the patient can be changed. For example, "A 50-year-old male patient complains of chest pain..." can be modified to "A middle-aged male office worker experiences squeezing pain in the precordial region after strenuous exercise...". Further, distracting information or background descriptions can be added to examine the model's ability to extract key information from complex text. Reverse questioning and negation transformation refer to changing positive questions into negative questions, or adjusting the negation logic of the options, to force the model to perform deep logical reasoning rather than simple keyword matching. For example, "Which of the following is the main symptom?" can be changed into the negative question "Which of the following is not a typical manifestation?"The terminology substitution involves replacing colloquial descriptions in the question stem with professional terminology, or vice versa, based on standard terminology dictionaries, such as medical terminology standards (e.g., SNOMED CT or ICD-10 dictionaries). For example, "hypertension" can be replaced with "persistently elevated systolic blood pressure," ensuring consistency in medical semantics. Numerical fine-tuning refers to adjusting some numerical values ​​in the question within the bounds of professional logic, ensuring that changes do not affect the final answer. For example, within the bounds of medical logic, values ​​in a test report can be fine-tuned (e.g., adjusting the white blood cell count from 11.5 to 12.0), ensuring that the numerical change does not alter the diagnostic conclusion. Numerical fine-tuning prevents the model from rote memorization of specific numbers and enhances its generalization ability. Next, in step 202, rule verification and readability evaluation are performed. The generated candidate questions are verified according to rules to check whether the format is standardized, whether the options are complete, and whether there are any sensitive words. Simultaneously, the fluency and clarity of the question stem are evaluated, and obscure or ambiguous generated results are eliminated. In one embodiment of the present invention, rule verification is performed using a rule-based verification system. Finally, in step 203, semantic consistency is checked. The generated candidate questions are semantically consistent with the corresponding seed questions in the seed question bank to determine if there is a shift in their professional meaning; that is, whether the newly generated question will cause a change in the standard answer. If a shift occurs, the verification fails, and the candidate questions that fail the verification are removed. In one embodiment of the invention, a finely tuned quality control model is used for semantic consistency checking.

[0025] Figure 3 This diagram illustrates a flowchart of an automatic generation method for evaluation questions of a large language model, according to another embodiment of the present invention. It is related to... Figure 2 The only difference in the illustrated embodiment is that it does not employ semantic consistency detection, but instead performs difficulty alignment and verification. Specifically, an automatic generation method for evaluation questions for large language models includes: First, in step 301, candidate questions are generated. Using the method described above, new candidate questions are generated specifically based on the type and function of the large language model to be evaluated. Next, in step 302, rule verification and readability evaluation are performed. The generated candidate questions are verified according to rules to check whether the format of the candidate questions is standardized, whether the options are complete, and whether there are any sensitive words, etc. At the same time, the fluency and clarity of the language of the question stem are evaluated, and generated results that are obscure or ambiguous are eliminated. Finally, in step 303, difficulty alignment and verification are performed. The generated candidate questions are aligned in difficulty with the corresponding seed questions in the seed question bank to quantitatively ensure the stability of the difficulty of the dynamic question bank. In one embodiment of the present invention, a prior model is used to solve the candidate questions and the corresponding seed questions in the seed question bank, respectively. Then, the accuracy rate of the prior model on the candidate questions and the seed questions is determined and denoted as P. new and P old And calculate the deviation between the two, Delta = |P old -P new The value is compared with a threshold. If the difference is less than the threshold, the verification passes; otherwise, the verification fails. In one embodiment of the invention, the threshold is 5%. Only when Delta is less than the threshold is the candidate question considered to be at the same difficulty level as the original seed question, and the candidate question can enter the candidate release pool. It should be understood that in other embodiments of the invention, the difficulty difference can also be determined based on other answer performance parameters. For example, a "model committee" composed of multiple different architecture models can be used, and its voting results or average accuracy rate can be used as the benchmark for difficulty assessment to further improve the robustness of the verification. In addition, if the prior model answers the original seed question correctly but the candidate question incorrectly, it indicates that the candidate question may have unclear wording or a significant increase in difficulty, and it needs to be eliminated or regenerated.

[0026] Figure 4 This diagram illustrates a flowchart of an automatic generation method for evaluation questions of a large language model, according to another embodiment of the present invention. It is related to... Figure 2 The difference in the illustrated embodiment is that, after semantic consistency detection passes, difficulty alignment and verification are performed. Specifically, an automatic generation method for evaluation questions of a large language model includes: First, in step 401, candidate questions are generated. Using the method described above, new candidate questions are generated specifically based on the type and function of the large language model to be evaluated. Next, in step 402, rule verification and readability evaluation are performed. The generated candidate questions are verified according to rules to check whether the format of the candidate questions is standardized, whether the options are complete, and whether there are any sensitive words, etc. At the same time, the fluency and clarity of the language of the question stem are evaluated, and generated results that are obscure or ambiguous are eliminated. Next, in step 403, semantic consistency detection is performed. The generated candidate questions are semantically consistent with the corresponding seed questions in the seed question bank to determine whether there is a drift in their professional meaning, that is, whether the newly generated question will cause the standard answer to change. If a drift occurs, the verification fails and the candidate questions that fail the verification are removed. Finally, in step 404, difficulty alignment and verification are performed. Candidate questions that pass the semantic consistency test are aligned in difficulty with the corresponding seed questions in the seed question bank to quantitatively ensure the stability of the difficulty of the dynamic question bank. Furthermore, if the prior model answers correctly on the original seed questions but incorrectly on the candidate questions, it indicates that the candidate questions may have unclear wording or a significant increase in difficulty, and they need to be removed or regenerated.

[0027] Based on the automatic update method described above Figure 5 This diagram illustrates the structure of an automatic update system for an evaluation question bank of a large language model, according to an embodiment of the present invention. Figure 5 As shown, an automatic update system for a large language model's evaluation question bank includes a question generation module 501, a question verification module 502, and an update module 503. The question generation module 502 generates new questions using methods described above, such as based on a predefined template or a seed question bank, through semantic transformation or other methods. The question verification module 502 is communicatively connected to the question generation module 501 and verifies the new questions generated by the question generation module, removing questions that fail verification. As mentioned earlier, the question verification module includes multiple sub-modules, where a first verification sub-module 521 is used for rule verification and readability evaluation, a second verification sub-module 522 for semantic consistency detection, and a third verification sub-module 523 for difficulty alignment and verification. At least one sub-module can be selected for verification at a time. The update module 503 replaces a specified proportion of questions in the evaluation question bank with newly verified questions at a fixed update cycle.

[0028] Based on the automatic update method described above, the present invention also provides an electronic device for updating evaluation questions of a large language model, comprising a memory and a processor, wherein the memory is configured to store a computer program that executes the automatic question generation method described above when the processor is running.

[0029] The present invention also provides a computer-readable storage medium for updating evaluation questions of a large language model, which stores a computer program that, when run on a processor, executes the question automatic generation method as described above.

[0030] Compared to existing static evaluation question banks that rely on manual maintenance, this invention dynamically generates and periodically changes the questions, ensuring a constantly evolving question format. This prevents the tested model from achieving high scores by simply memorizing specific question stems, forcing the model to truly understand the professional logic to solve the problems. This effectively prevents model cheating and overfitting, ensuring the authenticity of the evaluation results. In this invention, generative AI technology enables the infinite expansion and automatic maintenance of the question bank, meeting the needs of high-frequency model iteration evaluation, significantly reducing operational costs, and achieving low-cost, high-efficiency automated updates. Furthermore, the semantic transformation-based generation mechanism can construct richer scenarios and expressions than manually generated questions, covering more long-tail clinical cases, which helps to comprehensively evaluate the model's generalization ability and robustness. Simultaneously, an automated quality control module can intercept low-quality data in real time, ensuring that every published question undergoes rigorous logical and correctness verification, guaranteeing the reliability of the evaluation data. Moreover, by introducing a prior model for difficulty alignment, this invention quantitatively ensures the consistency of difficulty across question banks published in different periods, guaranteeing the vertical continuity and horizontal comparability of the evaluation results. This solves the problem of incomparable scores due to changes in questions in traditional methods, allowing developers to scientifically track the long-term ability changes of the model.

[0031] To verify the performance of the automatic update method in this invention, a basic question bank containing 500 seed questions was constructed, and a continuous evaluation was conducted on a large open-source medical model for four weeks. Two control groups were set up: Group A used a static question bank, and Group B used a dynamic question bank based on the automatic update method of this invention. The results showed that in Group A, as the number of evaluations increased, the score of the model increased from 60 to 95, exhibiting an abnormal linear upward trend, indicating overfitting to specific questions. In Group B, the score of the model remained within a reasonable range, oscillating between 60 and 65, and showed a slow and realistic increase with fine-tuning and optimization of the model itself. Through random sampling by human experts, the accuracy rate of the automatically generated questions in Group B in medical logic reached over 90%, and the difficulty deviation shown by the prior model validation was controlled within 3%. The experiment demonstrates that the question bank updated periodically using the automatic update method of this invention can effectively reflect the true capability of the model, and the quality and difficulty stability of the generated questions meet the requirements of practical applications.

[0032] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method for automatically generating semantically consistent assessment questions, characterized in that, include: Based on the functionality and technical field of the large language model to be evaluated, candidate questions are generated. as well as Perform semantic consistency checks on the candidate questions and remove those that fail the checks.

2. The automatic generation method as described in claim 1, characterized in that, Based on a seed question bank, candidate questions are generated through semantic transformation.

3. The automatic generation method as described in claim 2, characterized in that, The semantic transformations include: context reconstruction, reverse questioning and negation changes, terminology synonym substitution, and numerical fine-tuning.

4. The automatic generation method as described in claim 1, characterized in that, The semantic consistency detection includes: The generated candidate questions are semantically consistent with the corresponding seed questions in the seed question bank to determine whether the candidate questions will cause the standard answer to change. If so, the test fails.

5. The automatic generation method as described in claim 1, characterized in that, Also includes: Before performing semantic consistency detection, the candidate questions are validated according to rules to check whether the candidate question format is standardized, whether the options are complete, and whether there are any sensitive words.

6. The automatic generation method as described in claim 1, characterized in that, Also includes: Before performing semantic consistency detection, the candidate questions are evaluated for readability, and generated results that are obscure or ambiguous are eliminated.

7. An electronic device for automatically generating assessment questions based on semantic consistency, characterized in that, It includes a memory and a processor, wherein the memory is configured to store a computer program that executes the automatic generation method as described in any one of claims 1 to 6 when the processor is running.

8. A computer-readable storage medium for automatically generating assessment questions based on semantic consistency, characterized in that, The system contains a computer program that, when run on a processor, executes the automatic generation method as described above.

Images