Artificial intelligence model question solving ability evaluation method and system for dynamically synthesizing test questions

Abstract

The invention discloses an artificial intelligence model problem solving ability evaluation method and system for dynamically synthesizing test questions, and belongs to the field of artificial intelligence, and the method comprises the steps: through dynamically synthesizing the test questions, the defects of data leakage and static test data sets in the prior art are overcome, diversified and unpredictable test questions can be generated, and the test efficiency is improved. Therefore, the real evaluation of the problem solving ability of the artificial intelligence model is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and particularly relates to a method and system for evaluating the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions. Background Art

[0002] With the continuous development of artificial intelligence technology, especially its wide application in the fields of natural language processing (NLP), intelligent reasoning, decision support, etc., the evaluation of the problem-solving ability of artificial intelligence models has gradually become a key step in evaluating model performance. At present, the evaluation methods for artificial intelligence models mainly rely on the testing of standard test sets, and the problem-solving ability of the models is measured by comparing the performance of the models on these test sets.

[0003] In the existing evaluation methods, due to the problem of data leakage in the test data set, the evaluation results are often too high and cannot truly reflect the actual problem-solving ability of the model. In order to more accurately evaluate the performance of large artificial intelligence models in diverse tasks, the present invention proposes a method and system for evaluating the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, aiming to solve the problems of data leakage and inaccurate evaluation existing in the existing evaluation methods. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a method for evaluating the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, including:

[0005] Obtaining a large number of labeled high-quality evaluation data sets;

[0006] Rewriting the test questions in the evaluation data set based on a rewriting algorithm and a rewriting strategy to generate a newly synthesized test data set;

[0007] Obtaining the artificial intelligence model to be evaluated, inputting the newly synthesized test data set into the artificial intelligence model to be evaluated for calculation, and obtaining the model answer result;

[0008] Comparing and analyzing the model answer result with the standard answer to obtain the evaluation result.

[0009] Preferably, the data sources of the high-quality evaluation data set include but are not limited to public data sets, educational platforms, and online examination question banks;

[0010] The question structure of the high-quality evaluation data set is clear, and the problem-solving process and background information are complete.

[0011] Preferably, the rewriting strategy includes one or more of synonym replacement method, sentence pattern conversion method, information recombination method, detail supplementation method, context replacement method, reverse question method, concretization or abstraction method, format conversion method, emotion or stance conversion method.

[0012] Preferably, the process of generating the new synthetic test data set includes:

[0013] Input a question and a rewriting strategy into a rewriting algorithm to generate a new test question;

[0014] Repeat the generation of new test questions multiple times and integrate them to obtain the new synthetic test data set.

[0015] Preferably, the artificial intelligence model to be evaluated is a large-scale artificial intelligence model.

[0016] Preferably, the process of obtaining the model answer result further includes: evaluating the performance of the artificial intelligence model to be evaluated on the original test data set, and the evaluation metrics include but are not limited to accuracy, problem-solving time, and robustness.

[0017] Preferably, after obtaining the evaluation result, it further includes: optimizing and adjusting the artificial intelligence model to be evaluated according to the problem-solving result.

[0018] On the other hand, the present invention also provides an evaluation system for the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, including:

[0019] A data acquisition module for acquiring a large number of labeled high-quality evaluation data sets;

[0020] A question rewriting module for rewriting the test questions in the evaluation data set based on a rewriting algorithm and a rewriting strategy to generate a new synthetic test data set;

[0021] An evaluation module for acquiring an artificial intelligence model to be evaluated, inputting the new synthetic test data set into the artificial intelligence model to be evaluated for calculation, and obtaining a model answer result;

[0022] A result analysis and feedback module for comparing and analyzing the model answer result with the standard answer to obtain an evaluation result.

[0023] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, and the processor implements the method when executing the computing program.

[0024] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program implements the method when executed by a processor.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] Through dynamically synthesizing new test questions, the present invention effectively solves the problem of data leakage in existing evaluation methods. The model no longer relies on a fixed and reused test data set, avoiding the "memory effect" during the test process, thus ensuring that the evaluation results can truly reflect the performance of the model on unknown data. This improvement enhances the credibility and fairness of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0028] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail this application.

[0030] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0031] Embodiment 1

[0032] As Figure 1 shown, in this embodiment, an evaluation method for the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions is provided, including:

[0033] Obtain a large number of labeled high-quality evaluation data sets;

[0034] First, collect a large number of labeled high-quality evaluation data sets D. The data set D contains n test questions, D = {d1, d 2, d3... d n}. The question types and difficulties in the data set should be diverse, including both classic standard test questions and real data from actual application scenarios to ensure the comprehensiveness and representativeness of the evaluation. The data sources can include public data sets, educational platforms, online exam question banks, etc. The data set should ensure high-quality annotation, with clear question structures, complete problem-solving processes and background information, and be able to meet the needs of subsequent new question generation and model evaluation.

[0035] Rewrite the test questions in the evaluation data set based on a rewriting algorithm and a rewriting strategy to generate a newly synthesized test data set;

[0036] In this step, the collected dataset D is processed using the rewriting algorithm L to generate a new evaluation dataset N. A set of question rewriting prompts P is designed, where P contains multiple rewriting strategies, and P = {p1, p 2, p3... p m}. One question d i and one rewriting strategy p j are input into the rewriting algorithm L to generate a new test question m ij . The above steps are repeated multiple times to obtain a newly synthesized test dataset M:

[0037] M = L(P, D)

[0038] The generated new questions need to be rewritten in various ways to simulate the question variants that may occur in real scenarios. The rewriting strategies include the following:

[0039] Synonym replacement method:

[0040] Method description: By replacing keywords or phrases in the question with their synonyms, a new question is generated.

[0041] Example:

[0042] Original question: Which city is the capital of France?

[0043] Rewriting 1: Which is the main city of France?

[0044] Rewriting 2: Which city is the capital of France?

[0045] Sentence pattern conversion method:

[0046] Method description: By changing the sentence pattern structure of the question, such as changing from an interrogative sentence to a declarative sentence or a rhetorical question, a new question is generated.

[0047] Example:

[0048] Original question: How does photosynthesis occur?

[0049] Rewriting 1: Please explain how photosynthesis occurs.

[0050] Rewriting 2: What happens in photosynthesis?

[0051] Information recombination method:

[0052] Method description: By reorganizing the information order in the question, a new question is generated.

[0053] Example:

[0054] Original question: Who is the author of the book "1984"?

[0055] Rewrite 1: Who is the author of the book "1984"?

[0056] Rewrite 2: Who wrote the book "1984"?

[0057] Detail supplementation method:

[0058] Method description: Add some additional background information or context to the original question to make the question seem different while keeping the core content the same.

[0059] Example:

[0060] Original question: Who is the first president of the United States?

[0061] Rewrite 1: At the beginning of the founding of the United States, who is the first president?

[0062] Rewrite 2: After the independence of the United States, who is the first president?

[0063] Situation replacement method:

[0064] Method description: Generate new test questions by changing the situation or background involved in the question.

[0065] Example:

[0066] Original question: What is the boiling point of water?

[0067] Rewrite 1: What is the boiling point of ethanol?

[0068] Rewrite 2: What is the boiling point of water at sea level?

[0069] Reverse question method:

[0070] Method description: Convert the question into a reverse question, asking the model to answer the content related to but opposite to the original question.

[0071] Example:

[0072] Original question: What is the chemical formula of water?

[0073] Rewrite 1: What is the chemical formula of hydroxide?

[0074] Rewrite 2: What elements does water consist of and what is their ratio?

[0075] Concretization or abstraction method:

[0076] Method description: Make the question more challenging or adaptable to different answering scenarios by concretizing or abstracting the question.

[0077] Example:

[0078] Original question: What is climate change?

[0079] Rewriting 1 (Concretization): What are the main causes of climate change in urban areas?

[0080] Rewriting 2 (Abstraction): How do human activities affect global climate patterns?

[0081] Format Conversion Method:

[0082] Method Description: Convert the question from one format to another, such as converting a multiple-choice question to an open-ended question or changing a multiple-choice question to a single-choice question, etc.

[0083] Example:

[0084] Original question: Which of the following is a renewable energy source? A) Coal B) Solar energy C) Natural gas D) Nuclear energy Rewriting 1: Which energy source is renewable?

[0085] Rewriting 2: Among the following options, which one is a renewable energy source: coal, solar energy, natural gas, nuclear energy?

[0086] Emotion or Stance Conversion Method:

[0087] Method Description: Generate a new question by changing the tone or stance of the question.

[0088] Example:

[0089] Original question: What are the effects of smoking on health?

[0090] Rewriting 1: Why should smoking be prohibited in public places?

[0091] Rewriting 2: How does smoking have a negative impact on human health?

[0092] Obtain the artificial intelligence model to be evaluated, input the newly synthesized test data set into the artificial intelligence model to be evaluated for calculation, and obtain the model solution result;

[0093] Use the rewritten new test set M to evaluate the large model and test its performance on these dynamically generated questions. At the same time, evaluate the performance of the large model on the original test data set D. The evaluation metrics include accuracy rate, problem-solving time, robustness, etc.

[0094] Compare and analyze the model solution result with the standard answer to obtain the evaluation result.

[0095] By comparing the performance differences of the large model on the original data set D and the newly synthesized data set M, the problem-solving ability of the model can be comprehensively evaluated. The evaluation result will reveal the true ability of the model when facing dynamically generated questions with higher diversity and difficulty, providing a basis for further model optimization.

[0096] The present embodiment has the following technical effects:

[0097] Improve the accuracy and fairness of evaluation: By dynamically synthesizing new test questions, the present invention effectively solves the problem of data leakage in existing evaluation methods. The model no longer relies on fixed and reused test data sets, avoiding the "memory effect" of the model during the test process, thus ensuring that the evaluation results can truly reflect the performance of the model on unknown data. This improvement enhances the credibility and fairness of the evaluation results.

[0098] Enhance the diversity and comprehensiveness of evaluation: Traditional evaluation methods usually rely on predefined static data sets and are difficult to cover all possible task scenarios. However, the present invention ensures that the evaluation covers a variety of task types and application scenarios by dynamically generating test questions of different types and difficulties. The problem-solving ability of the model can be evaluated in a more diverse task background, thus achieving a more comprehensive ability evaluation.

[0099] Support multi-task evaluation and comprehensive ability assessment: The present invention can not only evaluate the model on a single task, but also evaluate the performance of the model on multiple tasks simultaneously, providing a comprehensive evaluation of the model's comprehensive ability. For example, the problem-solving abilities of the model in multiple tasks such as language understanding, reasoning, and generation can be accurately monitored. Compared with traditional single-task evaluation methods, the present invention provides a more detailed and multi-dimensional evaluation.

[0100] Improve the adaptability and flexibility of evaluation: The method of the present invention can dynamically adjust the test questions according to the real-time performance of the model, thus ensuring the matching of the test content with the model's ability. As the performance of the model improves, the difficulty and type of the evaluation tasks can be adjusted in real time to ensure the accuracy and flexibility of the evaluation. This dynamic adaptability enables the present invention to adapt to the evaluation needs of different stages and different types of models, avoiding the limitation that traditional evaluation methods are difficult to effectively reflect the performance of the optimized model.

[0101] Simplify the evaluation operation and improve efficiency: By automatically generating new test questions and evaluating the performance of the model, the present invention significantly simplifies the manual intervention process and reduces the workload of manually designing and adjusting questions in traditional evaluation methods. The automation and standardization of the evaluation process improve the evaluation efficiency, saving time and labor costs. In addition, with the dynamic generation of evaluation tasks, the evaluation process can respond more quickly to model changes and provide timely feedback.

[0102] Contribute to the optimization and iteration of artificial intelligence models: By continuously monitoring and evaluating the performance of the model in diverse tasks, the present invention can help developers quickly identify the deficiencies of the model and then carry out targeted optimizations. This evaluation method provides important support for the iterative upgrade of artificial intelligence models and helps to promote the continuous progress of model technology.

[0103] Example Two

[0104] In this embodiment, an artificial intelligence model problem-solving ability evaluation system for dynamically synthesizing test questions is provided, including:

[0105] Input the original data set: First, input an original data set, which contains questions and corresponding correct answers. The original data set can include various types of questions, such as multiple-choice questions, fill-in-the-blank questions, open-ended questions, etc.

[0106] Question rewriting module: Dynamically synthesize new questions by applying different rewriting methods (such as synonym replacement, sentence pattern conversion, information recombination, etc.). Each time a new test question is generated, a certain algorithm is used to control to ensure the difficulty and diversity of the questions.

[0107] Evaluation module: Input the generated test questions into the artificial intelligence model to obtain the problem-solving results of the model. By comparing the answers of the model with the correct answers, evaluate the problem-solving ability of the model.

[0108] Result analysis and feedback module: Evaluate the performance of the model in multiple tasks, conduct real-time analysis, and optimize or adjust the model performance according to the results.

[0109] The specific features of the technical solution of this embodiment mainly include the following aspects:

[0110] Question dynamic synthesis module:

[0111] Function description: The core function of this module is to dynamically generate new questions according to the original data set, avoiding the influence of a fixed data set. The question generation process is implemented through a rewriting algorithm, such as synonym replacement, sentence pattern conversion, information recombination, etc.

[0112] Working principle: Generate diverse question forms by semantically reconstructing the original questions in different ways. The specific rewriting methods include:

[0113] Synonym replacement: Replace keywords or phrases in the question with synonyms of the same or similar meaning.

[0114] Sentence pattern conversion: Change the grammar structure of the question, convert an interrogative sentence to a declarative sentence, or vice versa.

[0115] Information recombination: Change the information order in the question so that the question seems different but is essentially the same.

[0116] Detail supplement: Add relevant background information or details to the question to enrich the scenario and content of the question.

[0117] Evaluation module:

[0118] Functional description: This module is responsible for inputting the synthesized test questions into the artificial intelligence model to be evaluated, obtaining the model's answers to the questions, and comparing them with the correct answers.

[0119] Working principle: Automatically evaluate the model's answer results, and calculate indicators such as the model's accuracy rate and answering time. These indicators will be used as the basis for evaluating the model's problem-solving ability.

[0120] Technical implementation: During the evaluation process, use algorithms to automatically calculate the differences between the model's answers and the standard answers, and conduct quantitative analysis to generate the final evaluation results.

[0121] Result analysis and feedback module:

[0122] Functional description: This module is responsible for comprehensively evaluating the model's performance in multiple tasks and generating an evaluation report. The analysis module will also provide optimization suggestions for the model based on the model's answer results.

[0123] Working principle: Identify the advantages and disadvantages of the model through statistical analysis of the evaluation data. In multi-task evaluation, this module can analyze the model's performance in different tasks and different scenarios, providing valuable feedback information for model developers.

[0124] The following are the detailed implementation steps of the method in this embodiment:

[0125] Data preparation: Select an original dataset D containing various types of questions. This dataset can be a standard set of math questions, a set of language understanding questions, or any question bank suitable for evaluating large models.

[0126] Question generation: Apply the question rewriting module to rewrite the original questions in various ways. Each time a new question is generated, ensure the diversity of the question structure and content. The rewriting methods include synonym replacement, sentence pattern conversion, information recombination, etc. Generate a new test question dataset M.

[0127] Input to the model: Input the rewritten test questions into the artificial intelligence model to be evaluated to obtain the model's answers.

[0128] Answer comparison and evaluation: Compare the model's output answers with the standard answers, and calculate indicators such as the model's accuracy rate and speed. The evaluation indicators can be flexibly adjusted according to different task requirements.

[0129] 3.5 Feedback and optimization: Provide real-time feedback to the model based on the evaluation results. If it is found that the model performs poorly on certain types of questions, developers can use this information to optimize the model and conduct further training and improvement.

[0130] On the other hand, this embodiment also provides an evaluation system for the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, including:

[0131] A data acquisition module for acquiring a large number of labeled high-quality evaluation data sets;

[0132] A question rewriting module for rewriting the test questions in the evaluation data set based on a rewriting algorithm and a rewriting strategy to generate a new synthesized test data set;

[0133] An evaluation module for obtaining an artificial intelligence model to be evaluated, inputting the new synthesized test data set into the artificial intelligence model to be evaluated for calculation, and obtaining a model answer result;

[0134] A result analysis and feedback module for comparing and analyzing the model answer result with the standard answer to obtain an evaluation result.

[0135] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor. When the processor executes the computing program, the method is implemented.

[0136] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method is implemented.

[0137] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An evaluation method for the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, characterized in that, It includes: Obtain a large number of labeled high-quality evaluation data sets; Rewrite the test questions in the evaluation data set based on a rewriting algorithm and rewriting strategies to generate a new synthetic test data set; Obtain an artificial intelligence model to be evaluated, input the new synthetic test data set into the artificial intelligence model to be evaluated for calculation, and obtain a model answer result; Compare and analyze the model answer result with the standard answer to obtain an evaluation result.

2. The method according to claim 1, wherein The data sources of the high-quality evaluation data set include but are not limited to public data sets, educational platforms, and online exam question banks; The question structure of the high-quality evaluation data set is clear, and the problem-solving process and background information are complete.

3. The method according to claim 1, wherein The rewriting strategies include one or more of the synonym replacement method, sentence pattern conversion method, information recombination method, detail supplementation method, context replacement method, reverse question method, concretization or abstraction method, format conversion method, emotion or stance conversion method.

4. The method according to claim 1, characterized in that The process of generating the new synthetic test data set includes: Input a question and a rewriting strategy into the rewriting algorithm to generate a new test question; Repeat the generation of new test questions multiple times and integrate them to obtain the new synthetic test data set.

5. The method according to claim 1, wherein The artificial intelligence model to be evaluated is a large-scale artificial intelligence model.

6. The method according to claim 1, characterized in that The process of obtaining the model answer result further includes: evaluating the performance of the artificial intelligence model to be evaluated on the original test data set, and the evaluation metrics include but are not limited to accuracy, problem-solving time, and robustness.

7. The method according to claim 1, wherein After obtaining the evaluation result, it further includes: optimizing and adjusting the artificial intelligence model to be evaluated according to the problem-solving result.

8. An evaluation system for the problem-solving ability of an artificial intelligence model for dynamically synthesizing test questions, characterized in that, It includes: A data acquisition module for obtaining a large number of labeled high-quality evaluation data sets; A question rewriting module for rewriting the test questions in the evaluation data set based on a rewriting algorithm and rewriting strategies to generate a new synthetic test data set; An evaluation module for obtaining an artificial intelligence model to be evaluated, inputting the new synthetic test data set into the artificial intelligence model to be evaluated for calculation, and obtaining a model answer result; A result analysis and feedback module for comparing and analyzing the model answer result with the standard answer to obtain an evaluation result.

9. An electronic device, comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1-7.

Images