Method and device for evaluating model reasoning ability and storage medium
By constructing model inference ability evaluation data sets and evaluating the answers and steps of large language models, the problem of difficult to accurately evaluate inference ability in the existing technology is solved, and more comprehensive and accurate evaluation is achieved, especially in logical reasoning and instructions Follow the competence.
Patent Information
- Application Number
- CN202411172361.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
AI Technical Summary
It is difficult for the prior art to accurately evaluate the inference ability of large language models, especially in terms of logical reasoning and instruction compliance capabilities. Traditional benchmarks cannot distinguish whether the model really understands the problem or draws the answer through guessing.
By constructing a model inference ability evaluation dataset, input the large language model to be evaluated, obtain its output, including the answers to the question and the intermediate inference steps, and generate the evaluation results through the answer accuracy and step accuracy, using the answer accuracy, step accuracy and Indicators such as overall accuracy are evaluated, and combined with instructions to follow error rates and format conversion error rates, providing more comprehensive and accurate evaluations.
It improves the accuracy and reliability of the evaluation, can distinguish the performance of different models in complex problems, evaluate their abilities in multi-step reasoning tasks, and combines logical reasoning and instruction compliance capabilities to provide a deeper inference ability evaluation.
Smart Images

Figure CN119066381A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of large model evaluation, and specifically relates to a method, device and storage medium for evaluating model reasoning capability. Background Art
[0002] As the size of large language models continues to grow, reasoning ability has become a key capability. In order to evaluate the reasoning ability of the model, researchers have established many benchmarks:
[0003] (1) Early research mainly focused on logical reasoning, covering various forms of reasoning such as induction, deduction, and abduction, aiming to evaluate whether the model can infer the answer based on given conditions.
[0004] (2) Mathematical reasoning is another important area. The relevant benchmark tests include various forms such as word problems and theorem proofs. These problems require not only reasoning skills but also strong mathematical calculation skills.
[0005] (3) Knowledge-based reasoning, especially commonsense reasoning, is also an important focus. These benchmarks aim to determine whether the model has commonsense knowledge and can effectively use it for reasoning.
[0006] (4) Theory of mind reasoning goes a step further and examines whether the model can understand and integrate complex aspects of human cognition, such as thoughts and beliefs.
[0007] However, the Chain-of-Thought Prompting (CoT) method is difficult to accurately quantify the entire reasoning process. At the same time, traditional reasoning benchmarks often ignore the evaluation of the reasoning process, making it difficult to distinguish whether the model is guessing the answer or truly understanding and solving the problem, so it is impossible to accurately evaluate the model's reasoning ability.
[0008] The solution proposed in this application aims to fill this gap and provide a more comprehensive and accurate evaluation solution for the reasoning ability evaluation of large language models. Summary of the invention
[0009] In view of the above analysis, the embodiments of the present invention aim to provide a method, device and storage medium for evaluating the reasoning ability of a model, so as to provide a more comprehensive and accurate evaluation solution for the reasoning ability evaluation of a large language model.
[0010] In a first aspect of the present application, a method for evaluating model reasoning capability is provided, comprising:
[0011] Input the constructed model reasoning ability evaluation data set into the large language model to be evaluated, and obtain the output of the large language model to be evaluated, wherein the output at least includes the answer to the set problem and the intermediate reasoning steps in the process of solving the set problem;
[0012] Comparing the answer to the question with the reference answer to the set question to determine the answer accuracy, and comparing the intermediate reasoning steps with the reference steps of the set question to determine the step accuracy;
[0013] Based on the answer accuracy and / or the step accuracy, generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated;
[0014] Among them, the evaluation tasks in the model reasoning ability evaluation data set include execution tasks and planning tasks. The execution tasks involve one or more single-step reasoning processes, using clear rules to operate strings or states, and the operation results of each step are predictable; the planning tasks involve multi-step strategic decision-making, and the intermediate reasoning steps in the problem-solving process are obtained by introducing intermediate checkpoints or state variables.
[0015] Optionally, the evaluation task includes pre-designing a plurality of questions for each type of task, wherein the questions are constructed based on a plurality of different application scenarios; and the goal of the model reasoning capability evaluation data set is to evaluate the combination of the model's reasoning capability and instruction-following capability.
[0016] Optionally, the data in the model reasoning capability evaluation data set has Chinese version data and corresponding English version data.
[0017] Optionally, each rule type in the assessment task is divided into multiple subcategories, the questions in each subcategory are divided into multiple difficulty levels, and at least two different examples are provided for context learning;
[0018] The difficulty level is determined by the complexity of the rules involved and the number of reasoning steps required to solve the set problem.
[0019] Optionally, obtaining the output of the large language model to be evaluated includes:
[0020] Obtaining an output of the large language model to be evaluated, wherein the output is an output in a structured format generated after the output content is constrained by using constraint instructions;
[0021] For a single-step question, the output is a string list of the final answer to the question;
[0022] For questions involving multiple steps or complex reasoning, the output is a list of strings containing the answer to the question and a list of strings describing each step of the problem-solving process.
[0023] Optionally, generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated based on the answer accuracy and / or the step accuracy includes:
[0024] Based on the three indicators of answer accuracy, step accuracy, and answer and step accuracy, generate an evaluation result for evaluating the reasoning ability of the model to be evaluated;
[0025] The answer accuracy is used to evaluate the correctness of the answer to each question. A 1 for each answer indicates that the answer is correct, and a 0 for each answer indicates that the answer is wrong.
[0026] The step accuracy is used to evaluate the correctness of the intermediate reasoning steps, and is determined based on the percentage of similarity between the intermediate reasoning steps output by the large language model to be evaluated and the reference steps at the character level.
[0027] The answer and step accuracy is used to evaluate the overall accuracy of the answers and intermediate reasoning steps. It is calculated by combining the answer accuracy and the step accuracy with a logical 'AND' operation. When both the answer accuracy and the step accuracy are full marks, the corresponding answer and step accuracy is full marks, otherwise it is zero marks.
[0028] Optionally, generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated based on the answer accuracy and / or the step accuracy includes:
[0029] Based on the three indicators of answer accuracy, step accuracy, and answer and step accuracy, combined with two error indicators: instruction following error rate, format conversion error rate, and an overall accuracy indicator, all six indicators are used together to represent the evaluation results of the reasoning ability of the large language model to be evaluated;
[0030] The instruction following error rate indicates the proportion of all evaluation data in which the large language model to be evaluated cannot correctly follow the instructions to generate answers, and the format conversion error rate indicates the proportion of answers that cannot be generated in the correct format;
[0031] The overall accuracy rate represents the overall accuracy of the answers and steps under the premise that the large language model to be evaluated in all the evaluation data can correctly follow the instructions and perform the correct format conversion, that is, the answer and step accuracy rate divided by the number of questions that can correctly follow the instructions and perform the correct format conversion. In a second aspect of the present application, a device for evaluating the reasoning ability of a model is provided, including a memory and a processor, the memory storing a computer program, and the computer program, when executed by the processor, implements the method for evaluating the reasoning ability of a model according to any of the above-mentioned methods.
[0032] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for evaluating the model reasoning capability according to any one of the above-mentioned methods is implemented.
[0033] The method, device and storage medium for evaluating the model reasoning ability provided by the present application provide a comprehensive evaluation of the model reasoning ability by simultaneously evaluating the accuracy of the answer and the accuracy of the reasoning process. In this way, not only the correctness of the final output of the model is examined, but also whether the model follows the correct logic and rules during the reasoning process can be evaluated, thereby improving the accuracy and reliability of the evaluation. This method can effectively distinguish the performance differences of different models in solving complex problems, especially in multi-step reasoning tasks, and can better evaluate their performance in complex tasks. In addition, through the evaluation of execution tasks, the execution ability of the model under clear rules can be tested; through the evaluation of planning tasks, the prediction and planning ability of the model in a more complex environment can be tested; and logical reasoning and instruction following ability can be effectively combined. The solution provided by the present application accurately quantifies and scores the reasoning process, providing a more comprehensive and in-depth evaluation of the model's reasoning ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0035] Figure 1 A flowchart of a specific implementation method of the method for evaluating the model reasoning ability provided by the present application;
[0036] Figure 2 This is a structural block diagram of the device provided in this application for evaluating the model reasoning capability. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0038] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values and / or the values provided that will be recognized by those of ordinary skill in the art.
[0039] A flowchart of a specific implementation method of the method for evaluating the model reasoning ability provided in this application is as follows Figure 1 As shown, the method includes:
[0040] S101: Input the constructed model reasoning ability evaluation data set into the large language model to be evaluated, and obtain the output of the large language model to be evaluated, wherein the output at least includes an answer to a set problem and an intermediate reasoning step in the process of solving the set problem.
[0041] It is understandable that the model reasoning ability evaluation data set constructed in this application is carefully and specially designed, which is different from the common pre-training data set of the model and has the ability to examine rule execution and instruction following. This feature effectively reduces the interference of pre-training data on the evaluation results, making the evaluation more focused on the pure reasoning ability of the model.
[0042] By observing real-world reasoning problems, the evaluation tasks in the model reasoning ability evaluation dataset include execution tasks and planning tasks. The execution tasks involve one or more single-step reasoning processes, using clear rules to operate strings or states, and the operation results of each step are predictable; the planning tasks involve multi-step strategic decision-making, and the intermediate reasoning steps in the problem-solving process are obtained by introducing intermediate checkpoints or state variables.
[0043] Among them, the evaluation task includes pre-designing multiple questions for each type of task, and the questions are constructed based on multiple different application scenarios in the real world; the goal of the model reasoning ability evaluation data set is to evaluate the combination of the model's reasoning ability and instruction-following ability. In the embodiment of the present application, the evaluation tasks are adapted based on real-world scenarios, such as business decision analysis, technical fault diagnosis, etc., which are more practical and authentic than traditional abstract logic questions. This design makes the evaluation results better reflect the performance of the model in practical applications.
[0044] The dataset focuses on finding the correct solution path rather than optimizing efficiency. This reflects the common situation in real-world problems, where finding any effective solution is the main goal rather than pursuing the optimal solution. To reduce semantic ambiguity, the question design minimizes reliance on natural language, making the reasoning process more direct and avoiding the impact of uncertainty in natural language on the evaluation results.
[0045] The data in the model reasoning ability evaluation dataset has Chinese version data and corresponding English version data, making the dataset more widely applicable.
[0046] Each rule type in the evaluation task is divided into multiple subcategories, and the problems in each subcategory are divided into multiple difficulty levels, and at least two different examples are provided for contextual learning. The difficulty level is determined by the complexity of the rules involved and the number of reasoning steps required to solve the set problem. This grading method makes the evaluation results more discriminatory and can more accurately reflect the ability differences between different models. By providing at least two different examples, the model is helped to learn and adapt to new rules. These examples not only show the correct answer to the task, but also describe the solution process in detail. By learning these examples, the model can understand the application of the rules and apply them to the reasoning process of similar problems. This mechanism ensures that when the model encounters new rules, it can both correctly execute instructions and correctly infer results.
[0047] The model will generate outputs for these tasks, which include at least the final answer to the question and the intermediate reasoning steps generated by the model in the process of solving the problem. Obtaining the output of the large language model to be evaluated includes: obtaining the output of the large language model to be evaluated, the output is a structured format output generated after the output content is constrained by constraint instructions; for single-step problems, the output is a string list of the final answer to the question; for problems involving multiple steps or complex reasoning, the output is a string list of the answer to the question and a string list describing each step of the problem-solving process.
[0048] By setting clear output rules and formats, we can ensure that the output of different models or the same model at different times remains consistent. Specifically, we can use JSON format constraint instructions to constrain the output content according to different rule question types. JSON format data is easy for machines to interpret and process, which makes it possible to implement automated evaluation. Automated evaluation can quickly parse the output in JSON format and evaluate the correctness of the answer and the accuracy of the reasoning process according to the preset scoring criteria.
[0049] S102: Compare the answer to the question with the reference answer to the set question to determine the accuracy of the answer, and compare the intermediate reasoning step with the reference step of the set question to determine the accuracy of the step.
[0050] The evaluation content includes not only the correctness of the answer, but also the correctness of the process. Considering both the answer and the process effectively prevents the evaluation bias caused by the model guessing the correct answer. This mechanism ensures the accuracy and reliability of the evaluation results.
[0051] Answer accuracy: Compare the answers generated by the model with the reference answers to the set questions to determine the correctness of the answers.
[0052] Step Accuracy: The accuracy of the inference steps is determined by comparing the intermediate inference steps generated by the model with the reference steps of the setting problem.
[0053] S103: Based on the answer accuracy and / or the step accuracy, generate an evaluation result for evaluating the reasoning ability of the large language model to be evaluated.
[0054] Specifically, based on the answer accuracy and / or the step accuracy, generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated includes: based on three indicators: answer accuracy, step accuracy, and answer and step accuracy, generating an evaluation result for evaluating the reasoning ability of the model to be evaluated.
[0055] The answer accuracy is used to evaluate the correctness of the answer to each question. An evaluation result of 1 for each answer indicates that the answer is correct, and an evaluation result of 0 for each answer indicates that the answer is wrong.
[0056] The step accuracy is used to evaluate the correctness of the intermediate reasoning steps and is determined based on the percentage of similarity between the intermediate reasoning steps output by the large language model to be evaluated and the reference steps at the character level.
[0057] The answer and step accuracy is used to evaluate the overall accuracy of the answers and intermediate reasoning steps. It is calculated by combining the answer accuracy and the step accuracy with a logical 'AND' operation. When both the answer accuracy and the step accuracy are full marks, the corresponding answer and step accuracy is full marks, otherwise it is zero marks.
[0058] The final evaluation results include six indicators:
[0059] Based on the three described indicators of answer accuracy, step accuracy, and answer and step accuracy, as well as two error indicators: instruction following error rate, format conversion error rate, and an overall accuracy indicator.
[0060] Among them, the instruction following error rate and the format conversion error rate respectively represent the proportion of the large language model to be evaluated in all evaluation data that cannot correctly follow the instructions to generate answers, and the proportion of the large language model that cannot output in the correct format when generating answers.
[0061] The overall accuracy rate refers to the overall accuracy of the answers and steps in all the evaluation data, assuming that the large language model to be evaluated can correctly follow the instructions and perform the correct format conversion. Specifically, it is the accuracy rate of the answers and steps divided by the number of questions that can be correctly instructed to follow and perform the correct format conversion.
[0062] For example: Overall accuracy rate = answer and step accuracy rate / (1-instruction following error rate-format conversion error rate).
[0063] The method provided in this application provides a comprehensive evaluation of the model's reasoning ability by simultaneously evaluating the accuracy of the answer and the accuracy of the reasoning process. In this way, not only the correctness of the final output of the model is examined, but also whether the model follows the correct logic and rules during the reasoning process, thereby improving the accuracy and reliability of the evaluation. This method can effectively distinguish the performance differences of different models in solving complex problems, especially in multi-step reasoning tasks, and can better evaluate their performance in complex tasks. In addition, through the evaluation of execution-type tasks, the execution ability of the model under clear rules can be tested; through the evaluation of planning-type tasks, the prediction and planning capabilities of the model in a more complex environment can be tested; and logical reasoning and instruction-following capabilities can be effectively combined. The solution provided in this application accurately quantifies and scores the reasoning process, providing a more comprehensive and in-depth evaluation of the model's reasoning ability.
[0064] A key point in the implementation of this application is to build a data set that can test reasoning ability and instruction following ability. The process of building a model reasoning ability evaluation data set is further elaborated below.
[0065] The applicant found that real-life reasoning tasks have similarities with game mechanics, because real-life tasks often require following specific rules for reasoning, execution, or decision-making. Therefore, this application developed a new data set.
[0066] The dataset of this application is mainly divided into two areas: execution and planning. Execution refers to a deterministic single-step reasoning process. The model needs to apply clear rules to operate strings or states, and each step produces predictable results based on the current state and the applied rules. These tasks require the model to perform correct operations based on given information, simulating situations that are clearly indicated in reality. Planning categories are more complex and require long-term strategic thinking and multi-step decision-making, all of which are carried out in a rule-based environment. Planning problems require the model to analyze possible future states, formulate strategies, and determine the direction and steps of execution to reach a solution. The dataset of this application focuses on finding the correct solution path rather than optimizing efficiency, which is similar to many real-world situations, where finding any effective solution is the main goal.
[0067] After establishing the task categories, a team of experts designed questions for each type of task to ensure that the questions are innovative and can challenge the model's cross-domain reasoning ability. In order to reduce the semantic ambiguity associated with natural language reasoning, it is necessary to minimize the reliance on natural language. The problem design of this application makes it possible to evaluate reasoning ability more directly without the need for natural language inference during the reasoning process. In the execution category, it is necessary to ensure that each step is deterministic and verifiable, which facilitates accurate evaluation and avoids model guessing. In the planning field, due to its inherent uncertainty, intermediate checkpoints or state variables are introduced to more carefully evaluate the problem-solving process.
[0068] In order to facilitate accurate evaluation and simplify the matching process, the model is required to output a structured JSON format response. For single-step problems (level 0), the model only needs to output the final answer, and the evaluation is based only on the correctness of the answer. For problems involving multiple steps or complex reasoning (including level 1, level 2, level 3, and some level 0 problems), it is necessary to combine the evaluation of answers and processes. In all cases, the output JSON structure contains "answer", which is a list of strings representing the final solution; for multi-step problems, the output also includes "process", which is a list of strings that describes in detail the problem-solving process of each step.
[0069] In order to comprehensively evaluate the reasoning ability of the model, this application divides the benchmark structure into four different difficulty levels (0, 1, 2, and 3). The difficulty level depends on two key factors: the complexity of the rules involved and the number of reasoning steps required to reach the solution. Each level systematically introduces more rules and reasoning steps.
[0070] In addition, in order to evaluate the model's ability to learn rules and apply rules, the present application embodiment also develops two different examples for each problem. These examples include given questions, correct answers, step-by-step solutions, and detailed explanations. By providing these examples, not only the basic performance of the model is tested, but also its ability to learn from examples and apply new rules to similar problems is examined.
[0071] Initially, the questions in the dataset were designed in Chinese. However, the inventors of this application realized that this might bias the LLM, which is mainly trained on English data. In order to ensure the fairness and wide applicability of the benchmark, this application developed a comprehensive Chinese-English bilingual version.
[0072] The method provided in this application not only evaluates the correctness of the model's answers, but also evaluates whether the process by which the model obtains the answers is correct. According to the mandatory constraints in JSON format, the answers and reasoning steps of the rule game in a fixed format contained in the model's reply are extracted. The mandatory constraints are selected after experiments and are used to constrain the output format of the model when replying for subsequent operations. After the extracted answers and reasoning steps are converted in format, the accuracy of the answers is determined by comparing the consistency of the model's answers with the reference answers, and the consistency of the model's reasoning steps with the reference steps is ensured by comparing the consistency of the model's answers with the reference answers.
[0073] Specifically, this application defines three evaluation metrics to score each question:
[0074] (1) Answer accuracy (A-Acc): This metric evaluates the correctness of the answer to each question. The evaluation result of each answer is 0 or 1 to indicate whether the answer is correct. This metric is suitable for tasks with clear goals and single-step reasoning, reflecting whether the model can obtain the correct answer.
[0075] (2) Step Accuracy (P-Acc): This metric evaluates the correctness of the reasoning steps and is measured based on the percentage of similarity between the reasoning steps provided by the model and the expected reasoning steps at the character level. In some single-step reasoning problems that do not require a process, the step accuracy and answer accuracy are considered the same. By measuring the similarity at the character level, we can accurately analyze whether the model operates according to the expected logic and rules during the reasoning process, which is particularly critical for multi-step reasoning tasks.
[0076] (3) Answer and step accuracy (AP-Acc): This is a comprehensive indicator used to evaluate the overall accuracy of the answer and reasoning steps. It is calculated by combining the answer accuracy and the process accuracy. The answer and step accuracy of the question will be full marks only when both the answer and step accuracy are full marks, otherwise it will be zero marks. This is a more stringent evaluation standard that ensures that the model does not just "guess" the answer correctly in some cases, but draws conclusions through correct reasoning.
[0077] These metrics allow for a comprehensive and accurate assessment of the model’s performance, ensuring that it not only arrives at the right answers, but also provides the right reasoning.
[0078] The evaluation method provided in this application can be implemented on the LOGICGAME framework. The specific implementation of the LOGICGAME framework requires design and development in several key aspects, including the design of tasks, the establishment of a scoring system, and a comprehensive testing platform.
[0079] 1. Task design
[0080] (1) Scenario simulation: A series of logic game scenarios closely related to the real world are selected as task backgrounds, such as Sudoku, string operations, code cracking, etc. These tasks involve complex logical reasoning and instruction understanding.
[0081] (2) Multi-level difficulty setting: Design tasks of different difficulty levels for each scenario, ranging from basic information processing to advanced logical synthesis and judgment.
[0082] (3) Instruction set: Clear instructions and expected reasoning paths are defined for each task, which helps the model understand and execute specific task requirements.
[0083] 2. Evaluation system
[0084] Design evaluation rules to score each reasoning step and final answer proposed by the model based on the match with the unique reference answer. In addition to scoring a single step, a comprehensive evaluation of the correctness of the steps and final results of the entire reasoning process is required. Combining automated tools can ensure the accuracy, objectivity and reliability of the scoring.
[0085] 3. Test platform development
[0086] Develop a user-friendly testing platform that allows researchers and developers to easily upload models, set test parameters, and receive test results. The platform can manage a large number of tasks and model tests, including task allocation, progress tracking, and result aggregation. The platform can also provide tools to help users analyze and compare the performance of different models and identify strengths and weaknesses.
[0087] In addition, the present application also provides a device for evaluating the model reasoning ability, such as Figure 2 As shown in the structural block diagram of the device for evaluating the model reasoning ability provided in the present application, the device specifically includes a memory 21 and a processor 22, and the memory 21 stores a computer program. When the computer program is executed by the processor 22, it implements any of the above-mentioned methods for evaluating the model reasoning ability.
[0088] Specifically, by deploying equipment for evaluating the model's reasoning ability, it can become an automated operation platform. The platform has the advantages of being user-friendly, thus allowing researchers and developers to easily upload models, set test parameters and receive test results. When the evaluation platform is executed, a method for evaluating the model's reasoning ability is implemented. The specific implementation process can be referred to the description of the method part above, which will not be repeated here.
[0089] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for evaluating the model reasoning capability according to any of the above-mentioned methods is implemented.
[0090] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0091] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0092] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0093] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.
Claims
1. A method for evaluating the reasoning ability of a model, characterized in that: include: Input the constructed model reasoning ability evaluation data set into the large language model to be evaluated, and obtain the output of the large language model to be evaluated, wherein the output at least includes the answer to the set problem and the intermediate reasoning steps in the process of solving the set problem; Comparing the answer to the question with the reference answer to the set question to determine the answer accuracy, and comparing the intermediate reasoning steps with the reference steps of the set question to determine the step accuracy; Based on the answer accuracy and / or the step accuracy, generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated; Among them, the evaluation tasks in the model reasoning ability evaluation data set include execution tasks and planning tasks. The execution tasks involve one or more single-step reasoning processes, using clear rules to operate strings or states, and the operation results of each step are predictable; the planning tasks involve multi-step strategic decision-making, and the intermediate reasoning steps in the problem-solving process are obtained by introducing intermediate checkpoints or state variables.
2. The method for evaluating the model reasoning ability according to claim 1, characterized in that: The evaluation task includes pre-designing multiple questions for each type of task, and the questions are constructed based on multiple different application scenarios; the goal of the model reasoning ability evaluation data set is to evaluate the combination of the model's reasoning ability and instruction-following ability.
3. The method for evaluating the model reasoning ability according to claim 1, characterized in that: The data in the model reasoning capability evaluation data set has Chinese version data and corresponding English version data.
4. The method for evaluating the model reasoning ability according to claim 1, characterized in that: Each rule type in the assessment task is divided into multiple subcategories, the questions in each subcategory are divided into multiple difficulty levels, and at least two different examples are provided for context learning; The difficulty level is determined by the complexity of the rules involved and the number of reasoning steps required to solve the set problem.
5. The method for evaluating the model reasoning ability according to claim 1, characterized in that: The step of obtaining the output of the large language model to be evaluated includes: Obtaining an output of the large language model to be evaluated, wherein the output is an output in a structured format generated after the output content is constrained by using constraint instructions; For a single-step question, the output is a string list of the final answer to the question; For questions involving multiple steps or complex reasoning, the output is a list of strings containing the answer to the question and a list of strings describing each step of the problem-solving process.
6. The method for evaluating the model reasoning ability according to any one of claims 1 to 5, characterized in that: The step of generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated based on the answer accuracy and / or the step accuracy includes: Based on the three indicators of answer accuracy, step accuracy, and answer and step accuracy, generate an evaluation result for evaluating the reasoning ability of the model to be evaluated; The answer accuracy is used to evaluate the correctness of the answer to each question. A 1 for each answer indicates that the answer is correct, and a 0 for each answer indicates that the answer is wrong. The step accuracy is used to evaluate the correctness of the intermediate reasoning steps, and is determined based on the percentage of similarity between the intermediate reasoning steps output by the large language model to be evaluated and the reference steps at the character level. The answer and step accuracy is used to evaluate the overall accuracy of the answers and intermediate reasoning steps. It is calculated by combining the answer accuracy and the step accuracy with a logical 'AND' operation. When both the answer accuracy and the step accuracy are full marks, the corresponding answer and step accuracy is full marks, otherwise it is zero marks.
7. The method for evaluating the model reasoning ability according to claim 6, characterized in that: Generating an evaluation result for evaluating the reasoning ability of the large language model to be evaluated based on the answer accuracy and / or the step accuracy includes: Based on the three indicators of answer accuracy, step accuracy, and answer and step accuracy, combined with two error indicators: instruction following error rate, format conversion error rate, and an overall accuracy indicator, all six indicators are used together to represent the evaluation results of the reasoning ability of the large language model to be evaluated; The instruction following error rate indicates the proportion of all evaluation data in which the large language model to be evaluated cannot correctly follow the instructions to generate answers, and the format conversion error rate indicates the proportion of answers that cannot be generated in the correct format; The overall accuracy rate represents the overall accuracy of the answers and steps in all the evaluation data, under the premise that the large language model to be evaluated can correctly instruct to follow and perform correct format conversion, that is, the answer and step accuracy rate divided by the number of questions that can correctly instruct to follow and perform correct format conversion.
8. A device for evaluating the reasoning ability of a model, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for evaluating the model reasoning ability according to any one of claims 1 to 7 is implemented.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for evaluating the model reasoning capability according to any one of claims 1-7 is implemented.
Citation Information
Cited By
Language model evaluation method and device, equipment, medium and product
CN119337114A
Evaluation test method of multi-modal chart question and answer large model, medium and equipment
CN119760369A
Data processing method and device
CN119990333A
Data Processing Method and Apparatus
CN119990333B
Inference method, device and equipment, readable storage medium and program product
CN120046741A
