Method for obtaining evaluation result of large language model for evaluation question, and method and device for evaluating large language model

Abstract

The embodiment of the invention discloses a related method and a related device for evaluating a large language model. The method comprises the steps that an initial evaluation result of a to-be-tested model for an evaluation question is obtained based on an initial answer of the to-be-tested model for the evaluation question, the evaluation question has a corresponding reference answer, the reference answer comprises a reference reasoning result and one or more reference reasoning steps, the initial evaluation result comprises any one of reasoning correctness and reasoning error, and the one or more reference reasoning steps comprise one or more reference reasoning steps; the inference error includes an initial error inference step and supplemental knowledge information corresponding to the initial error inference step. And based on the supplementary knowledge information, obtaining an enhanced answer of the to-be-tested model for the evaluation question. The method further comprises the step of obtaining an enhanced evaluation result of the to-be-tested model for the evaluation question based on the reference answer, the initial answer and the enhanced answer, the enhanced evaluation result comprises a knowledge deficiency error and an inference deficiency error, and the knowledge ability and the inference ability of the large language model can be decoupled.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence technology, and in particular, relate to related methods, devices, computer program products, and electronic devices for evaluating large language models that can decouple the knowledge and reasoning capabilities of large language models. Background Art

[0002] In recent years, the rapid development of Large Language Models (LLMs) has revolutionized artificial intelligence. LLMs have demonstrated powerful natural language processing capabilities and inspired innovative applications across a wide range of fields, including scientific research, financial services, and content creation.

[0003] Current reasoning evaluation frameworks for LLMs cannot distinguish whether reasoning failures are due to knowledge gaps (e.g., unclear conceptual understanding) or defects in the reasoning process (e.g., missing reasoning steps), which limits the possibility of specifically optimizing the model. Summary of the Invention

[0004] In a first aspect of an embodiment of the present disclosure, a method for obtaining an evaluation result of a large language model for an evaluation question is provided. The method includes obtaining an initial evaluation result of the model to be tested for the evaluation question based on the initial answer of the model to be tested for the evaluation question, wherein the evaluation question has a corresponding reference answer, and the reference answer includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result includes any one of correct reasoning and incorrect reasoning, and the reasoning error includes an initial incorrect reasoning step and supplementary knowledge information corresponding to the initial incorrect reasoning step. Based on the supplementary knowledge information, an enhanced answer of the model to be tested for the evaluation question is obtained. In addition, the method also includes obtaining an enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer, and the enhanced answer, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

[0005] In a second aspect of an embodiment of the present disclosure, a method for evaluating a large language model is provided. The method includes determining a knowledge score and a reasoning score for the model under test based on an evaluation result set of the model under test for a set of evaluation questions, wherein the evaluation result set includes an evaluation result of the model under test for each evaluation question in the set of evaluation questions, the evaluation result includes an initial evaluation result and an enhanced evaluation result of the model under test for the corresponding evaluation question, the initial evaluation result includes either correct reasoning or incorrect reasoning, the reasoning error includes an initially incorrect reasoning step and supplementary knowledge information corresponding to the initially incorrect reasoning step, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

[0006] In a third aspect of the embodiments of the present disclosure, a device for obtaining an evaluation result of a large language model for an evaluation question is provided. The device includes an initial evaluation module for obtaining an initial evaluation result of the model to be tested for the evaluation question based on the initial answer of the model to be tested for the evaluation question, wherein the evaluation question has a corresponding reference answer, and the reference answer includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result includes any one of correct reasoning and incorrect reasoning, and the reasoning error includes an initial incorrect reasoning step and supplementary knowledge information. An enhanced answer acquisition module is used to obtain an enhanced answer of the model to be tested for the evaluation question based on the supplementary knowledge information. In addition, the device also includes an enhanced evaluation module for obtaining an enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer and the enhanced answer, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

[0007] In a fourth aspect of an embodiment of the present disclosure, a device for evaluating a large language model is provided. The device includes an evaluation score determination module configured to determine a knowledge score and a reasoning score of the model to be tested based on an evaluation result set of the model to be tested for a set of evaluation questions, wherein the evaluation result set includes an evaluation result of the model to be tested for each evaluation question in the set of evaluation questions, the evaluation result includes an initial evaluation result and an enhanced evaluation result of the model to be tested for the corresponding evaluation question, the initial evaluation result includes any one of correct reasoning and incorrect reasoning, the reasoning error includes an initially incorrect reasoning step and supplementary knowledge information corresponding to the initially incorrect reasoning step, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

[0008] In a fifth aspect of the disclosed embodiment, a computer program product is provided, comprising a computer program, which implements the method provided according to the first aspect or the second aspect when executed by a processor.

[0009] In a sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the method provided according to the first aspect or the second aspect.

[0010] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 shows an example environment in which various embodiments of the present disclosure may be implemented;

[0013] Figure 2 A flowchart of a method for obtaining evaluation results of a large language model for an evaluation topic according to some embodiments of the present disclosure is shown;

[0014] Figure 3 An example process of obtaining initial evaluation results according to some embodiments of the present disclosure is shown;

[0015] Figure 4 The data structure and corresponding relationships between the evaluation questions, reference answers, initial answers and initial evaluation results of some embodiments of the present disclosure are shown;

[0016] Figure 5 A schematic diagram of determining initial error reasoning steps in some embodiments of the present disclosure is shown;

[0017] Figure 6 An example process of obtaining enhanced answers according to some embodiments of the present disclosure is shown;

[0018] Figure 7 An example process of obtaining enhanced evaluation results according to some embodiments of the present disclosure is shown;

[0019] Figure 8 A schematic diagram showing the data structures and corresponding relationships between evaluation questions, reference answers, initial answers, initial evaluation results, enhanced answers, and enhanced evaluation results in some embodiments of the present disclosure;

[0020] Figure 9 A schematic diagram showing steps of determining and enhancing incorrect reasoning according to some embodiments of the present disclosure is shown;

[0021] Figure 10 The following shows prompt words for generating evaluation questions according to an embodiment of the present disclosure;

[0022] Figure 11 The reference reasoning results and prompt words of one or more reference reasoning steps for generating assessment questions in an embodiment of the present disclosure are shown;

[0023] Figure 12 Showing the prompt words for the knowledge points corresponding to the reference reasoning steps for generating the assessment questions in the embodiment of the present disclosure;

[0024] Figure 13The embodiment of the present disclosure shows prompt words for reasoning ability corresponding to reference reasoning steps for generating assessment questions;

[0025] Figure 14 Shows examples of assessment questions and reference answers according to an embodiment of the present disclosure;

[0026] Figure 15 shows another example environment in which embodiments of the present disclosure may be implemented;

[0027] Figure 16 The method flow for evaluating a large language model according to an embodiment of the present disclosure is shown;

[0028] Figure 17 A block diagram of an apparatus for obtaining evaluation results of a large language model for an evaluation topic according to some embodiments of the present disclosure is shown;

[0029] Figure 18 A block diagram of an apparatus 1800 for evaluating a large language model according to some embodiments of the present disclosure is shown;

[0030] Figure 19 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0032] The terms "including" and "having" and any variations thereof in this specification and claims and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or apparatuses. Depending on the context, the word "if" as used herein may be interpreted as "at..." or "when..." or "in response to determining" or "in response to detecting".

[0033] As mentioned above, large language models face challenges in complex reasoning tasks that require domain knowledge and complex reasoning. When relying solely on knowledge introduced during training or instruction, large language models struggle to achieve satisfactory performance in complex tasks such as financial decision-making. Reasoning capabilities are crucial for evaluating the understandability and applicability of LLMs. In existing technologies, performance metrics used to evaluate LLM capabilities often fail to distinguish between knowledge gaps and reasoning flaws in the model. This presents difficulties when engineers want to adjust the base model for targeted improvements. Therefore, accurately assessing the knowledge and reasoning capabilities of LLMs will help understand the model's shortcomings and support targeted optimization.

[0034] To this end, an embodiment of the present disclosure proposes a method for obtaining an evaluation result of a large language model for an evaluation question. In an embodiment of the present disclosure, based on the initial answer of the model to be tested for the evaluation question, the initial evaluation result of the model to be tested for the evaluation question is obtained, wherein the evaluation question has a corresponding reference answer, and the reference answer includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result includes any one of correct reasoning and incorrect reasoning, and the reasoning error includes an initial incorrect reasoning step and supplementary knowledge information. Based on the supplementary knowledge information, an enhanced answer of the model to be tested for the evaluation question is obtained. Furthermore, the method also includes obtaining an enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer and the enhanced answer, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

[0035] In this way, domain knowledge evaluation and reasoning ability evaluation can be effectively separated from a single evaluation task performance indicator, which helps to optimize the model in a targeted manner.

[0036] Figure 1 1 shows an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1As shown, environment 100 includes a computing unit 101, which can be any device with computing or processing capabilities. For example, computing unit 101 can include, but is not limited to, mobile phones, tablet computers, desktop computers, servers, etc. A method for obtaining evaluation results of a large language model for an evaluation question can be executed in computing unit 101. Computing unit 101 can call a model to be tested 102 and a referee model 103 deployed locally or remotely, and coordinate the coordination between the model to be tested 102 and the referee model 103 by inputting prompt words, etc. The model to be tested 102 is typically deployed on one or more computer devices and can generate output related to the input content based on the model input. The model to be tested 102 can be an LLM for various specialized domains, capable of handling complex reasoning tasks based on domain knowledge. In addition to specialized domain knowledge, such models also require reasoning capabilities and a certain degree of cognitive ability to solve complex reasoning problems. In one or more embodiments of the present disclosure, initial evaluation and enhanced evaluation can be performed based on the referee model 103 or the model to be tested 102, resulting in enhanced review results that independently represent the model's knowledge and reasoning capabilities. The referee model 103 can be another LLM model, which can complete the initial evaluation and enhanced evaluation of the model to be tested under the guidance of methods such as prompt words or thought chains. In one or more embodiments of the present disclosure, the referee model can be a traditional large language model such as qwen2.5-7b, GPT-4o, etc. Similarly, the referee model can also be deployed in one or more computer devices, and can generate output related to the input content based on the input of the model. Among them, the model to be tested and the referee model can be deployed in the same computer device, or in one or more different computer devices. The model to be tested and the referee model can also be deployed in the computing unit 101. In addition, the computing unit 101 can provide the evaluation questions 141 stored locally or remotely and the reference answers 142 corresponding to the evaluation questions 141 to the model to be tested 102 and the referee model 103, so that the model to be tested 102 and the referee model 103 can complete the tasks issued by the computing unit 101.

[0037] The computing unit 101 can obtain an initial evaluation result 144 of the model under test for the evaluation question 141 based on the initial answer 143 of the model under test 102. The reference answer 142 to the evaluation question includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result 144 of the model under test for the evaluation question includes two types: correct reasoning and incorrect reasoning. If the initial evaluation result 144 is an incorrect reasoning, the initial evaluation result further includes the initial incorrect reasoning step and supplementary knowledge information corresponding to the initial incorrect reasoning step. The supplementary knowledge information can be the supplementary knowledge content 1441' corresponding to the initial incorrect reasoning step itself, or information related to the supplementary knowledge content 1441'. The supplementary knowledge content 1441' can be retrieved or obtained through the supplementary knowledge information. If the initial evaluation result 144 is an incorrect reasoning, the computing unit 101 can inject the supplementary knowledge content 1441' into the model under test 102 based on the supplementary knowledge information to obtain an enhanced answer 145 of the model under test 102 for the evaluation question 141. Based on the determined enhanced answer 145, initial answer 143 and reference answer 142, the computing unit 101 can obtain the enhanced evaluation result 146 of the model to be tested 102 for the evaluation question 141. The enhanced evaluation result 146 further subdivides the reasoning errors of the model to be tested for the evaluation question into two types: insufficient knowledge errors and insufficient reasoning errors based on the initial evaluation result 144.

[0038] In this way, the reasoning errors of the model to be tested for the evaluation questions are divided into insufficient knowledge errors and insufficient reasoning errors, thereby separating the domain knowledge ability evaluation and reasoning ability evaluation from a single evaluation task performance indicator, which helps to optimize the model in a targeted manner.

[0039] The following combination Figure 2 The present invention describes a method for obtaining evaluation results of a large language model for an evaluation question according to an embodiment of the present invention. Figure 2 The flowchart of method 200 for obtaining the evaluation results of a large language model for an evaluation topic according to some embodiments of the present disclosure is shown. Method 200 can be executed at any suitable computing device, for example, method 200 can also be executed by computing unit 101. It should be understood that the numbers in the flowchart of method 200 do not represent the order in which these steps are executed. Some or all of these steps can be executed in parallel, or the order of execution can be interchanged, and this is not limited by the present disclosure. In addition, Figure 2 The methods in the drawings may also include additional steps not shown and / or may omit steps shown, and the scope of the present disclosure is not limited in this respect.

[0040] like Figure 2As shown, in box 202, method 200 can obtain an initial evaluation result 144 of the model to be tested for the evaluation question 141 based on the initial answer 143 of the model to be tested 102. The evaluation question has a corresponding reference answer 142, which includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result 144 includes any one of correct reasoning and incorrect reasoning. The reasoning error includes an initial incorrect reasoning step and supplementary knowledge information corresponding to the initial incorrect reasoning step. In box 204, method 200 can obtain an enhanced answer 145 of the model to be tested 102 for the evaluation question 141 based on the supplementary knowledge information. In box 206, method 200 can obtain an enhanced evaluation result 146 of the model to be tested for the evaluation question based on the reference answer 142, the initial answer 143, and the enhanced answer 145. The enhanced evaluation result includes two types: insufficient knowledge error and insufficient reasoning error.

[0041] Figure 3 An example process 300 at block 202 is shown according to some embodiments of the present disclosure. Figure 3As shown, at box 202, the evaluation question 341 can be input into the model to be tested 302, and the output of the model to be tested 302 is obtained as the initial answer 343 of the model to be tested to the evaluation question. Alternatively, the input of the model to be tested 302 can be generated based on the evaluation question 341, and the output of the model to be tested 302 is obtained as the initial answer 343 of the evaluation question. Alternatively, the evaluation question 341 can be input into the model to be tested 302, and the output of the model to be tested 302 is obtained, and the initial answer 343 of the model to be tested to the evaluation question is extracted from the output of the model to be tested 302. In one or more embodiments of the present disclosure, the model to be tested 302 is required to generate a solution process without format constraints when answering the evaluation question, so as to avoid cognitive bias caused by forcing the format to be consistent with the reference answer. In one or more embodiments of the present disclosure, an initial answer 343 can be input into the referee model 303. The referee model 303 then reviews the initial answer 343 based on the reference answer 342 of the evaluation question 341, and uses the output of the referee model 303 as the initial evaluation result 344 of the model under test 302 for the evaluation question. Alternatively, the initial evaluation result 344 of the model under test 302 for the evaluation question can be extracted based on the output of the referee model 303. In one or more embodiments of the present disclosure, the reference answer to the evaluation question can be injected into the referee model's knowledge base during its training phase, enabling the referee model 303 to compare the reference answer with the initial answer to complete the review of the initial answer and output the initial evaluation result 344. In one or more embodiments of the present disclosure, when generating the input for the referee model 303 based on the evaluation question 341, relevant information about the reference answer to the evaluation question (e.g., storage location index, storage link, etc.) or the reference answer itself can be attached, enabling the referee model 303 to compare the reference answer with the initial answer to complete the review of the initial answer and output the initial evaluation result 344. In one or more embodiments of the present disclosure, the initial incorrect reasoning step can be the first incorrect step in the reasoning process of the model under test for the test question. When determining whether the model under test 302 has made a reasoning error for the assessment question, the referee model 303 can be required to compare the reasoning steps of the initial answer with the reference answer, identify the first incorrect step in the reasoning process of the model under test 302 for the test question, and use this as the initial incorrect reasoning step. The referee model 303 can also specify the relevant information on the knowledge required to execute the initial incorrect reasoning step as supplementary knowledge information corresponding to the initial incorrect reasoning step.

[0042] In one or more embodiments of the present disclosure, one or more knowledge information (e.g., knowledge points) related to the assessment question may be shown in the reference answer, and the referee model may filter out the knowledge information corresponding to the initial incorrect reasoning step from the one or more knowledge information according to the initial incorrect reasoning step as supplementary knowledge information. Figure 4FIG4 shows the data structure and the corresponding relationship 400 between the evaluation questions 441, the reference answers 442, the initial answers 443 and the initial evaluation results 444 according to some embodiments of the present disclosure. Figure 4 As shown, in one or more embodiments of the present disclosure, to facilitate identification of initial incorrect reasoning steps and their corresponding supplementary knowledge information, each reference reasoning step of reference answer 444 may correspond to one or more pieces of knowledge information. Accordingly, the initial answer 443 of the model to be tested for evaluation question 441 may also include an initial reasoning result and one or more initial reasoning steps of the reasoning process that reflect the initial reasoning result.

[0043] At box 202, it can be determined whether the initial evaluation result is correct in reasoning based on the reference reasoning result of the reference answer 444 and the initial reasoning result of the initial answer 443. In one or more embodiments of the present disclosure, the initial answer 443 can be input into the referee model, and the referee model can judge whether the initial reasoning result in the initial answer 443 is correct based on the reference reasoning result of the reference answer 442 corresponding to the evaluation question 441. In one or more embodiments of the present disclosure, the evaluation question can be generated by extracting from a corpus that does not include answers by a large language model, and the reference reasoning steps of the evaluation question are at least two. In this way, it can be avoided that the model to be tested directly skips the reasoning step based on publicly released or Internet knowledge and directly retrieves the corresponding answer based on the evaluation question. Therefore, in one or more embodiments of the present disclosure, if the initial reasoning result is correct, the referee model can deem that the entire reasoning process of the model to be tested is correct, and the initial evaluation result is correct in reasoning. For example, calculation-type evaluation questions can judge whether the reasoning is correct by comparing the reference reasoning result with the calculation result in the initial reasoning result. Judgment-type evaluation questions can judge whether the reasoning is correct by comparing the reference reasoning result with the judgment result in the initial reasoning result. Question-answering-type questions can also judge whether the reasoning is correct by comparing the text content of the reference reasoning result and the initial reasoning result through the referee model. In one or more embodiments of the present disclosure, for evaluation questions whose initial evaluation results are correct reasoning, the evaluation of the question can be exited, and the corresponding evaluation results can only include the initial evaluation results. For evaluation questions whose initial evaluation results are incorrect reasoning, the referee model can compare one or more initial reasoning steps in the initial answer 443 with one or more reference reasoning steps in the reference answer 442 to find out the reference reasoning step corresponding to the first erroneous or different initial reasoning step in the initial answer reasoning process as the initial incorrect reasoning step. Figure 5 Schematic diagram 500 of determining initial error reasoning steps according to some embodiments of the present disclosure is shown. Figure 5As shown, in one or more embodiments of the present disclosure, the referee model can compare one or more enhanced reasoning steps 5452 in the initial answer with the reference reasoning step 5422 in the reference answer in order. If the referee model finds that the two steps are the same, it continues to compare the next step until it finds the first reference reasoning step that is different from the corresponding initial reasoning step (such as reference reasoning step 2 in the figure), and uses the reference reasoning step as the initial incorrect reasoning step. Based on the knowledge information corresponding to the initial incorrect reasoning step in the reference answer, supplementary knowledge information for the initial incorrect reasoning step can be found.

[0044] If the initial evaluation result of the model to be tested at block 202 is an inference error, the process may proceed to block 204 . Figure 6 An example process 600 at block 204 is shown according to some embodiments of the present disclosure. Figure 6 As shown, at block 204, the knowledge content 6441' required to perform the initial incorrect reasoning step can be injected into the model to be tested 602 (which may also be the model to be tested 302) based on the supplementary knowledge information in the initial evaluation result 645. For example, the knowledge content 6441' can be directly injected into the prompt word input to the model to be tested 602, and the model to be tested 602 can be asked to re-answer the test question 641 (which may also be the test question 341), and the output of the model to be tested 602 is obtained as the enhanced answer 645. Supplementary knowledge information can also be injected into the prompt word input to the model to be tested 602, and the model to be tested 602 can be asked to re-answer the test question 641. The model to be tested 602 will then answer the evaluation question based on the knowledge content 6441' obtained based on the supplementary knowledge information. In one or more embodiments of the present disclosure, the output of the model to be tested can be used as the enhanced answer 645, or the enhanced answer 645 of the model to be tested to the evaluation question can be extracted based on the output of the model to be tested. In one or more embodiments of the present disclosure, the structures of the enhanced answer and the initial answer should be similar, with the only difference being that the enhanced answer is obtained by reasoning after the knowledge content 6441' is added and injected into the model to be tested. Accordingly, the enhanced answer 645 of the model to be tested for the evaluation question 641 may also include an enhanced reasoning result and one or more enhanced reasoning steps of the reasoning process that reflect the enhanced reasoning result.

[0045] After obtaining the enhanced answer, the process may proceed to block 206 . Figure 7 An example process 700 at block 206 is shown according to some embodiments of the present disclosure. Figure 7As shown, in one or more embodiments of the present disclosure, the enhanced answer 745 (or enhanced answer 645) can be reviewed by the referee model 704 (or referee model 604) at box 206, thereby obtaining an enhanced evaluation result 746 of the model under test for the evaluation question. There are two necessary conditions for the correct reasoning of the initial reasoning answer of the model under test: first, the reasoning process of the model under test in answering the evaluation question is consistent with the reasoning process of the reference answer, and second, the model under test has sufficient relevant domain knowledge to ensure that each step of the reasoning conclusion of the model under test is correct. Therefore, the initial answer 743 and the enhanced answer 744 can be input into the referee model 703 to check whether the model under test prefers the external knowledge injected at box 204 (i.e., the knowledge content required to perform the initial incorrect reasoning step). It can be determined whether the relevant knowledge within the model under test is weak. If it is determined that the relevant knowledge within the model under test is weak, the enhanced evaluation result believes that the reasoning error of the model under test is caused by insufficient knowledge. Otherwise, the enhanced evaluation result believes that the reasoning error of the model under test is caused by insufficient reasoning. In this way, the knowledge ability and reasoning ability of the model to be tested can be decoupled, which helps to optimize the model to be tested in a targeted manner. In one or more embodiments of the present disclosure, when the result of the initial evaluation of the model to be tested for the evaluation question is a reasoning error, the initial wrong reasoning step is not found, that is, it is believed that the reasoning process of the model to be tested in obtaining the initial reasoning result is consistent with the reasoning process of the reference reasoning result, and therefore the enhanced evaluation result is directly determined to be insufficient knowledge. In one or more embodiments of the present disclosure, the reference reasoning result in the reference answer can also be used as the last reference reasoning step, and the corresponding reasoning ability and knowledge point can be assigned to it. When the result of the initial evaluation of the model to be tested for the evaluation question is a reasoning error, and when each initial reasoning step in the initial answer is consistent with the reference reasoning step of the reference answer, the initial wrong reasoning step is determined to be the last step to obtain the reference reasoning result.

[0046] In one or more embodiments of the present disclosure, the referee model can be used to compare the initial answer, the enhanced answer, and the reference answer to determine whether the enhanced answer is closer to the reference answer than the initial answer, for example, whether the enhanced answer is correctly reasoned (i.e., the enhanced reasoning result is consistent with the reference reasoning result of the reference answer), or whether the error steps of the enhanced answer are further back than those of the initial answer. Figure 8 and attached Figure 9 Describe the principle of enhanced review of enhanced answers by the referee model. Figure 8A schematic diagram 800 is shown of the data structures and corresponding relationships among an evaluation question 841, a reference answer 842, an initial answer 843, an initial evaluation result 844, an enhanced answer 845, and an enhanced evaluation result 846 according to some embodiments of the present disclosure. In one or more embodiments of the present disclosure, the enhanced reasoning result of the enhanced answer 845 can be compared with the reference reasoning result of the reference answer 842 (which can also be the reference answer 442) to see if they are consistent. If the enhanced reasoning result of the enhanced answer is consistent with the reference reasoning result of the reference answer, the enhanced evaluation result is insufficient knowledge. If the enhanced reasoning result of the enhanced answer is inconsistent with the reference reasoning result of the reference answer, the reference reasoning steps of the reference answer and the enhanced reasoning steps of the enhanced answer can be further compared by the referee model, and a reference reasoning step from the reference reasoning steps of the reference answer can be determined as the enhanced incorrect reasoning step of the model to be tested for the evaluation question. Figure 9 Schematic diagram 900 of determining enhanced error reasoning steps according to some embodiments of the present disclosure is shown. Figure 5 As shown, in one or more embodiments of the present disclosure, the referee model can be used to compare one or more enhanced reasoning steps 9452 in the initial answer with the reference reasoning step 9422 in the reference answer in order. If the referee model believes that the two steps are the same, it continues to compare the next step until the first reference reasoning step that is different from the corresponding enhanced reasoning step is found (such as reference reasoning step 4 in the figure), and the reference reasoning step is used as the enhanced incorrect reasoning step. Subsequently, the relative positions of the initial incorrect reasoning step and the enhanced incorrect reasoning step in the reasoning process of the reference reasoning answer are compared. If the enhanced incorrect reasoning step is later than the initial incorrect step, it indicates that the injection of supplementary knowledge content has, to a certain extent, made up for the knowledge deficiencies of the model to be tested when processing the evaluation questions, and the enhanced evaluation result should be a knowledge deficiency error. If the enhanced incorrect reasoning step is not later than the initial incorrect step, it indicates that the injection of supplementary knowledge content has not improved and made up for the ability of the model to be tested to process the evaluation questions, and the enhanced evaluation result should be a reasoning deficiency error.

[0047] In this way, the enhanced evaluation results divide the reasoning errors of the model under test for the evaluation questions into insufficient knowledge errors and insufficient reasoning errors, thereby separating the domain knowledge evaluation and reasoning ability evaluation from a single evaluation task performance indicator, which helps to optimize the model in a targeted manner.

[0048] In order to conduct a more refined evaluation of the ability of the large language model in facing reasoning tasks, in one or more embodiments of the present disclosure, at block 206, the reasoning errors in the enhanced evaluation results can also be divided into more fine-grained categories. For example, in one or more embodiments of the present disclosure, the reasoning ability of the model can be further divided into one or more of the five abilities of memory, understanding, application, analysis, and evaluation. Figure 4 As shown, in one or more embodiments of the present disclosure, a corresponding reasoning capability can be assigned to each reference reasoning step in the reference answer to the evaluation question. The number of reasoning capabilities corresponding to a reference reasoning step can be one or more. When the referee model performs an enhanced review, if the enhanced evaluation result of the model under test for a certain evaluation question is an insufficient reasoning error, the referee model can further find the reasoning capability corresponding to the initial incorrect reasoning step in the initial evaluation result of the model under test for the same evaluation question, and it can be considered that the model under test lacks the reasoning capability corresponding to the initial incorrect reasoning step.

[0049] In this way, we can conduct a more detailed evaluation of the ability of large language models in facing reasoning tasks, facilitating more targeted optimization of large language models.

[0050] The following is combined with Figure 10 -Attached Figure 14 Describes a method for constructing assessment questions and reference answers in one or more embodiments of the present disclosure. Figure 10 The prompt words 1000 for generating evaluation questions according to an embodiment of the present disclosure are shown. Figure 10 As shown, the prompt words can ensure that the generated evaluation topic at least includes the "sub-discipline name" that limits the application field corresponding to the evaluation topic, "certain key information" that limits the topic keywords, and "actual scenario" information that limits the application scenario of the topic. Figure 11 The diagram shows a reference reasoning result and prompt words 1100 of one or more reference reasoning steps for generating an assessment question according to an embodiment of the present disclosure. Figure 12 Shown are prompt words 1200 of knowledge points corresponding to reference reasoning steps for generating assessment questions according to an embodiment of the present disclosure. Figure 13The prompt words 1300 of the reasoning ability corresponding to the reference reasoning steps for generating evaluation questions according to an embodiment of the present disclosure are shown. In one or more embodiments of the present disclosure, the evaluation questions can be generated by an intelligent model based on a corpus that does not include answers. In one or more embodiments of the present disclosure, the corpus can be generated from textbooks in a professional field. Specifically, first, OCR (Optical Character Recognition) technology can be used to identify the text content and mathematical equations in the source material and convert them into a machine-readable format. Subsequently, irrelevant information in the source material can be eliminated by manually deleting non-essential components (for example, prefaces, appendices). Finally, standardization technology can be used to convert the cleaned data into structured data to ensure data compatibility and automated processing. In one or more embodiments of the present disclosure, an intelligent model is used to generate evaluation questions and reference answers based on a corpus under the guidance of custom-designed prompt words. Figure 14 An example 1400 of an assessment question and a reference answer according to an embodiment of the present disclosure is shown. Figure 14 Reference answer 1442 for assessment question 141 includes reference reasoning steps, reference reasoning results, knowledge points of the assessment question, and the corresponding reasoning abilities for each reference reasoning step. Reference answer 1442 specifies not only the corresponding reasoning abilities for each reference reasoning step but also the corresponding reasoning abilities for obtaining the reference reasoning result as the final step.

[0051] Figure 15 15 illustrates an example environment 1500 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, environment 1500 includes a computing unit 1501, which can be any device with computing or processing capabilities. For example, computing unit 1501 can include, but is not limited to, a mobile phone, a tablet computer, a desktop computer, a server, etc. The method for evaluating a large language model can be executed in computing unit 1501. Computing unit 1501 can obtain a set of evaluation questions 1502 stored locally or remotely and an evaluation result set 1503 corresponding to the evaluation question set 1502, and determine the knowledge score and reasoning score of the model to be tested based on the evaluation question set 1502 and the evaluation result set 1503.

[0052] The following combination Figure 16 The following describes a method flow for evaluating a large language model according to an embodiment of the present disclosure. Figure 161600 for evaluating a large language model according to some embodiments of the present disclosure. The method 1600 can be executed at any suitable computing device, for example, the method 1600 can also be executed by the computing unit 1501. It should be understood that the numbers in the flowchart of the method 1600 do not represent the order in which these steps are executed. Some or all of these steps can be executed in parallel, or the order of execution can be interchanged, and this is not limited by the present disclosure. In addition, Figure 16 The methods in the drawings may also include additional steps not shown and / or may omit steps shown, and the scope of the present disclosure is not limited in this respect.

[0053] like Figure 16 As shown, in box 1602, method 1600 can determine the knowledge score and reasoning score of the model to be tested based on the evaluation result set of the model to be tested for the evaluation question set, wherein the evaluation result set includes the evaluation results of the model to be tested for each evaluation question in the evaluation question set, and the evaluation results include the initial evaluation results and enhanced evaluation results of the model to be tested for the corresponding evaluation questions, the initial evaluation results include any one of correct reasoning and incorrect reasoning, the reasoning errors include the initial incorrect reasoning steps and the supplementary knowledge information corresponding to the initial incorrect reasoning steps, and the enhanced evaluation results include insufficient knowledge errors and insufficient reasoning errors.

[0054] In one or more embodiments of the present disclosure, the evaluation results of each evaluation question in the enhanced evaluation results can be obtained using method 200. In one or more embodiments of the present disclosure, the knowledge score KS of the model to be tested can be determined based on the sum of the number of knowledge points corresponding to the initial incorrect reasoning steps of the initial answers to all evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient knowledge errors and the sum of the number of knowledge points corresponding to all reference reasoning steps of each evaluation question in the evaluation question set, so that the knowledge score can be used to quantify the knowledge coverage of the model to be tested in the application field. For example, the calculation formula of the knowledge score KS is as follows:

[0055]

[0056] Where M represents the sum of the number of evaluation results with insufficient knowledge errors in the enhanced review evaluation results set. K′ i K represents the sum of the number of supplementary knowledge information in the preliminary evaluation results of the i-th evaluation question, whose errors are attributed to knowledge deficiencies. N is the total number of samples in the evaluation result set (i.e., the total number of evaluation results). i Represents the sum of the knowledge information corresponding to all reference reasoning steps of the i-th evaluation question.

[0057] In one or more embodiments of the present disclosure, the reasoning score RS of the model to be tested can be determined based on the sum of the number of evaluation questions in the evaluation result set whose initial evaluation results are correct reasoning and the sum of the number of evaluation results in the evaluation result set that do not contain knowledge errors, so that RS can reflect the reasoning ability of the model to be tested in the professional field. For example, the calculation formula of the reasoning score RS is as follows:

[0058]

[0059] where a i is the initial inference result of the model to be tested for the i-th evaluation question, and is the corresponding reference inference result. Therefore, the numerator of Formula 2 is the total number of evaluation results whose initial evaluation results in the evaluation result set are correct inference results. i ) represents the enhanced evaluation result of the evaluation question whose initial reasoning result is an incorrect reasoning, r(a i The output of ) can be knowledge error or reasoning error j , j∈{1,2,3,4,5}, where 1, 2, 3, 4, and 5 represent memory ability, comprehension ability, application ability, analysis ability, and evaluation ability, respectively. Therefore, the denominator in Formula 2 is the total number of evaluation results excluding knowledge errors.

[0060] In one or more embodiments of the present disclosure, the cognitive score CS of the model to be tested can also be determined based on the evaluation result set, and the cognitive score CS can include a score set of one or more of the following: memory ability score CS1, comprehension ability score CS2, application ability score CS3, analysis ability score CS4, and evaluation ability score CS5. In one or more embodiments of the present disclosure, the ability j score CS of the model to be tested can be determined based on the reasoning score RS, the sum of the number of reference reasoning steps corresponding to ability j in the reference answers of all evaluation questions in the evaluation question set, and the sum of the number of evaluation results in the evaluation result set whose enhanced evaluation results are insufficient reasoning ability errors and lack ability j. j For example, CS j The calculation formula is as follows:

[0061]

[0062] Where α∈(0,1) is a penalty coefficient. Different abilities j correspond to different α values, with lower levels of reasoning abilities j corresponding to larger penalty coefficients. In one or more embodiments of the present disclosure, cognitive abilities are categorized from low to high as follows: memory, comprehension, application, analysis, and evaluation. RS is used as a multiplier in Formula 3 because all cognitive deficiencies lead to reasoning failures.

[0063] In one or more embodiments of the present disclosure, the accuracy score Acc of the model to be tested may also be calculated. For example, the calculation formula of Acc is as follows:

[0064]

[0065] In this way, we can conduct a more fine-grained quantitative evaluation of reasoning ability while decoupling the knowledge ability to be tested and the reasoning ability.

[0066] Figure 17 A block diagram of an apparatus 1700 for obtaining evaluation results of a large language model for evaluation questions according to some embodiments of the present disclosure is shown. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Figure 17 As shown, apparatus 1700 includes an initial evaluation module 1701, configured to obtain an initial evaluation result of the model under test for the evaluation question based on the initial answer of the model under test for the evaluation question. The evaluation question has a corresponding reference answer, which includes a reference reasoning result and one or more reference reasoning steps. The initial evaluation result includes either correct reasoning or incorrect reasoning. Reasoning errors include initial incorrect reasoning steps and supplementary knowledge information corresponding to the initial incorrect reasoning steps. The apparatus also includes an enhanced answer acquisition module 1702, configured to obtain an enhanced answer of the model under test for the evaluation question based on the supplementary knowledge information. The apparatus also includes an enhanced evaluation module 1703, configured to obtain an enhanced evaluation result of the model under test for the evaluation question based on the reference answer, the initial answer, and the enhanced answer. The enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors. In one or more embodiments of the present disclosure, insufficient reasoning errors also include one or more reasoning capabilities that the model under test lacks for the evaluation question. The reference reasoning steps also have one or more corresponding reasoning capabilities. Reasoning capabilities include one or more of memory ability, comprehension ability, application ability, analysis ability, and evaluation ability. The enhanced evaluation module 1703 may also be configured to determine one or more reasoning deficiencies that the model to be tested lacks for the evaluation question based on the reasoning sub-capabilities corresponding to the initial incorrect reasoning steps in the initial answer in the initial evaluation result.

[0067] Figure 18The block diagram of the device 1800 for evaluating a large language model according to some embodiments of the present disclosure is shown. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Figure 18 As shown, the device 1800 includes an evaluation score determination module 1801, which is configured to determine the knowledge score and reasoning score of the model to be tested based on the evaluation result set of the model to be tested for the evaluation question set, wherein the evaluation result set includes the evaluation results of the model to be tested for each evaluation question in the evaluation question set, and the evaluation results include the initial evaluation results and enhanced evaluation results of the model to be tested for the corresponding evaluation question, the initial evaluation results include any one of correct reasoning and incorrect reasoning, the reasoning errors include the initial incorrect reasoning steps and the supplementary knowledge information corresponding to the initial incorrect reasoning steps, and the enhanced evaluation results include insufficient knowledge errors and insufficient reasoning errors. In one or more embodiments of the present disclosure, the insufficient reasoning errors of the enhanced evaluation results also include one or more reasoning capabilities that the model to be tested lacks for the evaluation questions. Accordingly, the evaluation score determination module 1801 is also configured to determine the cognitive score of the model to be tested based on the evaluation result set, and the cognitive score includes one or more of the memory ability score, the comprehension ability score, the application ability score, the analysis ability score, and the evaluation ability score. In one or more embodiments of the present disclosure, the evaluation results of the model to be tested for each evaluation topic in the evaluation topic set can be obtained according to method 200 or by using device 1700 .

[0068] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0069] Figure 19 1 shows a block diagram of an electronic device 1900 that can implement various embodiments of the present disclosure. Figure 19 As shown, electronic device 1900 includes a processor 1910, a disk drive 1920, an input / output interface 1930, a network interface 1940, and a memory 1950. The processor 1910, disk drive 1920, input / output interface 1930, network interface 1940, and memory 1950 can be communicatively connected via a communication bus 1960.

[0070] Among them, the processor 1910 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0071] The memory 1950 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage device, etc. The memory 1950 can store an operating system 1951 for controlling the operation of the electronic device 1900 and a basic input and output system (BIOS) 1952 for controlling the low-level operations of the electronic device 1900. In addition, a web browser 1953, a data storage management system 1954, etc. can also be stored. In short, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 1950 and is called and executed by the processor 1910.

[0072] The input / output interface 1930 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0073] The network interface 1940 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0074] The bus 1960 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1910 , disk drive 1920 , input / output interface 1930 , network interface 1940 , and memory 1950 ).

[0075] It should be noted that although the above device only shows the processor 1910, disk drive 1920, input / output interface 1930, network interface 1940, memory 1950, bus 1960, etc., in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the method of the present application, and does not necessarily include all the components shown in the figure.

[0076] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Likewise, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0078] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for obtaining evaluation results of a large language model for an evaluation question, including: Based on the initial answer of the model to be tested to the evaluation question, obtaining an initial evaluation result of the model to be tested for the evaluation question, wherein the evaluation question has a corresponding reference answer, the reference answer includes a reference reasoning result and one or more reference reasoning steps, the initial evaluation result includes any one of correct reasoning and incorrect reasoning, and the reasoning error includes an initial incorrect reasoning step and supplementary knowledge information corresponding to the initial incorrect reasoning step; Based on the supplementary knowledge information, obtaining an enhanced answer of the model to be tested for the evaluation question; as well as Based on the reference answer, the initial answer and the enhanced answer, an enhanced evaluation result of the model to be tested for the evaluation question is obtained, and the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

2. The method according to claim 1, wherein the reference reasoning step has one or more corresponding knowledge information, and the initial answer includes an initial reasoning result and one or more initial reasoning steps; And generating an initial evaluation result of the model to be tested for the evaluation question based on the initial answer of the model to be tested for the evaluation question includes: Determining whether the initial evaluation result is correct based on the reference reasoning result of the reference answer and the initial reasoning result of the initial answer; In response to determining that the initial evaluation result is not correct reasoning, comparing the reference reasoning steps of the reference answer and the initial reasoning steps of the initial answer, and determining a reference reasoning step from the reference reasoning steps of the reference answer as the initial incorrect reasoning step; and The supplementary knowledge information is determined based on the knowledge information corresponding to the initial erroneous reasoning step.

3. The method according to claim 2, wherein the enhanced answer comprises an enhanced reasoning result and one or more enhanced reasoning steps; And determining the enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer, and the enhanced answer includes: Based on the reference answer, the initial answer, and the enhanced answer, determining whether the enhanced answer is closer to the reference answer than the initial answer; as well as In response to determining that the enhanced answer is closer to the reference answer than the initial answer, the enhanced evaluation result is determined to be a knowledge deficiency error; otherwise, the enhanced evaluation result is determined to be a reasoning deficiency error.

4. The method according to claim 3, wherein the determining, based on the reference answer, the initial answer, and the enhanced answer, whether the enhanced answer is closer to the reference answer than the initial answer comprises: comparing whether the enhanced reasoning result of the enhanced answer is consistent with the reference reasoning result of the reference answer; In response to the enhanced reasoning result of the enhanced answer being consistent with the reference reasoning result of the reference answer, determining that the enhanced answer is closer to the reference answer than the initial answer; In response to determining that the enhanced reasoning result of the enhanced answer is inconsistent with the reference reasoning result of the reference answer, comparing the reference reasoning steps of the reference answer and the enhanced reasoning steps of the enhanced answer, and determining a reference reasoning step from the reference reasoning steps of the reference answer as the enhanced incorrect reasoning step of the model to be tested for the evaluation question; determining whether the enhanced error reasoning step is later than the initial error step; as well as In response to determining that the enhanced erroneous reasoning step is not further back than the initial erroneous step, determining that the enhanced answer is not closer to the reference answer than the initial answer.

5. The method according to claim 1, wherein the evaluation topic is generated by an intelligent model, and the evaluation topic at least includes an application field, topic keywords, and application scenarios.

6. The method according to claim 1, wherein the reference answers to the assessment questions are generated by an intelligent model.

7. The method according to any one of claims 1 to 6, wherein the insufficient reasoning error further comprises one or more reasoning abilities that the model to be tested lacks for the evaluation question, the reference reasoning step further comprises one or more corresponding reasoning abilities, the reasoning abilities comprising one or more of memory ability, comprehension ability, application ability, analysis ability, and evaluation ability; and determining the enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer, and the enhanced answer further comprises: Based on the reasoning sub-capabilities corresponding to the initial incorrect reasoning steps in the initial evaluation results, one or more reasoning capabilities that the model to be tested lacks for the evaluation question are determined.

8. Methods for evaluating large language models, including: Based on the evaluation result set of the model to be tested for the evaluation question set, the knowledge score and the reasoning score of the model to be tested are determined, wherein the evaluation result set includes the evaluation results of the model to be tested for each evaluation question in the evaluation question set, and the evaluation results include the initial evaluation results and enhanced evaluation results of the model to be tested for the corresponding evaluation questions, the initial evaluation results include any one of correct reasoning and incorrect reasoning, the reasoning errors include the initial incorrect reasoning steps and the supplementary knowledge information corresponding to the initial incorrect reasoning steps, and the enhanced evaluation results include insufficient knowledge errors and insufficient reasoning errors.

9. The method according to claim 8, wherein determining the knowledge score and the reasoning score of the model to be tested based on the evaluation result set of the test model for the evaluation question set comprises: The knowledge score of the model to be tested is determined based on the sum of the number of knowledge points corresponding to the initial incorrect reasoning steps of the initial answers to all evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient knowledge errors and the sum of the number of knowledge points corresponding to all reference reasoning steps of each evaluation question in the evaluation question set.

10. The method according to claim 8, wherein determining the knowledge score and the reasoning score of the model to be tested based on the evaluation result set of the test model for the evaluation question set comprises: The reasoning score of the model to be tested is determined based on the sum of the number of evaluation questions whose initial evaluation results in the evaluation result set are correct reasoning and the sum of the number of evaluation results in the evaluation result set that do not include knowledge errors.

11. The method according to claim 10, wherein the insufficient reasoning error further comprises one or more reasoning abilities of the model to be tested that are lacking with respect to the assessment topic, wherein the reasoning abilities include one or more of memory ability, comprehension ability, application ability, analysis ability, and evaluation ability; and wherein determining the knowledge score and reasoning score of the model to be tested based on the assessment result set of the model to be tested with respect to the assessment topic set comprises: Based on the evaluation result set, a cognitive score of the model to be tested is determined, and the cognitive score includes one or more of a memory ability score, a comprehension ability score, an application ability score, an analysis ability score, and an evaluation ability score.

12. The method according to claim 11, wherein determining the cognitive score of the model to be tested based on the evaluation result set comprises: The memory ability score of the model to be tested is determined based on the reasoning score, the sum of the number of reference reasoning steps corresponding to memory ability in the reference answers of all evaluation questions in the evaluation question set, and the sum of the number of evaluation results in the evaluation result set whose enhanced evaluation results are insufficient reasoning errors and lack of memory ability.

13. The method according to claim 11, wherein determining the cognitive score of the model to be tested based on the evaluation result set comprises: The comprehension ability score of the model to be tested is determined based on the reasoning score, the sum of the number of reference reasoning steps corresponding to comprehension ability in the reference answers of all evaluation questions in the evaluation question set, and the sum of the number of evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient reasoning errors and lack of comprehension ability.

14. The method according to claim 11, wherein determining the cognitive score of the model to be tested based on the evaluation result set comprises: The application ability score of the model to be tested is determined based on the reasoning score, the sum of the number of reference reasoning steps corresponding to application ability in the reference answers of all evaluation questions in the evaluation question set, and the number of evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient reasoning errors and lack of application ability.

15. The method according to claim 12, wherein determining the cognitive score of the model to be tested based on the evaluation result set comprises: The analytical ability score of the model to be tested is determined based on the reasoning score, the sum of the number of reference reasoning steps corresponding to analytical ability in the reference answers of all evaluation questions in the evaluation question set, and the number of evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient reasoning errors and lack of analytical ability.

16. The method according to claim 11, wherein determining the cognitive score of the model to be tested based on the evaluation result set comprises: The evaluation ability score of the model to be tested is determined based on the reasoning score, the sum of the number of reference reasoning steps corresponding to the evaluation ability in the reference answers of all evaluation questions in the evaluation question set, and the number of evaluation questions in the evaluation result set whose enhanced evaluation results are insufficient reasoning errors and lack of evaluation ability.

17. The method according to any one of claims 8-16, wherein the evaluation result of the model to be tested for the evaluation topic is obtained based on the method according to any one of claims 1-7.

18. A device for obtaining the evaluation results of a large language model for an evaluation question, comprising: An initial evaluation module is configured to obtain an initial evaluation result of the model to be tested for the evaluation question based on the initial answer of the model to be tested for the evaluation question, wherein the evaluation question has a corresponding reference answer, the reference answer includes a reference reasoning result and one or more reference reasoning steps, the initial evaluation result includes any one of correct reasoning and incorrect reasoning, and the reasoning error includes an initially incorrect reasoning step and supplementary knowledge information corresponding to the initially incorrect reasoning step; An enhanced answer acquisition module is configured to acquire an enhanced answer of the model to be tested for the evaluation question based on the supplementary knowledge information; as well as The enhanced evaluation module is configured to obtain an enhanced evaluation result of the model to be tested for the evaluation question based on the reference answer, the initial answer and the enhanced answer, wherein the enhanced evaluation result includes insufficient knowledge errors and insufficient reasoning errors.

19. The apparatus according to claim 18, wherein the insufficient reasoning error further comprises one or more reasoning capabilities that the model to be tested lacks for the assessment topic, the reference reasoning step further comprises one or more corresponding reasoning capabilities, the reasoning capabilities comprising one or more of memory capability, comprehension capability, application capability, analysis capability, and evaluation capability; and the enhanced assessment module is further configured to: Based on the reasoning sub-abilities corresponding to the initial incorrect reasoning steps in the initial answer in the initial evaluation result, one or more reasoning deficiencies that the model to be tested lacks for the evaluation question are determined.

20. A device for evaluating a large language model, comprising: An evaluation score determination module is configured to determine the knowledge score and reasoning score of the model to be tested based on the evaluation result set of the model to be tested for the evaluation question set, wherein the evaluation result set includes the evaluation results of the model to be tested for each evaluation question in the evaluation question set, and the evaluation results include the initial evaluation results and enhanced evaluation results of the model to be tested for the corresponding evaluation questions, the initial evaluation results include any one of correct reasoning and incorrect reasoning, the reasoning errors include the initial incorrect reasoning steps and the supplementary knowledge information corresponding to the initial incorrect reasoning steps, and the enhanced evaluation results include insufficient knowledge errors and insufficient reasoning errors.

21. The apparatus according to claim 20, wherein the insufficient reasoning error further comprises one or more lack of reasoning capabilities of the model to be tested for the assessment topic; and the assessment score determination module is further configured to: Based on the evaluation result set, a cognitive score of the model to be tested is determined, and the cognitive score includes one or more of a memory ability score, a comprehension ability score, an application ability score, an analysis ability score, and an evaluation ability score.

22. The device according to claim 20 or 21, wherein the evaluation result of the model to be tested for the evaluation topic is obtained based on the method according to any one of claims 1-7.

23. Computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 17.

24. Electronic equipment, including: one or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 17 are executed.

Images