Model evaluation method, device, electronic equipment, storage medium and program product

Abstract

This application provides a method, apparatus, electronic device, storage medium, and program product for model evaluation, applicable to the field of computer technology. The model evaluation method may include: obtaining a model to be evaluated, a target question for evaluating the model, and a list of evaluation questions for the target question; wherein the list of evaluation questions includes at least one sub-question for evaluating the quality of the answer to the target question, and the list is generated based on the target question using a large language model; inputting the target question into the model to be evaluated to obtain the target answer generated by the model; for each sub-question, evaluating the quality assessment result of the target answer corresponding to that sub-question using an evaluation model; and determining the performance evaluation result of the model to be evaluated based on the quality assessment results corresponding to at least one sub-question. This application embodiment can effectively evaluate the performance of a model.

Description

Technical Field

[0001] This application belongs to the field of computer technology. Specifically, this application relates to a method, apparatus, electronic device, storage medium, and program product for model evaluation. Background Technology

[0002] The advancements in large language models (LLMs) in recent years have been remarkable, driven by continuous technological progress and widespread adoption. The rapid emergence of new models and technologies has broadened their application scope, encompassing general text and vision models as well as models fine-tuned for domain-specific tasks. These models exhibit a range of capabilities and performance characteristics across diverse application scenarios. Therefore, effectively evaluating their capabilities is crucial for guiding their development.

[0003] Therefore, how to evaluate the performance of the model has become a key issue. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, storage medium, and program product for evaluating model performance. To achieve the above objective, the technical solutions provided by this application are as follows: Firstly, a method for model evaluation is provided, the method comprising: Obtain the model to be evaluated, the target question for evaluating the model to be evaluated, and the evaluation question list for the target question; wherein the evaluation question list contains at least one sub-question for evaluating the quality of the answer to the target question, and the question list is generated based on the target question through a large language model; The target question is input into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated. For each sub-question, the quality assessment result of the target answer corresponding to that sub-question is evaluated using an evaluation model; Based on the quality assessment results corresponding to the at least one sub-problem, the performance assessment results of the model to be evaluated are determined.

[0005] In one possible implementation, the list of evaluation questions for the target problem is generated in the following way: Obtain the target problem; Obtain first prompt information; wherein, the first prompt information includes target requirements, the target requirements being the requirements that the sub-problems to be generated by the large language model need to meet; Based on the first prompt information and the target question, and through the large language model, at least one sub-question of the target question that satisfies the target requirements is generated.

[0006] In another possible implementation, the step of inputting the target question into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated includes: Obtain at least one requirement for answering the target problem; The target question and the at least one question answering requirement are input into the model to be evaluated, and the target answer that satisfies each question answering requirement is generated by the model to be evaluated; The step of generating at least one sub-question of the target question that satisfies the target requirements based on the first prompt information and the target question, and through the large language model, includes: The first prompt information, the target question, and the answer requirements of each question are input into the large language model to generate at least one sub-question of the target question that satisfies the target requirements, wherein the at least one sub-question includes a sub-question corresponding to each answer requirement of the question.

[0007] In another possible implementation, the list of evaluation questions includes multiple sub-questions; The step of determining the performance evaluation result of the model to be evaluated based on the quality evaluation results corresponding to the at least one sub-problem includes: The quality assessment results corresponding to the multiple sub-problems are merged to obtain the merged quality assessment result. Based on the fused quality assessment results, the performance assessment results of the model to be evaluated are determined.

[0008] In another possible implementation, for each sub-question, evaluating the quality assessment result of the target answer corresponding to that sub-question through the evaluation model includes: The evaluation model predicts a first probability that the target answer matches the sub-question and a second probability that they do not. Determine the sum of probabilities between the first probability and the second probability; Determine the ratio between the first probability and the sum of the probabilities, and use the ratio as the quality assessment result of the target answer corresponding to the sub-question.

[0009] In another possible implementation, the step of fusing the quality assessment results corresponding to the multiple sub-problems to obtain a fused quality assessment result includes: Obtain the weights corresponding to each of the sub-problems; Based on the weights corresponding to each of the sub-problems, the quality assessment results corresponding to each of the sub-problems are weighted and fused to obtain the fused quality assessment result.

[0010] In another possible implementation, the weights corresponding to each of the sub-problems are determined in the following way: Determine the target application scenario for the model to be evaluated; Determine the degree of correlation between each of the sub-problems and the target application scenario; Based on the degree of correlation between each sub-problem and the target application scenario, the weight corresponding to each sub-problem is determined. The weight of a sub-problem is positively correlated with the degree of correlation between the sub-problem and the target application scenario.

[0011] In another possible implementation, for each sub-question, evaluating the quality assessment result of the target answer corresponding to that sub-question through the evaluation model includes: The target question, the sub-question, and the target answer are input into the evaluation model, which then evaluates the quality assessment result of the target answer corresponding to the sub-question, with reference to the target question.

[0012] In another possible implementation, the number of model parameters of the evaluation model is less than a preset parameter threshold.

[0013] In another possible implementation, the method further includes: Obtain the reference answer to the target question; The step of generating at least one sub-question for the target question based on the first prompt information and the target question, and through the large language model, includes: The first prompt, the target question, and the reference answer are input into the large language model, which then generates at least one sub-question for the target question by referring to the reference answer.

[0014] Secondly, a model evaluation apparatus is provided, the apparatus comprising: The first acquisition module is used to acquire a model to be evaluated, a target question for evaluating the model to be evaluated, and a list of evaluation questions for the target question; wherein, the list of evaluation questions includes at least one sub-question for evaluating the quality of the answer to the target question, and the list of questions is generated based on the target question through a large language model; The module is used to input the target question into the model to be evaluated, and obtain the target answer to the target question generated by the model to be evaluated; The evaluation module is used to evaluate the quality assessment result of the target answer corresponding to each sub-question using an evaluation model. The first determining module is used to determine the performance evaluation result of the model to be evaluated based on the quality evaluation result corresponding to the at least one sub-problem.

[0015] In one possible implementation, the apparatus further includes: a generation module, wherein, The evaluation question list for the target problem is generated by the generation module in the following way: Obtain the target problem; Obtain first prompt information; wherein, the first prompt information includes target requirements, the target requirements being the requirements that the sub-problems to be generated by the large language model need to meet; Based on the first prompt information and the target question, and through the large language model, at least one sub-question of the target question that satisfies the target requirements is generated.

[0016] In another possible implementation, when the obtaining module inputs the target question into the model to be evaluated and obtains the target answer to the target question generated by the model to be evaluated, it is specifically used for: Obtain at least one requirement for answering the target problem; The target question and the at least one question answering requirement are input into the model to be evaluated, and the target answer that satisfies each question answering requirement is generated by the model to be evaluated; Specifically, when the generation module generates at least one sub-question of the target question that satisfies the target requirements based on the first prompt information and the target question, and through the large language model, it is used to: The first prompt information, the target question, and the answer requirements of each question are input into the large language model to generate at least one sub-question of the target question that satisfies the target requirements, wherein the at least one sub-question includes a sub-question corresponding to each answer requirement of the question.

[0017] In another possible implementation, the list of evaluation questions includes multiple sub-questions; When the first determining module determines the performance evaluation result of the model to be evaluated based on the quality evaluation results corresponding to the at least one sub-problem, it is specifically used for: The quality assessment results corresponding to the multiple sub-problems are merged to obtain the merged quality assessment result. Based on the fused quality assessment results, the performance assessment results of the model to be evaluated are determined.

[0018] In another possible implementation, when the evaluation module evaluates the quality assessment result of the target answer corresponding to each sub-question using the evaluation model, it is specifically used for: The evaluation model predicts a first probability that the target answer matches the sub-question and a second probability that they do not. Determine the sum of probabilities between the first probability and the second probability; Determine the ratio between the first probability and the sum of the probabilities, and use the ratio as the quality assessment result of the target answer corresponding to the sub-question.

[0019] In another possible implementation, when the first determining module fuses the quality assessment results corresponding to the multiple sub-problems to obtain the fused quality assessment result, it is specifically used for: Obtain the weights corresponding to each of the sub-problems; Based on the weights corresponding to each of the sub-problems, the quality assessment results corresponding to each of the sub-problems are weighted and fused to obtain the fused quality assessment result.

[0020] In another possible implementation, the apparatus further includes: a second determining module, wherein, The weights corresponding to each of the sub-problems are determined by the second determining module in the following manner: Determine the target application scenario for the model to be evaluated; Determine the degree of correlation between each of the sub-problems and the target application scenario; Based on the degree of correlation between each sub-problem and the target application scenario, the weight corresponding to each sub-problem is determined. The weight of a sub-problem is positively correlated with the degree of correlation between the sub-problem and the target application scenario.

[0021] In another possible implementation, when the evaluation module evaluates the quality assessment result of the target answer corresponding to that sub-question using the evaluation model for each sub-question, it is specifically used for: The target question, the sub-question, and the target answer are input into the evaluation model, which then evaluates the quality assessment result of the target answer corresponding to the sub-question, with reference to the target question.

[0022] In another possible implementation, the number of model parameters of the evaluation model is less than a preset parameter threshold.

[0023] In another possible implementation, the apparatus further includes: a second acquisition module, wherein, The second acquisition module is used to acquire the reference answer corresponding to the target question; Specifically, when the generation module generates at least one sub-question for the target question based on the first prompt information and the target question, and through the large language model, it is used to: The first prompt, the target question, and the reference answer are input into the large language model, which then generates at least one sub-question for the target question by referring to the reference answer.

[0024] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the model evaluation method provided by any possible implementation of the first aspect.

[0025] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model evaluation method provided by any possible implementation of the first aspect.

[0026] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the model evaluation method provided by any possible implementation of the first aspect.

[0027] The beneficial effects of the technical solution provided in this application are as follows: This application provides a method, apparatus, electronic device, storage medium, and program product for model evaluation. In this application, the method involves obtaining a model to be evaluated and a target question for evaluating the model. Based on the target question, a large language model generates an evaluation question list containing at least one sub-question for evaluating the quality of the answer to the target question. The target question is then input into the model to be evaluated to obtain the target answer generated by the model. In this application, after obtaining the target answer generated by the model and the evaluation question list generated by the large language model, for each sub-question, the evaluation model can assess the quality evaluation result of the target answer corresponding to that sub-question. Based on the quality evaluation results corresponding to at least one sub-question, the performance evaluation result of the model to be evaluated is determined, thereby enabling the evaluation of the model's performance. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0029] Figure 1a This is a schematic diagram of an implementation environment in an embodiment of this application; Figure 1b This is a schematic diagram of another implementation environment in the embodiments of this application; Figure 1c This is a schematic diagram of yet another implementation environment in the embodiments of this application; Figure 2 This is a schematic diagram of a model evaluation method in an embodiment of this application; Figure 3 This is a flowchart illustrating a model evaluation method according to an embodiment of this application. Figure 4 This is an example diagram illustrating the overall framework of an automated evaluation scheme in an embodiment of this application; Figure 5 This is an example diagram illustrating a prompt message for generating a list of evaluation questions in an embodiment of this application; Figure 6 This is a schematic diagram of a model evaluation method for an application example in this application embodiment; Figure 7 This is a schematic diagram of a model evaluation device in an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0031] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0032] The relevant model evaluation schemes are divided into three main types: 1. Utilizing multiple-choice questions and keyword matching strategies: This approach evaluates the accuracy of the model's answers by designing a fixed set of questions and comparing the model's answers to predefined criteria. This method assesses model performance through multiple-choice questions and keyword matching. In MCQs (Multiple Choice Questions), the model needs to select the correct answer from multiple options, while in keyword matching, the model's answers are analyzed to determine if they contain specific keywords or phrases.

[0033] 2. Using LLM as an evaluator: In this approach, a large language model (LLM) is used as a "judge" to evaluate the responses of another model. This approach assumes that the judge model has high accuracy and comprehension capabilities, enabling it to correctly evaluate the responses of other models. LLM-as-a-Judge can provide a deeper evaluation because it considers multiple aspects of the response, not just keywords or predefined options.

[0034] 3. Evaluation model based on fine-tuning: This approach involves fine-tuning a pre-trained language model to more accurately evaluate answers for a specific task. The fine-tuned model can be optimized for specific evaluation criteria to improve accuracy. This fine-tuning requires generating highly refined data, including generating good and bad answers and scoring the final data. After obtaining the training data, fine-tuning is performed, and then the final model scores each model's answers to complete the final evaluation and comparison.

[0035] The inventors believe that the above-mentioned solutions have many defects, as detailed below: The first approach may not fully assess the model's comprehension ability because it relies on predefined options and keywords, potentially failing to capture the depth and creativity of the model's responses. Furthermore, it may be susceptible to generating seemingly reasonable but actually incorrect answers, as it does not evaluate the contextual relevance of the answers. Developing MCQs requires significant time and expertise and may suffer from the risk of memorizing answers rather than understanding them.

[0036] A key prerequisite for the second approach is that the LLM must be capable enough to fully understand the query and distinguish the quality of different responses. Therefore, many existing benchmarks tend to use the most powerful proprietary LLMs as evaluators. However, the cost of using these models for evaluation remains high, and there are other issues such as reproducibility and data privacy.

[0037] The third approach may require a large amount of labeled data and computational resources, which may not be suitable for all scenarios. Furthermore, the fine-tuning process may introduce the risk of overfitting, causing the model to perform well on specific datasets but poorly generalize to unseen data. There is also the possibility of inconsistencies between the evaluation criteria and actual human evaluation standards, leading to biased evaluation results. During fine-tuning, issues such as positional bias, knowledge bias, and format bias may arise, requiring specific technical means to address these problems and introducing additional resources.

[0038] To address the above issues, this application proposes a simple, reproducible, and accurate method for model evaluation. Figures 1a-1c This is a schematic diagram illustrating a possible implementation environment provided in an embodiment of this application. See also... Figures 1a-1c The implementation environment includes terminal 101 and server 102. Terminal 101 and server 102 are connected via a wireless or wired network.

[0039] In one possible implementation, such as Figure 1aAs shown, the model to be evaluated, the large language model, and the evaluation model can all be set in the server 102. The terminal 101 can provide the server 102 with target questions, etc. When the server 102 receives the target questions, it can generate a list of evaluation questions through the large language model, and generate a target answer for the target questions through the model to be evaluated, which will be output by the terminal 101. Furthermore, the terminal 101 can also provide the server 102 with the target answer output by the model to be evaluated and the list of evaluation questions generated by the large language model. By using the evaluation model and based on the target answer and the list of evaluation questions, the performance of the model to be evaluated is evaluated to obtain a performance evaluation result. The server 102 can provide the obtained performance evaluation result to the terminal 101 for display. In another possible implementation, such as Figure 1b As shown, the model to be evaluated and the large language model can be set in terminal 101, and the evaluation model can be set in server 102. Terminal 101 obtains the target question, etc., to generate the target answer through the model to be evaluated, and to generate the evaluation question list through the large language model. After obtaining the target answer and the evaluation question list, terminal 101 provides them to server 102, so that the evaluation model can obtain the performance evaluation result of the model to be evaluated based on the target answer and the evaluation question list. Server 102 provides the obtained performance evaluation result to terminal 101 for display. In another possible implementation, such as Figure 1c As shown, the model to be evaluated can be set in terminal 101, and the large language model and evaluation model can be set in server 102. Terminal 101 obtains the target question and generates the target answer through the model to be evaluated. It can also provide the target question and the generated target answer to server 102, so that server 102 can use the large language model to generate a target question list based on the target question, and use the evaluation model to evaluate based on the target answer and the target question list to obtain the performance evaluation result of the model to be evaluated. Server 102 provides the performance evaluation result of the model to be evaluated to terminal 101 for display. The above description of the implementation environment is only a few possible implementation methods and is not a limitation on the embodiments of this application. For example, the model to be evaluated, the large language model and the evaluation model can all be set in the terminal 101, and the server 102 is not required. That is, the implementation environment may not include the server 102. Therefore, as long as the implementation environment of the method shown in the embodiments of this application is implemented, it is within the protection scope of the embodiments of this application.

[0040] In the aforementioned implementation environment, this application provides a model evaluation method. First, we generate multiple sub-questions (i.e., a question list) for the current question-and-answer session using a high-quality industry model (such as a large language model). Next, we use a smaller model to answer these sub-questions, and the confidence probabilities of the smaller model's answers are collected. Finally, we integrate the confidence probabilities and output the final score for the current answer. This approach, for a single original question, only requires a larger, more expensive model to save the question list once; subsequent operations can be performed using a smaller model. Furthermore, by utilizing confidence probabilities, the instability of the smaller model is accounted for, resulting in a more reasonable final score. It is fast, cost-effective, and provides a certain level of assurance in its effectiveness. Specific model evaluation methods are detailed in the following embodiments.

[0041] This application provides a method for model evaluation, such as... Figure 2 As shown, the method is executed by an electronic device, which can be a server or a terminal device, and may include: Step S201: Obtain the model to be evaluated, the target problem for evaluating the model to be evaluated, and the list of evaluation problems for the target problem.

[0042] The evaluation question list includes at least one sub-question used to evaluate the quality of the answer to the target question. The evaluation question list is generated based on the target question using a large language model. In this embodiment, after obtaining the target question for evaluating the model to be evaluated, the target question is input into the large language model to obtain an evaluation question list containing at least one sub-question.

[0043] Furthermore, in the embodiments of this application, the model to be evaluated can be one or more, and if there are multiple models to be evaluated, these models can be different models or different versions of the same model, which is not limited in the embodiments of this application.

[0044] Step S202: Input the target question into the model to be evaluated, and obtain the target answer to the target question generated by the model to be evaluated.

[0045] Specifically, in this embodiment, the target question for evaluating the model to be evaluated can be obtained first, then a list of evaluation questions can be generated based on the target question and a large language model, and then the target question can be input into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated; alternatively, the target question for evaluating the model to be evaluated can be obtained first, then the target question can be input into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated, and then a list of evaluation questions can be generated based on the target question and a large language model; or, after obtaining the target question for evaluating the model to be evaluated, the target question can be used simultaneously to obtain a list of evaluation questions through a large language model, and then the target answer to the model to be evaluated can be generated through the model to be evaluated. This embodiment does not limit the specific execution order, and other possible execution orders are also within the protection scope of this embodiment.

[0046] Step S203: For each sub-question, evaluate the quality assessment result of the target answer corresponding to that sub-question using the evaluation model.

[0047] In this embodiment, the target answer of the model to be evaluated corresponds to the quality assessment result of a sub-question, which is used to characterize the degree of matching between the target answer of the model to be evaluated and the sub-question. In this embodiment, the quality assessment result can be characterized by a quality assessment score, wherein the quality assessment score of the target answer corresponding to a sub-question is positively correlated with the degree of matching between the target answer and the sub-question. That is, the more the target answer output by the model to be evaluated matches the sub-question, or the higher the degree of matching with the sub-question, the higher the quality assessment score of the target answer corresponding to the sub-question.

[0048] Specifically, in this embodiment of the application, after obtaining the list of evaluation questions and the target answer output by the model to be evaluated for the target question, the list of evaluation questions and the target answer output by the model to be evaluated for the target question are input into the evaluation model to obtain the quality evaluation result of the target answer for each sub-question.

[0049] For example, the models to be evaluated include Model 1 and Model 2. The target question generates target answer 1 through Model 1, and target answer 2 through Model 2. The evaluation question list includes sub-question 1 and sub-question 2. Based on the evaluation model, we can obtain the quality evaluation result 1-1 corresponding to sub-question 1 for target answer 1, and the quality evaluation result 1-2 corresponding to sub-question 2 for target answer 1. Based on the evaluation model, we can obtain the quality evaluation result 2-1 corresponding to sub-question 1 for target answer 2, and the quality evaluation result 2-2 corresponding to sub-question 2 for target answer 2. Figure 3 As shown.

[0050] For example, the models to be evaluated include Model A, Model B, and Model C. The target question generates target answer A through Model A, target answer B through Model B, and target answer C through Model C. The evaluation question list includes sub-question 1, sub-question 2, and sub-question 3. Based on the evaluation model, we can obtain the quality evaluation result A-1 corresponding to sub-question 1, A-2 corresponding to sub-question 2, and A-3 corresponding to sub-question 2. Based on the evaluation model, we can obtain the quality evaluation result B-1 corresponding to sub-question 1, B-2 corresponding to sub-question 2, and B-3 corresponding to sub-question 3. Based on the evaluation model, we can obtain the quality evaluation result C-1 corresponding to sub-question 1, C-2 corresponding to sub-question 2, and C-3 corresponding to sub-question 3.

[0051] Step S204: Determine the performance evaluation result of the model to be evaluated based on the quality evaluation result corresponding to at least one sub-problem.

[0052] In the embodiments of this application, if the evaluation question list contains a sub-question for evaluating the answer quality of the target question, that is, the evaluation model obtains a quality evaluation result for a sub-question, and this quality evaluation result is also the performance evaluation result of the model to be evaluated; if the evaluation question list contains multiple sub-questions for evaluating the answer quality of the target question, that is, the evaluation model can obtain quality evaluation results corresponding to multiple sub-questions respectively, and determine the performance evaluation result of the model to be evaluated based on the quality evaluation results corresponding to multiple sub-questions respectively.

[0053] For example, the target answer 1 of Model 1 corresponds to the quality assessment result 1-1 of sub-problem 1, the target answer 1 of Model 1 corresponds to the quality assessment result 1-2 of sub-problem 2, the target answer 2 of Model 2 corresponds to the quality assessment result 2-1 of sub-problem 1, the target answer 2 of Model 2 corresponds to the quality assessment result 2-2 of sub-problem 2, the performance assessment result of Model 1 is determined by the quality assessment result 1-1 and the quality assessment result 1-2, and the performance assessment result of Model 2 is determined by the quality assessment result 2-1 and the quality assessment result 2-2.

[0054] For example, for Model A, based on the evaluation model, we can obtain the quality evaluation result A-1 corresponding to sub-problem 1, the quality evaluation result A-2 corresponding to sub-problem 2, and the quality evaluation result A-3 corresponding to sub-problem 3 for the target answer A. The performance evaluation result of Model A is determined based on quality evaluation results A-1, A-2, and A-3. For Model B, based on the evaluation model, we can obtain the quality evaluation result B-1 corresponding to sub-problem 1, B-2 corresponding to sub-problem 2, and B-3 corresponding to sub-problem 3 for the target answer B. The performance evaluation result of Model B is determined based on quality evaluation results B-1, B-2, and B-3. For Model C, based on the evaluation model, we can obtain the quality evaluation result C-1 corresponding to sub-problem 1, C-2 corresponding to sub-problem 2, and C-3 corresponding to sub-problem 3 for the target answer C. The performance evaluation result of Model C is determined based on quality evaluation results C-1, C-2, and C-3.

[0055] This application provides a method for model evaluation. In this embodiment, a model to be evaluated and a target question for evaluating the model are obtained. Based on the target question, a large language model generates an evaluation question list containing at least one sub-question for evaluating the quality of the answer to the target question. The target question is then input into the model to be evaluated to obtain the target answer generated by the model. In this embodiment, after obtaining the target answer generated by the model and the evaluation question list generated by the large language model, for each sub-question, an evaluation model can assess the quality evaluation result of the target answer corresponding to that sub-question. Based on the quality evaluation results corresponding to at least one sub-question, the performance evaluation result of the model to be evaluated is determined, thereby enabling the evaluation of the model's performance. Furthermore, since the evaluation question list is generated by the large language model, the accuracy of each sub-question in the generated evaluation question list is high, resulting in high accuracy in subsequent performance evaluations of each model to be evaluated using other models. Moreover, it eliminates the need to perform performance evaluations of each model to be evaluated using a large language model; a single evaluation model suffices, thereby reducing the cost of model performance evaluation and improving the user experience.

[0056] Specifically, evaluating the question-answering ability of a large model is a complex task, whether using human evaluators or large model-based assessments. Human evaluation is often subjective, while large model-based assessments may suffer from difficulties such as comprehension of the questions, in-depth analysis of details, and weak logical reasoning abilities, leading to significant discrepancies in the final evaluation. Here, we follow the approach of creating a list of questions for a specific question (i.e., a list of evaluation questions for the target question) to guide the evaluation of the large model. Specifically, as shown in the above embodiments, the list of evaluation questions for the target question is generated using a large language model. In this embodiment, the list of evaluation questions for the target question is generated as follows: obtaining the target question; obtaining first prompt information; and based on the first prompt information and the target question, generating at least one sub-question that meets the target requirements using a large language model.

[0057] It should be noted that the first prompt information can be obtained before obtaining the target question, after obtaining the target question, or at the same time as obtaining the target question. This application embodiment does not impose any limitations on this. Specifically, in this embodiment, the first prompt information includes target requirements, which are the requirements that the sub-problems generated by the large language model must meet. In this embodiment, the target requirements may be generated by preset requirement rules.

[0058] For example, a pre-defined rule requirement might have the following characteristics: 1) relevance to the original question; 2) ability to effectively distinguish different responses; 3) independence from each other, functioning as independent questions. The target requirement derived from this pre-defined rule requirement could include: "Based on the above information, I need you to create a binary question list so that I can conduct an effective and accurate evaluation by answering several questions. Your questions should be concise and include any necessary key content and information (e.g., keywords, format, correct count, and values) that the user query or expects to display in the response. Your questions should consider not only the reference response but also all possible responses. Avoid creating repetitive, verbose, or vague questions. For example, you should ask 'Does this answer contain the correct answer...' instead of 'Is this answer correct?'." Figure 5 As shown. Generating an evaluation question list based on the above objectives ensures a comprehensive assessment of the response quality. Essentially, the question list can be considered a powerful knowledge analysis of a large model, subsequently prompting smaller evaluations of that large model.

[0059] Furthermore, after obtaining the first prompt information and the target question, at least one sub-question of the target question that satisfies the target requirements is generated based on the first prompt information and the target question.

[0060] For example, if the target question is "How can mining engineering be linked to renewable energy, and what is the future of this link, the importance and future of renewable energy, and what skills are needed for mining engineering graduate students to achieve this link and secure job and academic positions?", then based on this target question and the information described in the first part above, the generated target questions that meet the target requirements could include: "1. Does the response include examples of renewable energy technologies, such as wind turbines, solar panels, and energy storage systems? 2. Does the response clearly and in detail explain how mining engineering is linked to renewable energy, including its future potential? 3. Does the response mention the importance of renewable energy in mitigating climate change and reducing dependence on fossil fuels? 4. Does the response describe some academic or job positions related to mining engineering and renewable energy? 5. Is the answer clearly structured and organized, and can it be broken down into different topics, such as the link, importance, required skills, and opportunities?" Specifically, based on the first prompt information and the target question, and through a large language model, at least one sub-question of the target question that meets the target requirements is generated. Specifically, this may include: inputting the first prompt information, the target question, and the answer requirements of each question into the large language model to generate at least one sub-question of the target question that meets the target requirements.

[0061] At least one sub-problem includes a sub-problem corresponding to the solution requirements of each problem. In the embodiments of this application, the solution requirements of each problem are the solution requirements of the model to be evaluated when solving the target problem.

[0062] For example, if the target question is "What day of the week is two days after New Year's Day in 2025?", the corresponding answer requirements could include: "Please provide your thought process and answer, in JSON format." Furthermore, based on the aforementioned initial prompt, the target question, and the answer requirements for each question, at least one sub-question satisfying the target requirements could include: "1. Does the answer include the answer 'Friday'? 2. Does the answer follow JSON format? 3. Does the answer provide the analysis process?", specifically as follows... Figure 4 As shown.

[0063] Furthermore, to improve the accuracy of the generated evaluation question list, the first prompt information, the target question, and the answer requirements for each question are input into the large language model to generate at least one sub-question of the target question that meets the target requirements. Specifically, this may further include: obtaining a reference answer to the target question; inputting the first prompt information, the target question, the answer requirements for each question, and the reference answer to the target question into the large language model to generate at least one sub-question of the target question that meets the target requirements. In this embodiment, the reference answer to the target question conforms to the answer requirements for each question.

[0064] Specifically, in the embodiments of this application, the reference answer to the target question can be input by the target management object or generated by the target management object based on the conversation with the large language model. For example, the target management object can input the target question and the corresponding question answer requirements into the large language model to obtain the reference answer to the target question. Alternatively, it can generate a reference answer to the target question that meets the question answer requirements based on other more accurate models.

[0065] Furthermore, to ensure that the output format of the evaluation question list (sub-questions) generated by the large language model better meets the user's needs, the format of the output evaluation question list can be specified in the first prompt message, such as... Figure 5 As shown, the first prompt message may include requirements for the output format of the evaluation question list, as detailed below: ## Output Format Please output in the following format: 1.{question1} 2.{question2} 3.…” Furthermore, to improve the accuracy of the output list of evaluation questions, the first prompt message may also include: an example of the prompt message, such as... Figure 5 As shown, an example of a prompt message can be found below: # Dialogue between users and AI <|begin_of_history|> {history} <|end_of_history|> ## Current User Query <|begin_of_query|> {user_query} <|end_of_query|> ## Reference Response <|begin_of_reference_response|> {reference_response} <|end_of_reference_response|>.

[0066] Furthermore, the list of questions obtained through the above method, such as "Does the response contain the correct reasoning steps / final answer X?", can be very helpful when a small-scale evaluation of a large model struggles to determine all the key factors.

[0067] Furthermore, when generating a list of evaluation questions based on the above method and through a large language model, this process only needs to be executed once. Subsequently, the performance of each model to be evaluated can be evaluated using these sub-questions in the list of evaluation questions.

[0068] Furthermore, when generating the evaluation question list in the above manner, the number of sub-questions in the evaluation question list to be generated can be predefined, for example, 10, so that the number of sub-questions in the subsequently generated evaluation question list is 10. Alternatively, the number of sub-questions in the evaluation question list to be generated can not be predefined, and the evaluation question list can be directly generated by the large language model. The generated evaluation question list is determined by at least one of the first prompt information and the target question, or by combining the capabilities of the large language model. For example, if the capabilities of the large language model are high, the number of sub-questions in the generated evaluation question list may be more, and if the capabilities of the large language model are low, the number of sub-questions in the generated evaluation question list may be less.

[0069] Specifically, in step S202, the target question is input into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated. This may include: obtaining at least one question answer requirement for the target question; inputting the target question and at least one question answer requirement into the model to be evaluated, and generating a target answer that meets each question answer requirement through the model to be evaluated.

[0070] For example, if the target question is "What day of the week is two days after New Year's Day in 2025?", the corresponding answer requirements could include: "Please provide your thought process and answer in JSON format." The test models could be ModelA, ModelB, and ModelC. ModelA would output the answer as {'Thinking': '……', 'Answer': 'It's Friday'}; ModelB would output the answer as {'Thinking': '……', 'Answer': 'It's Wednesday'}; and ModelC would output the answer as {'Thinking': '……', 'Answer': 'It's Friday'}. Figure 4 As shown.

[0071] Furthermore, the target question, at least one question-answering requirement, and the target answer that satisfies each question-answering requirement can also be obtained from the historical sessions of each model under test. That is, as... Figure 4 As shown, the target question, at least one question answering requirement, and the corresponding answers for Model A, Model B, and Model C are obtained from historical sessions.

[0072] Furthermore, after obtaining the list of evaluation questions and the target answer output by the model to be tested, for each sub-question, the quality assessment result of the target answer corresponding to that sub-question is evaluated by the evaluation model. For example, if the generated list of evaluation questions contains 10 sub-questions, the quality assessment result of the target answer corresponding to that sub-question can be evaluated by the evaluation model for each of the 10 sub-questions to obtain the quality assessment result of the target answer relative to each of the 10 sub-questions; or, a certain number of sub-questions can be selected from the generated list of evaluation questions, such as 5 sub-questions, and the quality assessment result of the target answer corresponding to that sub-question can be evaluated by the evaluation model for each of the 5 sub-questions.

[0073] Specifically, in this embodiment, the method of selecting a certain number of sub-questions from the generated evaluation question list may include: the user selecting a certain number of sub-questions from the generated evaluation question list; or, based on the performance requirements of the model to be evaluated, selecting a certain number of sub-questions from the generated evaluation question list. Wherein, a higher performance requirement for the model to be evaluated results in a larger number of sub-questions selected from the generated evaluation list, and a lower performance requirement results in a smaller number of sub-questions selected from the generated evaluation list. In this embodiment, the performance requirements of the model to be evaluated may be related to at least one of the following: the number of parameters of the model to be evaluated and the application scenario to which the model to be evaluated is applicable. For example, a larger number of parameters in the model to be evaluated results in a higher performance requirement, while a smaller number of parameters results in a lower performance requirement. If the scenario to which the model to be evaluated is applicable is a scenario with higher requirements, such as medical consultation or legal consultation, then the performance requirement of the model to be evaluated is higher; if the scenario to which the model to be evaluated is applicable is a typical intelligent question-answering scenario, then the performance requirement of the model to be evaluated may be lower.

[0074] Specifically, each sub-question in the generated evaluation question list is used as a sub-question for subsequent evaluation of the model to be evaluated, or the sub-questions selected through the above method are used as sub-questions for subsequent evaluation of the model to be evaluated. Based on each sub-question, the evaluation model evaluates the quality evaluation result corresponding to each sub-question for the target answer. In this embodiment, for each sub-question, the evaluation model evaluates the quality evaluation result corresponding to the target answer for that sub-question. Specifically, this may include: inputting the target question, the sub-question, and the target answer into the evaluation model, and having the evaluation model evaluate the quality evaluation result corresponding to the target answer for that sub-question with reference to the target question. That is, to further improve the accuracy of the evaluation of the model to be tested, for each sub-question, when evaluating the quality evaluation result corresponding to the target answer output by the model to be tested for that sub-question, the evaluation model may also refer to the target question, and may even refer to the question answering requirements corresponding to the target question.

[0075] Specifically, for each sub-question, the quality assessment result of the target answer corresponding to the sub-question is evaluated by the evaluation model. This may include: predicting a first probability that the target answer matches the sub-question and a second probability that it does not match by the evaluation model; determining the sum of the probabilities between the first probability and the second probability; determining the ratio between the first probability and the sum of the probabilities, and setting the ratio as the quality assessment result of the target answer corresponding to the sub-question.

[0076] Specifically, lightweight LLMs exhibit high uncertainty, and relying solely on binary results such as "yes" or "no" can lead to significant errors in the final judgment. To mitigate this error, we introduce conditionally normalized probabilities as the basis for judgment, which posit the determinism of the outcome. Assuming the judgment LLM is parameterized by θ, and the target query has context x and a corresponding response y, the conditionally normalized probability of the question list question c is defined as: ; in, This indicates that the target answer corresponds to the quality assessment result of this sub-question. This represents the first probability that the target answer matches the subquestion. This represents the second probability that the target answer does not match the subquestion.

[0077] For example, for a certain sub-problem, the first probability that the target answer matches the sub-problem may be 85, and the second probability that the target answer does not match the sub-problem may be 14. Then the quality assessment result of the target answer for the sub-problem is 85.86.

[0078] For example, following the above embodiments, such as Figure 4As shown, the target answer output by Model A, through these three sub-questions "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 99, 83, and 85. That is, the target answer output by Model A corresponds to a quality assessment result of 99 for sub-question "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields a quality assessment result of 83, and a quality assessment result of 85 for sub-question "3. Does the answer provide the analysis process?". Furthermore, the target answer output by Model B, through these three sub-questions "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 32, 84, and 85. That is, the target answer output by Model B corresponds to a quality assessment result of 99 for sub-question "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 32, 84, and 85. The quality assessment results for the target answer of Model C through the sub-questions "1. Does the answer contain the answer Friday?" are 32, "2. Does the answer follow JSON format?", and "3. Does the answer provide the analysis process?" are 85. Furthermore, the quality assessment results for the target answer of Model C through these three sub-questions are: 96, 43, and 83. In other words, the quality assessment result for the target answer of Model C through the sub-question "1. Does the answer contain the answer Friday?" is 96, the quality assessment result for the target answer of Model B through the sub-question "2. Does the answer follow JSON format?" is 43, and the quality assessment result for the target answer of Model C through the sub-question "3. Does the answer provide the analysis process?" is 83.

[0079] Based on the above embodiments, the quality assessment results for each sub-question corresponding to the target answer can be obtained. Furthermore, the ultimate goal of this scheme is to assign a score to a specific response. After obtaining the normalized score in the previous stage, this stage uses the normalized scores from multiple question lists to obtain the final score. That is, it summarizes the scores of the degree to which the current response conforms to the question list, obtaining a final score. Then, responses from different large models can be scored and compared for automated evaluation.

[0080] Specifically, the evaluation question list can contain one or more sub-questions. If the number of parameters in the evaluation model is less than a preset parameter threshold, the model's understanding and reasoning abilities may not be strong enough. In this case, to improve the quality evaluation of the model under evaluation, the evaluation question list can contain multiple sub-questions. That is, when the evaluation question list contains multiple sub-questions, for a given model, the quality evaluation results corresponding to each sub-question for the target answer can be obtained.

[0081] Specifically, in this embodiment of the application, if the evaluation problem list includes multiple sub-problems, the performance evaluation result of the model to be evaluated is determined based on the quality evaluation result corresponding to at least one sub-problem. Specifically, this may include: fusing the quality evaluation results corresponding to the multiple sub-problems respectively to obtain a fused quality evaluation result; and determining the performance evaluation result of the model to be evaluated based on the fused quality evaluation result.

[0082] Specifically, the quality assessment result corresponding to a sub-problem can be represented by a quality assessment score. In this embodiment, the quality assessment results corresponding to multiple sub-problems are fused to obtain a fused quality assessment result. Based on the fused quality assessment result, the performance assessment result of the model to be evaluated is determined. Specifically, this may include: calculating the average value of the quality assessment scores corresponding to each sub-problem, and determining the average value of the calculated quality assessment scores as the performance assessment result of the model to be evaluated, as shown in the following formula: ; Where S represents the performance evaluation result of the model to be evaluated, s = [s1, s2, s3, ..., sn], and n is the number of subproblems in the problem list.

[0083] Specifically, in the embodiments of this application, the higher the average value of the calculated quality assessment score, the better the performance evaluation of the model under test. That is, the average value of the calculated quality assessment score is positively correlated with the performance of the corresponding model under test.

[0084] For example, continuing from the previous example, such as Figure 4As shown, the target answer output by Model A, through these three sub-questions "1. Does the answer contain the answer 'Friday'? 2. Does the answer follow JSON format? 3. Does the answer provide the analysis process?", yielded quality assessment results of 99, 83, and 85, respectively. Therefore, Model A's quality assessment score is 89. The target answer output by Model B, through these three sub-questions, yielded quality assessment results of 32, 84, and 85, respectively, resulting in a quality assessment score of 67. The target answer output by Model C, through these three sub-questions, yielded quality assessment results of 96, 43, and 83, respectively, resulting in a quality assessment score of 74. In other words, the quality assessment scores for Model A, Model B, and Model C are 89, 67, and 74, respectively. Therefore, Model A performs better than Model C, and Model C performs better than Model B.

[0085] Specifically, in order to further improve the accuracy of model evaluation and assess the matching degree between the model to be evaluated and the scenario in which the model is applied, another possible implementation involves fusing the quality evaluation results corresponding to multiple sub-problems to obtain a fused quality evaluation result. This can specifically include: obtaining the weights corresponding to each sub-problem; and weighting and fusing the quality evaluation results corresponding to each sub-problem based on the weights corresponding to each sub-problem to obtain a fused quality evaluation result.

[0086] For example, the quality assessment result corresponding to a sub-problem can be represented by a quality assessment score. The model to be evaluated includes Model 1, and the list of evaluation questions includes: sub-problem 1 and sub-problem 2. The weights corresponding to sub-problem 1 and sub-problem 2 are weight 1 and weight 2, respectively. Based on the evaluation model, the quality assessment score 1-1 corresponding to sub-problem 1 for target answer 1 and the quality assessment score 1-2 corresponding to sub-problem 2 for target answer 1 can be obtained. That is, the fused quality assessment result (score) corresponding to Model 1 = quality assessment score 1-1 × weight 1 + quality assessment score 1-2 × weight 2.

[0087] Specifically, the weights corresponding to each sub-problem are determined as follows: the target application scenario of the model to be evaluated is determined; the degree of correlation between each sub-problem and the target application scenario is determined; and the weights corresponding to each sub-problem are determined based on the degree of correlation between each sub-problem and the target application scenario.

[0088] The weight of a sub-problem is positively correlated with the degree of relevance of that sub-problem to the target application scenario.

[0089] In this embodiment of the application, the model (the model to be evaluated) is applied to intelligent question answering. However, intelligent questions involve different application scenarios. That is to say, the model to be evaluated can be applied to intelligent question answering in different scenarios. Therefore, the quality requirements or focus of the answers output by the model to be evaluated are different when applied to different application scenarios. For example, when the model to be evaluated is applied to the fields of medical consultation and legal consultation, in addition to focusing on whether the answers output by the model are correct, it is also necessary to focus on whether the answers output by the model use professional terminology, etc. As another example, when the model to be evaluated is applied to the field of customer service, in addition to focusing on whether the answers output by the model are correct, it is also necessary to focus on whether the answers output by the model are easy to understand and whether they are more in line with the customer's preferences, etc.

[0090] As shown above, in order to improve the accuracy of model quality assessment and enhance the user experience, when assessing the model's experience, the target application scenario of the model to be evaluated is determined, the degree of correlation between each sub-problem and the target application scenario is determined, and then the weights corresponding to each sub-problem are determined based on the degree of correlation between each sub-problem and the target application scenario. For example, when evaluating a model applied to medical and legal consulting fields, in addition to focusing on the correctness of the model's output answers, it is also necessary to consider whether the output answers use professional terminology. That is, the weights corresponding to sub-questions related to the correctness of the model's output answers and whether the model's output answers use certain professional terminology are better. For example, when evaluating the quality of a model applied to the legal consulting field, the large language model generates three sub-questions for the target question (e.g., what is the content of Article 22, Paragraph 2 of the Patent Law), namely sub-question 1, sub-question 2, and sub-question 3. Sub-question 1 indicates whether the target answer contains the content of "Article 22, Paragraph 2 of the Patent Law", sub-question 2 indicates whether the target answer uses legal professional terminology, and sub-question 3 indicates whether the target answer uses language that is easy for users to understand. Based on this, the weight information of sub-question 1 and sub-question 2 is greater than the weight information of sub-question 3. For example, the weight information of sub-question 1 is 50%, the weight information of sub-question 2 is 35%, and the weight information of sub-question 3 is 15%. Continuing as above, for example, if the model to be evaluated is applied in the customer service field, in addition to focusing on whether the model's output answer is correct, it is also necessary to focus on whether the model's output answer is easy to understand and whether it is more in line with customer preferences. The large language model generates three sub-questions for the target question (e.g., what does Article 22, Paragraph 2 of the Patent Law say?), namely sub-question 1, sub-question 2, and sub-question 3. Among them, sub-question 1 indicates whether the target answer contains the content of the legal provision, sub-question 2 indicates whether the target answer uses legal professional terminology, and sub-question 3 indicates whether the target answer uses language that is easy for users to understand. In this case, the weight of sub-question 1 and sub-question 3 is greater than the weight of sub-question 2. For example, the weight of sub-question 1 is 50%, the weight of sub-question 3 is 35%, and the weight of sub-question 2 is 15%.

[0091] Furthermore, after determining the weights corresponding to each sub-problem, the quality assessment results corresponding to each sub-problem are weighted and fused based on their respective weights to obtain the fused quality assessment result. For details on the method described in the above embodiment, please refer to the above embodiment.

[0092] Specifically, this paper introduces a model evaluation scheme through a concrete example. For instance, in large-model question-answering scenarios such as dialogue systems, given a dialogue or question, when it is necessary to evaluate the quality and extent of responses from different large models, automated LLM evaluation can play an important role in this process. The goal of this embodiment is to score the answers to the same question from different large models or different versions of the same large model. Specifically, a question-answer checklist is first generated using a proprietary and powerful LLM model (a powerful LLM is needed at this stage to ensure effectiveness). Then, a small model is used to evaluate the question list separately in a low-cost and reproducible manner, providing scores (using a very small LLM reduces costs and speeds up feedback; due to the uncertainty of the small model, normalized probabilities are used to obtain the scores for the current list of questions). Finally, the probabilities of all lists are summed to obtain the final score. Figure 6 As shown below: Step 601: Obtain the target question from the historical session. For example, the target question could be "What day of the week is two days after New Year's Day in 2025? Please provide your thought process and answer in JSON format." The historical session contains the requirements for each solution.

[0093] Step S602: Obtain the first prompt information; The first prompt message includes the target requirements, the requirements for the output format of the evaluation question list, and an example of the prompt message; Among them, the target requirements can be as follows: Figure 5 The text states, "Based on the information above, I need you to create a list of binary questions so that I can conduct an effective and accurate evaluation by answering several questions. Your questions should be concise and include any necessary key content and information (such as keywords, format, correct counts, and values) that the user queries or expects to display in the response. Your questions should consider not only the reference response but also all possible responses. Avoid creating repetitive, verbose, or vague questions. For example, you should ask 'Does this answer contain the correct answer...' instead of 'Is this answer correct?'." Among them, such as Figure 5 As shown, "## Output Format" Please output in the following format: 1.{question1} 2.{question2} 3.…” Among them, such as Figure 5 As shown, an example of a prompt message can be found below: # Dialogue between users and AI <|begin_of_history|> {history} <|end_of_history|> ## Current User Query <|begin_of_query|> {user_query} <|end_of_query|> ## Reference Response <|begin_of_reference_response|> {reference_response} <|end_of_reference_response|>.

[0094] It should be noted that step S601 can be executed before step S602, after step S602, or simultaneously with step S602. Figure 6 This is merely one possible example and is not intended to limit the embodiments of this application.

[0095] Step S603: Generate a list of evaluation questions based on the target questions and first prompt information in the historical sessions; The generated sub-questions (evaluation question list) that meet the target requirements may include: "1. Does the answer contain the answer 'Friday'? 2. Does the answer follow JSON format? 3. Does the answer provide the analysis process?", specifically as follows: Figure 4 As shown.

[0096] By generating a question list, this stage allows us to use a small model to quickly assess the degree to which different responses adhere to the question list. Because the small model has weak reading comprehension and logical reasoning abilities (its reasoning using COT (thought chain) is even worse than general evaluation methods), and exhibits significant inconsistency with consecutive answers (answering two consecutive questions simultaneously and then changing the order of the questions may yield drastically different results), our small model scoring stage mainly consists of the following two steps. Specifically, this may include: Individual answering: In the previous stage, we obtained a list of 5-10 questions. In this stage, we use the small model to answer the sub-questions (sub-questions in the evaluation question list) separately. Normalized probability: Given the high uncertainty of lightweight LLMs, relying solely on binary results such as "yes" or "no" may lead to significant errors in the final judgment, as detailed in steps S604 and S605 below.

[0097] Step S604: Input the target question into the model to be evaluated, and obtain the target answer to the target question generated by the model to be evaluated; For example, if the target question is "What day of the week is two days after New Year's Day in 2025?", the corresponding answer requirements could include: "Please provide your thought process and answer in JSON format." The test models could be ModelA, ModelB, and ModelC. ModelA would output the answer as {'Thinking': '……', 'Answer': 'It's Friday'}; ModelB would output the answer as {'Thinking': '……', 'Answer': 'It's Wednesday'}; and ModelC would output the answer as {'Thinking': '……', 'Answer': 'It's Friday'}. Figure 4 As shown.

[0098] Step S605: For each sub-question, evaluate the quality assessment result of the target answer corresponding to that sub-question using the evaluation model; like Figure 4As shown, the target answer output by Model A, through these three sub-questions "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 99, 83, and 85. That is, the target answer output by Model A corresponds to a quality assessment result of 99 for sub-question "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields a quality assessment result of 83, and a quality assessment result of 85 for sub-question "3. Does the answer provide the analysis process?". Furthermore, the target answer output by Model B, through these three sub-questions "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 32, 84, and 85. That is, the target answer output by Model B corresponds to a quality assessment result of 99 for sub-question "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 32, 84, and 85. The quality assessment results for the target answer of Model C through the sub-questions "1. Does the answer contain the answer Friday?" are 32, "2. Does the answer follow JSON format?", and "3. Does the answer provide the analysis process?" are 85. Furthermore, the quality assessment results for the target answer of Model C through these three sub-questions are: 96, 43, and 83. In other words, the quality assessment result for the target answer of Model C through the sub-question "1. Does the answer contain the answer Friday?" is 96, the quality assessment result for the target answer of Model B through the sub-question "2. Does the answer follow JSON format?" is 43, and the quality assessment result for the target answer of Model C through the sub-question "3. Does the answer provide the analysis process?" is 83.

[0099] Step S606: For each sub-problem to be evaluated, calculate the average quality assessment score of each sub-problem, and determine the average quality assessment score as the performance evaluation result of the model to be evaluated.

[0100] like Figure 4As shown, the target answer output by Model A, through these three sub-questions: "1. Does the answer contain the answer 'Friday'?; 2. Does the answer follow JSON format?; 3. Does the answer provide the analysis process?", yields quality assessment results of 99, 83, and 85. Therefore, Model A's quality assessment score is 89. The target answer output by Model B, through these three sub-questions, yields quality assessment results of 32, 84, and 85. Therefore, Model B's quality assessment score is 67. The target answer output by Model C, through these three sub-questions, yields quality assessment results of 96, 43, and 83. Therefore, Model C's quality assessment score is 74. In other words, the quality assessment scores for Model A, Model B, and Model C are 89, 67, and 74, respectively.

[0101] Specifically, through the embodiments of this application, large-scale model automated evaluation can be completed in a low-cost, reproducible, and secure manner, and the evaluation results are even better than those of larger models. The evaluation data used can be MT-bench or other evaluation data, such as questions containing 160 single / multi-round questions.

[0102] The evaluation metric here is the consistency comparison with the annotation results of crowdsourcing. Crowdsourcing uses multiple people to annotate the data to obtain a set of results that best reflects human preferences. Other models and people also annotate this set of data, sort the standard results, and calculate the consistency with the sorting order of the crowdsourcing data.

[0103] In the embodiments of this application, the evaluation consistency effect of the small model is better and is also close to the human annotation results.

[0104] Another point is that the method in this application embodiment consumes far fewer resources than other methods; and it has good security and reproducibility.

[0105] Based on the same principle as the model evaluation method provided in the embodiments of this application, the embodiments of this application provide a model evaluation apparatus, such as... Figure 7 As shown, the model evaluation device 70 may specifically include: a first acquisition module 71, an acquisition module 72, an evaluation module 73, and a first determination module 74, wherein, The first acquisition module 71 is used to acquire the model to be evaluated, the target question for evaluating the model to be evaluated, and the list of evaluation questions for the target question; wherein, the list of evaluation questions contains at least one sub-question for evaluating the quality of the answer to the target question, and the list of questions is generated based on the target question through a large language model; The module 72 is used to input the target question into the model to be evaluated and obtain the target answer to the target question generated by the model to be evaluated. Evaluation module 73 is used to evaluate the quality assessment result of the target answer corresponding to the sub-question using an evaluation model for each sub-question; The first determining module 74 is used to determine the performance evaluation result of the model to be evaluated based on the quality evaluation result corresponding to at least one sub-problem.

[0106] In one possible implementation of this application embodiment, the apparatus 70 further includes: a generation module, wherein... The evaluation question list for the target problem is generated by the generation module in the following way: Identify the target problem; Obtain the first prompt information; the first prompt information contains the target requirements, which are the requirements that the sub-problems to be generated by the large language model need to meet; Based on the initial prompt information and the target question, and through a large language model, at least one sub-question of the target question that meets the target requirements is generated.

[0107] In another possible implementation of this application embodiment, when obtaining the target answer generated by the target question through the model to be evaluated by inputting the target question into the model to be evaluated, the module 72 is specifically used for: Obtain at least one answer requirement for the target problem; Input the target question and at least one question's answer requirements into the model to be evaluated, and generate target answers that meet the answer requirements of each question through the model to be evaluated; Specifically, when the generation module generates at least one sub-problem of the target problem that meets the target requirements based on the first prompt information and the target problem, and through a large language model, it is used for: The first prompt information, the target question, and the requirements for answering each question are input into the large language model to generate at least one sub-question of the target question that meets the target requirements. The at least one sub-question includes sub-questions corresponding to the requirements for answering each question.

[0108] Another possible implementation of this application embodiment includes a list of sub-problems in the evaluation problem list; When determining the performance evaluation result of the model to be evaluated based on the quality evaluation result corresponding to at least one sub-problem, the first determining module 74 is specifically used for: The quality assessment results corresponding to multiple sub-problems are merged to obtain the merged quality assessment result. Based on the fused quality assessment results, the performance assessment results of the model to be evaluated are determined.

[0109] In another possible implementation of this application embodiment, when the evaluation module 73 evaluates the quality assessment result of the target answer corresponding to the sub-question using the evaluation model for each sub-question, it is specifically used for: The model is evaluated to predict the first probability of a match between the target answer and the subquestion, and the second probability of a mismatch. Determine the sum of the probabilities between the first probability and the second probability; Determine the ratio between the first probability and the sum of probabilities, and use this ratio as the quality assessment result of the target answer corresponding to the sub-problem.

[0110] In another possible implementation of this application embodiment, when the first determining module 74 fuses the quality assessment results corresponding to multiple sub-problems to obtain a fused quality assessment result, it is specifically used for: Obtain the weights corresponding to each sub-problem; Based on the weights corresponding to each sub-problem, the quality assessment results corresponding to each sub-problem are weighted and fused to obtain the fused quality assessment result.

[0111] In another possible implementation of this application embodiment, the apparatus 70 further includes: a second determining module, wherein... The weights corresponding to each sub-problem are determined by the second determining module in the following way: Determine the target application scenario for the model to be evaluated; Determine the degree of correlation between each sub-problem and the target application scenario; Based on the degree of correlation between each sub-problem and the target application scenario, the weight corresponding to each sub-problem is determined; The weight of a sub-problem is positively correlated with the degree of relevance of that sub-problem to the target application scenario.

[0112] In another possible implementation of this application embodiment, when the evaluation module 73 evaluates the quality assessment result of the target answer corresponding to the sub-question using the evaluation model for each sub-question, it is specifically used for: The target question, the sub-question, and the target answer are input into the evaluation model, which then evaluates the quality of the target answer corresponding to the sub-question, with reference to the target question.

[0113] Another possible implementation of this application embodiment is that the number of model parameters of the evaluation model is less than a preset parameter threshold.

[0114] In another possible implementation of this application embodiment, the device 70 further includes: a second acquisition module, wherein... The second acquisition module is used to obtain the reference answer corresponding to the target question; Specifically, when the generation module generates at least one sub-question for the target question based on the first prompt information and the target question, and through a large language model, it is used for: The first prompt, the target question, and the reference answer are input into the large language model, which then uses the reference answer to generate at least one sub-question for the target question.

[0115] This application provides a model evaluation apparatus. In this embodiment, the apparatus acquires a model to be evaluated and a target question for evaluating the model. Based on the target question, a large language model generates an evaluation question list containing at least one sub-question for evaluating the quality of the answer to the target question. The target question is then input into the model to be evaluated to obtain the target answer generated by the model. In this embodiment, after obtaining the target answer generated by the model and the evaluation question list generated by the large language model, for each sub-question, an evaluation model can assess the quality evaluation result of the target answer corresponding to that sub-question. Based on the quality evaluation results corresponding to at least one sub-question, the performance evaluation result of the model to be evaluated is determined, thereby enabling the evaluation of the model's performance. Furthermore, since the evaluation question list is generated by the large language model, the accuracy of each sub-question in the generated evaluation question list is high, resulting in high accuracy in subsequent performance evaluations of each model to be evaluated using other models. Moreover, it eliminates the need for performance evaluation of each model to be evaluated using a large language model; a single evaluation model suffices, thereby reducing the cost of model performance evaluation and improving the user experience.

[0116] It should be noted that the first acquisition module 71 and the second acquisition module can be the same acquisition module or different acquisition modules; the first determination module 74 and the second determination module can be the same determination module or different determination modules, and this is not limited in the embodiments of this application. The device of the embodiments of this application can execute the method provided in the embodiments of this application, and their implementation principles are similar. The actions performed by each module in the device of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For a detailed functional description of each module of the device, please refer to the description in the corresponding method shown above, which will not be repeated here.

[0117] Figure 8 A schematic diagram of the structure of an electronic device to which this application embodiment applies is shown, such as... Figure 8As shown, for example, the electronic device may be a server or a user terminal, and the electronic device may be used to implement the methods provided in any embodiment of this application.

[0118] like Figure 8 As shown, the electronic device 2000 may primarily include at least one processor 2001. Figure 8 The diagram shows components such as a memory 2002, a communication module 2003, and an input / output interface 2004. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 8 The structure of the electronic device 2000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0119] The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0120] Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0121] Electronic device 2000 can connect to a network via communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 2003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.

[0122] Electronic device 2000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 2004. Electronic device 2000 itself may have a display device, and other display devices can also be connected externally via interface 2004. Optionally, storage devices, such as hard drives, can also be connected via interface 2004 to store data from electronic device 2000, retrieve data from storage devices, or store data from storage devices into memory 2002. It is understood that input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 2004 can be a component of electronic device 2000 or an external device connected to electronic device 2000 when needed.

[0123] The bus 2005 used to connect the components may include a pathway for transmitting information between the components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc.

[0124] Optionally, for the solution provided in the embodiments of this application, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of this application.

[0125] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0126] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0127] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0128] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0129] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0130] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A method for evaluating a model, characterized in that, The method includes: Obtain the model to be evaluated, the target question for evaluating the model to be evaluated, and the evaluation question list for the target question; wherein the evaluation question list contains at least one sub-question for evaluating the quality of the answer to the target question, and the question list is generated based on the target question through a large language model; The target question is input into the model to be evaluated to obtain the target answer to the target question generated by the model to be evaluated. For each sub-question, the quality assessment result of the target answer corresponding to that sub-question is evaluated using an evaluation model; Based on the quality assessment results corresponding to the at least one sub-problem, the performance assessment results of the model to be evaluated are determined.

2. The method according to claim 1, characterized in that, The list of evaluation questions for the target problem was generated in the following way: Obtain the target problem; Obtain first prompt information; wherein, the first prompt information includes target requirements, the target requirements being the requirements that the sub-problems to be generated by the large language model need to meet; Based on the first prompt information and the target question, and through the large language model, at least one sub-question of the target question that satisfies the target requirements is generated.

3. The method according to claim 2, characterized in that, The step of inputting the target question into the model to be evaluated and obtaining the target answer to the target question generated by the model to be evaluated includes: Obtain at least one requirement for answering the target problem; The target question and the at least one question answering requirement are input into the model to be evaluated, and the target answer that satisfies each question answering requirement is generated by the model to be evaluated; The step of generating at least one sub-question of the target question that satisfies the target requirements based on the first prompt information and the target question, and through the large language model, includes: The first prompt information, the target question, and the answer requirements of each question are input into the large language model to generate at least one sub-question of the target question that satisfies the target requirements, wherein the at least one sub-question includes a sub-question corresponding to each answer requirement of the question.

4. The method according to claim 1, characterized in that, The evaluation question list includes multiple sub-questions; The step of determining the performance evaluation result of the model to be evaluated based on the quality evaluation results corresponding to the at least one sub-problem includes: The quality assessment results corresponding to the multiple sub-problems are merged to obtain the merged quality assessment result. Based on the fused quality assessment results, the performance assessment results of the model to be evaluated are determined.

5. The method according to claim 1, characterized in that, For each of the sub-questions, the evaluation of the quality assessment result of the target answer corresponding to that sub-question through the evaluation model includes: The evaluation model predicts a first probability that the target answer matches the sub-question and a second probability that they do not. Determine the sum of probabilities between the first probability and the second probability; Determine the ratio between the first probability and the sum of the probabilities, and use the ratio as the quality assessment result of the target answer corresponding to the sub-question.

6. The method according to claim 4, characterized in that, The step of fusing the quality assessment results corresponding to the multiple sub-problems to obtain the fused quality assessment result includes: Obtain the weights corresponding to each of the sub-problems; Based on the weights corresponding to each of the sub-problems, the quality assessment results corresponding to each of the sub-problems are weighted and fused to obtain the fused quality assessment result.

7. The method according to claim 6, characterized in that, The weights corresponding to each of the sub-problems are determined in the following way: Determine the target application scenario for the model to be evaluated; Determine the degree of correlation between each of the sub-problems and the target application scenario; Based on the degree of correlation between each sub-problem and the target application scenario, the weight corresponding to each sub-problem is determined. The weight of a sub-problem is positively correlated with the degree of correlation between the sub-problem and the target application scenario.

8. The method according to claim 1, characterized in that, For each of the sub-questions, the evaluation of the quality assessment result of the target answer corresponding to that sub-question through the evaluation model includes: The target question, the sub-question, and the target answer are input into the evaluation model, which then evaluates the quality assessment result of the target answer corresponding to the sub-question, with reference to the target question.

9. The method according to claim 4, characterized in that, The number of parameters in the evaluation model is less than a preset parameter threshold.

10. A device for model evaluation, characterized in that, The device includes: The first acquisition module is used to acquire a model to be evaluated, a target question for evaluating the model to be evaluated, and a list of evaluation questions for the target question; wherein, the list of evaluation questions includes at least one sub-question for evaluating the quality of the answer to the target question, and the list of questions is generated based on the target question through a large language model; The module is used to input the target question into the model to be evaluated, and obtain the target answer to the target question generated by the model to be evaluated; The evaluation module is used to evaluate the quality assessment result of the target answer corresponding to each sub-question using an evaluation model. The first determining module is used to determine the performance evaluation result of the model to be evaluated based on the quality evaluation result corresponding to the at least one sub-problem.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the model evaluation method according to any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the model evaluation method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the model evaluation method according to any one of claims 1 to 9.

Images