A method and related apparatus for evaluating the ability of large language models

Abstract

This application discloses a method and related apparatus for evaluating the capabilities of large language models, relating to the field of model evaluation technology. The method includes: obtaining the answers generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset; generating corresponding answer quality evaluation rules for each test question using a rule-based large model; combining the large language models in pairs, combining the two answers generated by the same model pair for each test question to obtain the answer pairs corresponding to each model pair for each test question; for each test question, evaluating the merits of the two answers contained in each answer pair according to the evaluation rules corresponding to that test question; and determining the capability evaluation result of each large language model based on the evaluation results of each answer pair. The evaluation method disclosed in this application can automatically evaluate the capabilities of multiple large language models, with high evaluation efficiency and objective and reliable evaluation results.

Description

Technical Field

[0001] This application relates to the field of model capability evaluation technology, and in particular to a method and related apparatus for evaluating the capability of large language models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful generalization capabilities and multi-task adaptability in the field of natural language processing. In recent years, a large number of advanced models with massive parameter scales and abundant training data have emerged in the industry and have been widely used in diverse scenarios such as machine translation, intelligent question answering, text generation, and semantic understanding. However, the resulting problem of scientifically evaluating model capabilities has become a key bottleneck restricting the industry's model selection.

[0003] Currently, the mainstream large language model capability assessment scheme is mainly a human-based assessment scheme. The basic process of this scheme is as follows: first, human judges review and score the answers generated by multiple large language models for each test question to obtain the answer quality scores of each language model for each test question. Then, statistical analysis is performed on the answer quality scores of each language model for each test question to obtain the capability assessment results of each language model.

[0004] However, current large language model ability assessment schemes are costly in terms of manpower due to the need to recruit and train high-level judges, and take a long time to assess because they require manual review of the massive amounts of answers generated by multiple large language models. Furthermore, the assessment results are easily affected by the judges' subjective perceptions, cultural backgrounds, and state of mind at the time, making it difficult to ensure consistency in the assessment criteria between different judges or even the same judge at different times. Therefore, the reliability and comparability of the assessment results are poor. Summary of the Invention

[0005] In view of this, this application provides a method and related apparatus for evaluating the capabilities of large language models, in order to solve the problems of high labor costs, long evaluation time, and poor reliability and comparability of evaluation results in existing large language model capability evaluation schemes. The technical solution is as follows:

[0006] The first aspect of this application provides a method for evaluating the capabilities of large language models, including:

[0007] A method for evaluating the capabilities of a large language model includes:

[0008] Obtain the answers generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset;

[0009] A large model is generated using rules to produce a dedicated evaluation rule for judging the quality of the answer for each test question, thus obtaining the dedicated evaluation rule corresponding to each test question;

[0010] Each of the large language models is paired up, and for each test question, the answers generated by the two large language models in the same model pair for the test question are combined to obtain the answer pair corresponding to each model pair for the test question.

[0011] For each test question, the merits of the two answers in the corresponding answer pair of each model are evaluated according to the specific evaluation rules for the test question.

[0012] Based on the evaluation results of the answers of each model to each test question, the capability evaluation results of each large language model are determined.

[0013] In one possible implementation, the test dataset covers multiple languages ​​and multiple task types;

[0014] Each test question in the test dataset belongs to one of the multiple task types and is expressed in one of the multiple languages.

[0015] In one possible implementation, the rule-based generation of a large model generates specific evaluation rules for each test question to assess the quality of the answers, resulting in specific evaluation rules for each test question, including:

[0016] For each of the test questions, a rule generation prompt is constructed for the test question. The rule generation prompt includes the test question and instructions to generate specific evaluation rules for the test question to judge the quality of the answer.

[0017] The rule-generated prompt is used to generate a large model, and the dedicated evaluation rule corresponding to the test question is output by the rule-generated large model.

[0018] In one possible implementation, obtaining the answer generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset includes:

[0019] For each test question in the test dataset, a response test prompt is constructed for the test question. The response test prompt includes the test question and a constraint instruction that limits the language of the output answer. The response test prompt instructs the test question to be answered in the language limited by the constraint instruction.

[0020] The response test prompt is input into each of the large language models in the set of large language models to be evaluated, and the answer generated by each of the large language models for the test question is obtained.

[0021] In one possible implementation, judging the merits of the two answers in each model pair corresponding to the test question based on the specific evaluation rules for the test question includes:

[0022] For each model pair, the order of the two answers in the answer pair corresponding to the model pair on the test question is randomized to obtain the target order.

[0023] Based on the target arrangement order, an answer comparison prompt is constructed, wherein the answer comparison prompt includes the test question, the dedicated evaluation rule corresponding to the test question, and the answer pair corresponding to the model on the test question, and the arrangement order of the two answers to be compared in the answer comparison prompt is the target arrangement order;

[0024] The answer is compared with the prompt and input into the evaluation model to obtain the evaluation result of the model on the test question.

[0025] In one possible implementation, the randomization of the order of the two answers in the answer pair corresponding to the test question in the model to obtain the target order includes:

[0026] Set the initial order of the two answers in the answer pair corresponding to the test question in the model;

[0027] A decision value is generated based on a random algorithm to determine whether to adjust the initial arrangement order;

[0028] Based on the decision values, determine the order in which the targets are arranged.

[0029] In one possible implementation, the answer comparison prompt includes constraint instructions to suppress the evaluation model's bias towards answer length.

[0030] In one possible implementation, the test dataset covers multiple languages;

[0031] The step of determining the capability evaluation results of each of the large language models based on the evaluation results of the answers corresponding to each of the test questions by each model includes:

[0032] Based on the evaluation results of the model's answers to each test question, the win rate of each large language model in each language is calculated.

[0033] Based on the win rate of each of the large language models in each of the languages, determine the ranking of each of the large language models in a single language and / or the overall ranking in the multiple languages.

[0034] In one possible implementation, the test dataset covers multiple task types;

[0035] The step of determining the capability evaluation results of each of the large language models based on the evaluation results of the answers corresponding to each of the test questions by each model includes:

[0036] Based on the evaluation results of the model's answers to each test question, the win rate of each large language model in each task type is calculated.

[0037] Based on the win rate of each of the large language models in each of the task types, determine the ranking of each of the large language models in a single task type and / or the overall ranking in the multiple task types.

[0038] One possible implementation method for evaluating the capabilities of large language models includes:

[0039] Obtain the updated model of the first target model for each test question, wherein the first target model is a large language model in the large language model set that has been updated;

[0040] The first target model in the model pair that contains the first target model in the original model pair is updated to the updated model of the first target model, thus obtaining the updated model pair;

[0041] For each test question, the answers generated by the two large language models contained in the same update model pair for the test question are combined to obtain the answer pair corresponding to each update model pair for the test question;

[0042] For each test question, the merits of the two answers in the corresponding answer pair of each updated model are evaluated according to the specific evaluation rules for the test question.

[0043] Based on the evaluation results of the updated model's answers to each test question and the evaluation results of the unupdated model's answers to each test question, the ability evaluation results of the updated model of the first target model and each unupdated large language model are determined.

[0044] One possible implementation method for evaluating the capabilities of large language models includes:

[0045] Obtain the answer generated by the second target model for each of the test questions, wherein the second target model is a newly added large language model to be evaluated;

[0046] The second target model is combined with each of the original large language models to obtain multiple new model pairs;

[0047] For each test question, the answers generated by the two large language models contained in the same new model pair for the test question are combined to obtain the answer pair corresponding to each new model pair for the test question;

[0048] For each test question, the merits of the two answers in the corresponding answer pair of each new model are evaluated according to the specific evaluation rules for the test question.

[0049] Based on the evaluation results of the answers of each newly added model to each test question, and the evaluation results of the answers of each existing model to each test question, the capability evaluation results of all large language models, including the second target model, are determined.

[0050] One possible implementation method for evaluating the capabilities of large language models includes:

[0051] When a major language model in the large language model set is discarded, the ability evaluation results of the major language models in the large language model set that are not discarded are re-determined based on the evaluation results of the corresponding answers of each target model pair on each test question. The target model pair is the model pair in each original model pair that does not contain the discarded major language model.

[0052] A second aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0053] The memory is used to store computer programs;

[0054] The processor is used to execute the computer program so that the electronic device can implement any of the above-mentioned large language model capability evaluation methods.

[0055] A third aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement any of the above-described large language model capability evaluation methods.

[0056] A fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement any of the above-described large language model capability evaluation methods.

[0057] By employing the above technical solution, the large language model capability evaluation method provided in this application is a novel automated method for evaluating the capability of large language models. This evaluation method introduces a dynamic rule generation mechanism, that is, it uses rule generation to generate corresponding dedicated evaluation rules for each test question in the large model. Based on this, all large language models are paired up, and for each test question, the merits of the two answers generated by the two large language models contained in the same model pair are evaluated according to the corresponding dedicated evaluation rules. Finally, based on the merit evaluation results of the answers corresponding to each model pair for each test question, the capability evaluation results of each large language model are determined. The large language model capability evaluation method provided in this application, on the one hand, eliminates the reliance on a large number of highly skilled human evaluators because the generation of evaluation rules, the comparison of answers, and the determination of the final capability evaluation results are all executed automatically, and the evaluation time is greatly shortened. On the other hand, since all answer pairs for the same test question are evaluated using the same evaluation rules, the standard drift of the same evaluator at different times or between different evaluators is avoided. Although the rules themselves may originate from the subjective understanding of the large model, once the rules are textualized and fixed, they become objective evaluation criteria, eliminating interference factors such as human emotions and fatigue, thereby greatly improving the reliability and comparability of the evaluation results. In addition, the large language model capability evaluation method provided in this application automates both the "standard setting" (evaluation rule generation) and "standard execution" (comparing and evaluating answers according to the evaluation rules), which makes large-scale, high-frequency model capability evaluation possible. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0059] Figure 1 A flowchart illustrating the large language model capability evaluation method provided in this application embodiment;

[0060] Figure 2 This is a flowchart illustrating the process of randomizing the order of two answers in a test question to obtain a target order, as provided in this embodiment of the application.

[0061] Figure 3 This is a schematic diagram of the structure of the large language model capability evaluation system provided in the embodiments of this application. Detailed Implementation

[0062] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0063] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0064] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0065] Currently, the mainstream evaluation schemes for large language models are mainly based on manual evaluation. However, these schemes suffer from high labor costs, long evaluation times, and poor reliability and comparability due to the susceptibility of results to subjective human factors. Therefore, research was conducted. The initial approach was to adopt a fixed-metric evaluation scheme, using preset indicators such as BLEU, ROUGE, and Accuracy to quantitatively score the answers generated by various language models for test questions. While fixed-metric evaluation schemes are efficient, they struggle to comprehensively and accurately reflect the performance of large language models in real-world application scenarios.

[0066] To address the shortcomings of evaluation schemes based on fixed indicators, this paper presents a more effective method for evaluating the capabilities of large language models through continuous research. The following embodiments will introduce the large language model capability evaluation method provided in this application.

[0067] Please see Figure 1 The diagram illustrates a flowchart of a large language model capability evaluation method provided in an embodiment of this application. This large language model capability evaluation method may include:

[0068] Step S101: Obtain the answer generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset.

[0069] The set of large language models to be evaluated includes multiple large language models to be evaluated, and the test dataset contains multiple test questions. The test dataset may also include relevant information for each test question, such as ID, language, task type, and reference answer.

[0070] The test dataset in this embodiment can cover multiple task types (such as text generation tasks, general question answering tasks, etc.) and multiple languages ​​(such as Chinese, English, etc.). A test dataset covering multiple task types and multiple languages ​​can be pre-built. After the test dataset is built, experts (such as language experts) can conduct quality checks on the test dataset to eliminate problematic data and supplement new data to ensure that the amount of data for each language and each task type meets the requirements.

[0071] It should be noted that this embodiment does not limit the test dataset to cover multiple task types and multiple languages. For example, the test dataset can cover one task type and multiple languages, multiple task types and one language, or one task type and one language. The coverage of languages ​​and task types in the test dataset can be determined according to the actual testing needs.

[0072] For each test question in the test dataset, an answer can be generated based on each large language model in the large language model set. Specifically, a response prompt can be constructed for the test question. The response prompt contains the test question and instructions to prompt the large language models to generate the corresponding answer. The response prompt constructed for the test question is input into each large language model in the large language model set to obtain the answer generated by each large language model in the set for the test question. It should be noted that if the answer to a test question fails to be obtained, it can be retried multiple times.

[0073] To avoid language mixing in the answers generated by the large language model, preferably, the response test prompt can include a constraint instruction to limit the language of the output answer, that is, the response test prompt instructs the large language model to answer the test question in the language limited by the constraint instruction.

[0074] In this embodiment, the format of the response test prompt can be {"id": "xxxx", "question": "xxxx", "query":"xxxx"}, where id is the unique identifier of the test question, question is the test question, and query is the instruction for the large language model to answer the test question in a limited language.

[0075] Step S102: Generate a large model using rules, generate a dedicated evaluation rule for each test question to judge the quality of the answer, and obtain the dedicated evaluation rule corresponding to each test question.

[0076] Specifically, for each test question in the test dataset, a rule generation prompt is first constructed for that test question. This prompt contains the test question and instructions to generate specific evaluation rules for assessing the quality of the answers. After obtaining the rule generation prompt for that test question, it is input into the rule generation model to obtain the specific evaluation rules for that test question, output by the model. Through this process, the specific evaluation rules for each test question in the test dataset can be obtained.

[0077] The format of the rule generation prompt can be {"id": "xxxx", "question": "xxxx", "query":"xxxx"}, where id is the unique identifier of the test question, question is the test question, and query is the prompt. The large-scale rule generation model generates special evaluation rules for the test question to judge the quality of the answer.

[0078] This embodiment utilizes a rule-generating large model to dynamically generate a set of evaluation rules for each test question in the test dataset. In other words, this embodiment automates the process of formulating evaluation criteria and customizes the questions.

[0079] Step S103: Combine each large language model in pairs. For each test question, combine the answers generated by the two large language models in the same model pair for that test question to obtain the answer pair corresponding to each model pair for that test question.

[0080] This embodiment combines the major language models in the large language model set in pairs to obtain one or more model pairs. Assuming there are N major language models in the large language model set, combining them in pairs without repetition yields C. N 2 For example, if there are three large language models A, B and C in the large language model set, then combining the three large language models in pairs will yield three model pairs (A, B), (A, C) and (B, C).

[0081] After obtaining the model pairs, the answers generated by the two large language models within the same model pair for each test question are combined to obtain the answer pair corresponding to each model pair for each test question. For example, the model pairs are (A, B), (A, C), and (B, C), and the test dataset contains three test questions, q1, q2, and q3. The two answers (result) generated by the model pair (A, B) for test question q1 are combined. A-q1 and result B-q1 By constructing answer pairs, we obtain the corresponding answer pairs (result) for the model pair (A, B) on test question q1. A-q1 , result B-q1 This can be used to generate two results from the model pair (A, C) for the test question q1. A-q1 and result C-q1 By constructing answer pairs, we obtain the answer pair (result) corresponding to the model pair (A, C) on test question q1. A-q1 , result C-q1 The model can generate two results for the test question q1 based on the model pair (B, C). B-q1 and result C-q1 By constructing answer pairs, we obtain the corresponding answer pair (result) for the model pair (B, C) on test question q1. B-q1 , result C-q1 Similarly, we can obtain the answer pair (result) for the model pair (A, B) on the test question q2. A-q2 , result B-q2 The model pair (A, C) corresponds to the answer pair (result) on test question q2. A-q2 ,result C-q2 ) and the corresponding answer pair (result) for model pair (B, C) on test question q2. B-q2 , result C-q2 It can also obtain the answer pair (result) for the model pair (A, B) on test question q3. A-q3 , result B-q3 The model pair (A,C) corresponds to the answer pair (result) on test question q3. A-q3 , result C-q3 ) and the corresponding answer pair (result) for model pair (B, C) on test question q3. B-q3 , result C-q3 ).

[0082] Step S104: For each test question, according to the specific evaluation rules corresponding to the test question, evaluate the merits of the two answers contained in the answer pair corresponding to the test question for each model.

[0083] In one possible implementation, for each test question in the test dataset, an evaluation model can be used to evaluate the merits of each model's answer pair for that test question based on the specific evaluation rules corresponding to that test question, so as to obtain the merit evaluation result of each model's answer for that test question.

[0084] Step S105: Determine the capability evaluation results of each large language model based on the evaluation results of the corresponding answers of each model on each test question.

[0085] After obtaining the evaluation results of the corresponding answers of each model on each test question, the win rate of each large language model can be calculated, and then the ranking of each large language model can be determined based on the win rate of each large language model.

[0086] The large language model capability evaluation method provided in this application introduces a dynamic rule generation mechanism and a pairwise comparison mechanism. Specifically, it uses rule generation to generate corresponding dedicated evaluation rules for each test question in the large model. Based on this, all large language models are paired up. For each test question, the merits of the two answers generated by the two large language models in the same model pair for the test question are evaluated according to the corresponding dedicated evaluation rules. Finally, the capability evaluation results of each large language model are determined based on the merit evaluation results of the answers corresponding to each model for each test question. The large language model capability evaluation method provided in this application has several advantages. First, because the generation of evaluation rules, the comparison of answers, and the determination of the final capability evaluation results are all executed automatically, the reliance on a large number of highly skilled human evaluators is eliminated, and the evaluation time is greatly shortened. Second, since all answer pairs for the same test question are evaluated using the same evaluation rules, the standard drift of the same evaluator at different times or between different evaluators is avoided. Although the rules themselves may originate from the subjective understanding of the large model, once the rules are textualized and fixed, they become objective evaluation criteria, eliminating interference factors such as human emotions and fatigue, thereby greatly improving the reliability and comparability of the evaluation results. In addition, the large language model capability evaluation method provided in this application automates both the "standard setting" (evaluation rule generation) and "standard execution" (comparing and evaluating answer pairs according to the evaluation rules), making large-scale, high-frequency model capability evaluation possible.

[0087] The above embodiments mention that, for each test question in the test dataset, a large evaluation model can be used to evaluate the merits of each model's answer pair for that test question based on the specific evaluation rules corresponding to that test question. Some embodiments of this application describe using a large evaluation model based on a test question q. i The corresponding dedicated evaluation rules are used to evaluate each model for the test problem q. i The specific implementation process of determining the merits of the two answers contained in the corresponding answer pair.

[0088] like Figure 2 As shown, using the large evaluation model, based on the test question q i The corresponding dedicated evaluation rules are used to evaluate each model for the test problem q. i The process of determining the merits of the two corresponding answer pairs can include:

[0089] Step S201: For each model pair, apply the model pair to the test question q. i The corresponding answer is used to randomize the order of the two answers to obtain the target order.

[0090] Considering that when the evaluation model judges two answers, its output may be affected by the order in which the answers appear. For example, the evaluation model may unconsciously be more inclined to think that the first answer is better (position bias) or pay more attention to the novelty of the second answer. If the evaluation model has order bias, it will affect the fairness of the evaluation results.

[0091] In view of this, this embodiment proposes to randomize the order of the two answers in the answer pair to be evaluated. Randomization allows the positional bias of the large evaluation model to cancel each other out among a large number of answer pairs, making the final evaluation result fairer. By controlling the potential interference factor of the answer presentation order, randomization eliminates or mitigates the order bias of the large evaluation model, making the evaluation result more reliable, unbiased, and statistically significant, thereby improving the scientificity and credibility of the entire evaluation process.

[0092] In one possible implementation, the model is applied to the test problem q. i The process of randomizing the order of the two answers in the corresponding answer set to obtain the target order may include:

[0093] Step S2011: Set the model for the test problem q i The initial order of the two answers contained in the corresponding answer pair above.

[0094] Suppose the model pair is (A, B), and the model pair (A, B) is used in the test question q. iThe corresponding answer pair above is (result) A-qi , result B-qi The result can be set. A-qi and result B-qi The initial arrangement order is result A-qi , result B-qi .

[0095] Step S2012: Based on a random algorithm, generate decision values ​​to determine whether to adjust the initial arrangement order.

[0096] After setting up the model pair (A, B) for test question q i The corresponding answer pair (result) A-qi , result B-qi After determining the initial order of the two answers, a decision value is generated based on a random algorithm to decide whether to adjust the initial order.

[0097] Step S2013: Determine the order of the targets based on the decision values.

[0098] In one possible implementation, a random integer within the interval [a, b] can be generated based on a random algorithm (the generated random integer is the decision value), where a and b are both integers, and b = a + 1. If the generated random integer is a, then it is determined that the initial arrangement order does not need to be adjusted, i.e., the initial arrangement order is maintained, and the initial arrangement order is determined as the target arrangement order. If the generated random integer is b, then it is determined that the initial arrangement order needs to be adjusted, and thus, the result is swapped. A-qi and result B-qi The position, the adjusted arrangement order is result B-qi , result A-qi The adjusted arrangement order is then determined to be the target arrangement order. Alternatively, the initial arrangement order can be adjusted if the random integer is 'a', and kept unchanged if the random integer is 'b'.

[0099] For example, a random integer in the interval [0, 1] can be generated based on a random algorithm. If the generated random integer is 0, it is determined that the initial arrangement order does not need to be adjusted, and the initial arrangement order is determined as the target arrangement order. If the generated random integer is 1, it is determined that the initial arrangement order needs to be adjusted, and the initial arrangement order is adjusted, and the adjusted arrangement order is determined as the target arrangement order.

[0100] The above randomization of the order of the two answers ensures that both answers have a 50% probability of appearing first, fully meeting the fairness requirement. Furthermore, since it only requires one random judgment and at most one swap, the execution efficiency is extremely high.

[0101] Step S202: Based on the order of the targets, formulate the test question q. i Construct an answer comparison prompt.

[0102] For testing question q i The constructed answer comparison prompt includes the test question q. i , Specific evaluation rules r corresponding to the test questions i The model is applicable to the test problem q. i The corresponding answer pair above, the answer comparison prompt is used to provide hints for evaluating the large model based on the test question q. i The corresponding specialized evaluation rules apply to this model in the test problem q. i The corresponding answer prompt evaluates the merits of the two answers. It should be noted that the order of the two answers to be compared in the answer comparison prompt is the target order.

[0103] In this embodiment, the format of the answer comparison prompt can be {"id":"01_model_a<>model_b","task_type":"text generation","language":"xx","question":"q"}} i ", "result_1": "result_a", "result_2": "result_b", "rule": "r i The expression `"query":"xxxx"}` represents the unique identifier of the model pair, `id` is the task type, `language` is the language, `question` is the test question, and `result_1` is the result of one model (model_a) in the model pair for the test question `q`. i The generated answer, result_2, is the other model (model_b) in the model pair for the test question q. i The generated answer uses `rule` as the specific evaluation rule corresponding to the test question and `query` as the hint for the large evaluation model based on the test question `q`. i The corresponding dedicated evaluation rules apply to the model pair (model_a, model_b) in the test question q. i The instruction that evaluates the merits of the two answers contained in (result_a, result_b) in the corresponding answer pair.

[0104] Optionally, the answer comparison prompt may also include test questions (q). i The reference answer can be used as a reference for evaluating the large model, and further, the test question q can be used as a reference. i The reference answer for the model in the test question qi The corresponding answer above evaluates the merits of the two answers contained therein.

[0105] Step S203: Input the answer comparison prompt into the evaluation model to obtain the model's performance on the test question q. i The corresponding answer's merits and demerits are evaluated.

[0106] After comparing the answers with the prompt and inputting it into the evaluation model, the evaluation model bases its evaluation on the test question q. i The corresponding dedicated evaluation rules apply to the model in the test problem q. i The corresponding answer is used to evaluate the merits of the two answers contained therein, and the output model is used to evaluate the performance of the test question q. i The corresponding evaluation results for the answer's merits. These evaluation results indicate which model is best suited for the test question q. i The generated answer is better.

[0107] Considering that evaluation models internalize a general human tendency in their training data—the tendency to associate longer, more detailed text with higher quality, greater effort, and more comprehensive knowledge—this latent, implicit "longer = better" association may still influence their judgments, even if they are instructed to judge fairly. To proactively combat and suppress this potential "length bias" in evaluation models and ensure that judgments focus on the substantive quality of the answers rather than superficial features, this application proposes the following strategies:

[0108] The answer comparison prompt introduces constraint instructions to suppress the evaluation model's bias towards answer length. For example, it might say, "Please focus on the content quality of the answers, strictly adhere to the provided evaluation rules, and do not judge an answer as better or worse simply because it is longer or shorter than another." These constraint instructions are explicit and adversarial, directly requiring the evaluation model not to show bias based on differences in answer length.

[0109] This application introduces constraint instructions in the answer comparison prompt to suppress the evaluation model's bias towards answer length, directly intervening in and guiding the evaluation logic of the evaluation model. This forces the evaluation model's attention to shift from misleading surface features (answer length) back to core quality dimensions related to true value (such as the accuracy, relevance, and logic of the answer), thereby fundamentally improving the fairness, accuracy, and robustness of the evaluation.

[0110] After obtaining the evaluation results of the answers of each model pair on each test question, the ability evaluation results of each large language model are determined based on the evaluation results of the answers of each model pair on each test question. In some embodiments of this application, the process of determining the ability evaluation results of each large language model based on the evaluation results of the answers of each model pair on each test question is described.

[0111] As mentioned in the above embodiments, the test dataset can cover multiple languages. In one possible implementation, the multilingual ability evaluation results of each large language model can be determined based on the evaluation results of the corresponding answers of each model on each test question.

[0112] Specifically, the process of determining the multilingual proficiency evaluation results of each large language model based on the evaluation results of each model's answers to each test question may include:

[0113] Step a1: Based on the evaluation results of each model's answers to each test question, calculate the win rate of each large language model in each language.

[0114] For example, if there are three large language models A, B and C in the large language model set, then combining the three large language models in pairs will yield three model pairs (A, B), (A, C) and (B, C). The test dataset contains four test questions q1, q2, q3 and q4. The test dataset covers Chinese (represented as zh) and English (represented as en). Among them, q1 and q3 are Chinese questions, and q2 and q4 are English questions. Since the evaluation is performed on the three model pairs and the four test questions, the total number of evaluations = 3 × 4 = 12. Accordingly, there are 12 evaluation results for the answer quality, as shown in Table 1 below.

[0115] Table 1. Evaluation results of the three models for each test question in each language.

[0116]

[0117] As shown in the table above, the evaluation results of the three models for the corresponding answers to Chinese questions are: q1(zh): A>B, A>C, B>C; q3(zh): B>A, C>A, B>C. The evaluation results of the three models for the corresponding answers to English questions are: q2(en): B>A, C>A, B>C; q4(en): A>B, A>C, C>B.

[0118] The win rates of each model on Chinese (zh) are calculated (the problems involved are q1 and q3): Each model participates in the evaluation 2×2=4 times on the Chinese problem. Among them, Model A has two wins and two losses, so the win rate of Model A on Chinese (zh) is 2 / 4=50%. Model B has three wins and one loss, so the win rate of Model B on Chinese (zh) is 3 / 4=75%. Model C has one win and three losses, so the win rate of Model C on Chinese (zh) is 1 / 4=25%.

[0119] Calculate the win rate of each model in English (en) (the problems involved are q2 and q4): Each model participates in the evaluation 2×2=4 times in English problems. Among them, model A has two wins and two losses, so the win rate of model A in English (en) is 2 / 4=50%. Model B has two wins and two losses, so the win rate of model B in English (en) is 2 / 4=50%. Model C has two wins and two losses, so the win rate of model C in English (en) is 2 / 4=50%.

[0120] Table 2 below summarizes the win rates of the three major language models across two languages:

[0121] Table 2. Win rates of the three major language models in two languages.

[0122]

[0123] Step a2: Based on the win rate of each large language model in each language, determine the ranking of each large language model in a single language and / or the comprehensive ranking in multiple languages.

[0124] This embodiment determines the ranking of each large language model in each language based on its win rate in each language. The single-language ranking of a model reveals its relative advantage in a specific language environment, which is extremely valuable for targeted user selection or domain applications.

[0125] In the example above, Model A has a win rate of 50% in Chinese, Model B has a win rate of 75% in Chinese, and Model C has a win rate of 25% in Chinese. Therefore, the ranking of the three models in Chinese is: Model B (1st place) > Model A (2nd place) > Model C (3rd place). Model A, Model B, and Model C all have a win rate of 50% in English, so Model A, Model B, and Model C are tied for first place in English.

[0126] This embodiment can also determine the overall win rate of each large language model across multiple languages ​​based on the win rate of each large language model in each language. Furthermore, based on the overall win rate of each large language model across multiple languages, the overall ranking of each large language model across multiple languages ​​can be determined. It should be noted that the overall ranking across multiple languages ​​reflects the overall multilingual capability of the model, which is a key indicator for evaluating the model's versatility and balance.

[0127] For the example above, we can sum the win rates of Model A in Chinese and English using weighted averages to obtain the overall win rate of Model A in both languages. Assuming that the weights for Chinese and English are the same, both being 0.5, then the overall win rate of Model A in both languages ​​is (50% + 50%) / 2 = 50%. Using the same method, we can obtain the overall win rate of Model B in both languages ​​as (75% + 50%) / 2 = 62.5%, and the overall win rate of Model C in both languages ​​as (25% + 50%) / 2 = 37.5%. Based on the overall win rates of the three models in both languages, we can determine the overall ranking of the three models in both languages ​​as: Model B (1st place) > Model A (2nd place) > Model C (3rd place).

[0128] As mentioned in the above embodiments, the test dataset can cover multiple task types. In one possible implementation, the multi-task capability evaluation results of each large language model can be determined based on the evaluation results of the superiority or inferiority of each model's answer to each test question.

[0129] Specifically, the process of determining the multi-task capability evaluation results of each large language model based on the evaluation results of each model's answers to each test question may include:

[0130] Step b1: Based on the evaluation results of each model's answers to each test question, calculate the win rate of each large language model in each task type.

[0131] For example, the large language model set contains three large language models A, B, and C. Combining these three large language models in pairs yields three model pairs: (A, B), (A, C), and (B, C). The test question set contains three test questions q1, q2, and q3. The test questions cover the task types of text generation (TG), logical reasoning (LR), and knowledge answering (KQ). Test question q1 belongs to the text generation task, test question q2 belongs to the logical reasoning task, and test question q3 belongs to the knowledge answering task. Since the evaluation is performed on the three model pairs and the three test questions, the total number of evaluations is 3 × 3 = 9. Correspondingly, there are 9 possible answer quality evaluations, as shown in Table 3 below.

[0132] Table 3. Evaluation results of the three models for each test question in each task type.

[0133]

[0134] As shown in the table above, the evaluation results of the three models for the corresponding answers in the text generation task are: q1(TG): A>B, C>A, B>C; the evaluation results of the three models for the corresponding answers in the logical reasoning task are: q2(LR): B>A, A>C, B>C; and the evaluation results of the three models for the corresponding answers in the knowledge question answering task are: q3(KQ): A>B, A>C, C>B.

[0135] The win rates of each model on the text generation task (the test question involved is q1) are calculated as follows: Each model participates in the evaluation twice on the test question of the text generation task. If model A wins once and loses once, then model A's win rate on the text generation task is 1 / 2 = 50%. If model B wins once and loses once, then model B's win rate on the text generation task is 1 / 2 = 50%. If model C wins once and loses once, then model C's win rate on the text generation task is 1 / 2 = 50%.

[0136] The win rate of each model on the logical reasoning task (the test question involved is q2) is calculated as follows: Each model participates in the evaluation twice on the test question of the logical reasoning task. If model A wins once and loses once, then model A's win rate on the logical reasoning task is 1 / 2 = 50%. If model B wins twice, then model B's win rate on the logical reasoning task is 2 / 2 = 100%. If model C loses twice, then model C's win rate on the logical reasoning task is 0 / 2 = 0%.

[0137] The win rates of each model on the knowledge question answering task (the test question involved is q3) are calculated as follows: Each model participates in the evaluation twice on the test question of the knowledge question answering task. If model A wins twice, then the win rate of model A on the knowledge question answering task is 2 / 2 = 100%. If model B loses twice, then the win rate of model B on the knowledge question answering task is 0 / 2 = 0%. If model C wins once and loses once, then the win rate of model C on the knowledge question answering task is 1 / 2 = 50%.

[0138] Table 4 below summarizes the win rates of the three major language models across the three task types:

[0139] Table 4. Win rates of the three major language models across three task types.

[0140]

[0141] Step b2: Based on the win rate of each large language model in each task type, determine the ranking of each large language model in a single task type and / or the comprehensive ranking in multiple task types.

[0142] This embodiment determines the ranking of each large language model for each task type based on its win rate. The ranking of a model in a single task type reveals its relative strength in that specific task, which is highly valuable for targeted user selection or domain applications.

[0143] In the example above, Model A, Model B, and Model C all have a 50% win rate in the text generation task, so they are tied for first place in this task. Model A has a 50% win rate in the logical reasoning task, Model B has a 100% win rate, and Model C has a 0% win rate. Therefore, the ranking of the three models in the logical reasoning task is: Model B (1st) > Model A (2nd) > Model C (3rd). Model A has a 100% win rate in the knowledge question answering task, Model B has a 0% win rate, and Model C has a 50.0% win rate. Therefore, the ranking of the three models in the knowledge question answering task is: Model A (1st) > Model C (2nd) > Model B (3rd). Thus, Model A is stronger in the knowledge question answering task, Model B is stronger in the logical reasoning task, and Model C has no outstanding performance in any task, showing no obvious characteristics.

[0144] This embodiment can also determine the overall win rate of each large language model across multiple task types based on the win rate of each large language model in each task type. Furthermore, based on the overall win rate of each large language model across multiple task types, the overall ranking of each large language model across multiple task types can be determined. It should be noted that the overall ranking of the large language model across multiple task types reflects the overall multi-task capability of the large language model.

[0145] For the example above, we can weight and sum the win rates of Model A in the text generation task, the logical reasoning task, and the knowledge question answering task to obtain the overall win rate of Model A in the three task types. Assuming that the weights for text generation, logical reasoning, and knowledge question answering tasks are all 1 / 3, then the overall win rate of Model A in the three task types is (50%+50%+100%) / 3=66.7%. Using the same method, we can obtain the overall win rate of Model B in the three task types as (50%+100%+0%) / 3=50%, and the overall win rate of Model C in the three task types as (50%+0%+50%) / 3=33.3%. Based on the overall win rates of the three models in the three task types, we can determine the overall ranking of the three models in the three task types as: Model A (1st place) > Model B (2nd place) > Model C (3rd place).

[0146] As the training methods for large language models are continuously optimized, the model iteration cycle is getting shorter and shorter. As a result, the release of models by various vendors is getting faster and faster. Therefore, the time for each round of model evaluation is very short. It is almost impossible to conduct a full evaluation in each iteration (full evaluation means that all new and old models regenerate answers on all questions, re-compare the answers pairwise, re-calculate the win rate, and re-rank). In view of this, this application proposes a non-full evaluation method.

[0147] Specifically, when a large language model in the large language model set is updated (the updated large language model is referred to as the first target model), the following non-full evaluation process is performed:

[0148] Obtain the answers generated by the updated model of the first target model for each test question; update the first target model in the model pair containing the first target model in the original model pair to the updated model of the first target model, to obtain the updated model pair; for each test question, combine the answers generated by the two large language models contained in the same updated model pair for the test question, to obtain the answer pair corresponding to each updated model pair for the test question; for each test question, use the evaluation large model and according to the special evaluation rules corresponding to the test question to evaluate the merits of the two answers contained in the answer pair corresponding to each updated model pair for the test question; based on the merit evaluation results of the answers corresponding to each updated model pair for each test question, and the merit evaluation results of the answers corresponding to each unupdated model pair for each test question, determine the capability evaluation results of the updated model of the first target model and each unupdated large language model.

[0149] For example, a large language model set contains three large language models A, B, and C. A, B, and C have been evaluated. Now, model A is updated, and the updated model is A'. The following steps are taken to obtain the answers generated by model A' for each test question in the test dataset: the original model pairs (A, B) and (A, C) are updated to (A', B) and (A', C). For each test question, the answers generated by the two large language models contained in the model pair (A', B) for that test question are combined to obtain the answer pair corresponding to the updated model pair (A', B) for that test question. This process yields the answer pair corresponding to the updated model pair (A', B) for each test question in the test dataset. Similarly, the answer pair corresponding to the updated model pair (A', C) for each test question in the test dataset can be obtained in the same way. For each test question, using the evaluated large model and according to the specific evaluation rules corresponding to that test question, the merits of the two answers contained in the answer pair corresponding to the updated model pair (A', B) for that test question are evaluated, and the updated model pair (A', B) is also evaluated. C) Based on the merits of the two answers in the corresponding answer pair for the test question, re-evaluate the merits of the answers (A', B) for each test question, the merits of the answers (A', C) for each test question, and the merits of the answers (B, C) for each test question, determine the win rates of the three large language models A', B, and C, and rank A', B, and C according to their win rates.

[0150] In practical applications, there may be situations where new large language models are added to the evaluation (the new models added to the evaluation are referred to as the second target model). To address this situation, this application also proposes a non-full evaluation method. The non-full evaluation process in this case includes:

[0151] Obtain the answers generated by the second target model for each test question; combine the second target model with each existing large language model to obtain multiple new model pairs; for each test question in the test dataset, combine the answers generated by the two large language models in the same new model pair for that test question to obtain the corresponding answer pair for each new model pair on that test question; for each test question, use the evaluation large model and, according to the specific evaluation rules corresponding to that test question, evaluate the merits of the two answers contained in the corresponding answer pair of each new model pair on that test question; based on the merit evaluation results of the answers corresponding to each new model pair on each test question, and the merit evaluation results of the answers corresponding to each existing model pair on each test question, determine the capability evaluation results of all large language models, including the first target model.

[0152] For example, a large language model set contains three large language models A, B, and C. A, B, and C have already been evaluated. A new model D is to be added to the evaluation. The process involves obtaining the answers generated by model D for each test question in the test dataset. Model D is then combined with the existing models A, B, and C to obtain three new model pairs: (D, A), (D, B), and (D, C). For each test question in the test dataset, the answers generated by the two large language models in the new model pair (D, A) for that test question are combined to obtain the answer pair corresponding to (D, A) for that test question. Similarly, the answer pairs corresponding to the new model pair (D, B) and the new model pair (D, C) for each test question in the test dataset can be obtained. Next, for each test question, the evaluation large model is used to evaluate the merits of the two answers in the answer pair corresponding to the new model pair (D, A) for that test question, according to the specific evaluation rules for that test question. B) Evaluate the merits of the two answers in the corresponding answer pair for the test question and the merits of the two answers in the corresponding answer pair for the test question (D, C). Then, based on the evaluation results of the original model pairs (A, B), (A, C), (B, C) and the newly added model pairs (D, A), (D, B), (D, C) for each test question in the test dataset, determine the win rates of the large language models A, B, C, and D, and rank A, B, C, and D according to their win rates.

[0153] In practical applications, there may be situations where a particular large language model in a large language model set is obsolete (the obsolete large language model is referred to as the third target model). In such cases, the following process can be performed:

[0154] Based on the evaluation results of the answers to each test question for each target model pair (model pairs that do not include obsolete large language models in each original model pair), the capability evaluation results of the major language models that are not obsolete in the large language model set are re-determined.

[0155] In practical applications, data such as model pairs during the evaluation process and the evaluation results of the answers for each model pair on each test question are usually stored. When the third target model is discarded, the model pairs containing the third target model can be deleted, and the evaluation results of the answers for each test question containing the third target model can also be deleted. Then, based on the remaining evaluation results, the win rates of all models other than the third target model are recalculated, and the models other than the third target model are ranked according to their win rates.

[0156] For example, there are four large language models in the large language model set: A, B, C, and D. A, B, C, and D have been evaluated. Now, model D is discarded. Then, the win rates of large language models A, B, and C are determined again based on the evaluation results of the model's answers to (A, B), (A, C), and (B, C) on each test question. A, B, and C are then ranked according to their win rates.

[0157] This application also provides a large language model capability evaluation system, such as... Figure 3 As shown, the large language model ability assessment system may include: an answer acquisition unit 301, an evaluation rule generation unit 302, an answer combination unit 303, an answer pair evaluation unit 304, and an ability assessment result determination unit 305.

[0158] Answer acquisition unit 301 is used to acquire the answer generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset.

[0159] The evaluation rule generation unit 302 is used to generate a large model using rules, generate a special evaluation rule for evaluating the quality of the answer for each test question, and obtain the special evaluation rule corresponding to each test question.

[0160] Answer combination unit 303 is used to combine the various large language models in pairs. For each test question, it combines the answers generated by the two large language models in the same model pair for the test question to obtain the answer pair corresponding to each model pair for the test question.

[0161] The answer pair evaluation unit 304 is used to evaluate the merits of the two answers contained in the answer pair corresponding to each model pair for each test question, based on the dedicated evaluation rules corresponding to the test question.

[0162] The ability assessment result determination unit 305 is used to determine the ability assessment results of each large language model based on the evaluation results of the corresponding answers of each model on each test question.

[0163] In one possible implementation, the test dataset covers multiple languages ​​and multiple task types, with each test question in the test dataset belonging to one of the multiple task types and expressed in one of the multiple languages.

[0164] In one possible implementation, the evaluation rule generation unit 302 uses a rule generation model to generate specific evaluation rules for evaluating the quality of answers for each test question. The process of obtaining specific evaluation rules for each test question may include:

[0165] For each test question, a rule generation prompt is constructed for the test question. The rule generation prompt contains the test question and instructions to generate specific evaluation rules for the test question to judge the quality of the answer.

[0166] The rule generation prompt is used as input to generate a large model, which in turn produces dedicated evaluation rules for the test questions.

[0167] In one possible implementation, the process by which the answer acquisition unit 301 acquires the answers generated by each large language model in the large language model set to be evaluated for each test question in the test dataset may include:

[0168] For each test question in the test dataset, a response test prompt is constructed for the test question. The response test prompt contains the test question and constraint instructions that limit the language of the output answer. The response test prompt instructs the test question to be answered in the language specified by the constraint instructions.

[0169] The response test prompt is input into each of the large language models in the set of large language models to be evaluated, and the answer generated by each large language model for the test question is obtained.

[0170] In one possible implementation, the process by which the answer pair evaluation unit 304 evaluates the merits of the two answers contained in the answer pair corresponding to each model pair on the test question, based on the specific evaluation rules corresponding to the test question, may include:

[0171] For each model pair, the order of the two answers in the corresponding answer pair on the test question is randomized to obtain the target order.

[0172] Based on the target order, an answer comparison prompt is constructed. The answer comparison prompt includes the test question, the specific evaluation rules corresponding to the test question, and the answer pair corresponding to the model pair on the test question. The order of the two answers to be compared in the answer comparison prompt is the target order.

[0173] The answers are compared with the prompt and input into the evaluation model to obtain the evaluation results of the model pair on the test question.

[0174] In one possible implementation, the process by which the answer pair evaluation unit 304 randomizes the order of the two answers in the answer pair corresponding to the test question to obtain the target order may include:

[0175] Set the initial order of the two answers in the answer pair corresponding to the test question for the model pair;

[0176] Based on a random algorithm, decision values ​​are generated to determine whether to adjust the initial arrangement order;

[0177] The order of the targets is determined based on the decision values.

[0178] In one possible implementation, the answer comparison prompt includes constraint instructions to suppress the evaluation model's bias towards answer length.

[0179] In one possible implementation, the test dataset covers multiple languages, and the process by which the competency assessment result determination unit 305 determines the competency assessment results of each large language model based on the evaluation results of each model's answers to each test question can include:

[0180] Based on the evaluation results of each model's answers to each test question, the win rate of each large language model in each language is calculated.

[0181] Based on the win rate of each large language model in each language, determine the ranking of each large language model in a single language and / or the overall ranking in multiple languages.

[0182] In one possible implementation, the test dataset covers multiple task types, and the process by which the capability evaluation result determination unit 305 determines the capability evaluation results of each large language model based on the evaluation results of each model's answers to each test question can include:

[0183] Based on the evaluation results of each model's answers to each test question, the win rate of each large language model in each task type is calculated.

[0184] Based on the win rate of each large language model in each task type, determine the ranking of each large language model in a single task type and / or the overall ranking in multiple task types.

[0185] In one possible implementation, the answer acquisition unit 301 is also used to acquire the answer generated by the updated model of the first target model for each test question, wherein the first target model is a large language model that has been updated in the large language model set.

[0186] The answer combination unit 303 is also used to update the first target model in the model pair containing the first target model in the original model pair to the updated model of the first target model, so as to obtain an updated model pair. For each test question, the answers generated by the two large language models contained in the same updated model pair for the test question are combined to obtain the answer pair corresponding to each updated model pair for the test question.

[0187] The answer pair evaluation unit 304 is also used to evaluate the merits of the two answers contained in the answer pair corresponding to each updated model pair on the test question, based on the dedicated evaluation rules corresponding to the test question.

[0188] The ability assessment result determination unit 305 is also used to determine the ability assessment results of the updated model of the first target model and each unupdated large language model based on the evaluation results of the updated model's answers to each test question and the evaluation results of the unupdated model's answers to each test question.

[0189] In one possible implementation, the answer acquisition unit 301 is also used to acquire the answer generated by the second target model for each test question, wherein the second target model is a newly added large language model to be evaluated.

[0190] The answer combination unit 303 is also used to combine the second target model with each existing large language model to obtain multiple new model pairs. For each test question, the answers generated by the two large language models contained in the same new model pair for the test question are combined to obtain the answer pair corresponding to each new model pair for the test question.

[0191] The answer pair evaluation unit 304 is also used to evaluate the merits of the two answers contained in the answer pair corresponding to each new model pair on the test question, based on the dedicated evaluation rules corresponding to the test question.

[0192] The ability assessment result determination unit 305 is also used to determine the ability assessment results of all large language models, including the second target model, based on the evaluation results of the answers of each new model to each test question and the evaluation results of the answers of each original model to each test question.

[0193] In one possible implementation, the capability assessment result determination unit 305 is also used to re-determine the capability assessment results of the major language models that are not abandoned in the large language model set when a major language model in the large language model set is abandoned, based on the evaluation results of the superiority or inferiority of the answers corresponding to each target model pair on each test question. Here, the target model pair is the model pair that does not contain the abandoned major language model in each of the original model pairs.

[0194] The large language model capability evaluation system provided in this application embodiment can automatically evaluate the capabilities of multiple large language models (such as multilingual capabilities), eliminating the reliance on a large number of highly skilled human evaluators and significantly shortening the evaluation time. Because the large language model capability evaluation system provided in this application embodiment dynamically generates specific evaluation rules for each test question, all answer pairs for the same test question are evaluated using the same evaluation rules. Therefore, it avoids standard drift by the same evaluator at different times or between different evaluators, thereby greatly improving the reliability and comparability of the evaluation results. Furthermore, the large language model capability system provided in this application embodiment automates both the "standard setting" (evaluation rule generation) and "standard execution" (comparing and evaluating answer pairs according to the evaluation rules) of the evaluation, and introduces a non-full-scale evaluation method, making large-scale, high-frequency model capability evaluation possible.

[0195] This application also provides an electronic device, which may include: at least one processor and a memory connected to the processor.

[0196] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; the memory may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0197] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the large language model capability evaluation method provided in the above embodiments.

[0198] This application also provides a computer storage medium that carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device is able to implement the large language model capability evaluation method provided in the above embodiments.

[0199] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the large language model capability evaluation method provided in the above embodiments.

[0200] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0202] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0203] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for evaluating the capabilities of a large language model, characterized in that, include: Obtain the answers generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset; A large model is generated using rules to produce a dedicated evaluation rule for judging the quality of the answer for each test question, thus obtaining the dedicated evaluation rule corresponding to each test question; Each of the large language models is paired up, and for each test question, the answers generated by the two large language models in the same model pair for the test question are combined to obtain the answer pair corresponding to each model pair for the test question. For each test question, the merits of the two answers in the corresponding answer pair of each model are evaluated according to the specific evaluation rules for the test question. Based on the evaluation results of the superiority or inferiority of the answers of each model to each test question, the capability evaluation results of each of the large language models are determined. The step of generating a large model using rules generates specific evaluation rules for each test question to assess the quality of the answer, resulting in specific evaluation rules for each test question, including: For each of the test questions, a rule generation prompt is constructed for the test question. The rule generation prompt includes the test question and instructions to generate specific evaluation rules for the test question to judge the quality of the answer. The rule-generated prompt is used to generate a large model, and the dedicated evaluation rule corresponding to the test question is output by the rule-generated large model.

2. The method for evaluating the ability of a large language model according to claim 1, characterized in that, The test dataset covers multiple languages ​​and multiple task types; Each test question in the test dataset belongs to one of the multiple task types and is expressed in one of the multiple languages.

3. The method for evaluating the ability of a large language model according to claim 1, characterized in that, The process of obtaining the answers generated by each large language model in the set of large language models to be evaluated for each test question in the test dataset includes: For each test question in the test dataset, a response test prompt is constructed for the test question. The response test prompt includes the test question and a constraint instruction that limits the language of the output answer. The response test prompt instructs the test question to be answered in the language limited by the constraint instruction. The response test prompt is input into each of the large language models in the set of large language models to be evaluated, and the answer generated by each of the large language models for the test question is obtained.

4. The method for evaluating the ability of a large language model according to claim 1, characterized in that, The evaluation of the merits of each model's answer pair for the test question, based on the specific evaluation rules corresponding to the test question, includes: For each model pair, the order of the two answers in the answer pair corresponding to the model pair on the test question is randomized to obtain the target order. Based on the target arrangement order, an answer comparison prompt is constructed, wherein the answer comparison prompt includes the test question, the dedicated evaluation rule corresponding to the test question, and the answer pair corresponding to the model on the test question, and the arrangement order of the two answers to be compared in the answer comparison prompt is the target arrangement order; The answer is compared with the prompt and input into the evaluation model to obtain the evaluation result of the model on the test question.

5. The method for evaluating the ability of a large language model according to claim 4, characterized in that, The step of randomizing the order of the two answers in the answer pair corresponding to the test question in the model to obtain the target order includes: Set the initial order of the two answers in the answer pair corresponding to the test question in the model; A decision value is generated based on a random algorithm to determine whether to adjust the initial arrangement order; Based on the decision values, determine the order in which the targets are arranged.

6. The method for evaluating the ability of a large language model according to claim 4, characterized in that, The answer comparison prompt includes constraint instructions to suppress the evaluation model's bias towards answer length.

7. The method for evaluating the ability of a large language model according to claim 1, characterized in that, The test dataset covers multiple languages; The step of determining the capability evaluation results of each of the large language models based on the evaluation results of the answers corresponding to each of the test questions by each model includes: Based on the evaluation results of the model's answers to each test question, the win rate of each large language model in each language is calculated. Based on the win rate of each of the large language models in each of the languages, determine the ranking of each of the large language models in a single language and / or the overall ranking in the multiple languages.

8. The method for evaluating the ability of a large language model according to claim 1, characterized in that, The test dataset covers a variety of task types; The step of determining the capability evaluation results of each of the large language models based on the evaluation results of the answers corresponding to each of the test questions by each model includes: Based on the evaluation results of the model's answers to each test question, the win rate of each large language model in each task type is calculated. Based on the win rate of each of the large language models in each of the task types, determine the ranking of each of the large language models in a single task type and / or the overall ranking in the multiple task types.

9. The method for evaluating the capabilities of a large language model according to any one of claims 1 to 8, characterized in that, Also includes: Obtain the updated model of the first target model for each test question, wherein the first target model is a large language model in the large language model set that has been updated; The first target model in the model pair that contains the first target model in the original model pair is updated to the updated model of the first target model, thus obtaining the updated model pair; For each test question, the answers generated by the two large language models contained in the same update model pair for the test question are combined to obtain the answer pair corresponding to each update model pair for the test question; For each test question, the merits of the two answers in the corresponding answer pair of each updated model are evaluated according to the specific evaluation rules for the test question. Based on the evaluation results of the updated model's answers to each test question and the evaluation results of the unupdated model's answers to each test question, the ability evaluation results of the updated model of the first target model and each unupdated large language model are determined.

10. The method for evaluating the ability of a large language model according to any one of claims 1 to 8, characterized in that, Also includes: Obtain the answer generated by the second target model for each of the test questions, wherein the second target model is a newly added large language model to be evaluated; The second target model is combined with each of the original large language models to obtain multiple new model pairs; For each test question, the answers generated by the two large language models contained in the same new model pair for the test question are combined to obtain the answer pair corresponding to each new model pair for the test question; For each test question, the merits of the two answers in the corresponding answer pair of each new model are evaluated according to the specific evaluation rules for the test question. Based on the evaluation results of the answers of each newly added model to each test question, and the evaluation results of the answers of each existing model to each test question, the capability evaluation results of all large language models, including the second target model, are determined.

11. The method for evaluating the capabilities of a large language model according to any one of claims 1 to 8, characterized in that, Also includes: When a major language model in the large language model set is discarded, the ability evaluation results of the major language models in the large language model set that are not discarded are re-determined based on the evaluation results of the corresponding answers of each target model pair on each test question. The target model pair is the model pair in each original model pair that does not contain the discarded major language model.

12. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the large language model capability evaluation method as described in any one of claims 1 to 11.

13. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the large language model capability evaluation method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the large language model capability assessment method as described in any one of claims 1 to 11.

Images