Model evaluation method and device

Abstract

The embodiment of the invention provides a model evaluation method and device, and the method comprises the steps: obtaining at least one evaluation data pair, and enabling the evaluation data pair to comprise a to-be-evaluated question and a to-be-evaluated answer obtained based on a to-be-evaluated model; the depolarization score and the preference score of the at least one evaluation data pair are determined, the depolarization score is used for indicating whether the to-be-evaluated answer in the corresponding evaluation data pair answers the subject and demand question of the to-be-evaluated question, and the preference score is used for indicating the score of the to-be-evaluated answer in the corresponding evaluation data pair under the preference scoring rule; determining a target score of the at least one evaluation data pair according to the depolarization score and the preference score of the at least one evaluation data pair; and determining an evaluation result of the to-be-evaluated model according to the target score of the at least one evaluation data pair. The technical performance of the to-be-assessed model is considered, the performance of the to-be-assessed model in the aspect of meeting the user preference is also considered, and the to-be-assessed model is assessed more comprehensively, more accurately and more flexibly.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and more particularly to a model evaluation method and device. Background Art

[0002] With the rapid development of computer technology, machine learning, and deep learning technology, machine learning models have been widely used in many fields such as natural language processing, machine translation, speech synthesis, and image generation. Large language models have been born accordingly. Large language models have been proven to have excellent performance in understanding and generating natural language texts. Large language model technology has developed rapidly, and it has shown strong capabilities in instruction-following, problem-solving, and open-ended chatting, which has brought greater challenges to how to evaluate the processing capabilities of such large language models.

[0003] In the prior art, the evaluation questions and the answers of the large language model are generally directly input into the evaluation model to obtain the correctness evaluation results of the large language model. The correctness of the answers of the large language model is only evaluated, which cannot adapt to different application scenarios. The evaluation results are not comprehensive, and the accuracy and flexibility are poor. Therefore, a more comprehensive, accurate and flexible model evaluation solution is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a model evaluation method. One or more embodiments of this specification also relate to a model evaluation device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a model evaluation method is provided, comprising: Acquire at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on a model to be evaluated; Determine a debiased score and a preference score of the at least one evaluation data pair, wherein the debiased score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the question to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule; determining a target score for the at least one evaluation data pair based on the debiased score and the preference score for the at least one evaluation data pair; An evaluation result of the model to be evaluated is determined according to the target score of the at least one evaluation data pair.

[0006] According to a second aspect of an embodiment of this specification, a model evaluation device is provided, comprising: An acquisition module is configured to acquire at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on a model to be evaluated; A first determination module is configured to determine a debiasing score and a preference score of the at least one evaluation data pair, wherein the debiasing score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the question to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule; A second determination module is configured to determine a target score of the at least one evaluation data pair based on the debiased score and the preference score of the at least one evaluation data pair; The third determination module is configured to determine the evaluation result of the model to be evaluated according to the target score of the at least one evaluation data pair.

[0007] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned model evaluation method are implemented.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned model evaluation method are implemented.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above-mentioned model evaluation method when executed by a processor.

[0010] The embodiments of this specification provide a model evaluation method, which realizes determining the preference score of the answer to be evaluated in at least one evaluation data pair under the preference scoring rule, superimposing the debiased score of at least one evaluation data pair, indicating whether the answer to be evaluated in the evaluation data pair answers the main question of the question to be evaluated, so as to obtain the evaluation result of the model to be evaluated. In this way, human preferences are defined and quantified, and the corresponding evaluation results can be automatically output for the preferences, which not only considers the technical performance of the model to be evaluated, but also considers the performance of the model to be evaluated in meeting user needs and preferences, and conducts a more comprehensive, accurate and flexible evaluation of the model to be evaluated, thereby improving the evaluation accuracy of the model to be evaluated, thereby accelerating the iteration of the effect of the model to be evaluated and the research and development of products. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of a model evaluation method provided by one embodiment of this specification; Figure 2 is a schematic diagram of a processing process of model evaluation provided by an embodiment of this specification; Figure 3 is a process flow chart of a model evaluation method provided by an embodiment of this specification; Figure 4 is a structural schematic diagram of a model evaluation device provided by an embodiment of this specification; Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0012] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0013] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0015] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] First, the terms involved in one or more embodiments of this specification are explained.

[0017] Large Language Models (LLMS): refers to a type of language processing model with a large number of parameters. These models are trained through deep learning technology and can show excellent performance in a variety of natural language processing tasks. In recent years, with the growth of computing resources and the advancement of algorithms, large language models have become one of the important breakthroughs in the field of natural language processing. Large language models usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. Large language models can also be called cornerstone models / foundation models. Large language models are pre-trained using large-scale unlabeled corpora to produce pre-trained models with more than 100 million parameters. Such models can adapt to a wide range of downstream tasks and have good generalization capabilities. For example, large-scale language models (LLM) and multi-modal pre-training models have shown strong capabilities in instruction-following, problem-solving and open-ended chatting, and can be applied to many fields such as natural language processing, machine translation, speech synthesis, image generation, etc.

[0018] Human preference: refers to the inclination or preference that an individual or group shows for a specific thing, activity, attribute or other option. This preference can be broad, such as a preference for a certain answer style, or specific, such as the preference for a certain brand. Human preferences are formed by a combination of factors, including but not limited to cultural background, personal experience, social influence and personal characteristics.

[0019] It should be noted that with the rapid development of artificial intelligence technology, large language models have played an important role in a variety of application scenarios. However, there are still many challenges in evaluating large language models, especially from the perspective of multiple human preferences. For example, in the application scenario of news writing, large language models can generate coherent and accurate news articles. From the perspective of multiple human preferences, different readers may have different preferences for the style, content and opinions of news. Therefore, evaluating the performance of large language models in this scenario becomes complicated. For example, a news article may be well received by some readers, but controversial among other readers. In the application scenario of customer service chatbots, large language models can provide accurate and timely customer service. However, different customers may have different expectations for the quality, speed and attitude of the service. Therefore, evaluating the performance of large language models in this scenario also needs to take into account these multiple human preferences. For example, some customers may prefer quick and direct answers, while others may prefer detailed and patient answers. In the application scenario of educational tutoring, large language models can provide personalized learning suggestions and feedback. However, different students may have different preferences for teaching methods, speed, and difficulty. Some students may prefer detailed explanations, while others prefer concise answers. Therefore, evaluating the performance of large language models in this scenario also needs to take into account these diverse human preferences.

[0020] From the above, it can be seen that from the perspective of multiple human preferences, evaluating the processing capabilities of a large language model is a complex task. It is necessary to consider not only the technical performance of the model, but also the performance of the model in meeting different user needs and preferences. Therefore, the embodiments of this specification provide a model evaluation method and device to define and quantify multiple human preferences, and abstract an evaluation paradigm that can automatically produce corresponding evaluation results for people with different preferences, so as to better evaluate the performance of large language models in different application scenarios.

[0021] In this specification, a model evaluation method is provided. This specification also relates to a model evaluation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0022] See also Figure 1 , Figure 1 A flow chart of a model evaluation method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0023] Step 102: Obtain at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on a model to be evaluated.

[0024] Among them, the model to be evaluated can be a large language model in various application scenarios. For example, in the application scenario of news writing, the model to be evaluated is a large language model that can generate coherent and accurate news releases; in the application scenario of customer service chatbots, the model to be evaluated is a large language model that provides accurate and timely customer service; in the application scenario of educational tutoring, the model to be evaluated is a large language model that provides personalized learning suggestions and feedback.

[0025] In actual implementation, at least one question to be evaluated can be obtained, and each question to be evaluated can be input into the model to be evaluated to obtain the answer to be evaluated corresponding to each question to be evaluated. The answer to be evaluated is the processing result given by the model to be evaluated for the question to be evaluated. The answer to be evaluated represents the analysis and processing capability of the model to be evaluated for the question to be evaluated. Therefore, a question to be evaluated and a corresponding answer to be evaluated can be used as an evaluation data pair for subsequent evaluation of the analysis and processing capability of the model to be evaluated.

[0026] In a specific implementation, at least one question to be evaluated may be obtained from a local database, a third-party platform and / or the Internet, and the at least one question to be evaluated may be a question in the same or different application scenarios.

[0027] As an example, m questions to be evaluated can be obtained, and the first question to be evaluated q1 is input into the model to be evaluated, and the answer to be evaluated a1 corresponding to the first question to be evaluated q1 can be obtained, and (q1, a1) is used as the first evaluation data pair; then, the second question to be evaluated q2 is input into the model to be evaluated, and the answer to be evaluated a2 corresponding to the second question to be evaluated q2 can be obtained, and (q2, a2) is used as the second evaluation data pair; ...; the mth question to be evaluated qm is input into the model to be evaluated, and the answer to be evaluated am corresponding to the mth question to be evaluated qm can be obtained, and (qm, am) is used as the mth evaluation data pair. Thus, the at least one evaluation data pair obtained can constitute the evaluation set F={(q1, a1), (q2, a2), ..., (qm, am)}.

[0028] Step 104: Determine a debiased score and a preference score for at least one evaluation data pair, wherein the debiased score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the topic to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule.

[0029] Among them, the main need problem usually refers to the core and most critical part of the question, that is, the main need or goal of solving this question. The preference scoring rule refers to the scoring rule corresponding to the needs or preferences of a specific group of people. For example, the preference scoring rule can be that the richer the answer, the higher the score; the neater the answer format, the higher the score; the more citations are given to the answer, the higher the score, etc.

[0030] It should be noted that the debiased score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the topic to be evaluated, and does not include human preference, that is, the objective scoring result; while the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule, that is, the subjective scoring result under human preference.

[0031] As an example, suppose the question to be evaluated is "Please analyze the ages of Li Bai and Du Fu", the main question of the question to be evaluated is "You need to answer that Li Bai is 11 years older than Du Fu", and other information such as the styles and evaluations of Li Bai and Du Fu are auxiliary information; suppose the question to be evaluated is "Given a right triangle ABC, where angle C=90°, AC=3cm, BC=4cm, find the length of the hypotenuse AB, and calculate the area of ​​the triangle", the main question is "Find the length of the hypotenuse AB and the area of ​​the triangle ABC", and the properties of the right triangle, the given angles and the lengths of the two sides are auxiliary information; suppose the question to be evaluated is "Design a science experiment course plan suitable for junior high school students, aiming to improve students' hands-on ability and scientific interest", the main question is "Design a science experiment course plan", and the students' age characteristics, existing scientific knowledge level, available teaching resources, etc. are auxiliary information.

[0032] In actual implementation, the evaluation data pair can be scored based on whether the answer to be evaluated in the evaluation data pair answers the main question of the topic to be evaluated, so as to obtain a debiased score for the evaluation data pair, considering only the technical performance of the model but not human preferences. Moreover, the answer to be evaluated in the evaluation data pair can be scored based on the preference scoring rules to obtain a preference score, so as to indicate whether the answer to be evaluated in the evaluation data pair meets human needs or preferences, so as to subsequently combine the debiased score and the preference score to jointly evaluate the model to be evaluated, thereby improving the comprehensiveness, accuracy and flexibility of the evaluation.

[0033] In an optional implementation of this embodiment, determining a debiasing score and a preference score of at least one evaluation data pair includes: Inputting at least one evaluation data pair into a debiased scoring model to obtain a debiased score for at least one evaluation data pair; Based on the preference scoring rules, a preference score of at least one evaluation data pair is respectively determined.

[0034] In actual implementation, a debiased scoring model can be pre-trained, and at least one evaluation data pair can be input into the debiased scoring model to obtain the debiased score of at least one evaluation data pair. The debiased scoring model can be obtained through the following training process: obtaining sample questions, sample answers, and corresponding debiased scoring labels, wherein the debiased scoring label is a score that is manually annotated considering only whether the sample answer answers the main question of the sample question; inputting the sample questions and sample answers into the initial debiased scoring model to obtain a predicted debiased score, wherein the initial debiased scoring model can be a pre-trained large language model or other pre-trained scoring model; calculating the model loss value based on the debiased scoring label and the predicted debiased score, adjusting the parameters of the initial debiased scoring model in reverse according to the model loss value, and returning to execute the operation steps of obtaining the sample questions, sample answers, and corresponding debiased scoring labels until the training stop condition is reached to obtain a trained debiased scoring model.

[0035] In addition, the preference score of at least one evaluation data pair can be determined based on the pre-configured preference scoring rules. For example, assuming that the preference scoring rule is that the richer the answer, the higher the score, then the richer the content of the answer to be evaluated of the evaluation data pair or the more characters there are, the higher the preference score. For example, if the question to be evaluated in the evaluation data pair is "Please analyze the ages of Li Bai and Du Fu", if the answer to be evaluated is "Li Bai (701-762), Du Fu (712-770), Li Bai is 11 years older than Du Fu", the preference score can be 7 points; if the answer to be evaluated is "Li Bai (701-762), Du Fu (712-770), Li Bai is 11 years older than Du Fu, Li Bai and Du Fu are famous poets, and both have a high status in literary achievements and influence", the preference score can be 8 points.

[0036] It should be noted that there can be multiple preference scoring rules, that is, different preference types can correspond to different preference scoring rules. The preference scoring can indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rules corresponding to different preference types, so as to indicate the pros and cons of the answer to be evaluated under different preference types.

[0037] In the embodiments of the present specification, the debiased scoring model obtained by pre-training can be used to determine the debiased scoring of at least one evaluation data pair, and only whether the question or answer is a main need is considered for scoring, without considering human preferences, and then the preference scoring is performed based on the preference scoring rules, considering whether the answer to be evaluated meets human needs or preferences. Using the debiased scoring model and preference scoring rules obtained by pre-training, the debiased score and preference score can be obtained quickly and accurately, respectively, and the determination process is simple, without the need for complex calculation operations, thereby improving the evaluation efficiency.

[0038] Of course, in practical applications, debiasing scoring rules can also be predefined, and then at least one evaluation data pair can be scored based on the debiasing scoring rules to obtain a debiased score; in addition, a corresponding preference scoring model can be pre-trained based on preference scoring rules, and the preference score of at least one evaluation data pair can be determined based on the preference scoring model. Alternatively, the debiasing score and preference score can also be obtained using a programming language or other scoring algorithms, and the embodiments of this specification are not limited to this.

[0039] In an optional implementation of this embodiment, based on the preference scoring rule, determining the preference score of at least one evaluation data pair respectively includes: determining a first evaluation data pair from the at least one evaluation data pair; Scoring the first evaluation data pair based on the preference scoring rules corresponding to each preference type, respectively, to obtain the type preference score of the first evaluation data pair under each preference type; Determining a preference score for the first evaluation data pair based on the type preference scores of the first evaluation data pair under each preference type; Continue to determine the first evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a preference score for each evaluation data pair.

[0040] Among them, preference type refers to the dimension corresponding to the demand or preference. For example, preference type can include illusion, redundancy, format, security, etc. Each preference type can be defined with a corresponding preference scoring rule.

[0041] In the field of model evaluation, hallucination refers to the situation where the answer to be evaluated generated by the model to be evaluated does not match the provided context or constraints, that is, hallucination refers to the situation where the model to be evaluated generates content that contradicts the input information, facts or logic when generating the answer to be evaluated. Redundancy usually refers to unnecessary repetition in the answer to be evaluated generated by the model to be evaluated. This repetition may be repetition of vocabulary, phrases or entire sentences, or even similar or identical information in different paragraphs or documents. Format usually refers to the structured form or standard style of the answer to be evaluated output by the model to be evaluated, ensuring that the content generated by the model to be evaluated meets specific requirements and is convenient for subsequent processing, parsing or display. Security can usually include multiple aspects, aiming to ensure that the application of the model to be evaluated in different scenarios will not have a negative impact on users, systems or other aspects, such as whether it contains sensitive content.

[0042] In actual implementation, any one of the at least one evaluation data pair can be determined as the first evaluation data pair, and then the first evaluation data pair can be scored based on the preference scoring rules corresponding to each preference type to obtain the type preference score of the first evaluation data pair under each preference type, and then the preference score of the first evaluation data pair can be determined based on the type preference score of the first evaluation data pair under each preference type. After that, the next evaluation data pair in the at least one evaluation data pair can be scored until the preference scores of each evaluation data pair are obtained.

[0043] For example, suppose n preference types are configured and corresponding preference scoring rules are defined respectively. For the i-th evaluation data pair (qi, ai), the preference score obtained under the preference scoring rule corresponding to the first preference type is A1, the preference score obtained under the preference scoring rule corresponding to the second preference type is A2, and so on. The preference score obtained under the preference scoring rule corresponding to the n-th preference type is A1. n Based on the preference scores A1, A2, ..., A n The preference score Score_ai corresponding to the i-th evaluation data pair (qi, ai) can be determined.

[0044] In the embodiments of the present specification, the first evaluation data pair can be scored based on the preference scoring rules corresponding to each preference type, and the type preference scores under each preference type can be obtained. The preference scores of the first evaluation data pair are then determined in combination with the type preference scores under each preference type. This defines and quantifies multiple human preferences, and can automatically output corresponding preference evaluation results for multiple preferences. This takes into account the performance of the model to be evaluated in meeting the different needs and preferences of users, thereby facilitating a more comprehensive, accurate and flexible evaluation of the model to be evaluated in the future.

[0045] In an optional implementation of this embodiment, determining the preference score of the first evaluation data pair based on the type preference score of the first evaluation data pair under each preference type includes: Get the type weight corresponding to each preference type; Based on the type preference scores and type weights of the first evaluation data pair under each preference type, a preference score of the first evaluation data pair is determined.

[0046] In actual implementation, the corresponding type weight can be configured for each preference type based on the actual application scenario or business needs or the experience of the staff, and the sum of the type weights corresponding to each preference type is 1. The type preference scores of the first evaluation data pair under each preference type are weighted and summed to obtain the preference score of the first evaluation data pair.

[0047] Specifically, the preference score of the first evaluation data pair can be calculated using the following formula (1): (1) in, represents the preference score corresponding to the i-th evaluation data pair (qi, ai); - Respectively represent the type weights corresponding to the 1st preference type to the nth preference type; - They respectively represent the preference scores obtained under the preference scoring rule corresponding to the 1st preference type - under the preference scoring rule corresponding to the nth preference type.

[0048] In the embodiments of the present specification, the type preference scores of the evaluation data under each preference type can be weighted and summed to obtain the corresponding preference scores, and an evaluation paradigm for the preference scores of the evaluation data under different preference types can be abstracted, and multiple human preferences can be defined and quantified. The corresponding preference evaluation results are automatically output for the multiple preferences, and the performance of the model to be evaluated in satisfying different preference types of users is taken into consideration.

[0049] Of course, in actual implementation, in addition to the above-mentioned implementation method of weighted summing the type preference scores of the evaluation data pairs under each preference type to obtain the corresponding preference score, other calculation methods can also be used to calculate the type preference scores of the evaluation data pairs under each preference type, such as directly summing the type preference scores of the evaluation data pairs under each preference type to obtain the corresponding preference score; or, calculating the geometric mean, weighted geometric mean, etc. of the type preference scores of the evaluation data pairs under each preference type to obtain the corresponding preference score, and the embodiments of this specification do not limit this.

[0050] In an optional implementation of this embodiment, before determining the preference score of at least one evaluation data pair based on the preference scoring rule, the method further includes: Obtain at least one preference type corresponding to at least one user group; Configure preference scoring rules corresponding to at least one preference type respectively.

[0051] It should be noted that different user groups have different dimensions of needs or preferences, that is, different user groups focus on different things. For example, technical personnel such as R&D or maintenance may pay more attention to the format of the answers to be evaluated output by the model to be evaluated, while operations personnel may pay more attention to the illusions and redundancies of the answers to be evaluated output by the model to be evaluated; or, user group A may prefer answers with rich content, while user group B may prefer answers with concise content.

[0052] Therefore, in actual implementation, at least one preference type corresponding to at least one user group can be obtained, and then based on the business characteristics of the corresponding user group, the preference scoring rule corresponding to the preference type can be configured. In specific implementation, a user group can correspond to at least one preference type, and a preference type can be configured with at least one preference scoring rule.

[0053] In the embodiments of the present specification, corresponding preference scoring rules can be defined for each preference type, and each preference scoring rule can define its preference characteristics, so as to obtain preference scores based on the preference scoring rules, quantify human preferences of different user groups, and define corresponding preference scoring rules, so as to automatically output corresponding preference evaluation results for multiple preferences, consider the performance of the model to be evaluated when satisfying different preference types of users, and make the evaluation more comprehensive, efficient and accurate.

[0054] It should be noted that in addition to determining the corresponding preference type based on the user group and then further determining the preference scoring rules, the preference scoring rules under the corresponding preference type can also be configured based on the application field of the model to be evaluated, the type of question to be evaluated, etc., to adapt to the actual business scenario, making the preference scoring more comprehensive and flexible.

[0055] Step 106: Determine a target score for at least one evaluation data pair based on the debiased score and the preference score of at least one evaluation data pair.

[0056] It should be noted that after obtaining the debiased score and preference score of at least one evaluation data pair, the debiased score and the preference score can be aligned to obtain the target score of at least one evaluation data pair, thereby achieving alignment of the objective scoring result and the subjective preference scoring result, and obtaining a more comprehensive and accurate scoring result.

[0057] In actual implementation, for any evaluation data pair, the debiased score and the preference score can be directly summed to achieve alignment, and the target score of the evaluation data pair can be obtained. Alternatively, the debiased score and the preference score can be aligned by weighted summation, geometric mean, weighted geometric mean and other algorithms to obtain the target score of the evaluation data pair, which is not limited in the embodiments of this specification.

[0058] In an optional implementation of this embodiment, determining a target score of at least one evaluation data pair according to a debiasing score and a preference score of at least one evaluation data pair includes: determining a second evaluation data pair from the at least one evaluation data pair; Obtaining a preference weight for the second evaluation data pair; determining a target score for the second evaluation data pair based on the debiased score, the preference score, and the preference weight of the second evaluation data pair; Continue to determine the second evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a target score for each evaluation data pair.

[0059] The preference weight refers to the degree of influence of the preference score of the evaluation data pair on the target score. The larger the preference weight, the greater the influence of the preference score on the target score, and the smaller the influence of the debiased score on the target score; the smaller the preference weight, the smaller the influence of the preference score on the target score, and the greater the influence of the debiased score on the target score. The preference weight can be configured based on the actual application scenario, business needs, or staff experience.

[0060] In actual implementation, any one of the at least one evaluation data pair can be determined as the second evaluation data pair, the preference weight of the second evaluation data pair is obtained, and the debiased score and preference score of the second evaluation data pair are weighted and summed to obtain the target score of the second evaluation data pair. After that, the debiased score and preference score of the next evaluation data pair in the at least one evaluation data pair are weighted and summed until the target score of each evaluation data pair is obtained. Among them, the preference weights of at least one evaluation data pair can be the same or different, that is, a corresponding preference weight can be configured for the overall model to be evaluated as the preference weight of each evaluation data pair, or different preference weights can be configured for different evaluation data pairs.

[0061] Specifically, the target score of each evaluation data pair can be calculated using the following formula (2): (2) in, represents the target score of the i-th evaluation data pair (qi, ai); represents the preference score corresponding to the i-th evaluation data pair (qi, ai); represents the debiased score corresponding to the i-th evaluation data pair (qi, ai); Represents the preference weight of the i-th evaluation data pair (qi, ai).

[0062] In the embodiments of this specification, the debiased scores and preference scores of the evaluation data pair can be weighted and summed to obtain the target score of the evaluation data pair, so as to achieve alignment between the objective scoring results and the subjective preference scoring results, thereby making the evaluation results of the model to be evaluated more comprehensive and accurate.

[0063] In an optional implementation of this embodiment, obtaining the preference weight of the second evaluation data pair includes: Determining the question type of the question to be evaluated and / or the answer type of the answer to be evaluated in the second evaluation data pair; According to the question type and / or the answer type, a preference weight of the second evaluation data pair is determined.

[0064] It should be noted that different question types and / or different answer types have different requirements for the degree of influence of preference scores on target scores. For example, for questions that require objective answers, the influence of preference scores on target scores should be smaller; for questions without objective standards, the influence of preference scores on target scores should be larger; or, for situations where there are objective facts about the answer to be evaluated, the influence of preference scores on target scores should be smaller; for situations where there are no objective facts about the answer to be evaluated, the influence of preference scores on target scores should be larger.

[0065] Therefore, in actual implementation, corresponding preference weights can be configured in advance for different question types and / or different answer types; then, the question type of the question to be evaluated and / or the answer type of the answer to be evaluated in the second evaluation data pair can be determined to determine the preference weight of the second evaluation data pair. Specifically, if more than one preference weight is determined based on the question type and the answer type, the average of the more than one preference weights can be used as the preference weight of the second evaluation data pair.

[0066] In the embodiments of the present specification, the corresponding preference weights can be determined based on the question type of the question to be evaluated and / or the answer type of the answer to be evaluated in the second evaluation data. Different question types and / or different answer types can correspond to different preference weights, thereby being more adaptable to a variety of different application scenarios and improving the flexibility and accuracy of model evaluation.

[0067] In an optional implementation of this embodiment, the model evaluation method further includes: Get test data pairs and test models; Perform tests according to the test model and the test data to obtain test results; Adjust preference weights based on test results.

[0068] Specifically, the test data pair refers to a test question and a corresponding test answer for testing, wherein the test answer is the answer output by the test model after processing and analyzing the test question. The test model is annotated with a corresponding evaluation tag based on actual application conditions or the experience of the staff.

[0069] In actual implementation, the test data pairs can be tested based on the test model, and the test scores of each test data pair can be obtained by combining the above preference weight calculation, and then the test labels of the test model can be determined based on the test scores of each test data pair. According to the difference between the test label and the evaluation label, the preference weights are continuously adjusted after multiple tests, and the test label with the smallest difference between the test label and the evaluation label is selected as the final preference weight. In specific implementation, the type weights corresponding to the above preference types can also be adjusted based on the test data pairs and the test model through multiple test trials.

[0070] In the embodiments of this specification, preference weights and / or type weights corresponding to each preference type can be adjusted through multiple test experiments based on test data pairs and test models to make the weight configuration better, thereby ensuring the accuracy of model evaluation.

[0071] Step 108: Determine an evaluation result of the model to be evaluated based on the target score of at least one evaluation data pair.

[0072] It should be noted that the answer to be evaluated in each evaluation data pair is the answer output by the model to be evaluated after analyzing and processing the evaluation question. The answer to be evaluated in the evaluation data pair represents the processing and analysis capability of the model to be evaluated on the evaluation question. Therefore, after obtaining the target score of at least one evaluation data pair, the overall evaluation result of the model to be evaluated can be determined by combining the target score of at least one evaluation data pair.

[0073] In an optional implementation of this embodiment, determining the evaluation result of the model to be evaluated according to the target score of at least one evaluation data pair includes: Determining a model score of the model to be evaluated based on a target score of at least one evaluation data pair; determining a model capability level of the model to be evaluated based on a target score of at least one evaluation data pair; The model score and / or model capability level is used as the evaluation result of the model to be evaluated.

[0074] In actual implementation, the target score of at least one evaluation data pair may be summed to obtain the model score of the model to be evaluated; or, the geometric mean of the target score of at least one evaluation data pair may be calculated as the model score of the model to be evaluated. Of course, other data calculation methods may also be used to process the target score of at least one evaluation data pair to obtain the model score of the model to be evaluated, such as the harmonic mean, which is not limited in the embodiments of this specification.

[0075] In a possible implementation, the model score of the model to be evaluated can be calculated using the following formula (3): (3) in, Represents the model score of the model to be evaluated; represents the target score of the i-th evaluation data pair; Indicates the number of data pairs to be evaluated.

[0076] In addition, the model capability level of the model to be evaluated can also be determined based on the target score of at least one evaluation data pair, wherein the model capability level is used to indicate the quality of the processing capability of the model to be evaluated. For example, the model capability level can be excellent, medium, poor, or level one, level two, level three, ..., etc. The higher the level, the better the processing capability.

[0077] Specifically, different scoring ranges and capability levels corresponding to the scoring ranges may be preconfigured, and the target scores of at least one evaluation data pair may be summed, the geometric mean may be calculated, or other data calculation methods may be used to obtain a comprehensive score of at least one evaluation data pair. The calculation method of the comprehensive score may be the same as or different from the calculation method of the model score of the model to be evaluated. Then, the target scoring range corresponding to the comprehensive score is determined, and the target capability level corresponding to the target scoring range is used as the model capability level of the model to be evaluated.

[0078] It should be noted that the model score of the model to be evaluated can be used as the evaluation result of the model to be evaluated, the model capability level of the model to be evaluated can be used as the evaluation result of the model to be evaluated, or the model score and model capability level of the model to be evaluated can be used together as the evaluation result of the model to be evaluated.

[0079] For example, assuming that a scoring range and capability level correspondence table as shown in Table 1 is pre-configured, assuming that the comprehensive score of at least one evaluation data pair is 85, and the model score of the model to be evaluated is also 85, then based on Table 1, it can be determined that the model capability level of the model to be evaluated is excellent, and the final evaluation result of the model to be evaluated is 85 points and the model capability level is excellent.

[0080] Table 1 Correspondence between scoring range and ability level

[0081] It should be noted that the model score and / or model capability level can be used as the evaluation result of the model to be evaluated. The model evaluation results are richer and more flexible, can adapt to a variety of different application scenarios, and improve the model evaluation effect.

[0082] For example, Figure 2 is a schematic diagram of a model evaluation process provided by an embodiment of this specification, such as Figure 2As shown, the question to be evaluated is input into the model to be evaluated to obtain the corresponding answer to be evaluated. The question to be evaluated and the answer to be evaluated are input into the debiasing scoring model to obtain the corresponding debiasing score. The illusion score corresponding to the question to be evaluated and the answer to be evaluated is determined based on the illusion scoring rule, the redundant score corresponding to the question to be evaluated and the answer to be evaluated is determined based on the redundant scoring rule, the format score corresponding to the question to be evaluated and the answer to be evaluated is determined based on the format scoring rule, ..., the security score corresponding to the question to be evaluated and the answer to be evaluated is determined based on the security scoring rule, and the preference score corresponding to the question to be evaluated and the answer to be evaluated is determined based on the illusion score, the redundant score, the format score, ..., the security score. According to the debiasing score and the preference score, the target score of the question to be evaluated and the answer to be evaluated is determined, and the evaluation result of the model to be evaluated is determined based on the target score.

[0083] In the embodiments of this specification, a model evaluation method is provided, which realizes determining the preference score of the answer to be evaluated in at least one evaluation data pair under the preference scoring rule corresponding to at least one preference type, superimposing the debiased score of at least one evaluation data pair, indicating whether the answer to be evaluated in the evaluation data pair answers the main question of the question to be evaluated, so as to obtain the evaluation result of the model to be evaluated. In this way, multiple human preferences are defined and quantified, and the corresponding evaluation results can be automatically output for multiple preferences, which not only considers the technical performance of the model to be evaluated, but also considers the performance of the model to be evaluated in meeting different user needs and preferences, and conducts a more comprehensive, more accurate and more flexible evaluation of the model to be evaluated, thereby improving the evaluation accuracy of the model to be evaluated, thereby accelerating the iteration of the effect of the model to be evaluated and the research and development of the product.

[0084] The following combination Figure 3 , taking the application of the model evaluation method provided in this specification in the question-answering scenario in the content application platform as an example, the model evaluation method is further explained. Figure 3 A processing flow chart of a model evaluation method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0085] Step 302: obtaining at least one question based on the historical question-and-answer records of the content application platform, inputting the at least one question into the large language model to be evaluated, and obtaining corresponding answers to each question.

[0086] Step 304: Input each question and the corresponding answer into the pre-trained debiased scoring model to obtain a debiased score for each question and the corresponding answer.

[0087] Step 306: For any question and corresponding answer, score the question and the corresponding answer based on the preference scoring rules corresponding to each preference type to obtain the type preference score of the question and the corresponding answer under each preference type; and perform weighted summation of the type preference scores of the question and the corresponding answer under each preference type according to the type weights corresponding to each preference type to obtain the preference score of the question and the corresponding answer.

[0088] Step 308: For any question and corresponding answer, according to the preference weights of the question and the corresponding answer, the debiased score and the preference score of the question and the corresponding answer are weightedly summed to obtain the target score of the question and the corresponding answer.

[0089] Step 310: Sum the target scores of each question and the corresponding answer to obtain the model score of the large language model to be evaluated.

[0090] A model evaluation method is provided in an embodiment of the present specification, which defines and quantifies multiple human preferences and can automatically output corresponding scores for multiple preferences. It not only takes into account the technical performance of the large language model to be evaluated, but also takes into account the performance of the large language model to be evaluated in meeting the needs and preferences of different users in question-and-answer scenarios. It conducts a more comprehensive, accurate, and flexible evaluation of the large language model to be evaluated, improves the evaluation accuracy of the large language model to be evaluated, and thus accelerates the iteration of the effect of the large language model to be evaluated and the research and development of products.

[0091] Corresponding to the above method embodiment, this specification also provides a model evaluation device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a model evaluation device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises: The acquisition module 402 is configured to acquire at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on the model to be evaluated; A first determination module 404 is configured to determine a debiased score and a preference score of at least one evaluation data pair, wherein the debiased score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the question to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule; A second determination module 406 is configured to determine a target score of at least one evaluation data pair based on the debiased score and the preference score of at least one evaluation data pair; The third determination module 408 is configured to determine the evaluation result of the model to be evaluated according to the target score of at least one evaluation data pair.

[0092] Optionally, the first determining module 404 is further configured to: Inputting at least one evaluation data pair into a debiased scoring model to obtain a debiased score for at least one evaluation data pair; Based on the preference scoring rules, a preference score of at least one evaluation data pair is respectively determined.

[0093] Optionally, the first determining module 404 is further configured to: determining a first evaluation data pair from the at least one evaluation data pair; Scoring the first evaluation data pair based on the preference scoring rules corresponding to each preference type, respectively, to obtain the type preference score of the first evaluation data pair under each preference type; Determining a preference score for the first evaluation data pair based on the type preference scores of the first evaluation data pair under each preference type; Continue to determine the first evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a preference score for each evaluation data pair.

[0094] Optionally, the first determining module 404 is further configured to: Get the type weight corresponding to each preference type; Based on the type preference scores and type weights of the first evaluation data pair under each preference type, a preference score of the first evaluation data pair is determined.

[0095] Optionally, the device further includes a configuration module configured to: Obtain at least one preference type corresponding to at least one user group; Configure preference scoring rules corresponding to at least one preference type respectively.

[0096] Optionally, the second determining module 406 is further configured to: determining a second evaluation data pair from the at least one evaluation data pair; Obtaining a preference weight for the second evaluation data pair; determining a target score for the second evaluation data pair based on the debiased score, the preference score, and the preference weight of the second evaluation data pair; Continue to determine the second evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a target score for each evaluation data pair.

[0097] Optionally, the second determining module 406 is further configured to: Determining the question type of the question to be evaluated and / or the answer type of the answer to be evaluated in the second evaluation data pair; According to the question type and / or the answer type, a preference weight of the second evaluation data pair is determined.

[0098] Optionally, the device further includes an adjustment module configured to: Get test data pairs and test models; Perform tests according to the test model and the test data to obtain test results; Adjust preference weights based on test results.

[0099] Optionally, the third determining module 408 is further configured to: Determining a model score of the model to be evaluated based on a target score of at least one evaluation data pair; determining a model capability level of the model to be evaluated based on a target score of at least one evaluation data pair; The model score and / or model capability level is used as the evaluation result of the model to be evaluated.

[0100] A model evaluation device is provided in an embodiment of the present specification, including an acquisition module, a first determination module, a second determination module, and a third determination module. The definition and quantification of multiple human preferences are achieved through the interactive cooperation of multiple modules, and the corresponding evaluation results can be automatically output for multiple preferences. Not only the technical performance of the model to be evaluated is considered, but also the performance of the model to be evaluated in meeting different user needs and preferences is considered. The model to be evaluated is evaluated in a more comprehensive, accurate, and flexible manner, which improves the evaluation accuracy of the model to be evaluated, thereby accelerating the iteration of the effect of the model to be evaluated and the research and development of products.

[0101] The above is a schematic scheme of a model evaluation device of this embodiment. It should be noted that the technical scheme of the model evaluation device and the technical scheme of the above-mentioned model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the model evaluation device can be referred to the description of the technical scheme of the above-mentioned model evaluation method.

[0102] Figure 5 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0103] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0104] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0105] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.

[0106] The processor 520 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model evaluation method.

[0107] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned model evaluation method.

[0108] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model evaluation method.

[0109] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned model evaluation method.

[0110] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned model evaluation method.

[0111] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned model evaluation method.

[0112] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] Computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0114] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0115] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0116] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to specific implementation methods. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can understand and use this specification well. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on the model to be evaluated; Determine a debiased score and a preference score of the at least one evaluation data pair, wherein the debiased score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the question to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule; determining a target score for the at least one evaluation data pair based on the debiased score and the preference score for the at least one evaluation data pair; An evaluation result of the model to be evaluated is determined according to the target score of the at least one evaluation data pair.

2. The model evaluation method according to claim 1, characterized in that: Determining a debiasing score and a preference score for the at least one evaluation data pair comprises: Inputting the at least one evaluation data pair into a debiased scoring model to obtain a debiased score for the at least one evaluation data pair; Based on the preference scoring rule, a preference score of the at least one evaluation data pair is respectively determined.

3. The model evaluation method according to claim 2, characterized in that: The step of respectively determining the preference score of the at least one evaluation data pair based on the preference score rule comprises: determining a first evaluation data pair from the at least one evaluation data pair; Scoring the first evaluation data pair based on the preference scoring rules corresponding to each preference type, respectively, to obtain a type preference score of the first evaluation data pair under each preference type; Determining a preference score for the first evaluation data pair based on the type preference scores of the first evaluation data pair under each preference type; Continue to determine the first evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a preference score for each evaluation data pair.

4. The model evaluation method according to claim 3, characterized in that: The determining the preference score of the first evaluation data pair based on the type preference score of the first evaluation data pair under each preference type includes: Obtaining the type weight corresponding to each preference type; Based on the type preference scores and type weights of the first evaluation data pair under each preference type, a preference score of the first evaluation data pair is determined.

5. The model evaluation method according to claim 3, characterized in that: Before respectively determining the preference score of the at least one evaluation data pair based on the preference score rule, the method further includes: Obtain at least one preference type corresponding to at least one user group; The preference scoring rules corresponding to the at least one preference type are configured respectively.

6. The model evaluation method according to any one of claims 1 to 5, characterized in that: Determining a target score of the at least one evaluation data pair according to the debiasing score and the preference score of the at least one evaluation data pair comprises: determining a second evaluation data pair from the at least one evaluation data pair; Obtaining a preference weight for the second evaluation data pair; determining a target score for the second evaluation data pair according to the debiased score, the preference score, and the preference weight of the second evaluation data pair; Continue to determine the second evaluation data pair from the at least one evaluation data pair until the last evaluation data pair in the at least one evaluation data pair, and obtain a target score for each evaluation data pair.

7. The model evaluation method according to claim 6, characterized in that: The obtaining the preference weight of the second evaluation data pair includes: Determining a question type of a question to be evaluated and / or an answer type of an answer to be evaluated in the second evaluation data pair; A preference weight of the second evaluation data pair is determined according to the question type and / or answer type.

8. The model evaluation method according to claim 6, characterized in that: The method further comprises: Get test data pairs and test models; Performing a test based on the test model and the test data to obtain a test result; The preference weight is adjusted according to the test result.

9. The model evaluation method according to any one of claims 1 to 5, characterized in that: The step of determining the evaluation result of the model to be evaluated according to the target score of the at least one evaluation data pair comprises: Determining a model score of the model to be evaluated according to a target score of the at least one evaluation data pair; Determining a model capability level of the model to be evaluated according to a target score of the at least one evaluation data pair; The model score and / or model capability level is used as the evaluation result of the model to be evaluated.

10. A model evaluation device, characterized in that: The device comprises: An acquisition module is configured to acquire at least one evaluation data pair, wherein the evaluation data pair includes a question to be evaluated and an answer to be evaluated obtained based on a model to be evaluated; A first determination module is configured to determine a debiasing score and a preference score of the at least one evaluation data pair, wherein the debiasing score is used to indicate whether the answer to be evaluated in the corresponding evaluation data pair answers the main question of the question to be evaluated, and the preference score is used to indicate the score of the answer to be evaluated in the corresponding evaluation data pair under the preference scoring rule; A second determination module is configured to determine a target score of the at least one evaluation data pair based on the debiased score and the preference score of the at least one evaluation data pair; The third determination module is configured to determine the evaluation result of the model to be evaluated according to the target score of the at least one evaluation data pair.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the model evaluation method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the model evaluation method described in any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the model evaluation method according to any one of claims 1 to 9.

Images