Model evaluation network construction method, model evaluation method and device

Abstract

The embodiment of the invention provides a construction method of a model evaluation network, and a model evaluation method and device. The construction method of the model evaluation network comprises the steps of obtaining at least one sample topic and a corresponding annotation evaluation rule; constructing a preference sample based on the annotation evaluation rule, and performing preference optimization on the candidate rule generation model to obtain a target rule generation model; obtaining a reference answer of the at least one sample question, and training the initial evaluation model based on a labeled evaluation result and a generated evaluation result corresponding to the reference answer to obtain a target evaluation model; and obtaining a model evaluation network according to the target rule generation model and the target evaluation model. The target rule generation model obtained by preference optimization can generate the evaluation rule closer to the annotation preference, the evaluation result of the target evaluation model obtained by training based on the annotation evaluation result and the generation evaluation result is more accurate, the stability of the model evaluation network is ensured, and a more accurate and more effective evaluation result can be obtained.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a method for constructing a model evaluation network, a model evaluation method and a device. Background Art

[0002] With the rapid development of computer technology, machine learning, and deep learning technology, machine learning models have been widely used in many fields such as natural language processing, machine translation, speech synthesis, and image generation. Large language models have been born accordingly. Large language models have been proven to have excellent performance in understanding and generating natural language texts. Large language model technology has developed rapidly, and it has shown strong capabilities in instruction-following, problem-solving, and open-ended chatting, which has brought greater challenges to how to evaluate the processing capabilities of such large language models.

[0003] In the prior art, a certain number of evaluation sets are generally established for various capabilities of large language models, such as mathematics, reasoning, knowledge question answering, creation, etc., and then these evaluation sets are used to ask questions to the large language model to be evaluated, and its processing capabilities are evaluated based on the correctness of the large language model's answers. However, in the above evaluation method, the basic facts of different questions are diverse, and the accuracy and effectiveness of model evaluation are poor. Therefore, a more accurate and effective model evaluation solution is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a method for constructing a model evaluation network. One or more embodiments of this specification also relate to a model evaluation method, a device for constructing a model evaluation network, a model evaluation device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a method for constructing a model evaluation network is provided, comprising: Obtain at least one sample question and a corresponding annotation evaluation rule; Constructing preference samples based on the annotation evaluation rules, optimizing the preference of the candidate rule generation model, and obtaining the target rule generation model; Obtaining a reference answer to the at least one sample question, and based on the annotated evaluation result and the generated evaluation result corresponding to the reference answer, training the initial evaluation model to obtain a target evaluation model; A model evaluation network is obtained according to the target rule generation model and the target evaluation model.

[0006] According to a second aspect of the embodiments of this specification, a model evaluation method is provided, the method comprising: Obtain at least one question to be tested, and obtain an answer to be evaluated output by the model to be evaluated for the at least one question to be tested; Generate a model according to the target rules in the model evaluation network to obtain the target evaluation rule corresponding to the at least one test question; According to the at least one test question and the corresponding answer to be evaluated and the target evaluation rule, the evaluation result of the model to be evaluated is obtained using the target evaluation model in the model evaluation network, wherein the model evaluation network is obtained based on the above-mentioned model evaluation network construction method.

[0007] According to a third aspect of an embodiment of this specification, a device for constructing a model evaluation network is provided, including: A first acquisition module is configured to acquire at least one sample topic and a corresponding annotation evaluation rule; A preference optimization module is configured to construct a preference sample based on the annotation evaluation rule, perform preference optimization on the candidate rule generation model, and obtain a target rule generation model; A first training module is configured to obtain a reference answer to the at least one sample question, and based on the annotated evaluation result and the generated evaluation result corresponding to the reference answer, train the initial evaluation model to obtain a target evaluation model; The first acquisition module is configured to obtain a model evaluation network according to the target rule generation model and the target evaluation model.

[0008] According to a fourth aspect of the embodiments of this specification, a model evaluation device is provided, the device comprising: A second acquisition module is configured to acquire at least one question to be tested, and acquire an answer to be evaluated output by the model to be evaluated for the at least one question to be tested; A second obtaining module is configured to generate a model according to a target rule in the model evaluation network to obtain a target evaluation rule corresponding to the at least one test question; The third acquisition module is configured to obtain the evaluation result of the model to be evaluated by using the target evaluation model in the model evaluation network according to the at least one test question and the corresponding answer to be evaluated and the target evaluation rule, wherein the model evaluation network is obtained based on the above-mentioned model evaluation network construction method.

[0009] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the above-mentioned model evaluation network construction method or the steps of the model evaluation method are implemented.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the above-mentioned model evaluation network construction method or the steps of the model evaluation method.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned model evaluation network construction method or the steps of the model evaluation method.

[0012] The embodiments of this specification provide a method for constructing a model evaluation network, which realizes the construction of preference samples based on the annotated evaluation rules corresponding to at least one sample question, performs preference optimization on the candidate rule generation model, obtains the target rule generation model, and enables the target rule generation model to align with the annotated evaluation rules; and, based on the annotated evaluation results and generated evaluation results corresponding to the reference answer, better training data can be constructed to improve the training efficiency and training accuracy of the target evaluation model. The target rule generation model obtained by preference optimization can produce evaluation rules that are closer to the annotated preferences, and the evaluation results of the target evaluation model obtained by training based on the annotated evaluation results and generated evaluation results are more accurate. Based on the target rule generation model and the target evaluation model, a corresponding model evaluation network can be constructed to evaluate the model to be evaluated, thereby ensuring the stability of the model evaluation network, obtaining more accurate and more effective evaluation results, and there is no need to call a third-party evaluation model, thus avoiding data leakage and reducing the calling cost.

[0013] The embodiments of this specification also provide a model evaluation method, which can obtain at least one question to be tested, and obtain the answer to be evaluated output by the model to be evaluated to the at least one question to be tested, and then generate a model based on the target rules obtained by the above preference optimization to obtain the target evaluation rules corresponding to the at least one question to be tested; and, based on the at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rules, the evaluation result of the model to be evaluated is obtained by using the target evaluation model obtained by the above training, so that more accurate and more effective evaluation results can be obtained, and there is no need to call a third-party evaluation model, which avoids data leakage and reduces the calling cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of a processing process of model evaluation based on a closed source model provided by an embodiment of this specification; Figure 2is a flowchart of a method for constructing a model evaluation network provided by an embodiment of this specification; Figure 3 is a flow chart of a model evaluation method provided by an embodiment of this specification; Figure 4a It is a schematic diagram of a processing process of a method for constructing a model evaluation network provided by an embodiment of this specification; Figure 4b It is a schematic diagram of a processing process of a model evaluation method provided by an embodiment of this specification; Figure 5 It is a structural diagram of a device for constructing a model evaluation network provided by an embodiment of this specification; Figure 6 is a structural schematic diagram of a model evaluation device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0015] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0016] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] First, the terms involved in one or more embodiments of this specification are explained.

[0020] Large Language Models (LLMS): refers to a type of language processing model with a large number of parameters. These models are trained through deep learning technology and can show excellent performance in a variety of natural language processing tasks. In recent years, with the growth of computing resources and the advancement of algorithms, large language models have become one of the important breakthroughs in the field of natural language processing. Large language models usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. Large language models can also be called cornerstone models / foundation models. Large language models are pre-trained using large-scale unlabeled corpora to produce pre-trained models with more than 100 million parameters. Such models can adapt to a wide range of downstream tasks and have good generalization capabilities. For example, large-scale language models (LLM) and multi-modal pre-training models have shown strong capabilities in instruction-following, problem-solving and open-ended chatting, and can be applied to many fields such as natural language processing, machine translation, speech synthesis, image generation, etc.

[0021] Fully supervised fine-tuning (SFT): is a model training technique in the field of machine learning, which is usually used to further optimize pre-trained models. Pre-trained models are usually initially trained on a large-scale dataset to learn general feature representations. Then, through SFT, these pre-trained models can be fine-tuned on smaller, domain-specific datasets to better adapt to the needs of specific tasks.

[0022] Reject Sampling: It is a random algorithm for generating samples from a probability distribution. It is a type of Monte Carlo method. It is usually used when it is difficult to directly sample from the target distribution. The core idea of ​​this method is to indirectly sample from the target distribution through an auxiliary distribution. The rejection sampling involved in the embodiments of this specification usually refers to sampling K samples on the basis of a fine-tuned model (which may be SFT fine-tuning or fine-tuning by reinforcement learning PPO algorithm, etc.), and then we have a rejection or acceptance function to filter the samples generated by the model sampling to select samples that meet our target distribution, and then fine-tune the model.

[0023] Human preference: refers to the inclination or preference that an individual or group shows for a specific thing, activity, attribute or other option. This preference can be broad, such as a preference for a certain answer style, or specific, such as the preference for a certain brand. Human preferences are formed by a combination of factors, including but not limited to cultural background, personal experience, social influence and personal characteristics.

[0024] Preference learning is a type of machine learning task whose goal is to learn a model from the preference information of users or systems that can predict or explain the preferences of individuals or groups. It mainly refers to aligning the output of LLMs to human preferences through reinforcement learning, etc., to better support the practical application of LLMs. Preference learning can be applied to many scenarios, such as recommendation systems, sorting tasks, personalized services, etc.

[0025] Evaluation Model: It can be a large language model used to evaluate the quality of content generated by other models. In other words, the evaluation model can be used to evaluate the analytical processing capabilities of other models.

[0026] LLM-as-a-judge: refers to using a large language model as an evaluation model to evaluate the quality of content generated by other models.

[0027] Direct preference optimization (DPO) is a method used in reinforcement learning and machine learning to learn from preferences. It focuses on learning a policy so that the behavior generated by the policy can best reflect the user's preferences. Compared with traditional reinforcement learning methods, DPO does not directly optimize the reward function, but learns directly from the user's preference feedback. The purpose is to make the policy or behavior generated by the model more in line with the user's expectations, providing a simpler and more direct learning method. The direct preference optimization algorithm uses the mapping relationship between the reward function and the optimal policy to prove that this restricted reward maximization problem can be accurately optimized through a single-stage policy training. In essence, it solves a classification problem on human preference data. The direct preference optimization algorithm is stable, lightweight in performance and computational cost, and does not require fitting a reward model. Direct preference optimization attempts to directly optimize preference comparisons to better preserve and utilize the original preference attributes. The working principle of the direct preference optimization algorithm is to increase the logarithmic probability of the preferred sample and reduce the logarithmic probability of the non-preferred sample response. It combines a dynamic weighting mechanism to avoid the model degradation problem encountered when only using the probability ratio target.

[0028] It should be noted that the rapid development of large language model technology has led to its widespread use. It has shown strong capabilities in instruction-following, problem-solving, and open-ended chatting, which poses a greater challenge to how to evaluate such open-ended cognitive abilities. The general evaluation method is generally to establish a certain number of evaluation sets for various capabilities of the model, such as mathematics, reasoning, knowledge question-answering, and creation, and then use these evaluation sets to ask questions to each large language model (LLMs), and judge the ability level of each LLMs based on the correctness of their answers and their consistency with human preferences.

[0029] Evaluating the capabilities of large language models in open-ended question answering scenarios is a major challenge. Metric-based automated evaluations provide speed and convenience, but due to the diversity of ground truth, the accuracy and effectiveness of the evaluations are often insufficient. In contrast, manual evaluations provide reliable evaluations but require a lot of resources.

[0030] In one possible implementation, LLM-as-a-judge attempts to strike a balance between automated and manual evaluation. Specifically, proprietary evaluation LLMs can be used to evaluate the responses of other models or compare the responses of different models. These proprietary evaluation LLMs can be closed-source models provided by third parties. Figure 1is a schematic diagram of a processing process of model evaluation based on a closed source model provided by an embodiment of this specification, such as Figure 1 As shown, an evaluation set M can be obtained, each evaluation question in the evaluation set M can be input into the model to be evaluated, the answer corresponding to each evaluation question can be obtained, each evaluation question and the corresponding answer can be input into a closed source model provided by a third party, and the evaluation result of the model to be evaluated can be obtained.

[0031] Since the update iteration of the internal version of the closed-source model is opaque, it not only affects the stability of the automated evaluation, but may also affect its fairness, and faces greater version uncertainty; secondly, the data to be evaluated needs to be uploaded to its cloud system through API (Application Programming Interface) calls to generate a response, which faces a great risk of data leakage; in addition, renting closed-source models provided by third parties also faces high calling costs.

[0032] Therefore, the embodiments of this specification provide a method for constructing a model evaluation network and a model evaluation method, which use a direct preference optimization algorithm to align human scoring rules to obtain a target rule generation model, construct high-quality training data through a rejection sampling method, and obtain a target evaluation model through training. The model evaluation network is independently constructed based on the target rule generation model and the target evaluation model, which ensures the stability of the model evaluation network, can obtain more accurate and more effective evaluation results, and does not need to call a third-party evaluation model, thereby avoiding data leakage and reducing the calling cost.

[0033] In this specification, a method for constructing a model evaluation network is provided. This specification also involves a model evaluation method, a device for constructing a model evaluation network, a model evaluation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0034] See also Figure 2 , Figure 2 A flowchart of a method for constructing a model evaluation network provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0035] Step 202: Obtain at least one sample question and corresponding annotation evaluation rules.

[0036] The sample questions are questions used to build a model evaluation network. The at least one sample question may come from different application scenarios. For example, in a news writing scenario, the sample question may be "Please give the press release corresponding to XXXX"; in a knowledge question answering scenario, the sample question may be "Please analyze the ages of Li Bai and Du Fu", etc. The labeled evaluation rules may refer to detailed evaluation rules labeled for the sample questions. For example, the labeled evaluation rules corresponding to the sample questions may be "Out of 10 points, answer feature A and get 2 points; answer feature B and get 3 points; answer feature C and get 5 points".

[0037] In actual implementation, the labeling evaluation rules can be manually labeled or directly obtained from the scoring criteria of other third-party question banks.

[0038] It should be noted that at least one sample question can be obtained from a local database, a third-party platform and / or the Internet, and then detailed scoring rules can be manually annotated for each sample question to obtain an annotated evaluation rule corresponding to the at least one sample question. The at least one sample question and the corresponding annotated evaluation rule can be used for the subsequent construction of a model evaluation network.

[0039] In an optional implementation of this embodiment, after obtaining at least one sample question and the corresponding annotation evaluation rule, the method further includes: According to at least one sample question and a corresponding annotation evaluation rule, an initial rule generation model is trained to obtain a candidate rule generation model.

[0040] Among them, the initial rule generation model is an open source large language model obtained through preliminary training on a large-scale dataset; the candidate rule generation model is a model adapted to the rule generation task generated in the fully supervised fine-tuning stage.

[0041] In actual implementation, based on at least one sample question and the corresponding annotation evaluation rule, the initial rule generation model can be fully supervised and fine-tuned to obtain a candidate rule generation model that is suitable for the rule generation task. Specifically, at least one sample question and prompt information can be input into the initial rule generation model. The prompt information is used to instruct the initial rule generation model to process the input data in a corresponding manner. The initial rule generation model can output the corresponding prediction evaluation rule. Based on the prediction evaluation rule and the annotation evaluation rule of the sample question, the model loss value can be calculated. According to the model loss value, the parameters of the initial rule generation model are adjusted in reverse. The above steps are repeated until the training stop condition is reached to obtain a candidate rule generation model that has been trained.

[0042] In the embodiments of the present specification, the pre-trained initial rule generation model can be directly subjected to fully supervised fine-tuning training based on at least one sample question and the corresponding annotation evaluation rule to obtain a candidate rule generation model, so that various rule generation tasks can be processed based on the candidate rule generation model to meet the requirements of the rule generation tasks, with a high training rate and good training effect.

[0043] Step 204: construct preference samples based on the labeled evaluation rules, perform preference optimization on the candidate rule generation model, and obtain the target rule generation model.

[0044] Among them, preference samples refer to training samples that can represent human preferences, and preference optimization refers to the preference learning of the candidate rule generation model on preference samples so that the generated evaluation rules can align with human preferences. The target rule generation model is used to output the corresponding detailed evaluation rules for the input questions.

[0045] It should be noted that each sample question and the corresponding annotated evaluation rule can be used as a preference sample, and the candidate rule generation model can be optimized based on the preference sample to obtain a target rule generation model, which can generate evaluation rules that are more in line with human preferences.

[0046] In an optional implementation of this embodiment, preference samples are constructed based on the labeled evaluation rules, and preference optimization is performed on the candidate rule generation model to obtain the target rule generation model, including: Determine a generation evaluation rule corresponding to at least one sample question using a candidate rule generation model; Constructing preference samples based on the annotation evaluation rules and constructing rejection samples based on the generation evaluation rules; The preferred samples and rejected samples are used as optimized sample pairs, and the candidate rule generation model is optimized to obtain the target rule generation model.

[0047] In actual implementation, the candidate rule generation model can be the model obtained in the fully supervised fine-tuning stage. The candidate rule generation model can adapt to the rule generation task. Therefore, at least one sample question can be input into the candidate rule generation model to obtain the generation evaluation rule output by the model. Each sample question and the corresponding annotation evaluation rule are used as preference samples, and each sample question and the corresponding generation evaluation rule are used as rejection samples. Then, based on the direct preference optimization algorithm, the logarithmic probability of the preference samples (manually annotated annotation evaluation rules) is increased and the logarithmic probability of the rejection samples (generation evaluation rules automatically generated by the model) is reduced. The candidate rule generation model is directly optimized to obtain the target rule generation model.

[0048] Specifically, direct preference optimization uses preference samples and rejection samples as preference data pairs. The preference data pairs can be used as input data to represent the preference degree of preference samples relative to rejection samples. The optimization goal of direct preference optimization is to fit the two data of the preference data pairs (i.e., preference samples and rejection samples) to the greatest extent possible. It is necessary to define an appropriate loss function or optimization target to minimize the difference between the model prediction and the actual preference. By using the preference data pairs corresponding to multiple sample questions as training data sets for training, direct preference optimization adjusts the model parameters to make them fit the preference data pairs to the greatest extent possible. The training process may involve optimization algorithms, such as gradient descent or evolutionary algorithms, to minimize the loss.

[0049] In the embodiments of the present specification, manually annotated annotation evaluation rules can be used as preference samples, and the generated evaluation rules automatically generated by the model can be used as rejection samples to construct preference data pairs. The candidate rule generation model obtained in the fully supervised fine-tuning stage can be optimized and trained using direct preference optimization to solve detail problems that cannot be solved by domain fine-tuning. Since further training is carried out on the basis of domain fine-tuning, there is no need to change the domain fine-tuning data. At the same time, by comparing the manually annotated annotation evaluation rules and the generated evaluation rules automatically generated by the model, detail problems can be well solved, and the target rule generation model obtained by optimization can generate detailed evaluation rules that align with human preferences.

[0050] Step 206: Obtain a reference answer to at least one sample question, and based on the annotated evaluation results and generated evaluation results corresponding to the reference answers, train the initial evaluation model to obtain a target evaluation model.

[0051] Among them, the annotated evaluation results are the detailed evaluation results manually annotated for the evaluation and analysis of the reference answers; the generated evaluation results are the detailed evaluation results automatically generated by the model for the reference answers.

[0052] In actual implementation, the reference answer is the answer to the sample question. The sample question can be processed and analyzed using a trained large language model or a closed-source model provided by other service providers to obtain the corresponding reference answer. Alternatively, questions with standard answers can be directly obtained from other third-party platforms or the Internet as sample questions, and the standard answers carried can be used as reference answers.

[0053] It should be noted that after obtaining the reference answer of at least one sample question, the annotated evaluation results and generated evaluation results corresponding to the reference answer can be obtained, and the initial evaluation model can be trained to obtain the target evaluation model by combining the annotated evaluation results and the generated evaluation results, so that the trained target evaluation model can output the evaluation results of the model to be evaluated.

[0054] In an optional implementation of this embodiment, obtaining a reference answer to at least one sample question includes: Acquire at least one reference processing model, wherein the at least one reference processing model has a different model processing capability and / or model scale; Based on at least one reference processing model, at least one reference answer corresponding to each sample question is obtained.

[0055] Among them, model processing capacity usually refers to the various performance indicators and technical characteristics of the model when performing its intended tasks, including not only the prediction accuracy of the model, but also multiple aspects of the model in practical applications, such as processing speed, memory consumption, parallel processing capabilities, etc. Model scale usually refers to the size and complexity of the model, which can include the number of parameters, hierarchical structure, computational complexity, storage requirements, amount of training data, model architecture, model complexity, etc.

[0056] In actual implementation, at least one reference processing model with different model processing capabilities and / or model scales can be obtained, and each sample question can be input into at least one reference processing model to obtain at least one reference answer corresponding to each sample question. That is, one sample question can obtain at least one reference answer.

[0057] In the embodiments of the present specification, reference processing models with different model processing capabilities and / or model scales are used to obtain at least one reference answer corresponding to each sample question. Due to the different model processing capabilities and / or model scales, these reference answers have a good diversity distribution, which can improve the accuracy of subsequent model training.

[0058] In an optional implementation of this embodiment, based on the annotated evaluation results and the generated evaluation results corresponding to the reference answers, the initial evaluation model is trained to obtain the target evaluation model, including: A first training sample is constructed based on the annotated evaluation result corresponding to the reference answer of the first sample question, and a second training sample is constructed based on the generated evaluation result corresponding to the reference answer of the second sample question, wherein the first sample question is a set proportion of sample questions in at least one sample question, and the second sample question is the remaining sample questions in at least one sample question except the first sample question; According to the labeled evaluation rules, the first training sample and the second training sample, the initial evaluation model is trained to obtain the target evaluation model.

[0059] The set ratio is a custom configured ratio based on actual business needs, such as 20%, 40%, etc. The larger the set ratio is, the more data needs to be manually labeled, the higher the labor cost and time cost of labeling, and the higher the accuracy will be.

[0060] In actual implementation, a set proportion of sample questions can be selected from at least one sample question as the first sample question. For the first sample question, the annotation evaluation results corresponding to the reference answers are manually annotated, and a first training sample is constructed based on the annotation evaluation results. The remaining sample questions in the at least one sample question are used as the second sample questions, and other trained large language models or closed-source models provided by other service providers are used to automatically generate corresponding generation evaluation results for the reference answers to the second sample questions, and a second training sample is constructed based on the generation evaluation results.

[0061] That is to say, the first training sample is the manually annotated detailed evaluation result of the reference answer to the first sample question, and the second training sample is the reference answer to the second sample question. The detailed evaluation results automatically generated by the model are combined with the annotated evaluation rules, the first training sample and the second training sample to train the initial evaluation model and obtain the target evaluation model.

[0062] For example, 20% of the data is selected from at least one sample question, and its reference answers are manually annotated to obtain detailed annotation evaluation results, such as "the reference answer includes feature A, gets 2 points; does not include feature B, gets no points; includes feature F, gets 5 points, and the total score of the reference answer is 7 points". For the remaining 80% of the data in at least one sample question, its reference answers are input into a closed-source model provided by another third party to obtain the generated evaluation results automatically generated by the model, and each sample question and the corresponding annotation evaluation results or generated evaluation results are constructed to obtain a training sample set.

[0063] In the embodiments of the present specification, a portion of sample questions are manually annotated to obtain detailed annotation evaluation results, while the other portion is automatically generated by a model with detailed generation evaluation results. Training samples are constructed based on the two portions of data, and the initial evaluation model is trained to obtain a target evaluation model. The source of training samples is richer, and the amount of data that requires manual annotation is reduced, thereby saving the time and cost of obtaining training samples and improving the efficiency of model training.

[0064] In an optional implementation of this embodiment, constructing a first training sample based on the annotation evaluation result corresponding to the reference answer of the first sample question includes: Selecting a set proportion of first sample questions from at least one sample question; Obtaining the annotation evaluation analysis and annotation evaluation score corresponding to the reference answer of the first sample question; A first training sample is constructed based on the annotation evaluation analysis and the annotation evaluation scores.

[0065] It should be noted that the evaluation results may include detailed evaluation analysis and a final evaluation score. The detailed evaluation analysis may be a step-by-step or sentence-by-sentence analysis of the reference answer, such as “The reference answer includes feature A, and gets 2 points; does not include feature B, and gets no points; includes feature F, and gets 5 points”, and the final evaluation score is “The reference answer gets a total score of 7 points”.

[0066] In an embodiment of the present specification, a set proportion of first sample questions can be selected from at least one sample question, and a detailed annotation evaluation analysis and a final annotation evaluation score can be manually annotated for the reference answer of each first sample question. The annotated evaluation analysis and the annotated evaluation score are used as the first training sample, and the detailed evaluation analysis and the final evaluation score are annotated to construct the first training sample, so that the target evaluation model obtained through training can output the evaluation analysis and the final evaluation score for the input answer, thereby realizing the evaluation of the model to be evaluated, and the evaluation result is more comprehensive, thereby improving the evaluation effect.

[0067] In an optional implementation of this embodiment, the method further includes: Obtain the annotated evaluation score corresponding to the reference answer of the second sample question; Accordingly, constructing a second training sample based on the generated evaluation result corresponding to the reference answer of the second sample question includes: Using the reference evaluation model to obtain a generated evaluation analysis and a generated evaluation score corresponding to a reference answer to the second sample question; If the generated evaluation score corresponding to the reference answer of the second sample question is consistent with the corresponding annotated evaluation score, the second sample question is used as the retained sample question; A second training sample is constructed based on the generated evaluation analysis and generated evaluation scores corresponding to the reference answers to the retained sample questions.

[0068] It should be noted that when constructing the second training sample based on the remaining second sample questions, a rejection sampling algorithm can be used, that is, a rejection function is configured to filter and screen the samples generated by the sampling, and samples that meet the distribution of the rejection function are determined as the second training samples, and then the model is fine-tuned in combination with the first training samples.

[0069] In actual implementation, for the remaining second sample questions except the first sample questions in at least one sample question, only the annotated evaluation score corresponding to the reference answer can be manually annotated without annotating the detailed evaluation analysis. Using a closed-source model that has been trained or provided by other third parties as a reference evaluation model, each second sample question and the corresponding reference answer are input into the reference evaluation model, and the generation evaluation analysis and generation evaluation score corresponding to the reference answer of the second sample question are automatically generated based on the reference evaluation model. Compare the generation evaluation score corresponding to the reference answer of the second sample question with the corresponding annotated evaluation score. If they are consistent, it means that the generation evaluation analysis and generation evaluation score generated by the reference evaluation model are relatively reliable, and the sample data can be retained; if they are inconsistent, it means that the generation evaluation analysis and generation evaluation score generated by the reference evaluation model are unreliable, and the sample data is screened out. The generation evaluation analysis and generation evaluation score corresponding to the reference answer of each retained sample question are used as the second training sample.

[0070] For example, obtain a set of questions N and the corresponding reference answers, which includes at least one sample question. In order to reduce the cost of manual annotation, perform detailed evaluation analysis and manual annotation of evaluation scores for the reference answers of 20% of the sample questions. For the remaining data, only the final evaluation score is manually annotated. Use a closed-source model that has been trained or provided by a third party to automatically generate evaluation analysis and evaluation scores for the reference answers of the remaining sample questions. If the automatically generated evaluation score is equal to the manually annotated evaluation score, retain this data as the second training sample; if not, screen out this data, and use the rejection sampling algorithm to screen and construct detailed evaluation analysis and evaluation scores for the remaining 80% of the data as the second training data.

[0071] It should be noted that the final target evaluation model can be obtained by fine-tuning the self-developed or open-source large language model using the manually annotated annotation evaluation rules, manually annotated annotation evaluation results, and the generated evaluation results automatically generated by the model. Specifically, the annotation evaluation rules and sample questions corresponding to the sample questions can be used as input data and input into the initial evaluation model to obtain the corresponding predicted evaluation results. The annotation evaluation results or generated evaluation results corresponding to the sample questions can be used as labels for the input data, and the parameters of the initial evaluation model can be adjusted until the training stop conditions are met to obtain the target evaluation model.

[0072] In the embodiments of this specification, high-quality training data can be constructed through the rejection sampling method, and the target evaluation model can be obtained through training, thereby ensuring the training efficiency and training accuracy of the target evaluation model, and thus obtaining more accurate and more effective evaluation results.

[0073] Step 208: Obtain a model evaluation network according to the target rule generation model and the target evaluation model.

[0074] It should be noted that the direct preference optimization algorithm can be used to align human scoring rules to obtain the target rule generation model, and the rejection sampling method can be used to construct high-quality training data and train the target evaluation model. The model evaluation network is self-developed based on the target rule generation model and the target evaluation model to ensure the stability of the model evaluation network.

[0075] The embodiments of this specification provide a method for constructing a model evaluation network, which realizes the construction of preference samples based on the annotated evaluation rules corresponding to at least one sample question, performs preference optimization on the candidate rule generation model, obtains the target rule generation model, and enables the target rule generation model to align with the annotated evaluation rules; and, based on the annotated evaluation results and generated evaluation results corresponding to the reference answer, better training data can be constructed to improve the training efficiency and training accuracy of the target evaluation model. The target rule generation model obtained by preference optimization can produce evaluation rules that are closer to the annotated preferences, and the evaluation results of the target evaluation model obtained by training based on the annotated evaluation results and generated evaluation results are more accurate. Based on the target rule generation model and the target evaluation model, a corresponding model evaluation network can be constructed to evaluate the model to be evaluated, thereby ensuring the stability of the model evaluation network, obtaining more accurate and more effective evaluation results, and there is no need to call a third-party evaluation model, thus avoiding data leakage and reducing the calling cost.

[0076] See also Figure 3 , Figure 3 A flow chart of a model evaluation method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0077] Step 302: Obtain at least one question to be tested, and obtain an answer to be evaluated output by the model to be evaluated for the at least one question to be tested.

[0078] Among them, at least one question to be tested is used to evaluate the answering ability of the model to be evaluated. The at least one question to be tested is different from the at least one sample question mentioned above, and the at least one question to be tested can come from the same or different application scenarios.

[0079] In actual implementation, at least one question to be tested is input into the model to be evaluated, and a corresponding answer to be evaluated can be obtained, which represents the analysis and processing capability of the model to be evaluated for the question to be tested.

[0080] Step 304: Generate a model according to the target rules in the model evaluation network to obtain a target evaluation rule corresponding to at least one test question.

[0081] In actual implementation, at least one test question is input into the target rule generation model, and the corresponding target evaluation rules can be output respectively.

[0082] Step 306: Based on at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rule, the target evaluation model in the model evaluation network is used to obtain the evaluation result of the model to be evaluated, wherein the model evaluation network is obtained based on the above-mentioned model evaluation network construction method.

[0083] It should be noted that at least one question to be tested, as well as the answers to be evaluated and target evaluation rules corresponding to each question to be tested, can be input into the target evaluation model. The target evaluation model, in turn, processes and analyzes the answers to be evaluated based on the target evaluation rules corresponding to each question to be tested, obtains the single-question score of the answer to be evaluated, and then comprehensively determines and outputs the evaluation results of the model to be evaluated based on the single-question scores of each question to be tested.

[0084] The embodiments of this specification provide a model evaluation method, which can obtain at least one question to be tested, and obtain the answer to be evaluated output by the model to be evaluated to the at least one question to be tested, and then generate a model based on the target rules obtained by the above preference optimization to obtain the target evaluation rules corresponding to the at least one question to be tested; and, according to the at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rules, the evaluation result of the model to be evaluated is obtained by using the target evaluation model obtained by the above training, so that more accurate and more effective evaluation results can be obtained, and there is no need to call a third-party evaluation model, which avoids data leakage and reduces the calling cost.

[0085] The above is a schematic scheme of a model evaluation method of this embodiment. It should be noted that the technical scheme of the model evaluation method and the technical scheme of the method for constructing the model evaluation network described above are of the same concept, and the details not described in detail in the technical scheme of the model evaluation method can be found in the description of the technical scheme of the method for constructing the model evaluation network described above.

[0086] The following combination Figure 4a and attached Figure 4b , taking the application of the model evaluation method provided in this specification in the question-answering scenario in the content application platform as an example, the model evaluation method is further explained. Figure 4a A schematic diagram of a process of constructing a model evaluation network provided by an embodiment of the present specification is shown. Figure 4b A schematic diagram of the processing process of a model evaluation method provided by an embodiment of this specification is shown.

[0087] like Figure 4a As shown, an evaluation set N is obtained based on the historical question and answer records of the content application platform, and the evaluation set N includes at least one sample question. Each sample question in the evaluation set N is input into at least one reference processing model to obtain a reference answer for each sample question.

[0088] Manually annotate the labeling evaluation rules of each sample question in the evaluation set N; train the initial rule generation model based on each sample question and the corresponding labeling evaluation rules to obtain the candidate rule generation model (fully supervised fine-tuning stage). Input each sample question in the evaluation set N into the candidate rule generation model, and use the candidate rule generation model to output the generation evaluation rules corresponding to each sample question; use the labeling evaluation rules as the preference samples and the generation evaluation rules as the rejection samples, and use direct preference optimization to perform preference learning on the candidate rule generation model to obtain the target rule generation model.

[0089] For the reference answers of each sample question, select 20% for manual annotation, and annotate the corresponding annotation evaluation analysis and annotation evaluation score. For the remaining 80%, manually annotate the corresponding annotation evaluation score. In addition, the remaining 80% of sample questions and reference answers are input into the reference evaluation model to obtain the generated evaluation analysis and generated evaluation score output by the model. If the annotation evaluation score and the generated evaluation score are consistent, they are used as retained sample questions. If they are inconsistent, the data is screened out, which is the rejection sampling algorithm. The annotation evaluation rules, the manually annotated annotation evaluation analysis and annotation evaluation scores, and the generated evaluation analysis and generated evaluation scores corresponding to the retained sample questions are used as training data sets to train the initial evaluation model to obtain the target evaluation model.

[0090] Based on the above target rule generation model and target evaluation model, a model evaluation network is obtained.

[0091] like Figure 4b As shown, an evaluation set P different from the evaluation set N can be obtained based on the historical question and answer records of the content application platform, and the evaluation set P includes at least one question to be tested. The at least one question to be tested is input into the target rule generation model, and the corresponding target evaluation rule is obtained question by question. In addition, at least one question to be tested can also be input into the model to be evaluated, and the corresponding answer to be evaluated can be obtained question by question. Then, at least one question to be tested, and the answer to be evaluated corresponding to each question to be tested, and the target evaluation rule are input into the target evaluation model. The target evaluation model can output the evaluation result of the model to be evaluated, and the evaluation result can include a detailed evaluation analysis and a final evaluation score.

[0092] The embodiments of this specification provide a model evaluation method, which can align human scoring rules with the help of a direct preference optimization algorithm to obtain a target rule generation model, construct high-quality training data through a rejection sampling method, obtain a target evaluation model through training, and construct a model evaluation network based on the target rule generation model and the target evaluation model to ensure the stability of the model evaluation network. Then, at least one question to be tested is obtained, and the answer to be evaluated output by the model to be evaluated for at least one question to be tested is obtained, and the target rule generation model obtained based on the above preference optimization is used to obtain the target evaluation rule corresponding to the at least one question to be tested; and, according to at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rule, the evaluation result of the model to be evaluated is obtained using the target evaluation model obtained through the above training, so that a more accurate and more effective evaluation result can be obtained, and there is no need to call a third-party evaluation model, which avoids data leakage and reduces the calling cost.

[0093] Corresponding to the above method embodiment, this specification also provides an embodiment of a device for constructing a model evaluation network. Figure 5 FIG. 1 is a schematic diagram showing a structure of a device for constructing a model evaluation network provided by an embodiment of the present specification. Figure 5 As shown, the device comprises: A first acquisition module 502 is configured to acquire at least one sample topic and a corresponding annotation evaluation rule; The preference optimization module 504 is configured to construct a preference sample based on the annotation evaluation rule, perform preference optimization on the candidate rule generation model, and obtain a target rule generation model; A first training module 506 is configured to obtain a reference answer to at least one sample question, and based on the annotated evaluation result and the generated evaluation result corresponding to the reference answer, train the initial evaluation model to obtain a target evaluation model; The first obtaining module 508 is configured to obtain a model evaluation network according to the target rule generation model and the target evaluation model.

[0094] Optionally, the preference optimization module 504 is further configured to: Determine a generation evaluation rule corresponding to at least one sample question using a candidate rule generation model; Constructing preference samples based on the annotation evaluation rules and constructing rejection samples based on the generation evaluation rules; The preferred samples and rejected samples are used as optimized sample pairs, and the candidate rule generation model is optimized to obtain the target rule generation model.

[0095] Optionally, the device further includes a second training module configured to: According to at least one sample question and a corresponding annotation evaluation rule, an initial rule generation model is trained to obtain a candidate rule generation model.

[0096] Optionally, the first training module 506 is further configured to: A first training sample is constructed based on the annotated evaluation result corresponding to the reference answer of the first sample question, and a second training sample is constructed based on the generated evaluation result corresponding to the reference answer of the second sample question, wherein the first sample question is a set proportion of sample questions in at least one sample question, and the second sample question is the remaining sample questions in at least one sample question except the first sample question; According to the labeled evaluation rules, the first training sample and the second training sample, the initial evaluation model is trained to obtain the target evaluation model.

[0097] Optionally, the first training module 506 is further configured to: Selecting a set proportion of first sample questions from at least one sample question; Obtaining the annotation evaluation analysis and annotation evaluation score corresponding to the reference answer of the first sample question; A first training sample is constructed based on the annotation evaluation analysis and the annotation evaluation scores.

[0098] Optionally, the device further includes a third acquisition module, configured to: Obtain the annotated evaluation score corresponding to the reference answer of the second sample question; Accordingly, the first training module 506 is further configured to: Using the reference evaluation model to obtain a generated evaluation analysis and a generated evaluation score corresponding to a reference answer to the second sample question; If the generated evaluation score corresponding to the reference answer of the second sample question is consistent with the corresponding annotated evaluation score, the second sample question is used as the retained sample question; A second training sample is constructed based on the generated evaluation analysis and generated evaluation scores corresponding to the reference answers to the retained sample questions.

[0099] Optionally, the first training module is further configured to: Acquire at least one reference processing model, wherein the at least one reference processing model has a different model processing capability and / or model scale; Based on at least one reference processing model, at least one reference answer corresponding to each sample question is obtained.

[0100] The embodiment of this specification provides a device for constructing a model evaluation network, including a first acquisition module, a preference optimization module, a first training module, and a first acquisition module. Through the interaction of multiple modules, a preference sample is constructed based on the labeled evaluation rules corresponding to at least one sample question, and the candidate rule generation model is optimized for preference to obtain a target rule generation model, so that the target rule generation model can align with the labeled evaluation rules; and based on the labeled evaluation results and generated evaluation results corresponding to the reference answer, better training data can be constructed to improve the training efficiency and training accuracy of the target evaluation model. The target rule generation model obtained by preference optimization can produce evaluation rules that are closer to the labeled preferences. The evaluation results of the target evaluation model obtained by training based on the labeled evaluation results and generated evaluation results are more accurate. Based on the target rule generation model and the target evaluation model, a corresponding model evaluation network can be constructed for evaluating the model to be evaluated, which ensures the stability of the model evaluation network, and can obtain more accurate and more effective evaluation results. There is no need to call a third-party evaluation model, which avoids data leakage and reduces the calling cost.

[0101] The above is a schematic scheme of a device for constructing a model evaluation network of this embodiment. It should be noted that the technical scheme of the device for constructing the model evaluation network and the technical scheme of the method for constructing the model evaluation network are of the same concept, and the details not described in detail in the technical scheme of the device for constructing the model evaluation network can be found in the description of the technical scheme of the method for constructing the model evaluation network.

[0102] Corresponding to the above method embodiment, this specification also provides a model evaluation device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a model evaluation device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: The second acquisition module 602 is configured to acquire at least one question to be tested, and acquire an answer to be evaluated output by the model to be evaluated for the at least one question to be tested; The second obtaining module 604 is configured to generate a model according to the target rule in the model evaluation network, and obtain a target evaluation rule corresponding to at least one test question; The third obtaining module 606 is configured to obtain an evaluation result of the model to be evaluated by using the target evaluation model in the model evaluation network according to at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rule, wherein the model evaluation network is obtained based on the above-mentioned model evaluation network construction method.

[0103] The embodiment of the present specification provides a model evaluation device, including a second acquisition module, a third acquisition module, and a third acquisition module. Through the interaction of multiple modules, at least one question to be tested can be obtained, and the answer to be evaluated output by the model to be evaluated to at least one question to be tested can be obtained, and then a target rule generation model obtained based on the above preference optimization can be obtained to obtain a target evaluation rule corresponding to at least one question to be tested; and, based on at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rule, an evaluation result of the model to be evaluated is obtained using the target evaluation model obtained by the above training, so that a more accurate and more effective evaluation result can be obtained, and there is no need to call a third-party evaluation model, thereby avoiding data leakage and reducing the calling cost.

[0104] The above is a schematic scheme of a model evaluation device of this embodiment. It should be noted that the technical scheme of the model evaluation device and the technical scheme of the above-mentioned model evaluation network construction method and model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the model evaluation device can be referred to the description of the technical scheme of the above-mentioned model evaluation network construction method and model evaluation method.

[0105] Figure 7 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0106] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0107] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0108] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0109] Among them, the processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the above-mentioned model evaluation network construction method or model evaluation method steps.

[0110] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned model evaluation network construction method or model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the above-mentioned model evaluation network construction method or model evaluation method technical scheme.

[0111] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-mentioned model evaluation network construction method or model evaluation method steps.

[0112] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned model evaluation network construction method or model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned model evaluation network construction method or model evaluation method.

[0113] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned model evaluation network construction method or the steps of the model evaluation method.

[0114] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned model evaluation network construction method or model evaluation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned model evaluation network construction method or model evaluation method.

[0115] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] Computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0117] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and models involved are not necessarily required by the embodiments of this specification.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for constructing a model evaluation network, characterized in that: The method comprises: Obtain at least one sample question and a corresponding annotation evaluation rule; Constructing preference samples based on the annotation evaluation rules, optimizing the preference of the candidate rule generation model, and obtaining the target rule generation model; Obtaining a reference answer to the at least one sample question, and based on the annotated evaluation result and the generated evaluation result corresponding to the reference answer, training the initial evaluation model to obtain a target evaluation model; A model evaluation network is obtained according to the target rule generation model and the target evaluation model.

2. The method for constructing a model evaluation network according to claim 1, characterized in that: The step of constructing a preference sample based on the annotation evaluation rule, performing preference optimization on the candidate rule generation model, and obtaining the target rule generation model includes: Determining a generation evaluation rule corresponding to the at least one sample question using the candidate rule generation model; constructing a preference sample based on the annotation evaluation rule, and constructing a rejection sample based on the generation evaluation rule; The preferred samples and the rejected samples are used as optimized sample pairs, and the candidate rule generation model is subjected to preference optimization to obtain a target rule generation model.

3. The method for constructing a model evaluation network according to claim 1 or 2, characterized in that: After obtaining at least one sample topic and the corresponding annotation evaluation rule, the method further includes: The initial rule generation model is trained according to the at least one sample question and the corresponding annotation evaluation rule to obtain the candidate rule generation model.

4. The method for constructing a model evaluation network according to claim 1, characterized in that: The step of training the initial evaluation model to obtain the target evaluation model based on the annotated evaluation result corresponding to the reference answer and the generated evaluation result includes: Constructing a first training sample based on the annotated evaluation result corresponding to the reference answer of the first sample question, and constructing a second training sample based on the generated evaluation result corresponding to the reference answer of the second sample question, wherein the first sample question is a set proportion of sample questions in the at least one sample question, and the second sample question is the remaining sample questions in the at least one sample question except the first sample question; According to the labeled evaluation rules, the first training samples and the second training samples, the initial evaluation model is trained to obtain a target evaluation model.

5. The method for constructing a model evaluation network according to claim 4, characterized in that: The step of constructing a first training sample based on the annotation evaluation result corresponding to the reference answer of the first sample question includes: Selecting a set proportion of first sample topics from the at least one sample topic; Obtaining a marked evaluation analysis and a marked evaluation score corresponding to the reference answer to the first sample question; The first training sample is constructed based on the annotation evaluation analysis and the annotation evaluation score.

6. The method for constructing a model evaluation network according to claim 4, characterized in that: The method further comprises: Obtaining the marked evaluation score corresponding to the reference answer of the second sample question; Accordingly, the step of constructing a second training sample based on the generated evaluation result corresponding to the reference answer to the second sample question includes: Obtaining a generated evaluation analysis and a generated evaluation score corresponding to a reference answer to the second sample question using a reference evaluation model; If the generated evaluation score corresponding to the reference answer of the second sample question is consistent with the corresponding annotated evaluation score, the second sample question is used as a retained sample question; The second training sample is constructed according to the generated evaluation analysis and generated evaluation score corresponding to the reference answer of the retained sample question.

7. The method for constructing a model evaluation network according to claim 1, characterized in that: The step of obtaining a reference answer to at least one sample question includes: Acquire at least one reference processing model, wherein the at least one reference processing model has different model processing capabilities and / or model scales; Based on the at least one reference processing model, at least one reference answer corresponding to each sample question is obtained.

8. A model evaluation method, characterized in that: The method comprises: Obtain at least one question to be tested, and obtain an answer to be evaluated output by the model to be evaluated for the at least one question to be tested; Generate a model according to the target rules in the model evaluation network to obtain the target evaluation rule corresponding to the at least one test question; According to the at least one question to be tested and the corresponding answer to be evaluated and the target evaluation rule, the evaluation result of the model to be evaluated is obtained by using the target evaluation model in the model evaluation network, wherein the model evaluation network is obtained based on the construction method of the model evaluation network described in any one of claims 1 to 7 above.

9. A device for constructing a model evaluation network, characterized in that: The device comprises: A first acquisition module is configured to acquire at least one sample topic and a corresponding annotation evaluation rule; A preference optimization module is configured to construct a preference sample based on the annotation evaluation rule, perform preference optimization on the candidate rule generation model, and obtain a target rule generation model; A first training module is configured to obtain a reference answer to the at least one sample question, and based on the annotated evaluation result and the generated evaluation result corresponding to the reference answer, train the initial evaluation model to obtain a target evaluation model; The first acquisition module is configured to obtain a model evaluation network according to the target rule generation model and the target evaluation model.

10. A model evaluation device, characterized in that: The device comprises: A second acquisition module is configured to acquire at least one question to be tested, and acquire an answer to be evaluated output by the model to be evaluated for the at least one question to be tested; A second obtaining module is configured to generate a model according to a target rule in the model evaluation network to obtain a target evaluation rule corresponding to the at least one test question; The third acquisition module is configured to obtain the evaluation result of the model to be evaluated by using the target evaluation model in the model evaluation network according to the at least one test question and the corresponding answer to be evaluated and the target evaluation rule, wherein the model evaluation network is obtained based on the construction method of the model evaluation network described in any one of claims 1 to 7 above.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the method for constructing a model evaluation network described in any one of claims 1 to 7 or the steps of the model evaluation method described in claim 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the method for constructing a model evaluation network as described in any one of claims 1 to 7 or the steps of the model evaluation method as described in claim 8.

13. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the method for constructing a model evaluation network as described in any one of claims 1 to 7 or the steps of the model evaluation method as described in claim 8.

Images