Large model effect evaluation method and device, storage medium and computer equipment

Abstract

The invention provides a large model effect evaluation method and device, a storage medium and computer equipment. Specifically, the to-be-evaluated large model is a large language model constructed according to the target prompt statement. According to the method, the to-be-evaluated text can be generated according to the session reply text of the to-be-evaluated large model for the at least one to-be-evaluated data set, and the to-be-evaluated text is evaluated by using the target judgment model corresponding to the target evaluation index, so that the effect evaluation result of the to-be-evaluated large model is determined. The to-be-evaluated text is a comprehensive result of combined action of the large language model, the target prompt statement and the to-be-evaluated data set, so that the method can cover three perspectives of the large model, the prompt statement and the data set at the same time, multi-angle and comprehensive evaluation is realized, and the effect and the service capability of the large model can be accurately obtained. Moreover, automatic scoring is carried out by adopting the target judgment model, so that the evaluation subjectivity and the labor cost can be greatly reduced, and large-model evaluation can be efficiently realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence application technology, and in particular to a method, apparatus, storage medium and computer equipment for evaluating the performance of large models. Background Technology

[0002] With the development of AI (Artificial Intelligence) technology, large-scale AI models are increasingly being applied in scenarios such as question answering, dialogue, and character simulation, demonstrating enormous potential and value across various fields. To provide guidance on large-scale model selection and optimization, it is necessary to evaluate the effectiveness of large-scale AI models in practice to determine their performance. However, the effectiveness of large-scale models depends on the model itself, the prompts, and the dataset. Current technologies lack an integrated, automated evaluation scheme for these three elements, making it difficult to accurately and efficiently determine the model's performance and business capabilities. Summary of the Invention

[0003] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the technical deficiency in the prior art that makes it difficult to accurately and efficiently derive the effects and business capabilities of large models.

[0004] In a first aspect, embodiments of this application provide a method for evaluating the performance of large models, including:

[0005] Identify the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement;

[0006] Obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of the at least one dataset to be evaluated;

[0007] Construct a target-based judging model based on pre-specified target evaluation indicators;

[0008] The text to be evaluated is evaluated using the target judge model to obtain a judge score corresponding to the target evaluation index.

[0009] The performance evaluation results of the large model to be evaluated are obtained based on the judges' scores.

[0010] In some embodiments, obtaining the session response text of the large model to be evaluated for the target dataset, and generating the evaluation text based on the session response text, includes:

[0011] If the target dataset is a question set dataset, then extract the text of each first question from the target dataset;

[0012] Each of the first question texts is sequentially input into the large model to be evaluated to obtain the first conversation response text of the large model to be evaluated for each of the first question texts;

[0013] Each of the first question texts and each of the first conversation response texts are combined into a multi-turn dialogue text, and the multi-turn dialogue text is used as the evaluation text corresponding to the target dataset.

[0014] In some embodiments, obtaining the session response text of the large model to be evaluated for the target dataset, and generating the evaluation text based on the session response text, includes:

[0015] If the target dataset is a question-answer pair dataset, then extract each second question text and the expected answer text corresponding to each second question text from the target dataset.

[0016] Each of the second question texts is input into the large model to be evaluated to obtain the second conversation response text of the large model to be evaluated for each of the second question texts;

[0017] The text to be evaluated is generated based on each of the second session response texts and each of the expected response texts.

[0018] In some embodiments, obtaining the session response text of the large model to be evaluated for the target dataset includes:

[0019] Based on the number of large models to be evaluated N and the number of datasets to be evaluated M, N×M evaluation subtasks are generated; wherein, the evaluation subtasks are used to obtain the conversation response text of a large model to be evaluated for a dataset to be evaluated;

[0020] The N×M evaluation subtasks are distributed to N task queues; wherein, each of the N task queues corresponds one-to-one with one of the N large models to be evaluated.

[0021] If N is greater than or equal to 2, the evaluation subtasks in the N task queues are executed concurrently to obtain the session response text of the large model to be evaluated for the target dataset.

[0022] In some embodiments, obtaining the performance evaluation result of the large model to be evaluated based on the judge's score includes:

[0023] For each of the target evaluation indicators, determine the target judge score corresponding to that target evaluation indicator;

[0024] For each of the target evaluation indicators, if the number of target judge scores is greater than or equal to 3, the target judge scores are filtered according to a preset extreme value elimination method, and the single indicator score corresponding to the target evaluation indicator is obtained based on the filtered target judge scores.

[0025] The effect evaluation result is obtained based on the single indicator score corresponding to each of the target evaluation indicators.

[0026] In some embodiments, obtaining the effect evaluation result based on the single indicator score corresponding to each of the target evaluation indicators includes:

[0027] Obtain the weight coefficient corresponding to each of the target evaluation indicators;

[0028] The scores of each individual indicator are weighted and summed according to each of the weighting coefficients to obtain the total score.

[0029] The effect evaluation result is generated based on the total effect score.

[0030] In some embodiments, constructing the target referee model based on pre-specified target evaluation metrics includes:

[0031] Obtain the referee prompt statements corresponding to the target evaluation indicators;

[0032] Determine the referee model identifier;

[0033] The target referee model is constructed based on the referee prompt statement and the referee model identifier.

[0034] Secondly, embodiments of this application provide a device for evaluating the effects of large models, including:

[0035] The data determination module is used to determine the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement;

[0036] The evaluation text generation module is used to obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of the at least one dataset to be evaluated;

[0037] The referee model construction module is used to construct a target referee model based on pre-specified target evaluation indicators.

[0038] The referee score acquisition module is used to evaluate the text to be evaluated through the target referee model and obtain a referee score corresponding to the target evaluation index.

[0039] The evaluation result acquisition module is used to obtain the effect evaluation result of the large model to be evaluated based on the judge's score.

[0040] Thirdly, embodiments of this application provide a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the evaluation method for the large model effect described in any of the above embodiments.

[0041] Fourthly, embodiments of this application provide a computer device, which includes: one or more processors, and a memory;

[0042] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the large model effect evaluation method described in any of the above embodiments.

[0043] In the large model performance evaluation methods, apparatuses, storage media, and computer devices provided in some embodiments of this application, the large model to be evaluated is a large language model constructed based on target prompt statements. This application can obtain the conversational response text of the large model to be evaluated for at least one dataset to be evaluated, generate the text to be evaluated accordingly, and use the target judge model corresponding to the target evaluation index to evaluate the text to be evaluated, thereby determining the performance evaluation result of the large model to be evaluated. In this application, the text to be evaluated is a comprehensive result of the combined effect of the large language model, the target prompt statements, and the dataset to be evaluated. Determining the performance of the large model based on the text to be evaluated can simultaneously cover three perspectives: the large model, the prompt statements, and the dataset, thereby achieving multi-angle and comprehensive evaluation, and thus accurately determining the performance and business capabilities of the large model. Furthermore, by using a target judge model for automatic scoring, this application can significantly reduce the subjectivity and manual cost of evaluation, contributing to large-scale, high-precision, and high-efficiency large model evaluation. Therefore, this application can accurately and efficiently determine the performance and business capabilities of large models, and can provide objective quantitative references for model selection, prompt statement optimization and dataset improvement, thereby guiding the application optimization of AI. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts illustrating a method for evaluating the performance of large models in some embodiments;

[0046] Figure 2 Here is a system architecture diagram of the evaluation system in some embodiments;

[0047] Figure 3 This is a second flowchart illustrating the evaluation method for large model performance in some embodiments;

[0048] Figure 4 This is a schematic diagram of the structure of the evaluation device for large model effects in some embodiments;

[0049] Figure 5 This is a diagram of the internal structure of a computer device in some embodiments. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The technical terms used in this application are explained below.

[0052] Large models: Large Language Models (LLMs) are natural language generation models with a large number of parameters, such as GPT, BELLE, and ChatGLM. They are usually implemented based on the Transformer architecture and have powerful text understanding and generation capabilities.

[0053] Prompt statements: These are input statements or rule information used to guide large models to produce input statements or rules with specific styles, content, or structures. They can include system prompts, user instructions, etc.

[0054] Dialogue dataset: Includes multi-turn interaction corpora between users and large models, used to test the multi-turn understanding and memory capabilities of large models.

[0055] Question-and-answer pair evaluation: This involves checking metrics such as the accuracy and relevance of a single round of input and output.

[0056] Problem set evaluation: The process of comprehensively evaluating a large model based on a set of related problem texts.

[0057] Dialogue Flow Evaluation: An evaluation method based on multi-turn dialogue flows, used to evaluate complex abilities of large models, such as persona consistency, fluency of expression, and empathy.

[0058] In some embodiments, this application provides a method for evaluating the performance of large models. The following embodiments illustrate this method by applying it to a computer device. It is understood that the computer device described in this application is a device with data processing capabilities, and may be, but is not limited to, various servers, server clusters, tablet computers, laptops, desktop computers, smartphones, etc., and this application does not impose specific limitations on it.

[0059] like Figure 1 As shown, the evaluation method for the large model effect provided in this application may include the following steps:

[0060] S102: Determine the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement.

[0061] It is understandable that the large model to be evaluated can be a large language model that has been trained. By inputting the target prompt (i.e., the target prompt) into the large language model that has been trained, the large model to be evaluated can be obtained.

[0062] The dataset to be evaluated refers to the dataset that needs to be evaluated. Its type can be, but is not limited to, multi-turn dialogue text datasets, single question-and-answer dialogue datasets, question sets, custom text, SD (Stable Diffusion) graph-text datasets, audio datasets, etc. It is understood that the number of datasets to be evaluated can be determined according to the actual situation. This application does not impose a specific limit on this, as long as there is more than or equal to one dataset to be evaluated.

[0063] In this step, the computer device can determine the large model to be evaluated and at least one dataset to be evaluated. For example, the computer device can determine the target prompt statement and the model identifier / call interface corresponding to the model to be evaluated, thereby determining the large model to be evaluated based on the target prompt statement and the model identifier / call interface.

[0064] S104: Obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of at least one dataset to be evaluated.

[0065] In this step, for each dataset to be evaluated, the computer device can obtain the session response text of the large model to be evaluated for that dataset, and construct the evaluation text corresponding to that dataset based on the session response text output by the large model to be evaluated. It should be noted that if the number of datasets to be evaluated is K, where K is a positive integer greater than 1, then this application can obtain K evaluation texts, and these K evaluation texts correspond one-to-one with the K datasets to be evaluated.

[0066] It can be understood that the conversation response text refers to the response text output by the large model to be evaluated, specifically for the dataset to be evaluated. The computer device can use the target dataset as input text for the large model to be evaluated, enabling the model to perform inference based on the target dataset and output the conversation response text corresponding to the target dataset.

[0067] S106: Construct a target evaluation model based on pre-specified target evaluation indicators.

[0068] The target evaluation metrics can be one or more pre-determined evaluation metrics, and can be built-in metrics or custom metrics defined by the testers; this application does not impose specific restrictions on this. For example, the target evaluation metrics may include any one or any combination of metrics such as accuracy, model score, persona consistency, dialogue fluency, and empathy.

[0069] The target judge model is used to evaluate the performance of the large model and the dataset to be evaluated under the target evaluation metrics. For example, when there are P target evaluation metrics, the computer device can construct P target judge models according to the target evaluation metrics, with each of the P target judge models corresponding one-to-one with the P target evaluation metrics, where P is a positive integer.

[0070] In this step, given the target evaluation metric, the computer device can construct a target referee model to facilitate subsequent evaluation of the large model's performance under the target evaluation metric. For example, when the target evaluation metric is a mathematical indicator such as accuracy, the computer device can construct the target referee model based on the calculation formula corresponding to the target evaluation metric. Furthermore, when the target evaluation metric is an indicator with a certain degree of subjectivity, such as empathy or persona consistency, this application can construct the target referee model using the indicator prompts corresponding to the target evaluation metric and a pre-determined large model.

[0071] S108: The text to be evaluated is evaluated using the target judge model to obtain the judge score corresponding to the target evaluation index.

[0072] In this step, the computer device can use the text to be evaluated as the evaluation object and perform evaluation using the target judge model to obtain a judge score. This judge score is used to reflect the performance of the large model to be evaluated and the target dataset under the target evaluation metric.

[0073] It should be noted that if multiple texts to be evaluated are obtained, such as K texts to be evaluated, the computer device can use the target judge model to score each of the K texts to be evaluated, thereby obtaining K judge scores corresponding to the target evaluation index.

[0074] S110: Obtain the performance evaluation results of the large model to be evaluated based on the judges' scores.

[0075] In this step, given the scores of each judge, this application can determine the performance evaluation result of the large model to be evaluated. For example, this application can compare the judge scores of different datasets to be evaluated under the same target evaluation index, and derive the performance evaluation result based on the comparison results.

[0076] For example, a computer device can determine a comparison model. The comparison model and the model to be evaluated may have different model structures, different model parameters, or be constructed based on different prompts. This application can obtain the evaluation text of the model to be evaluated and the comparison model for the same evaluation dataset, and compare the judge scores of the evaluation texts corresponding to different models under the same target evaluation index to obtain the effect evaluation results.

[0077] For example, this application may pre-set a scoring effect mapping relationship, and the computer equipment may determine the effect evaluation result based on the scores of each judge corresponding to the large model to be evaluated.

[0078] In this application, the evaluation text is a comprehensive result of the combined effects of a large language model, target prompts, and the evaluation dataset. Determining the performance of the large model based on the evaluation text simultaneously covers three perspectives: the large model, the prompts, and the dataset. This allows for multi-faceted and comprehensive evaluation, leading to accurate results regarding the model's performance and business capabilities. Furthermore, by employing a target-judgment model for automatic scoring, this application significantly reduces evaluation subjectivity and manual costs, facilitating large-scale, high-precision, and high-efficiency large model evaluation. Therefore, this application can accurately and efficiently determine the performance and business capabilities of the large model, providing objective quantitative references for model selection, prompt optimization, and dataset improvement, thereby guiding the optimization of AI applications.

[0079] In some embodiments, obtaining the session response text of the large model to be evaluated to the target dataset, and generating the evaluation text based on the session response text, includes:

[0080] Step A1: If the target dataset is a question set dataset, then extract the text of each first question from the target dataset;

[0081] Step A3: Input each first question text into the large model to be evaluated in sequence to obtain the first conversation response text of the large model to be evaluated for each first question text;

[0082] Step A5: Combine the texts of each first question and each first conversation response into a multi-turn dialogue text, and use the multi-turn dialogue text as the evaluation text corresponding to the target dataset.

[0083] In this embodiment, the question set dataset includes multiple interrelated question texts. If the target dataset is a question set dataset, the computer device can extract the question texts from the target dataset to obtain each first question text. The computer device can use each first question text as input to a large model, input it into the large model to be evaluated, and obtain the response text of the large model to be evaluated for each first question text, thereby obtaining each first conversation response text. The computer device can assemble each first question text and each first conversation response text in a question-and-answer format to obtain multi-turn dialogue text that can simulate the dialogue between the user and the large model to be evaluated. This multi-turn dialogue text is the evaluation text corresponding to the question set dataset.

[0084] For example, when the target dataset includes both guiding question text and follow-up question text, the computer device can first input the guiding question text into the large model to be evaluated and obtain the guiding response text output by the large model. Then, the computer device can input the follow-up question text into the large model to be evaluated and obtain the follow-up response text output by the large model. The computer device can use a preset combination format to assemble the text in the order of guiding question text, guiding response text, follow-up question text, and follow-up response text, thereby obtaining the evaluation text corresponding to the question set dataset.

[0085] This embodiment can automatically assemble the response text and question set of the large model to be evaluated into a context-rich dialogue sequence. This allows for context-enhanced evaluation in subsequent assessments and enables intelligent metrics such as persona consistency and empathy based on the dialogue sequence. In this way, context and complex dialogue flows can be integrated into the metric system, further improving the accuracy of the evaluation results.

[0086] In some embodiments, obtaining the session response text of the large model to be evaluated to the target dataset, and generating the evaluation text based on the session response text, includes:

[0087] Step B1: If the target dataset is a question-answer pair dataset, then extract each second question text and the expected answer text corresponding to each second question text from the target dataset;

[0088] Step B3: Input each second question text into the large model to be evaluated to obtain the second conversation response text of the large model to be evaluated for each second question text;

[0089] Step B5: Generate the evaluation text based on the response texts of each second session and the expected response texts.

[0090] Specifically, a question-answer pair dataset may include multiple question-answer pairs, each consisting of a question text and an expected answer text corresponding to that question text. For example, the question text is "How many hours are there in a day?", and the expected answer text is "24".

[0091] If the target dataset is a question-answer pair dataset, the computer device can extract the question text and answer text from each question-answer pair to obtain each second question text and each expected answer text. The computer device can then use each second question text as input to a large model to be evaluated, obtaining the model's response to each second question text, i.e., the second conversation response text. Based on each expected answer text and each second conversation response text, the computer device can generate evaluation text to facilitate subsequent evaluation based on the expected answer text, thereby further improving the accuracy of the evaluation results.

[0092] For example, this application can combine each expected answer text and each second session reply text one by one according to a preset combination format, so as to combine the expected answer text and the second session reply text of the same question text together and obtain the text to be evaluated.

[0093] For example, this application can assemble the texts of each second question and each second conversation response into conversation information in a question-and-answer format, and use the conversation information and each expected answer text as text to be evaluated.

[0094] In some embodiments, if the target dataset is a multi-turn dialogue dataset, the computer device can directly assemble the dialogue information of the multi-turn dialogue dataset into historical conversation information and use the historical conversation information as the text to be evaluated.

[0095] In some embodiments, obtaining the session response text of the large model to be evaluated to the target dataset includes:

[0096] Step C1: Based on the number of large models to be evaluated N and the number of datasets to be evaluated M, generate N×M evaluation subtasks; where each evaluation subtask is used to obtain the conversation response text of a large model to be evaluated for a dataset to be evaluated;

[0097] Step C3: Distribute the N×M evaluation subtasks to N task queues; where each of the N task queues corresponds one-to-one with one of the N large models to be evaluated.

[0098] Step C5: If N is greater than or equal to 2, then concurrently execute the evaluation subtasks in the N task queues to obtain the session response text of the large model to be evaluated for the target dataset.

[0099] Specifically, in practical application scenarios, the number of large models to be evaluated can be one or more, and the number of datasets to be evaluated can also be one or more. In this case, the computer device can generate multiple evaluation sub-tasks based on the number of large models to be evaluated and the number of datasets to be evaluated. For example, when the number of large models to be evaluated is N and the number of datasets to be evaluated is M, the computer device can generate N×M evaluation sub-tasks. Furthermore, if each large model to be evaluated includes c benchmark large models and o comparison large models, then this application can generate (o+c)×M evaluation sub-tasks.

[0100] When multiple evaluation subtasks are generated, the computer device can distribute tasks according to the large model to be evaluated described in each evaluation subtask. Specifically, the computer device can maintain N task queues, with each of the N task queues corresponding one-to-one with the N large models to be evaluated. Each task queue is used to maintain the task scheduling status of one large model to be evaluated. The computer device can distribute evaluation subtasks corresponding to the same large model to be evaluated into the same task queue, and distribute evaluation subtasks corresponding to different large models to be evaluated into different task queues to achieve task distribution.

[0101] When there are multiple large models to be evaluated, considering that the operation of different large models does not affect each other, the computer can concurrently execute the evaluation subtasks corresponding to N large models to be evaluated, so as to achieve concurrent evaluation of N large models. This can improve evaluation efficiency.

[0102] In some embodiments, the performance evaluation results of the large model to be evaluated are obtained based on the judges' scores, including:

[0103] For each of the target evaluation indicators, determine the target judge score corresponding to that target evaluation indicator;

[0104] For each of the target evaluation indicators, if the number of target judge scores is greater than or equal to 3, the target judge scores are filtered according to a preset extreme value elimination method, and the single indicator score corresponding to the target evaluation indicator is obtained based on the filtered target judge scores.

[0105] The effectiveness evaluation results are obtained based on the individual indicator scores corresponding to each target evaluation indicator.

[0106] In this embodiment, for each target evaluation indicator, the computer device can obtain the single indicator score corresponding to that target evaluation indicator according to steps D1 to D5:

[0107] Step D1: Using multiple different target referee models, obtain multiple referee scores under the target evaluation index, that is, obtain multiple target referee scores.

[0108] Step D3: If the target evaluation indicator corresponds to three or more judge scores, the computer device employs a supervision mechanism to monitor the multi-judge model, preventing high coupling of judge model data and further improving the accuracy of the evaluation results. Specifically, the supervision mechanism can filter and screen the multiple target judge scores according to a preset extreme value removal method, removing extreme values ​​and retaining non-extreme values. In some examples, the computer device can remove the highest and lowest scores from each target judge score.

[0109] Step D5: The computer device can obtain the individual indicator score corresponding to the target referee indicator based on the non-extreme value scores. For example, the computer device can calculate the average of each non-extreme value score and use this average as the individual indicator score corresponding to the target referee indicator.

[0110] After obtaining the individual index scores corresponding to each target evaluation index, the computer device can determine the model performance of the large model to be evaluated, so as to evaluate the target prompt statement, the large model itself and / or the dataset.

[0111] In some embodiments, the effect evaluation result is obtained based on the single indicator score corresponding to each target evaluation indicator, including:

[0112] Obtain the weight coefficient corresponding to each target evaluation indicator;

[0113] The scores of each individual indicator are weighted and summed according to each weight coefficient to obtain the total score.

[0114] The effectiveness evaluation results are generated based on the total effectiveness score.

[0115] In this embodiment, when multiple target evaluation indicators are applied simultaneously, the computer device can perform a weighted summation of the individual indicator scores corresponding to each target evaluation indicator to obtain a total effect score, and generate an effect evaluation result based on the total effect score. This enables multi-indicator, multi-dimensional evaluation, further improving the accuracy of the evaluation results.

[0116] It is understandable that by adjusting the weighting coefficients corresponding to the target evaluation indicators, the proportion of the target evaluation indicators in the total performance score can be adjusted. The weighting coefficients can be adjusted.

[0117] In some embodiments, constructing a target referee model based on pre-specified target evaluation metrics includes:

[0118] Obtain the referee's prompts corresponding to the target evaluation indicators;

[0119] Determine the referee model identifier;

[0120] Construct a target referee model based on the referee's prompts and the referee model identifier.

[0121] In this embodiment, a target judge model can be constructed by using the judge prompt statements and large model corresponding to the target evaluation indicators. The target judge model can then be used to evaluate custom indicators, supporting highly scalable custom business indicators that are plug-and-play, thereby improving the flexibility of the evaluation.

[0122] In some examples, the computer device can provide a referee model setting page, allowing testers to easily set evaluation scenarios and configure default parameters for the referee model. The computer device can then construct referee models for different evaluation scenarios based on the scenarios and parameters set by the testers on the referee model setting page.

[0123] In some embodiments, this application provides an evaluation system for performing the steps of the method described in any embodiment. The system architecture of the evaluation system may be as follows: Figure 2 and Figure 3 As shown, it includes a dataset management module, an evaluation object management module, an evaluation task engine, a model referee scoring module, a results analysis and visualization module, and an evaluation management platform / front-end. The functions supported by each engine / module are as follows: Figure 2 As shown.

[0124] The task scheduler refers to dividing the large models to be evaluated and the datasets to be evaluated into multiple evaluation subtasks (multiple evaluation subtasks constitute a whole evaluation task), and then distributing the evaluation subtasks to the corresponding task queues for execution. The large models to be evaluated are evaluated concurrently. For a detailed description of the task scheduler, please refer to the above explanation of the generation and concurrent scheduling of evaluation subtasks; this application will not repeat it further.

[0125] Various errors may occur during model invocation and response. Error handlers can handle different error types uniformly, such as timeout retry, parsing failure retry, and fast skip.

[0126] The operational logic of batch automatic evaluation scheduling can be found in the task scheduler. The difference is that batch automatic evaluation scheduling is used to perform multiple evaluation tasks simultaneously.

[0127] A multi-dimensional scoring engine refers to using a corresponding judge model to score the model's response after obtaining the large model's response text, and supports combining all questions and model responses in a single dataset for further scoring.

[0128] Specifically, the dataset management module can manage various datasets required for evaluation, including question sets, question-answer pairs, and multi-turn dialogue datasets. It provides a dataset management page that allows testers to create datasets, create dataset versions, and record each dataset through version control, supporting backtracking between different dataset versions. By providing a convenient and quick dataset management page, testers can quickly manage evaluation datasets. Simply set the dataset name and type, and then import the dataset.

[0129] The evaluation task engine can construct and execute evaluation tasks based on the evaluation object and evaluation metrics. Specifically, the evaluation task engine provides a concise evaluation task creation page to facilitate testers in setting evaluation scenarios, evaluation types, evaluation methods, large models to be evaluated, datasets to be evaluated, and other data.

[0130] Upon receiving an evaluation instruction, the evaluation task engine can create an evaluation task and calculate the number of evaluation subtasks. For details on the calculation method for the number of evaluation subtasks, please refer to the description in the above embodiments. If the dataset to be evaluated is a question-answer pair dataset, the evaluation task engine can extract the context and the first question text, construct a conversation scenario, obtain the response text of the large model to be evaluated in response to the context and the first question text, and extract the expected answer text.

[0131] If the dataset to be evaluated is a question set dataset, the evaluation task engine can extract the context and the second question text, construct a conversation scenario, obtain the response text of the large model to be evaluated for the context and the second question text, and assemble it into multi-turn dialogue text.

[0132] If the dataset to be evaluated is a dialogue dataset, the context and dialogue content can be extracted and assembled into historical conversation information.

[0133] The model referee scoring module can construct referee models for different indicators and scenarios based on preset referee information templates, and use the referee models to score, thereby determining the total effect score.

[0134] Furthermore, during the evaluation process, the evaluation system can output evaluation information details in a log to record information such as data content, current evaluation round, current evaluation object, current subtask, and evaluation progress.

[0135] This embodiment constructs an automated evaluation system that simultaneously targets large models, prompt words, and datasets, integrates multi-level context, and supports rich automated and custom evaluation metrics, thereby achieving accurate and objective reflection of multi-dimensional effects and business capabilities.

[0136] The following describes the large model effect evaluation device provided in the embodiments of this application. The large model effect evaluation device described below and the large model effect evaluation method described above can be referred to in correspondence.

[0137] In some embodiments, such as Figure 4 As shown, this application provides a large model effect evaluation device 400, including:

[0138] The data determination module 402 is used to determine the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement;

[0139] The evaluation text generation module 404 is used to obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of the at least one dataset to be evaluated;

[0140] The referee model construction module 406 is used to construct a target referee model based on pre-specified target evaluation indicators.

[0141] The referee score acquisition module 408 is used to evaluate the text to be evaluated through the target referee model and obtain a referee score corresponding to the target evaluation index.

[0142] The evaluation result acquisition module 410 is used to obtain the effect evaluation result of the large model to be evaluated based on the judge's score.

[0143] In some embodiments, the evaluation text generation module 404 of this application includes:

[0144] The first question text extraction unit is used to extract each first question text from the target dataset if the target dataset is a question set dataset;

[0145] The first response text acquisition unit is used to sequentially input each of the first question texts into the large model to be evaluated, and obtain the first conversation response text of the large model to be evaluated for each of the first question texts;

[0146] The combination unit is used to combine each of the first question texts and each of the first conversation response texts into a multi-turn dialogue text, and to use the multi-turn dialogue text as the evaluation text corresponding to the target dataset.

[0147] In some embodiments, the evaluation text generation module 404 of this application includes:

[0148] The second question text extraction unit is used to extract each second question text and the expected answer text corresponding to each second question text from the target dataset if the target dataset is a question-answer pair dataset.

[0149] The second response text acquisition unit is used to input each of the second question texts into the large model to be evaluated, and obtain the second conversation response text of the large model to be evaluated for each of the second question texts;

[0150] The evaluation text generation unit is used to generate the evaluation text based on each of the second session response texts and each of the expected response texts.

[0151] In some embodiments, the evaluation text generation module 404 of this application includes:

[0152] The subtask generation unit is used to generate N×M evaluation subtasks based on the number of large models to be evaluated N and the number of datasets to be evaluated M; wherein, the evaluation subtask is used to obtain the conversation response text of a large model to be evaluated to a dataset to be evaluated;

[0153] The subtask distribution unit is used to distribute N×M evaluation subtasks to N task queues respectively; wherein, the N task queues correspond one-to-one with the N large models to be evaluated;

[0154] The subtask execution unit is used to concurrently execute the evaluation subtasks in the N task queues if N is greater than or equal to 2, in order to obtain the session response text of the large model to be evaluated for the target dataset.

[0155] In some embodiments, the evaluation result acquisition module 410 of this application includes:

[0156] The first scoring acquisition unit is used to determine the target judge score corresponding to each of the target evaluation indicators.

[0157] The second scoring unit is used to filter each of the target evaluation indicators according to a preset extreme value elimination method when the number of target judge scores is greater than or equal to 3, and obtain the single indicator score corresponding to the target evaluation indicator based on the filtered target judge scores.

[0158] The result acquisition unit is used to obtain the effect evaluation result based on the single indicator score corresponding to each of the target evaluation indicators.

[0159] In some embodiments, the result acquisition unit of this application includes:

[0160] The weight acquisition unit is used to acquire the weight coefficient corresponding to each of the target evaluation indicators;

[0161] The weighted summation unit is used to perform weighted summation on the scores of each single indicator according to each of the weight coefficients to obtain the total effect score;

[0162] The result generation unit is used to generate the effect evaluation result based on the total effect score.

[0163] In some embodiments, the referee model construction module 406 of this application includes:

[0164] The statement acquisition unit is used to acquire the referee prompt statement corresponding to the target evaluation index;

[0165] The identifier acquisition unit is used to determine the referee model identifier;

[0166] The model building unit is used to build the target referee model based on the referee prompt statement and the referee model identifier.

[0167] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the evaluation method for the large model effect as described in any embodiment.

[0168] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform steps of the evaluation method for large model effects as described in any embodiment.

[0169] Indicatively, Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. In one example, the computer device can be a server. (Refer to...) Figure 5 The computer device 900 includes a processing component 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions, such as application programs, that can be executed by the processing component 902. The application programs stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 902 is configured to execute instructions to perform the steps of the methods described in any of the above embodiments.

[0170] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0171] Those skilled in the art will understand that the internal structure of the computer device shown in this application is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0172] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0173] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0174] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the performance of a large model, characterized in that, include: Identify the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement; Obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of the at least one dataset to be evaluated; Construct a target judging model based on pre-specified target evaluation indicators; The text to be evaluated is evaluated using the target judge model to obtain a judge score corresponding to the target evaluation index. The performance evaluation results of the large model to be evaluated are obtained based on the judges' scores.

2. The method according to claim 1, characterized in that, The step of obtaining the conversation response text of the large model to be evaluated for the target dataset, and generating the evaluation text based on the conversation response text, includes: If the target dataset is a question set dataset, then extract the text of each first question from the target dataset; Each of the first question texts is sequentially input into the large model to be evaluated to obtain the first conversation response text of the large model to be evaluated for each of the first question texts; Each of the first question texts and each of the first conversation response texts are combined into a multi-turn dialogue text, and the multi-turn dialogue text is used as the evaluation text corresponding to the target dataset.

3. The method according to claim 1, characterized in that, The step of obtaining the conversation response text of the large model to be evaluated for the target dataset, and generating the evaluation text based on the conversation response text, includes: If the target dataset is a question-answer pair dataset, then extract each second question text and the expected answer text corresponding to each second question text from the target dataset. Each of the second question texts is input into the large model to be evaluated to obtain the second conversation response text of the large model to be evaluated for each of the second question texts; The text to be evaluated is generated based on each of the second session response texts and each of the expected response texts.

4. The method according to claim 1, characterized in that, The step of obtaining the conversation response text of the large model to be evaluated for the target dataset includes: Based on the number of large models to be evaluated N and the number of datasets to be evaluated M, N×M evaluation subtasks are generated; wherein, the evaluation subtasks are used to obtain the conversation response text of a large model to be evaluated for a dataset to be evaluated; The N×M evaluation subtasks are distributed to N task queues; wherein, each of the N task queues corresponds one-to-one with one of the N large models to be evaluated. If N is greater than or equal to 2, the evaluation subtasks in the N task queues are executed concurrently to obtain the session response text of the large model to be evaluated for the target dataset.

5. The method according to any one of claims 1 to 4, characterized in that, The process of obtaining the performance evaluation results of the large model to be evaluated based on the judges' scores includes: For each of the target evaluation indicators, determine the target judge score corresponding to that target evaluation indicator; For each of the target evaluation indicators, if the number of target judge scores is greater than or equal to 3, the target judge scores are filtered according to a preset extreme value elimination method, and the single indicator score corresponding to the target evaluation indicator is obtained based on the filtered target judge scores. The effect evaluation result is obtained based on the single indicator score corresponding to each of the target evaluation indicators.

6. The method according to claim 5, characterized in that, The step of obtaining the effect evaluation result based on the single indicator score corresponding to each of the target evaluation indicators includes: Obtain the weight coefficient corresponding to each of the target evaluation indicators; The scores of each individual indicator are weighted and summed according to each of the weighting coefficients to obtain the total score. The effect evaluation result is generated based on the total effect score.

7. The method according to any one of claims 1 to 4, characterized in that, The construction of the target referee model based on pre-specified target evaluation indicators includes: Obtain the referee prompt statements corresponding to the target evaluation indicators; Determine the referee model identifier; The target referee model is constructed based on the referee prompt statement and the referee model identifier.

8. A device for evaluating the effect of a large model, characterized in that, include: The data determination module is used to determine the large model to be evaluated and at least one dataset to be evaluated; wherein, the large model to be evaluated is a large language model constructed based on the target prompt statement; The evaluation text generation module is used to obtain the conversation response text of the large model to be evaluated to the target dataset, and generate the evaluation text based on the conversation response text; wherein, the target dataset is any one of the at least one dataset to be evaluated; The referee model construction module is used to construct a target referee model based on pre-specified target evaluation indicators. The referee score acquisition module is used to evaluate the text to be evaluated through the target referee model and obtain a referee score corresponding to the target evaluation index. The evaluation result acquisition module is used to obtain the effect evaluation result of the large model to be evaluated based on the judge's score.

9. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the evaluation method for the large model effect as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the evaluation method for the large model effect as described in any one of claims 1 to 7.

Images