Large model evaluation method, device, equipment and medium

Abstract

The embodiment of the invention provides a large model evaluation method and device, equipment and a medium, which are used for solving the problems of low accuracy and poor effect of large model evaluation in related technologies. In the embodiment of the invention, the electronic equipment obtains any pre-constructed sample problem, and inputs the sample problem into a to-be-evaluated large model; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option in the options corresponding to the at least two candidate answers; obtaining an output option output by the large model, and obtaining a standard option stored for the sample problem; and based on whether the output option corresponding to each sample problem is consistent with the standard option, evaluating the capability of the large model. The electronic equipment evaluates the large model based on the options output by the large model and the standard options, so that the accuracy and effect of evaluating the large model can be improved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a large model evaluation method, device, equipment and medium. Background Art

[0002] With the continuous progress of society, the application scope of big models is becoming increasingly wide. However, the big models commonly seen in the current market mainly rely on public data for training, so they show good performance in general fields. But when companies actually use big models, they pay more attention to the problem-solving ability of big models in vertical business scenarios, that is, certain specific professional or industry categories, and the performance of big models in general fields is not equivalent to their capabilities in vertical fields. In view of this, when selecting an applicable big model, it is particularly important to conduct a comprehensive assessment of its performance in specific fields.

[0003] Common large model evaluation methods include word overlap rate-based methods, word vector-based methods, other deep learning-based methods, and manual evaluation methods. The word overlap rate-based method mainly calculates the word overlap rate between the generated text and the reference text as the evaluation standard for model quality. The more classic representatives include the Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), METEOR, etc. The word vector-based method is a word vector-based method that calculates the similarity between the generated text and the reference text in different forms to compare the quality of the model, such as Greedy Matching, etc. The other deep learning-based methods mainly use supervised learning methods to simulate the human evaluation process through models, such as Generative Adversarial Network (GAN) and ADEM. The above three are automatic evaluation methods, and more reliable manual evaluation methods can also be used. Secondly, although the common large model automatic evaluation method saves time and effort, the evaluation effect is not good. The manual evaluation method is expensive and highly subjective. Summary of the invention

[0004] The embodiments of the present application provide a large model evaluation method, device, equipment and medium, which are used to solve the problem of low accuracy and poor effect in the evaluation of large models in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a large model evaluation method, the method comprising:

[0006] Obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question;

[0007] The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options.

[0008] In a second aspect, the present application also provides a large model evaluation method, the method comprising:

[0009] A processing module is used to obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question;

[0010] The evaluation module is used to evaluate the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

[0011] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0012] A memory for storing program instructions;

[0013] A processor is used to call the program instructions stored in the memory, and execute the steps included in any of the above-mentioned methods for large model evaluation according to the obtained program instructions.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for large model evaluation as described in any one of the above items.

[0015] In an embodiment of the present application, the electronic device obtains any pre-constructed sample question and inputs the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines the output option from the options corresponding to the at least two candidate answers; obtains the output options output by the large model, and obtains the standard options saved for the sample question; and evaluates the capability of the large model based on whether the output options and standard options corresponding to each sample question are consistent. The electronic device evaluates the large model based on the options and standard options output by the large model, which can improve the accuracy and effect of the evaluation of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a large model evaluation method provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a scoring process for evaluating the evaluation capability of a large model provided in an embodiment of the present application;

[0019] Figure 3 A structural diagram of a large model evaluation device provided in an embodiment of the present application;

[0020] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0022] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0023] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0025] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0026] In order to solve the problem of low accuracy and poor effect in the evaluation of large models in the related art, the embodiments of the present application provide a large model evaluation method, device, equipment and medium.

[0027] The large model evaluation method includes: an electronic device obtains any sample question of a pre-constructed sample question set, and inputs the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtains the output options output by the large model, and obtains the standard options saved for the sample question; and evaluates the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

[0028] Embodiment 1:

[0029] Figure 1 A schematic diagram of a large model evaluation method provided in an embodiment of the present application, the process includes the following steps:

[0030] S101: Obtain any sample question from a pre-constructed sample question set, and input the sample question into a large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question.

[0031] The large model evaluation method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0032] In order to accurately and effectively evaluate the large model and thus select a suitable large model, a pre-constructed sample question set is stored in the electronic device, wherein the sample question set includes multiple sample questions. The electronic device can obtain any sample question in the pre-constructed sample question set. After obtaining the sample question, the electronic device inputs the sample question into the large model to be evaluated.

[0033] It should be noted that the sample question carries at least two candidate answers and options corresponding to each candidate answer, so that the large model to be evaluated determines the output option from the options corresponding to the at least two candidate answers. The options can be letters A, B, or numbers such as 1, 2, etc.

[0034] After the sample question is input into the large model to be evaluated, the electronic device can obtain output options output by the large model.

[0035] In order to evaluate the large model accurately and effectively, each sample question is stored with standard options, and the electronic device can determine the standard options stored for the sample question.

[0036] It should be noted that in order to make the big model only output options without outputting additional content, the sample questions also contain instructions. By giving clear instructions, the language comprehension ability of the big model is stimulated, allowing the big model to output options that meet the specifications.

[0037] Table 1 is a schematic diagram of detailed information of a sample question provided in an embodiment of the present application:

[0038]

[0039]

[0040] Table 1

[0041] The first column of Table 1 is the fields corresponding to the sample questions, the second column of Table 1 is the definition of the fields, and the third column of Table 1 is the examples. As can be seen from Table 1, the sample questions contain "instructions". "Instructions" represent instructions. By giving clear instructions, the language comprehension ability of the big model is stimulated, and the big model is allowed to output options that meet the specifications. It should be noted that here the big model can be required to only output options corresponding to the answers; the corresponding example can be "answer questions according to the context and only output correct options".

[0042] Moreover, as can be seen from Table 1, the sample questions contain "input", where "input" represents the context and specific questions, and the corresponding example can be ""Context: Natural gas itself is colorless and odorless. It is light blue when completely burned and yellow or red when not completely burned. Natural gas refers to all gases that exist naturally in nature, including gases formed by various natural processes in the atmosphere, hydrosphere and lithosphere (including oilfield gas, gas field gas, mud volcano gas, coalbed methane and biogenic gas, etc.). \nReasons for the appearance of red or yellow flames: \n1. Blockage of the air outlet \nLong-term use of gas stoves can easily cause oil stains to block the air outlet, resulting in incomplete combustion. If adjustment is required, only the oil stains on the air outlet need to be cleaned. The fire hole of the gas stove is easily blocked by rice residue dust, which can often be dredged with wire or an old toothbrush. \n2. Air door adjustment \nThere are two movable slide rails on the left and right sides below the gas stove. After igniting the gas stove, move the slider to adjust the flame state to a blue flame. The size of the lower baffle depends on the upward force of the flame. If the flame is straight and there is a whooshing sound, the damper is too large; if the flame is soft and unstable, the damper is too small. \n3. Poor smoke exhaust\nThe gas stove can only continue to burn completely when the smoke is exhausted quickly. If the ventilation is poor, the smoke will remain around the flame, resulting in insufficient oxygen and incomplete combustion. Open the kitchen window or increase the intensity of the range hood. \nExtended information: \nWhen the natural gas content in the air reaches a certain level, it will suffocate people. Natural gas is not as toxic as carbon monoxide, and it is basically harmless to the human body. However, if the concentration of natural gas is high and the oxygen in the air is not enough to sustain life, it will still cause death. After all, natural gas cannot be used for human breathing. Natural gas as a fuel can also cause casualties due to explosions. \nQuestion: Why does the gas stove flame appear red or yellow? ". Among them, \n is used for line breaks to improve the readability of the text.

[0043] As shown in Table 1, the sample questions contain "option", "option" is the candidate answer and the corresponding option. The corresponding examples are "A: caused by blockage of the air outlet and incomplete combustion; \nB: improper adjustment of the damper, resulting in red or yellow flames; \nC: poor ventilation and incomplete combustion when smoke exhaust is not smooth; \nD: The air pressure in the kitchen environment is too low, causing the flame to change color; \nE: Option A, Option B and Option C are all correct; \nF: Option A, Option B and Option D are all correct; \nG: Option A, Option B, Option C and Option D are all correct".

[0044] Moreover, as can be seen from Table 1, this sample question has a corresponding "output", "output" is a standard option, and the corresponding example is "E". This sample question also has a corresponding "id", "id" is an identifier, and the corresponding example is "context question answering_1".

[0045] S102: Evaluate the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

[0046] After obtaining the output options of the large model for each sample question and obtaining the standard options saved corresponding to each sample question, the electronic device can evaluate the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

[0047] In a possible implementation, the electronic device may determine the number of sample questions whose output options and representation options are consistent in each sample question, and determine the ratio of the number to the total number of sample questions as the score for evaluating the capability of the large model.

[0048] The method provided in the embodiment of the present application directly focuses on the consistency of the options, without the need to compare and analyze a large amount of text word by word, thereby improving the evaluation efficiency and accuracy. Moreover, the present application designs the data as an option class, and only needs to match the output options corresponding to the large model with the standard options to evaluate the large model, which completely solves the problems of poor traditional automatic evaluation and high manual evaluation costs.

[0049] In an embodiment of the present application, the electronic device obtains any pre-constructed sample question and inputs the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines the output option from the options corresponding to the at least two candidate answers; obtains the output options output by the large model, and obtains the standard options saved for the sample question; and evaluates the capability of the large model based on whether the output options and standard options corresponding to each sample question are consistent. The electronic device evaluates the large model based on the options and standard options output by the large model, which can improve the accuracy and effect of the evaluation of the large model.

[0050] Embodiment 2:

[0051] In order to accurately and effectively evaluate the large model, based on the above embodiment, in the embodiment of the present application, the ability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options, including:

[0052] For each pre-saved evaluation capability, determine that the capability corresponding to each sample question is each target sample question of the evaluation capability; determine the score for evaluating the evaluation capability of the large model based on whether the output options corresponding to each target sample question are consistent with the standard options, as well as the pre-saved consistent and inconsistent scores.

[0053] Since in actual scenarios, large models need to demonstrate various capabilities, electronic devices can accurately evaluate the different capabilities of large models according to specific evaluation requirements.

[0054] The big model can determine the score of each pre-saved evaluation capability of the big model. Each evaluation capability includes contextual learning, contextual question and answer, knowledge question and answer, calculation, reasoning, summary, question generation, text classification, semantic analysis, multi-round dialogue, etc. In order to determine the score of the big model corresponding to different evaluation capabilities, the electronic device stores the evaluation capability corresponding to each sample question. Therefore, the electronic device can determine the corresponding capability in each sample question, which is each target sample question of the evaluation capability. After determining each target sample question, the electronic device can evaluate the capability of the big model in the evaluation capability based on each determined target sample question.

[0055] Specifically, the electronic device can determine the score for evaluating the evaluation capability of the large model based on whether the output options of the large model and the standard options corresponding to each target sample question are consistent when outputting each target sample question, as well as the scores corresponding to the consistency and inconsistency stored in advance. For each target sample question, if the output answer corresponding to the target sample question is consistent with the standard answer, the electronic device can determine the sub-score of the target sample question to be the score corresponding to the consistency; otherwise, determine the sub-score corresponding to the target sample question to be the score corresponding to the inconsistency. The sum of the sub-scores corresponding to each target sample question is determined as the score for evaluating the evaluation capability of the large model.

[0056] Figure 2 A schematic diagram of a process for determining a score for evaluating the evaluation capability of a large model provided in an embodiment of the present application, the process comprising the following steps:

[0057] Figure 2 The following is an example of determining the score of a certain assessment ability.

[0058] S201: Determine the capability corresponding to each sample problem as each target sample problem for the evaluation capability.

[0059] S202: For each target sample question, determine the score for evaluating the large model based on the target sample question according to whether the output option corresponding to the target sample question is consistent with the standard option, and the pre-saved consistent and inconsistent scores.

[0060] S203: Evaluate the score of the large model according to each target sample question, and evaluate the score of the evaluation capability of the large model.

[0061] Table 2 is a schematic diagram of the information included in the sample questions corresponding to each evaluation capability provided in the embodiment of the present application:

[0062]

[0063]

[0064] Table 2

[0065] The first column in Table 2 is the evaluation capability, and the second column in Table 2 is the instruction examples contained in the corresponding sample questions. As can be seen from Table 2, the instruction examples contained in the sample questions corresponding to the evaluation capability of "context learning" can be "Please judge whether the text is a question sentence based on the example, and only output the correct option. Example: (1) In the quality procurement and intelligent purchasing enterprise, how do buyers download images in batches: True; (2) Hello, robot Xiaozhi is happy to serve you: False"; the instruction examples contained in the sample questions corresponding to the evaluation capability of "contextual question and answer" can be "Please answer the question based on the context, and only output the correct option"; the instruction examples contained in the sample questions corresponding to the evaluation capability of "knowledge question and answer" can be "Please answer the question and only output the correct option"; the instruction examples contained in the sample questions corresponding to the evaluation capability of "calculation" can be "Please answer the question and only output the correct option"; the instruction examples contained in the sample questions corresponding to the evaluation capability of "reasoning" can be "Please answer the question based on the context, and only output the correct option"; the instruction examples contained in the sample questions corresponding to the evaluation capability of "summary" can be "Based on the context, select the most appropriate answer from the following options to summarize The sample questions corresponding to the evaluation capability of "question generation" may include "please select the question corresponding to the given answer according to the context, and only output the option corresponding to the question, and do not output specific text information"; the sample questions corresponding to the evaluation capability of "knowledge question answering" may include "please answer the question and only output the correct option"; the sample questions corresponding to the evaluation capability of "text classification" may include "please identify the intention of the text description and only output the correct option"; the sample questions corresponding to the evaluation capability of "semantic analysis" may include "please read the article and select the description that is inconsistent and contradictory with the original content from the given options. Only output the corresponding option, do not output redundant content"; the sample questions corresponding to the evaluation capability of "multi-round dialogue" may include "please select the next least likely dialogue content from the given options based on background knowledge and a dialogue between the user and the source model. Only output the option, do not output the rest of the content".

[0066] Because large models store a vast amount of knowledge and have the "emergence ability" of large models, large models are capable of performing a variety of tasks and are widely used in more and more actual business scenarios. In order to understand and analyze industry needs, we refer to mainstream large model evaluation benchmarks, such as the General Language Understanding Evaluationbenchmark, Massive Multitask Language Understanding (MMLU), Machine Translation Benchmark (MT-bench, etc.), to clarify the evaluation capabilities of large models required by the industry. In the embodiment of this application, taking gas as an example, only the 10 evaluation capabilities shown in Table 2 are selected as the benchmark.

[0067] Embodiment 3:

[0068] In order to accurately and effectively evaluate the large model, based on the above embodiments, in the embodiment of the present application, the ability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options, including:

[0069] For each sample question, determine the score of the large model based on the sample question according to whether the output option corresponding to the sample question is consistent with the standard option, and the pre-saved consistent and inconsistent scores; and obtain the difficulty level saved for the sample question, and determine the target weight corresponding to the difficulty level of the sample question according to the correspondence between the saved level and the weight;

[0070] According to the score of the large model given by each sample question and the target weight corresponding to each sample question, a target score for evaluating the ability of the large model is determined.

[0071] In practical applications, large models need to be able to identify and handle problems of different difficulty levels. In order to be more in line with actual application scenarios, the evaluation of large models should be based on sample problems of various difficulty levels. It is worth noting that whether the large model can accurately identify and properly handle these problems of different difficulty levels is crucial to the display of its overall capabilities. In view of this, when electronic equipment conducts large model capability evaluation, it can set reasonable target scores based on the difficulty level corresponding to each sample problem.

[0072] For each sample question, the electronic device can determine the score for evaluating the large model based on the sample question according to whether the output option corresponding to the sample question is consistent with the standard option, as well as the pre-saved consistent and inconsistent scores. Specifically, if the output option is consistent with the standard option, the pre-saved consistent score is determined as the score for evaluating the large model based on the sample question. If the output option is inconsistent with the standard option, the pre-saved inconsistent score is determined as the score for evaluating the large model based on the sample question. When determining the score for evaluating the large model based on the sample question, the electronic device also obtains the difficulty level saved for the sample question. In order to accurately and effectively evaluate, the electronic device locally pre-saves the corresponding relationship between the difficulty level and the weight. The electronic device can determine the target weight corresponding to the difficulty level of the sample question.

[0073] After determining the score of each sample question for evaluating the large model and the target weight corresponding to the difficulty level of each sample question, the electronic device can determine the target score for evaluating the ability of the large model based on the score of each sample question for evaluating the large model and the target weight corresponding to each sample question. Specifically, the electronic device can determine, for each sample question, the product of the score of the sample question for evaluating the large model and the target weight corresponding to the sample question, and determine the sum of each product obtained as the target score for evaluating the ability of the large model.

[0074] Embodiment 4:

[0075] In order to accurately and effectively construct a sample question set, based on the above embodiments, in the embodiment of the present application, the sample question set is constructed in the following manner:

[0076] Receive a build instruction carrying the fields to be built;

[0077] Acquire a knowledge document in the knowledge base of the field to be constructed; send the knowledge document to a preset device, and the knowledge document is used to determine a sample problem;

[0078] Receive each sample question returned by the preset device, add each sample question to a sample question set, and save it.

[0079] In order to accurately and effectively construct a sample question set, the electronic device can receive a construction instruction carrying the field to be constructed. When business personnel need to construct a sample question set in a specific field, they can select the field they want to construct through the preset interface on the preset device and simply click a preset button such as "Build". This operation immediately triggers the electronic device to receive a construction instruction containing the field to be constructed. The electronic device responds to this instruction, and the electronic device automatically retrieves and obtains the relevant knowledge documents in the knowledge base of the field, and then sends these documents to the preset device for reference by the business personnel.

[0080] The business personnel construct sample questions based on the received knowledge document. The constructed sample questions are transmitted back to the electronic device through the preset device. Once the electronic device receives the sample questions from the preset device, the electronic device can add each sample question to the sample question set and save the sample question set corresponding to the added sample question.

[0081] Table 3 shows the requirements for business personnel to construct sample questions:

[0082]

[0083] Table 3

[0084] The first column in Table 3 is the fields included or corresponding to the sample questions, and the second column in Table 3 is the design requirements corresponding to the fields. As can be seen from Table 3, the design requirements for instruction are that the instructions for the same task should be as consistent as possible; the design requirements for input are: diversified sources of context and question corpus (such as Haoqi.com, Xinao Energy, Pan-Energy Business, etc.), diversified context and question splicing methods (such as context + question, question + context), diversified context and question length (short-long), and diversified question difficulty (easy-difficult); the design requirements for option are diversified number of options (few-many), diversified answer lengths corresponding to options (short-long), and diversified option difficulty (diverse wrong options-from easy to difficult); the design requirements for output are: the format is uppercase English letters, options are separated by \n, only options, no other characters; the design requirements for id are: customizable.

[0085] In an embodiment of the present application, the electronic device can generate a sample question set for unstructured, semi-structured and structured business data using a large model or manual method; specifically, question and answer (QA) data pairs can be manually extracted from multiple sources based on business data; the QA data pairs are converted into option questions, in which the correct answer is randomly selected as one of the options, and other incorrect answer options can be generated through model generation and manual construction, thereby ensuring the diversity of options. In addition, the difficulty of the data is graded to facilitate a more fine-grained evaluation of the large model.

[0086] It should be noted that in the field of artificial intelligence, the evaluation of large models is a crucial step, which can provide important reference for the development and optimization of models. This mainly involves two aspects: one is the construction of evaluation data sets, and the other is the selection of evaluation methods. In the prior art, common data sets based on large model evaluation mainly include GLUE, MMLU, etc., among which BIGbench consists of 204 tasks, and the task topics involve problems in the fields of linguistics, child development, mathematics, common sense reasoning, biology, physics, social bias, software development, etc. GLUE includes nine NLU tasks such as natural language inference, text entailment, sentiment analysis, and semantic similarity. SuperCLUE evaluates models from three dimensions: basic capabilities (chatting, dialogue, encyclopedia and knowledge, etc.), professional capabilities (idioms, poetry, literature, etc.), and Chinese characteristic capabilities (algebra, astronomy, clinical knowledge, etc.). C-EVAL and MMLU cover basic mathematics, computer science, history, law, etc., focusing on the evaluation of model capabilities in disciplines. Of course, the public large model evaluation sets are far more than these. The above benchmarks are more about the evaluation of general field capabilities. The embodiment of the present application provides a method for constructing a new sample question set.

[0087] Embodiment 5:

[0088] In order to evaluate the large model accurately and effectively, based on the above embodiments, in the embodiment of the present application, the receiving of each constructed sample question returned by the preset device includes:

[0089] receiving each constructed sample question returned by the preset device and the evaluation capability saved for each sample question;

[0090] After receiving each constructed sample question returned by the preset device and before adding each sample question to the sample question set, the method further includes:

[0091] Determine whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced; if so, execute the subsequent step of adding each sample question to the sample question set.

[0092] In order to evaluate large models accurately and effectively, the sample problem set for evaluating large models should include sample problems for evaluating different capabilities, and the number of sample problems corresponding to different evaluation capabilities should be balanced.

[0093] Specifically, when constructing sample questions, business personnel will clearly identify the capabilities to be assessed for each question, and attach the corresponding assessment capabilities when submitting these questions to the electronic device. Therefore, the electronic device will receive each sample question that has been constructed and returned by the preset device, and the assessment capabilities saved for each sample question.

[0094] The electronic device determines whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and determines whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced. If the evaluation capability corresponding to each received sample question covers each pre-saved capability, and the number of sample questions corresponding to each evaluation capability is balanced, the subsequent step of adding each sample question to the sample question set is executed.

[0095] Specifically, the electronic device can count the number of sample questions corresponding to each evaluation capability and determine whether the difference between these numbers is less than a preset threshold. If the difference in the number of sample questions of all evaluation capabilities is less than the preset threshold, it is determined that the number of sample questions of each evaluation capability has reached a balanced state.

[0096] If the evaluation capability corresponding to each sample question does not cover each pre-saved capability, or if the number of sample questions corresponding to each evaluation capability in each sample question is unbalanced, the electronic device can send a preset reminder message, specifically, send the reminder message to the preset device to inform the business personnel of each sample question that needs to be optimized and constructed. In one possible implementation, if the number of sample questions corresponding to each evaluation capability in each sample question is unbalanced, the electronic device can return the number of sample questions corresponding to each evaluation capability and the corresponding evaluation capability to the preset device, so that the business personnel can determine which evaluation field needs to have sample questions added.

[0097] Embodiment 6:

[0098] In order to accurately and effectively construct a sample question set, based on the above embodiments, in the embodiment of the present application, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0099] Determine whether there are separators between the options corresponding to each candidate answer contained in each received sample question. If so, perform the subsequent step of adding each sample question to the sample question set.

[0100] In the embodiment of the present application, in order to facilitate the recognition of the large model, there should be a preset separator between the options corresponding to each candidate answer. Therefore, after receiving each constructed sample question, the electronic device determines whether there is a separator between the options corresponding to each candidate answer contained in each received sample question, wherein the separator can be "\n". If there is a separator between the options corresponding to each candidate answer contained in each sample question, the subsequent step of adding each sample question to the sample question set is executed.

[0101] If there is no separator between the options corresponding to each candidate answer included in a sample question, the electronic device can add the separator at the position where the separator does not exist, and perform the subsequent step of adding each sample question to the sample question set.

[0102] It should be noted that, for the industry capability list, the embodiment of the present application specifies the format of the sample problem set as an instruction data set. The so-called sample ent refers to a type of guiding instruction used in training or using a large model, such as problem description, context information, examples, structural templates, thought chains, etc. By providing clear prompts or instructions, the large model can better understand people's intentions and requirements, and produce outputs that are more in line with human instruction requirements, thereby stimulating the understanding ability of the language model. Studies have shown that instruction prompts are closely related to model capabilities (mainly reasoning ability). From the simplest text prompts (Textual Prompt) and text prompts (CoT) with thought chains, to code prompts (Program of Thoughts), tree prompts (Tree of Thoughts), graph prompts (KG of Thoughts) and other more structured prompts, and even directly using logical rules as prompts, as the representation level increases, the model capability can be improved. Therefore, it is necessary to first construct reasonable sample problems to facilitate the model to understand different tasks and take correct actions. A good instruction should meet the following principles:

[0103] a) Clear numbering: Each sample question has a clear number, which helps to improve the organization and reference of the reading. If you need to discuss a specific rule, you can directly refer to its number;

[0104] b) Directness and simplicity: Each sample question is written in direct and clear language, concise and easy to understand;

[0105] c) Use imperative tone: Most sample questions appear in the form of imperatives, such as "You must..." and "Please...". This tone helps to enhance the authority and necessity of the rules;

[0106] d) Clear role definition: Clear role definition helps to set the subsequent rules;

[0107] e) Rules for specific scenarios: Some rules are targeted at specific situations or issues, such as the handling of copyrighted content, responses to developer-related questions, etc.

[0108] f) Guidance on format and style: In addition to content, the rules also provide guidance on format and style, such as using Markdown format, including programming language names, etc.

[0109] Embodiment 7:

[0110] In order to evaluate the large model accurately and effectively, based on the above embodiments, in the embodiment of the present application, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0111] For each sample question received, determine whether each sample question contains any pre-saved target information based on each pre-saved target information, wherein each target information is used to prompt the large model to output an option from the options corresponding to each answer contained in the sample question; if so, execute the subsequent step of adding each sample question to the sample question set.

[0112] In order to evaluate the large model accurately and effectively, the electronic device can prompt the large model to output options and not output other content. Therefore, each sample question should contain any target information that prompts the large model to output options. Specifically, the electronic device can pre-save each target information, where the target information can also be called an instruction. Each target information is used to prompt the large model to output an option in the options corresponding to each answer contained in the sample question. For example, the target information can be "output the options in the above options and do not output other content". The electronic device can determine whether each sample question contains any target information based on each pre-saved target information for each sample question received. If each sample question contains any pre-saved target information, the subsequent step of adding each sample question to the sample question set is performed.

[0113] If there is a sample question that does not contain any pre-saved target information, it is determined that the sample question is abnormal. The electronic device can output the sample question to a preset device, prompt the business personnel that the sample question is abnormal, and delete the sample question.

[0114] In the embodiment of the present application, the evaluation set is reviewed by multiple people in a cross-over manner (that is, the person who prepared a certain capability evaluation set does not participate in the review of the capability), and the specific review requirements are as follows:

[0115] a) Textual correctness: The text has no grammatical errors and the answers to the questions are correct and cannot conflict with the facts;

[0116] b) Format consistency: For the same type, the format of each sample question under this type is consistent;

[0117] c) Ability diversity: sample questions must cover the 10 assessment abilities mentioned above, and the number of samples for each ability must be balanced;

[0118] d) Business diversity: Data sources must cover the main business areas of the enterprise;

[0119] e) Format variability: Each field should be kept diverse according to design requirements;

[0120] f) Privacy security: Real information that appears in the assessment set, such as name, address, and phone number, needs to be desensitized to avoid privacy leakage;

[0121] g) Option non-ambiguity: All ambiguous options should be deleted;

[0122] h) The options are confusing: the wrong options should not be too simple, but confusing;

[0123] i) Example diversity: Regarding contextual learning capabilities, the details are as follows: Diversity of example task types, such as extraction, matching, judgment, etc.; the number of examples increases from small to large; iii. Diversity of example task answers, including wrong answers and correct answers. It should be noted that the answer in the example can be wrong, but the task type cannot change.

[0124] The format of the sample questions in the sample question set in the embodiment of the present application is a choice question type, so it is only necessary to match the output options of the large model with the standard options. Instructions can be used to require special characters before and after the options for easy matching. Among them, the cost of constructing a sample question set containing options is much higher than that of a QA data set. If a lower-cost QA data format is used, it is recommended to use a large model to evaluate the large model. A method based on word overlap rate can also be used, which does not require model training and is simple and fast.

[0125] Large model evaluation has always been a difficult problem in the industry. Traditional evaluation indicators are inaccurate and manual evaluation costs are high. The embodiment of the present application designs the data into option questions. By designing reasonable instructions, the large model only outputs the correct options. In this way, the large model can be evaluated by matching the model output options with the standard options.

[0126] Embodiment 8:

[0127] Based on the same inventive concept, the present application embodiment provides a large model evaluation device, please refer to Figure 3, the device comprises:

[0128] The processing module 301 is used to obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question;

[0129] The evaluation module 302 is used to evaluate the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

[0130] In one possible implementation, the evaluation module 302 is specifically used to determine, for each pre-saved evaluation capability, the capability corresponding to each sample problem as each target sample problem of the evaluation capability, and determine the score for evaluating the evaluation capability of the large model based on whether the output options corresponding to each target sample problem are consistent with the standard options, as well as the pre-saved consistent and inconsistent scores.

[0131] In a possible implementation, the evaluation module 302 is specifically used to determine, for each sample question, a score for the large model based on the sample question according to whether the output options corresponding to the sample question are consistent with the standard options, as well as pre-saved consistent and inconsistent scores; and obtain the difficulty level saved for the sample question, and determine the target weight corresponding to the difficulty level of the sample question according to the correspondence between the saved level and the weight; and determine the target score for evaluating the ability of the large model according to the score of the large model for each sample question and the target weight corresponding to each sample question.

[0132] In a possible implementation, the processing module 301 is also used to construct the sample question set in the following manner: receiving a construction instruction carrying a domain to be constructed; obtaining a knowledge document in a knowledge base of the domain to be constructed; sending the knowledge document to a preset device, where the knowledge document is used to determine sample questions; receiving each sample question returned by the preset device, adding each sample question to the sample question set and saving it.

[0133] In a possible implementation manner, the processing module 301 is specifically configured to receive each constructed sample question returned by the preset device, and the evaluation capability saved for each sample question;

[0134] The processing module 301 is also used to determine whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced; if so, executing the subsequent step of adding each sample question to the sample question set.

[0135] In a possible implementation, the processing module 301 is further used to determine whether there are separators between the options corresponding to each candidate answer contained in each received sample question, and if so, execute the subsequent step of adding each sample question to the sample question set.

[0136] In a possible implementation, the processing module 301 is also used to determine, for each sample question received, whether each sample question contains any pre-saved target information based on each pre-saved target information, wherein each target information is used to prompt the large model to output an option from the options corresponding to each answer contained in the sample question; if so, executing the subsequent step of adding each sample question to the sample question set.

[0137] Embodiment 9:

[0138] Based on the same inventive concept, the present application embodiment provides an electronic device, which can implement the function of the above intention update, please refer to Figure 4 The device includes a processor 401, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0139] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:

[0140] Obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question;

[0141] The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options.

[0142] In a possible implementation, the evaluating the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options includes:

[0143] For each pre-saved evaluation capability, determine that the capability corresponding to each sample question is each target sample question of the evaluation capability; determine the score for evaluating the evaluation capability of the large model based on whether the output options corresponding to each target sample question are consistent with the standard options, as well as the pre-saved consistent and inconsistent scores.

[0144] In a possible implementation, the evaluating the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options includes:

[0145] For each sample question, determine the score of the large model based on the sample question according to whether the output option corresponding to the sample question is consistent with the standard option, and the pre-saved consistent and inconsistent scores; and obtain the difficulty level saved for the sample question, and determine the target weight corresponding to the difficulty level of the sample question according to the correspondence between the saved level and the weight;

[0146] According to the score of the large model given by each sample question and the target weight corresponding to each sample question, a target score for evaluating the ability of the large model is determined.

[0147] In a possible implementation, the sample question set is constructed in the following manner:

[0148] Receive a build instruction carrying the fields to be built;

[0149] Acquire a knowledge document in the knowledge base of the field to be constructed; send the knowledge document to a preset device, and the knowledge document is used to determine a sample problem;

[0150] Receive each sample question returned by the preset device, add each sample question to a sample question set, and save it.

[0151] In a possible implementation manner, the receiving each constructed sample question returned by the preset device includes:

[0152] receiving each constructed sample question returned by the preset device and the evaluation capability saved for each sample question;

[0153] After receiving each constructed sample question returned by the preset device and before adding each sample question to the sample question set, the method further includes:

[0154] Determine whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced; if so, execute the subsequent step of adding each sample question to the sample question set.

[0155] In a possible implementation manner, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0156] Determine whether there are separators between the options corresponding to each candidate answer contained in each received sample question. If so, perform the subsequent step of adding each sample question to the sample question set.

[0157] In a possible implementation manner, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0158] For each sample question received, determine whether each sample question contains any pre-saved target information based on each pre-saved target information, wherein each target information is used to prompt the large model to output an option from the options corresponding to each answer contained in the sample question; if so, execute the subsequent step of adding each sample question to the sample question set.

[0159] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0160] The communication interface 402 is used for communication between the electronic device and other devices.

[0161] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0162] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0163] Embodiment 10:

[0164] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0165] Obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question;

[0166] The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options.

[0167] In a possible implementation, the evaluating the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options includes:

[0168] For each pre-saved evaluation capability, determine that the capability corresponding to each sample question is each target sample question of the evaluation capability; determine the score for evaluating the evaluation capability of the large model based on whether the output options corresponding to each target sample question are consistent with the standard options, as well as the pre-saved consistent and inconsistent scores.

[0169] In a possible implementation, the evaluating the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options includes:

[0170] For each sample question, determine the score of the large model based on the sample question according to whether the output option corresponding to the sample question is consistent with the standard option, and the pre-saved consistent and inconsistent scores; and obtain the difficulty level saved for the sample question, and determine the target weight corresponding to the difficulty level of the sample question according to the correspondence between the saved level and the weight;

[0171] According to the score of the large model given by each sample question and the target weight corresponding to each sample question, a target score for evaluating the ability of the large model is determined.

[0172] In a possible implementation, the sample question set is constructed in the following manner:

[0173] Receive a build instruction carrying the fields to be built;

[0174] Acquire a knowledge document in the knowledge base of the field to be constructed; send the knowledge document to a preset device, and the knowledge document is used to determine a sample problem;

[0175] Receive each sample question returned by the preset device, add each sample question to a sample question set, and save it.

[0176] In a possible implementation manner, the receiving each constructed sample question returned by the preset device includes:

[0177] receiving each constructed sample question returned by the preset device and the evaluation capability saved for each sample question;

[0178] After receiving each constructed sample question returned by the preset device and before adding each sample question to the sample question set, the method further includes:

[0179] Determine whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced; if so, execute the subsequent step of adding each sample question to the sample question set.

[0180] In a possible implementation manner, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0181] Determine whether there are separators between the options corresponding to each candidate answer contained in each received sample question. If so, perform the subsequent step of adding each sample question to the sample question set.

[0182] In a possible implementation manner, after receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes:

[0183] For each sample question received, determine whether each sample question contains any pre-saved target information based on each pre-saved target information, wherein each target information is used to prompt the large model to output an option from the options corresponding to each answer contained in the sample question; if so, execute the subsequent step of adding each sample question to the sample question set.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A large model evaluation method, characterized in that: The method comprises: Obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question; The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options.

2. The method according to claim 1, characterized in that The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options, including: For each pre-saved evaluation capability, determine the corresponding capability in each sample question as each target sample question of the evaluation capability; determine the score for evaluating the evaluation capability of the large model based on whether the output options corresponding to each target sample question are consistent with the standard options, as well as the pre-saved consistent and inconsistent scores.

3. The method according to claim 1 or 2, characterized in that: The capability of the large model is evaluated based on whether the output options corresponding to each sample question are consistent with the standard options, including: For each sample question, determine the score of the large model based on the sample question according to whether the output option corresponding to the sample question is consistent with the standard option, and the pre-saved consistent and inconsistent scores; and obtain the difficulty level saved for the sample question, and determine the target weight corresponding to the difficulty level of the sample question according to the correspondence between the saved level and the weight; According to the score of the large model given by each sample question and the target weight corresponding to each sample question, a target score for evaluating the ability of the large model is determined.

4. The method according to claim 1, characterized in that: The sample problem set is constructed in the following way: Receive a build instruction carrying the fields to be built; Acquire a knowledge document in the knowledge base of the field to be constructed; send the knowledge document to a preset device, and the knowledge document is used to determine a sample problem; Receive each sample question returned by the preset device, add each sample question to a sample question set, and save it.

5. The method according to claim 4, characterized in that The receiving each sample problem returned by the preset device after the construction is completed includes: receiving each constructed sample question returned by the preset device and the evaluation capability saved for each sample question; After receiving each constructed sample question returned by the preset device and before adding each sample question to the sample question set, the method further includes: Determine whether the evaluation capability corresponding to each received sample question covers each pre-saved capability, and whether the number of sample questions corresponding to each evaluation capability in each sample question is balanced; if so, execute the subsequent step of adding each sample question to the sample question set.

6. The method according to claim 4, characterized in that After receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes: Determine whether there are separators between the options corresponding to each candidate answer contained in each received sample question. If so, perform the subsequent step of adding each sample question to the sample question set.

7. The method according to claim 4, characterized in that After receiving each constructed sample question and before adding each sample question to the sample question set, the method further includes: For each sample question received, determine whether each sample question contains any pre-saved target information based on each pre-saved target information, wherein each target information is used to prompt the large model to output an option from the options corresponding to each answer contained in the sample question; if so, execute the subsequent step of adding each sample question to the sample question set.

8. A large model evaluation method, characterized in that: The method comprises: A processing module is used to obtain any sample question of the pre-constructed sample question set, and input the sample question into the large model to be evaluated; wherein the sample question carries at least two candidate answers and options corresponding to the candidate answers, so that the large model determines an output option from the options corresponding to the at least two candidate answers; obtain the output option output by the large model, and obtain the standard option saved for the sample question; The evaluation module is used to evaluate the capability of the large model based on whether the output options corresponding to each sample question are consistent with the standard options.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for large model evaluation according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the large model evaluation method according to any one of claims 1 to 7.

Images