Model evaluation methods, devices, equipment and media

Abstract

The disclosed embodiments relate to a model evaluation method, apparatus, device and medium, the method comprising: determining a candidate test set group; wherein the candidate test set group corresponds to a test field one-to-one, and the candidate test set group includes multiple candidate test sets of different difficulty levels; testing a model to be tested based on multiple candidate test sets to obtain a test result; determining the test result that matches the difficulty of the model to be tested as the target test result, and using the candidate test set corresponding to the target test result as the target test set; and determining the evaluation result that matches the corresponding difficulty level of the model to be tested in the test field according to the difficulty level corresponding to the target test set. The disclosed embodiments avoid the influence of the uncertainty of the model's answer to a single question on the evaluation result, and improve the accuracy of the final evaluation result from multiple aspects such as the pertinence of the field and the stability of the evaluation.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a model evaluation method, device, equipment and medium. Background Art

[0002] The performance of the model can be determined by evaluating the model through the test set. However, the model's answers are random. For the same question, multiple answers may produce different results. Sometimes the answer is correct, and sometimes it is wrong, which will lead to inaccurate evaluation results of the model. How to improve the accuracy of model evaluation is an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a model evaluation method, device, equipment and medium.

[0004] The present disclosure provides a model evaluation method, including:

[0005] Determine a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels;

[0006] Testing the model to be tested based on the multiple candidate test sets to obtain a test result;

[0007] Determine the test result that matches the difficulty of the model to be tested as the target test result, and use the candidate test set corresponding to the target test result as the target test set;

[0008] According to the difficulty level corresponding to the target test set, an evaluation result of the model to be tested matching the corresponding difficulty level in the test domain is determined.

[0009] The present disclosure also provides a model evaluation device, including:

[0010] A first determination module is used to determine a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels;

[0011] A testing module, used to test the model to be tested based on the multiple candidate test sets to obtain a test result;

[0012] A second determination module is used to determine the test result that matches the difficulty of the model to be tested as the target test result, and to use the candidate test set corresponding to the target test result as the target test set;

[0013] The third determination module is used to determine, according to the difficulty level corresponding to the target test set, an evaluation result of whether the model to be tested matches the corresponding difficulty level in the test domain.

[0014] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the model evaluation method provided in the embodiment of the present disclosure.

[0015] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the model evaluation method provided by the embodiment of the present disclosure.

[0016] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages: the model evaluation solution provided in the embodiments of the present disclosure, the method includes: determining a candidate test set group; wherein the candidate test set group corresponds to the test field one-to-one, and the candidate test set group includes multiple candidate test sets of different difficulty levels; testing the model to be tested based on the multiple candidate test sets to obtain test results; determining the test result that matches the difficulty of the model to be tested as the target test result, and using the candidate test set corresponding to the target test result as the target test set; according to the difficulty level corresponding to the target test set, determining the evaluation result of the model to be tested that matches the corresponding difficulty level under the test field. By adopting the above technical scheme, a corresponding candidate test set group is set up for the test field, and the candidate test set group includes multiple candidate test sets with different difficulty levels. The evaluation result of the model to be tested in the corresponding test field is determined according to the candidate test set that matches the difficulty of the model to be tested. The test field dimension is refined, and the evaluation result of the model to be tested is determined based on the overall performance of the model to be tested in the candidate test set corresponding to the difficulty level, thereby avoiding the influence of the uncertainty of the model's answer to a single question on the evaluation result, and improving the accuracy of the final evaluation result from multiple aspects such as field specificity and evaluation stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1A flowchart of a model evaluation method provided in an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of another model evaluation method provided by an embodiment of the present disclosure;

[0021] Figure 3 A flowchart of another model evaluation method provided in an embodiment of the present disclosure;

[0022] Figure 4 A flowchart of another model evaluation method provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of a model evaluation device provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0027] In order to solve the above problems, the embodiments of the present disclosure provide a model evaluation method, which is introduced below in conjunction with specific embodiments.

[0028] Figure 1 The present invention provides a flow chart of a model evaluation method provided by an embodiment of the present invention. The model evaluation method can be applied to a model evaluation device. The model evaluation device can be implemented by software and / or hardware. The model evaluation device can generally be integrated in an electronic device. Figure 1 As shown, the model evaluation method includes:

[0029] Step 101, determine a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels.

[0030] Among them, the number of candidate test set groups can be one or more. When the model to be tested is evaluated in multiple test fields, there can be a corresponding candidate test set group for each test field. The model to be tested can be a neural network model to be evaluated for performance testing. This embodiment does not limit the type of the model to be tested. For example, the model to be tested can be a large language model. This embodiment does not limit the language type of the data processed by the model to be tested. For example, the language type can include Chinese. The test field can be a specific field range for testing the model to be tested. This embodiment does not limit the test field. For example, the test field can include one or more of Chinese, mathematics, English, physics, chemistry, medicine, and architecture. The difficulty level can characterize the difficulty of answering the candidate questions in the candidate test set. It can be understood that when the performance of the model to be tested remains unchanged, the greater the difficulty level, the smaller the probability that the model to be tested will answer correctly. This embodiment does not limit the difficulty level. For example, the difficulty level can include simple, medium, difficult, etc., or the difficulty level can include one or more of elementary school, junior high school, high school, university, and major.

[0031] In some embodiments of the present disclosure, the candidate test set group may be pre-set, and the model evaluation device obtains the pre-set candidate test set.

[0032] In some embodiments of the present disclosure, determining a candidate test set group includes: determining difficulty level labels and test field labels of candidate questions; grouping candidate questions with the same difficulty level labels and the same test field labels into the same candidate test set; and grouping candidate test sets corresponding to the same test field labels into the same candidate test set group.

[0033] The candidate questions may be questions constituting a candidate test set, and the candidate questions may be candidate questions for testing the model to be tested. This embodiment does not limit the source of the candidate questions. For example, the candidate questions may include entrance examination questions, stage test questions, common sense questions, brain teasers, etc. The difficulty level label may be a label for marking the difficulty level of the candidate questions. The test field label may be a label for marking the test field of the candidate questions.

[0034] In this embodiment, the candidate questions in the question bank are manually labeled with difficulty level labels and test field labels. The model evaluation device obtains the candidate questions labeled with difficulty level labels and test field labels, and divides the candidate questions with the same difficulty level labels and the same test field labels into the same candidate test set. The difficulty level recorded in the difficulty level label of the candidate question in the candidate test set is used as the difficulty level of the candidate test set, and the test field recorded in the test field label of the candidate question in the candidate test set is used as the test field of the candidate test set. Candidate test sets with the same test field are divided into the same candidate test set group.

[0035] In the above solution, the candidate test sets and candidate test set groups are automatically generated according to the difficulty level labels and test field labels, which lays a foundation for subsequent model evaluation.

[0036] In some embodiments of the present disclosure, determining a candidate test set group includes: determining the difficulty level label, test field label, and question type label of the candidate questions; wherein the question type label includes an objective question label and a subjective question label; classifying the candidate questions with the same difficulty level label, the same test field label, and the same question type label into the same candidate test set; and classifying the candidate test sets corresponding to the same test field label and the same question type label into the same candidate test set group. Thus, candidate questions of different question types are classified into different candidate test sets, achieving a more fine-grained classification of questions.

[0037] Step 102: Test the model to be tested based on multiple candidate test sets to obtain test results.

[0038] The test result may be a result obtained by testing the model to be tested on one or more candidate test sets, and the test result corresponds to the candidate test set. This embodiment does not limit the type of the test result, for example, the test result may be the accuracy.

[0039] In an embodiment of the present disclosure, after determining multiple candidate test sets in the candidate test set group, the model evaluation device can apply some or all of the multiple candidate test sets to test the model to be tested to obtain the test results of the model to be tested on the corresponding candidate test sets.

[0040] In an optional implementation, the model evaluation device can select a candidate test set from multiple candidate test sets as the first candidate test set, apply the first candidate test set to test the model to be tested, and obtain the corresponding test result. Subsequently, if the test set result is between the first threshold and the second threshold, the first candidate test set is used as the target test set. If the test set result is not between the first threshold and the second threshold, the next candidate test set to test the model to be tested is determined based on the test result, and the difficulty level of the candidate test set is iterated cyclically based on the candidate test set until a target test set that matches the difficulty of the model to be tested is determined.

[0041] Figure 2 A flow chart of another model evaluation method provided in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, in some embodiments of the present disclosure, before testing the model to be tested based on the multiple candidate test sets to obtain the test results, the model evaluation method further includes:

[0042] Step 201 , obtaining historical judgment results of the model to be tested for historical questions in the test field, and determining the basic level of the model to be tested according to the historical judgment results.

[0043] Among them, the historical questions can be questions that the model to be tested has completed answering within the historical time period, and the historical time period can be a time period before the current test of the model to be tested. The historical judgment result can be the correctness judgment result of the historical question, and the historical judgment result can include correct and incorrect. The basic level can be the basic difficulty level based on which the difficulty matching of the model to be tested is subsequently performed, and the basic level can be understood as the difficulty level matched by the model to be tested based on the prediction of the historical judgment result.

[0044] In this embodiment, the model evaluation device can obtain the historical questions of the model to be tested in the same field of the candidate test set group, as well as the historical judgment results of the historical questions. And the historical questions are labeled with difficulty level labels. Under each difficulty level label, the correct proportion of the historical judgment results corresponding to the historical questions (i.e., the correct rate) is counted, and the difficulty level with a correct rate greater than the second threshold is determined as the candidate difficulty level, and the maximum difficulty level in the candidate difficulty level is determined as the basic level of the model to be tested.

[0045] For example, if the accuracy rate corresponding to history questions with a medium difficulty level is greater than the second threshold, and the accuracy rate corresponding to history questions with a difficult difficulty level is not greater than the second threshold, the basic level of the model to be tested in the test field is determined to be medium.

[0046] Step 202: Determine, among multiple candidate test sets, a candidate test set whose difficulty level is consistent with the basic level as the first candidate test set; wherein the first candidate test set is the first candidate test set to test the model to be tested.

[0047] In this embodiment, after determining the basic level of the model to be tested, the model evaluation device can determine a candidate test set whose corresponding difficulty level is the same as the basic level, and determine the candidate test set as the first candidate test set. In the subsequent process of testing and evaluating the model to be tested, the first candidate test set is first used to test and evaluate the model to be tested.

[0048] In the above scheme, the first candidate test set for testing and evaluating the model to be tested is determined based on the historical judgment results of the model to be tested, and the potential range of the evaluation results of the model to be tested is determined more accurately. The difficulty level is iterated based on the first candidate test set, which reduces the computing resources consumed for evaluating the model to be tested.

[0049] Step 103: determine the test result that matches the difficulty of the model to be tested as the target test result, and use the candidate test set corresponding to the target test result as the target test set.

[0050] The target test set may be a candidate test set with the highest difficulty level that matches the difficulty of the model to be tested.

[0051] In this embodiment, after the test results of the candidate test sets are determined, the candidate test sets whose test results are greater than the second threshold and whose difficulty level is the highest are used as the target test sets.

[0052] Optionally, if the model to be tested is tested based on the multiple candidate test sets to obtain test results, including: testing the model to be tested according to the multiple candidate test sets to obtain multiple test results corresponding to the multiple candidate test sets, wherein the candidate test sets correspond to the test results one-to-one. That is, all candidate test sets have been used to test the model to be tested. Correspondingly, the test result that matches the difficulty of the model to be tested is determined as the target test result, and the candidate test set corresponding to the target test result is used as the target test set, including: determining the test result that matches the difficulty of the model to be tested in multiple candidate test sets, determining the test result as the target test result, and using the candidate test set corresponding to the target test set result as the target test set.

[0053] In some embodiments of the present disclosure, the test result is the test result of the current test set among multiple candidate test sets, wherein the current test set may be a candidate test set that is currently testing and evaluating the model to be tested, and the current test set may be one of the multiple candidate test sets. That is, each time a candidate test set is used to test the model to be tested, a test result corresponding to the candidate test set is obtained. If the test result is subsequently between the first threshold and the second threshold, the test result is determined as the target test result. If the test result is not between the first threshold and the second threshold, the difficulty level of the candidate test set for testing the model to be tested is iterated cyclically based on the test result until the target test result is determined.

[0054] Accordingly, the test result that matches the difficulty of the model to be tested is determined as the target test result, and the candidate test set corresponding to the target test result is used as the target test set, including:

[0055] If the test result is not less than the first threshold, the candidate test set with a difficulty level greater than the current test set is determined as the new current test set, and the calculation of the new test result is returned; if the test result is not greater than the second threshold, the candidate test set with a difficulty level less than the current test set is determined as the new current test set, and the calculation of the new test result is returned; wherein the first threshold is greater than the second threshold; if the test result is between the first threshold and the second threshold, the test result is determined as the target test result, and the current test set is determined as the target test set.

[0056] The first threshold may be a pre-set threshold for the test result of upgrading the difficulty level of the current test set. This embodiment does not limit the first threshold, for example, the first threshold may be 90% or 70%. The second threshold may be a pre-set threshold for the test result of reducing the difficulty level of the current test set. This embodiment does not limit the second threshold, for example, the first threshold may be 50%.

[0057] In this embodiment, it is determined whether the test result is between the first threshold and the second threshold. If the test result is not less than the first threshold, it means that the difficulty level of the current test set is low, and the candidate test set whose difficulty level is adjacent to the current test set and greater than the current test set is determined as the new current test set. Further, return to step 102 to test the model to be tested according to the new current test set, obtain a new test result, and determine whether the new test result is between the first threshold and the second threshold.

[0058] Furthermore, if the test result is not greater than the second threshold, indicating that the difficulty level of the current test set is too high, a candidate test set whose difficulty level is adjacent to the current test set and less than the current test set is determined as a new current test set. Further, returning to step 102, the model to be tested is tested according to the new current test set to obtain a new test result, and determining whether the new test result is between the first threshold and the second threshold.

[0059] Furthermore, if the test result is between the first threshold and the second threshold, it means that the difficulty level of the current test set matches the model to be tested, and the current test set is determined as the target test set, and the subsequent step 104 is executed.

[0060] For example, after determining the first candidate test set among multiple candidate test sets whose difficulty level is closest to the performance of the model to be tested. Use the first candidate test set as the current test set to test the model to be tested to obtain the accuracy. If the accuracy is not less than the first threshold, determine the candidate test set with a difficulty level greater than the current test set as the new current test set, and determine the new accuracy to determine whether the new accuracy is between the first threshold and the second threshold. If the accuracy is not greater than the second threshold, determine the candidate test set with a difficulty level less than the current test set as the new current test set, and determine the new accuracy to determine whether the new accuracy is between the first threshold and the second threshold. If the accuracy is between the first threshold and the second threshold, determine the current test set as the target test set.

[0061] In the above scheme, for different test fields, the difficulty level of the current test set is dynamically adjusted according to the performance of the model to be tested in the corresponding test field, so that the corresponding test set is tailored for the model to be tested according to the performance of the model to be tested. By looping through steps 102 and 103, the model to be tested is tested with candidate test sets in a targeted manner, and the target test set matched by the model to be tested is efficiently determined. Compared with the method of testing the model to be tested on a full set of questions, the embodiment of the present disclosure can use fewer questions to more accurately evaluate the performance of the model to be tested.

[0062] Step 104, according to the difficulty level corresponding to the target test set, determine the evaluation result of the model to be tested that matches the corresponding difficulty level in the test domain.

[0063] Among them, the evaluation results can characterize the performance of the model to be tested in the test field.

[0064] In this embodiment, the model evaluation device can determine the difficulty level of the target test set under the test field, and use the difficulty level as the evaluation result of the model to be tested. Alternatively, according to the difficulty level, a query is performed in a pre-set difficulty result relationship to obtain the evaluation result corresponding to the difficulty level, and the evaluation result is used as the evaluation result of the model to be tested; wherein the difficulty result relationship is used to record the corresponding relationship between the difficulty level for the test set and the evaluation result for the model.

[0065] Optionally, in some embodiments of the present disclosure, if the same candidate test set group includes candidate test sets corresponding to the same test domain label and the same question type label, then according to the difficulty level corresponding to the target test set, determining the evaluation result of the model to be tested matching the corresponding difficulty level under the test domain includes:

[0066] For the target test set marked with the same test domain label, obtain the objective question difficulty level of the target test set marked with the objective question label; obtain the subjective question difficulty level of the target test set marked with the subjective question label. Perform a weighted sum of the objective difficulty level and the subjective difficulty level to obtain the evaluation result of the model to be tested in the test domain.

[0067] The model evaluation method provided by the embodiment of the present disclosure includes: determining a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes multiple candidate test sets of different difficulty levels; testing the model to be tested based on the multiple candidate test sets to obtain the test result; determining the test result matching the difficulty of the model to be tested as the target test result, and using the candidate test set corresponding to the target test result as the target test set; and determining the evaluation result of the model to be tested that matches the corresponding difficulty level under the test field according to the difficulty level corresponding to the target test set. By adopting the above technical scheme, a corresponding candidate test set group is set for the test field, and the candidate test set group includes multiple candidate test sets of different difficulty levels. The evaluation result of the model to be tested in the corresponding test field is determined according to the candidate test set matching the difficulty of the model to be tested, and the test field dimension is refined, and the evaluation result of the model to be tested is determined based on the overall performance of the candidate test set corresponding to the difficulty level of the model to be tested, thereby avoiding the influence of the uncertainty of the model's answer to a single question on the evaluation result, and improving the accuracy of the final evaluation result from multiple aspects such as the pertinence of the field and the stability of the evaluation.

[0068] In the related art, the test set for testing the model to be tested includes multiple-choice questions with static data. The model to be tested predicts the results of the multiple-choice questions. For each multiple-choice question, the model to be tested is required to select an option. However, due to the uncertainty of the answer of the model to be tested, it may obtain an answer different from the available options, resulting in failure to extract the answer, and it is judged that the answer of the model to be tested is incorrect, which affects the evaluation result of the model to be tested.

[0069] Figure 3 A flow chart of another model evaluation method provided in the embodiment of the present disclosure is as follows: Figure 3 As shown, in some embodiments of the present disclosure, the model to be tested is tested based on multiple candidate test sets to obtain test results, including:

[0070] Step 301, input the first current question into the model to be tested to obtain an initial answer result; wherein the first current question is an objective question in the current test set, and the current test set is a candidate test set currently testing the model to be tested.

[0071] Among them, the initial answer result can be the answer result obtained by the model to be tested when answering the first current question for the first time.

[0072] In this embodiment, the question currently answered by the model to be tested in the current test set is the first current question, and the first current question is an objective question. The model evaluation device can input the first current question into the model to be tested, and the model to be tested answers the first current question to obtain an initial answer result output by the model to be tested.

[0073] Step 302: If the data type of the initial answer result is not an option type, then obtain the background knowledge text associated with the initial answer result and the stem of the first current question.

[0074] The option type may be a multiple-choice question option type. The background knowledge text may be background knowledge associated with the initial answer result under the question stem dimension of the first current question.

[0075] In this embodiment, the model evaluation device can determine whether the data type of the initial answer result is an option type. If not, the initial answer result is fed back to the user. The user searches the initial answer result for background knowledge in the question stem dimension of the first current question to obtain the background knowledge text, and inputs the background knowledge text into the model evaluation device, and the model evaluation device obtains the background knowledge text. Alternatively, the model evaluation device can perform semantic analysis on the initial answer result, extract the object text in the initial answer result, and perform semantic analysis on the question stem of the first current question, extract the background text in the question stem, and perform retrieval processing based on the object text and the background text to obtain the background knowledge text.

[0076] Step 303: input the background knowledge text and the first current question into the model to be tested to obtain an adjusted answer result.

[0077] Among them, the adjusted answer result can be the answer result obtained by the model to be tested re-answering the first current question in combination with the background knowledge text.

[0078] In this embodiment, the model evaluation device can input the background knowledge text and the first current question as input data into the model to be tested. The model to be tested answers the first current question in combination with the background knowledge text to obtain an adjusted answer result output by the model to be tested.

[0079] In some embodiments of the present disclosure, inputting the background knowledge text and the first current question into the model to be tested includes: dividing the background knowledge text into multiple text blocks, converting the segmented text blocks into multiple vectors and storing them in a vector database through an embedding model, and establishing a corresponding relationship between the vector database and the first current question. When the model to be tested subsequently answers the first current question, the model to be tested can query the vector corresponding to the background knowledge text from the vector database and return the answer result. Therefore, the model to be tested combines the background knowledge text when answering the first current question, thereby improving the accuracy of the answer result.

[0080] Step 304: If the adjusted answer result is consistent with the standard option of the first current question, determine that the first judgment result of the first current question is passed, and determine the test result according to the first judgment result.

[0081] The standard option may be a standard answer to the first current question. The first judgment result may be a judgment result of the correctness of the answer to the first current question.

[0082] In this embodiment, it is determined whether the type of the adjusted answer result is an option type. If so, it is determined whether the adjusted answer result is consistent with the standard option of the first current question. If so, it is determined that the first judgment result of the first current question is passed, and the accuracy rate is statistically analyzed based on the first judgment result to obtain the test result.

[0083] In the above scheme, when the model to be tested answers objective questions but does not output options, the initial answer result output by the model to be tested and the background knowledge related to the question stem are obtained, and the model to be tested is made to answer the question for the second time in combination with the background knowledge to obtain an adjusted answer result, thereby reducing the randomness of the answer to the question by the model to be tested, and determining the correctness of the answer result generated by the model to be tested when text search can be performed, so that the final test result is more accurate.

[0084] In some embodiments of the present disclosure, the model evaluation method also includes: if the data type of the adjusted answer result is not an option type, determining a first judgment result of the first current question based on the text similarity between the adjusted answer result and the standard text corresponding to the standard option.

[0085] The text similarity may be the semantic similarity between texts.

[0086] In this embodiment, it is determined whether the type of the adjusted answer result is an option type. If not, it means that the model to be tested still does not output the answer options of the first current question as required, then the standard text under the standard option is obtained, and the similarity between the adjusted answer result and the standard text is calculated. If the similarity is greater than the similarity threshold, the first judgment result of the first current question is determined to be passed. If the similarity is not greater than the similarity threshold, the first judgment result of the first current question is determined to be failed.

[0087] In the above scheme, the adjusted answer result calculated for text similarity with the standard text is the adjusted answer result determined by the model to be tested in combination with background knowledge. The second judgment result represents the text correctness of the answer result generated by the model to be tested when text search is possible, making the final test result more accurate.

[0088] In the related art, the test set for testing the model to be tested includes subjective questions of open-domain question answering. The subjective questions are written manually, and the subjective questions can be subjective questions in the form of multiple rounds of question answering. Through the test of subjective questions, the question-answering ability of the model to be tested in real scenarios, the production ability of open-ended questions, the ability to follow instructions and multiple rounds of interaction, etc. can be determined. In the test based on subjective questions, the answers of the model to be tested are scored by the model with better performance, but lengthy answers will get higher scores when the model is scored, resulting in lower fairness and accuracy of the evaluation.

[0089] Figure 4 A flow chart of another model evaluation method provided in the embodiment of the present disclosure is as follows: Figure 4 As shown, in some embodiments of the present disclosure, the model to be tested is tested based on multiple candidate test sets to obtain test results, including:

[0090] Step 401, input the second current question into the model to be tested to obtain the answer result of the subjective question; wherein the second current question is a subjective question in the current test set.

[0091] The subjective question answer result may be the result of the model to be tested answering the second current question. The subjective question may be a question of unstructured data type, where unstructured data refers to data without a predefined data model or format, and the unstructured data may include one or more of natural language text, images, audio, and video.

[0092] In this embodiment, the question currently answered by the model to be tested in the current test set is the second current question, and the second current question is a subjective question. The model evaluation device can input the second current question into the model to be tested, and the model to be tested answers the second current question, and the model evaluation device obtains the subjective question answer result output by the model to be tested.

[0093] Step 402, determine the second judgment result of the second current question based on the text similarity between the answer result of the subjective question and the reference answer result, and determine the test result based on the second judgment result; wherein the reference answer result is the answer result of the domain fine-tuning model of the test domain for the second current question.

[0094] The domain fine-tuning model may be a model obtained by training a preset model whose performance exceeds a preset performance threshold in a test domain. The preset model may be a super model with good performance. For example, if the test domain is chemistry, the domain fine-tuning model may be a model obtained by training the preset model in the chemistry domain.

[0095] In this embodiment, the model evaluation device can input the second current question into the domain fine-tuning model to obtain the reference answer result output by the domain fine-tuning model. And calculate the text similarity between the subjective answer result and the reference answer result. If the text similarity is greater than the similarity threshold, the second judgment result of the second current question is determined to be passed. If the text similarity is not greater than the similarity threshold, the second judgment result of the second current question is determined to be failed.

[0096] In the above scheme, the model with better performance is avoided from directly scoring the answers output by the model to be tested. Instead, the model with better performance is trained in the test domain to obtain a domain fine-tuned model, and the text similarity between the domain fine-tuned model and the answer results output by the model to be tested for the same subjective question is compared to obtain the corresponding judgment result, thereby improving the fairness and accuracy of the judgment result.

[0097] Optionally, for the second current question for which a human answer exists, the model evaluation device may determine a second judgment result of the second current question based on the text similarity between the subjective question answer result and the human answer result, and determine the test result based on the second judgment result.

[0098] In some embodiments of the present disclosure, the model to be tested includes a first model to be tested and a second model to be tested having the same evaluation results in the same test field. In order to determine the target model to be tested with better performance between the first model to be tested and the second model to be tested, the model evaluation device can obtain the discrimination question corresponding to the evaluation result in the test field, obtain the first discrimination answer result of the first model to be tested to the discrimination question, and obtain the second discrimination answer result of the second model to be tested to the discrimination question. Determine the target discrimination answer result with the highest score between the first discrimination answer result and the second discrimination answer result, and determine the model to be tested corresponding to the target discrimination answer result as the target model to be tested. Among them, the discrimination question can be a question used to distinguish the performance of the model, and a question for which a small performance difference of the model for the discrimination question will cause a large change in the judgment result.

[0099] The model evaluation method provided by the embodiment of the present disclosure addresses the problem of randomness in the answers of the model to be tested. A target test set that matches the difficulty of the model to be tested is determined in an adaptive manner, and the evaluation result of the model to be tested is determined based on the overall performance of the model to be tested on the target test set, thereby improving the accuracy of model evaluation. In addition, there is no need to test the model to be tested on all questions, and the ability of the model to be tested can be more accurately estimated with fewer questions, which reduces the time cost consumed in data processing and improves the efficiency of model evaluation.

[0100] In addition, corresponding judgment result determination processes are set up for objective questions and subjective questions respectively, which further optimizes the inaccurate test results caused by the instability and randomness of the output of the model to be tested during the model evaluation process, greatly improving the fairness and reliability of the final test results.

[0101] Figure 5 This is a schematic diagram of the structure of a model evaluation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and the device can also be integrated in an electronic device. Figure 5 As shown, the model evaluation device comprises:

[0102] A first determination module 501 is used to determine a candidate test set group; wherein the candidate test set group corresponds to a test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels;

[0103] A testing module 502 is used to test the model to be tested based on the multiple candidate test sets to obtain a test result;

[0104] A second determination module 503 is used to determine the test result that matches the difficulty of the model to be tested as a target test result, and to use the candidate test set corresponding to the target test result as a target test set;

[0105] The third determination module 504 is used to determine, according to the difficulty level corresponding to the target test set, an evaluation result of whether the model to be tested matches the corresponding difficulty level in the test domain.

[0106] Optionally, the first determining module 501 is configured to:

[0107] Determine the difficulty level label and test area label of the candidate questions;

[0108] Classify candidate questions with the same difficulty level label and the same test field label into the same candidate test set;

[0109] The candidate test sets corresponding to the same test field label are divided into the same candidate test set group.

[0110] Optionally, the testing module 502 is used to:

[0111] Inputting a first current question into the model to be tested to obtain an initial answer result; wherein the first current question is an objective question in a current test set, and the current test set is the candidate test set currently testing the model to be tested;

[0112] If the data type of the initial answer result is not an option type, obtaining background knowledge text associated with the initial answer result and the stem of the first current question;

[0113] Inputting the background knowledge text and the first current question into the model to be tested to obtain an adjusted answer result;

[0114] If the adjusted answer result is consistent with the standard option of the first current question, the first judgment result of the first current question is determined to be passed, and the test result is determined according to the first judgment result.

[0115] Optionally, the model evaluation device further comprises:

[0116] The fourth determination module is used to determine the first judgment result of the first current question according to the text similarity between the adjusted answer result and the standard text corresponding to the standard option if the data type of the adjusted answer result is not an option type.

[0117] Optionally, the testing module 502 is used to:

[0118] Inputting a second current question into the model to be tested to obtain a subjective question answer result; wherein the second current question is a subjective question in a current test set, and the current test set is the candidate test set currently testing the model to be tested;

[0119] According to the text similarity between the answer result of the subjective question and the reference answer result, a second judgment result of the second current question is determined, and the test result is determined according to the second judgment result; wherein the reference answer result is the answer result of the domain fine-tuning model of the test domain for the second current question.

[0120] Optionally, the test result is a test result of a current test set among the multiple candidate test sets. Accordingly, the second determining module 503 is configured to:

[0121] If the test result is not less than the first threshold, determining the candidate test set with a difficulty level greater than the current test set as a new current test set, and returning to calculate a new test result;

[0122] If the test result is not greater than a second threshold, determining the candidate test set with a difficulty level less than the current test set as a new current test set, and returning to calculate a new test result; wherein the first threshold is greater than the second threshold;

[0123] If the test result is between the first threshold and the second threshold, the test result is determined as the target test result, and the current test set is determined as the target test set.

[0124] Optionally, the model evaluation device further comprises:

[0125] An acquisition module, used for acquiring historical judgment results of the model to be tested for historical questions in the test field, and determining the basic level of the model to be tested according to the historical judgment results;

[0126] The fifth determination module is used to determine, among the multiple candidate test sets, the candidate test set whose difficulty level is consistent with the basic level as the first candidate test set; wherein the first candidate test set is the first candidate test set to test the model to be tested.

[0127] The model evaluation device provided in the embodiments of the present disclosure can execute the model evaluation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Figure 6 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and a memory 602 .

[0129] The processor 601 may be a central processing unit (CPU) or other forms of processing units having a model evaluation capability and / or an instruction execution capability, and may control other components in the electronic device 600 to perform desired functions.

[0130] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may run the program instructions to implement the model evaluation method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0131] In one example, the electronic device 600 may further include: an input device 603 and an output device 604 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0132] In addition, the input device 603 may also include, for example, a keyboard, a mouse, etc.

[0133] The output device 604 can output various information to the outside, including the determined distance information, direction information, etc. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0134] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device 600 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 600 may also include any other appropriate components.

[0135] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the model evaluation method provided by the embodiment of the present disclosure.

[0136] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0137] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the model evaluation method provided by the embodiment of the present disclosure.

[0138] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0139] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0140] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model evaluation method, characterized in that: include: Determine a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels; wherein the difficulty level represents the difficulty of answering the candidate questions in the candidate test set; The model to be tested is tested based on the multiple candidate test sets to obtain a test result; wherein the test result corresponds to the candidate test set, and the test result is the accuracy; Determine the test result that matches the difficulty of the model to be tested as the target test result, and use the candidate test set corresponding to the target test result as the target test set; According to the difficulty level corresponding to the target test set, determine an evaluation result of the model to be tested matching the corresponding difficulty level in the test domain; wherein the evaluation result represents the performance of the model to be tested in the test domain; The step of determining a candidate test set group includes: Determine the difficulty level label and test field label of the candidate questions; wherein the test field label is used to mark the test field of the candidate questions; classify the candidate questions with the same difficulty level label and the same test field label into the same candidate test set; classify the candidate test sets corresponding to the same test field label into the same candidate test set group; The step of testing the model to be tested based on the multiple candidate test sets to obtain the test results includes: Inputting a first current question into the model to be tested to obtain an initial answer result; wherein the first current question is an objective question in a current test set, and the current test set is the candidate test set currently testing the model to be tested; Determine whether the data type of the initial answer result is an option type. If the data type of the initial answer result is not an option type, obtain the background knowledge text associated with the initial answer result and the stem of the first current question; wherein, obtaining the background knowledge text associated with the initial answer result and the stem of the first current question includes: feeding back the initial answer result to the user, and obtaining the background knowledge text input by the user; or, performing semantic analysis on the initial answer result, extracting the object text in the initial answer result, performing semantic analysis on the stem of the first current question, extracting the background text in the stem, and performing retrieval processing based on the object text and the background text to obtain the background knowledge text; Inputting the background knowledge text and the first current question into the model to be tested to obtain an adjusted answer result; If the adjusted answer result is consistent with the standard option of the first current question, the first judgment result of the first current question is determined to be passed, and the test result is determined according to the first judgment result.

2. The method according to claim 1, characterized in that The method further comprises: If the data type of the adjusted answer result is not an option type, the first judgment result of the first current question is determined according to the text similarity between the adjusted answer result and the standard text corresponding to the standard option.

3. The method according to claim 1, characterized in that The testing of the model to be tested based on the multiple candidate test sets to obtain the test result includes: Inputting the second current question into the model to be tested to obtain a subjective question answer result; wherein the second current question is a subjective question in the current test set; According to the text similarity between the answer result of the subjective question and the reference answer result, a second judgment result of the second current question is determined, and the test result is determined according to the second judgment result; wherein the reference answer result is the answer result of the domain fine-tuning model of the test domain for the second current question.

4. The method according to claim 1, characterized in that: The test result is the test result of the current test set among the multiple candidate test sets. Accordingly, determining the test result that matches the difficulty of the model to be tested as the target test result, and taking the candidate test set corresponding to the target test result as the target test set, includes: If the test result is not less than the first threshold, determining the candidate test set with a difficulty level greater than the current test set as a new current test set, and returning to calculate a new test result; If the test result is not greater than a second threshold, determining the candidate test set with a difficulty level less than the current test set as a new current test set, and returning to calculate a new test result; wherein the first threshold is greater than the second threshold; If the test result is between the first threshold and the second threshold, the test result is determined as the target test result, and the current test set is determined as the target test set.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining historical judgment results of the model to be tested for historical questions in the test field, and determining the basic level of the model to be tested according to the historical judgment results; Among the multiple candidate test sets, the candidate test set whose difficulty level is consistent with the basic level is determined as the first candidate test set; wherein the first candidate test set is the first candidate test set to test the model to be tested.

6. A model evaluation device, characterized in that: include: A first determination module is used to determine a candidate test set group; wherein the candidate test set group corresponds to the test field one by one, and the candidate test set group includes a plurality of candidate test sets of different difficulty levels; wherein the difficulty level represents the difficulty of answering the candidate questions in the candidate test set; A testing module, used to test the model to be tested based on the multiple candidate test sets to obtain a test result; wherein the test result corresponds to the candidate test set, and the test result is a correct rate; A second determination module is used to determine the test result that matches the difficulty of the model to be tested as the target test result, and to use the candidate test set corresponding to the target test result as the target test set; A third determination module is used to determine, according to the difficulty level corresponding to the target test set, an evaluation result of whether the model to be tested matches the corresponding difficulty level in the test domain; wherein the evaluation result represents the performance of the model to be tested in the test domain; The step of determining a candidate test set group includes: Determine the difficulty level label and test field label of the candidate questions; wherein the test field label is used to mark the test field of the candidate questions; classify the candidate questions with the same difficulty level label and the same test field label into the same candidate test set; classify the candidate test sets corresponding to the same test field label into the same candidate test set group; The step of testing the model to be tested based on the multiple candidate test sets to obtain the test results includes: Inputting a first current question into the model to be tested to obtain an initial answer result; wherein the first current question is an objective question in a current test set, and the current test set is the candidate test set currently testing the model to be tested; Determine whether the data type of the initial answer result is an option type. If the data type of the initial answer result is not an option type, obtain the background knowledge text associated with the initial answer result and the stem of the first current question; wherein, obtaining the background knowledge text associated with the initial answer result and the stem of the first current question includes: feeding back the initial answer result to the user, and obtaining the background knowledge text input by the user; or, performing semantic analysis on the initial answer result, extracting the object text in the initial answer result, performing semantic analysis on the stem of the first current question, extracting the background text in the stem, and performing retrieval processing based on the object text and the background text to obtain the background knowledge text; Inputting the background knowledge text and the first current question into the model to be tested to obtain an adjusted answer result; If the adjusted answer result is consistent with the standard option of the first current question, the first judgment result of the first current question is determined to be passed, and the test result is determined according to the first judgment result.

7. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the model evaluation method described in any one of claims 1-5 above.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the model evaluation method described in any one of claims 1 to 5.

Images