Model evaluation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model evaluation method and device, electronic equipment and a storage medium. The electronic equipment can obtain an evaluation test data set; based on the evaluation test data set, the to-be-evaluated model is evaluated through a preset evaluation mode, an evaluation result is obtained, and the preset evaluation mode comprises at least one of a consistency evaluation mode, an understanding ability evaluation mode and a knowledge mastery degree evaluation mode; the consistency evaluation mode is used for evaluating the consistency between prediction answers output by the to-be-evaluated model for the same test data based on different cue words, and the understanding ability evaluation mode is used for evaluating the accuracy of the prediction answers output by the to-be-evaluated model for multiple pieces of rewriting test data; the knowledge mastery degree evaluation mode is used for evaluating the accuracy of prediction answers output by the to-be-evaluated model for the in-depth test data corresponding to the test data. Therefore, the consistency, the understanding ability and the knowledge mastering degree of the large language model can be evaluated.

Description

Technical Field

[0001] The present invention relates to the field of model evaluation technology, and in particular to a model evaluation method, device, electronic equipment and storage medium. Background Art

[0002] In the field of artificial intelligence, especially in natural language processing technology, large language models (LLMs) have become an important tool for interactive dialogue and information generation. Large language models have the ability to understand and generate human language by learning a large amount of sample data. However, current large language models have the following problems:

[0003] First, in the actual human-computer interaction process, the response content and quality of the large language model are not completely autonomous, but are largely guided and influenced by the content input by the user. Second, when answering objective questions with standard answers, the large language model may rely on probabilistic strategies to guess, rather than drawing conclusions based on in-depth understanding and reasoning. Third, in real situations, there is often a problem of data contamination, that is, the test set data is mixed into the training data, resulting in the accuracy of the large language model being overestimated in actual applications. Therefore, a method is needed to accurately evaluate the above-mentioned problems of large language models. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a model evaluation method, device, electronic device and storage medium to evaluate the consistency, comprehension ability and knowledge mastery of a large language model. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a model evaluation method, the method comprising:

[0006] Acquire an evaluation test data set, wherein the evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied;

[0007] Based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method;

[0008] The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, where the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, where the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

[0009] Optionally, the preset evaluation method is the consistency evaluation method, and the test data is a test question;

[0010] The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes:

[0011] Obtain at least one test question, and obtain a plurality of different prompt words corresponding to the test question;

[0012] For the test question, according to a preset combination method, a plurality of different prompt words corresponding to the test question are respectively combined with the test question to obtain a plurality of combination results;

[0013] Inputting the plurality of combined results into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test question included in each combined result based on the indication of the prompt word included in the combined result;

[0014] Based on the differences between the multiple predicted answers corresponding to the test question, a consistency evaluation result corresponding to the model to be evaluated is determined.

[0015] Optionally, the step of obtaining a plurality of different prompt words corresponding to the test question includes:

[0016] Inputting a prompt word generation instruction into a preset large language model, so that the preset large language model generates a plurality of candidate prompt words based on the prompt word generation instruction, wherein the prompt word generation instruction is used to instruct the generation of prompt information of an answer method corresponding to the test question;

[0017] When the candidate prompt word meets the prompt word condition, the candidate prompt word is used as the prompt word corresponding to the test question, wherein the prompt word condition indicates that the candidate prompt word can indicate the answer method corresponding to the test question of the model to be evaluated when it meets the expression method of natural language;

[0018] In the case that the candidate prompt word does not meet the prompt word condition, a modified prompt word generation instruction is obtained, and the process returns to the step of inputting the prompt word generation instruction into the preset large language model until the generated candidate prompt word meets the prompt word condition.

[0019] Optionally, the step of determining the consistency evaluation result corresponding to the model to be evaluated based on the difference between the multiple predicted answers corresponding to the test question includes at least one of the following:

[0020] In the case where the test question is an objective question, based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer, determining the consistency evaluation result corresponding to the model to be evaluated;

[0021] When the test question is a subjective question and there is a standard answer, based on the similarity between the multiple predicted answers corresponding to the test question and the standard answer, determine the consistency evaluation result corresponding to the model to be evaluated;

[0022] When the test question is a subjective question and there is no standard answer, a preset judging model is used to determine the consistency evaluation result corresponding to the model to be evaluated from multiple preset judging dimensions, wherein the preset judging dimensions are used to reflect the degree of match between the predicted answer and the test question.

[0023] Optionally, the step of determining the consistency evaluation result corresponding to the model to be evaluated based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer includes:

[0024] If the multiple predicted answers corresponding to the test question are all consistent with the standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined to be the first preset result; if there is a predicted answer different from the standard answer among the multiple predicted answers corresponding to the test question, the consistency evaluation result corresponding to the model to be evaluated is determined to be the second preset result;

[0025] The step of determining the consistency evaluation result corresponding to the model to be evaluated from multiple preset judgment dimensions by using a preset judgment model includes:

[0026] Inputting the test question and the multiple predicted answers corresponding to the test question into a preset judging model, so that the preset judging model generates a score corresponding to each predicted answer in each preset judging dimension, wherein the preset judging dimension includes at least one of the correctness of the predicted answer, the correlation between the predicted answer and the corresponding test data, the language coherence of the predicted answer, the detail level of the predicted answer, and the logic of the predicted answer;

[0027] For each preset judging dimension, calculate the average value of the score corresponding to each predicted answer in the preset judging dimension; based on the average value corresponding to each preset judging dimension and the preset weight, perform weighted summation on the average values ​​to obtain the consistency evaluation result corresponding to the model to be evaluated.

[0028] Optionally, the consistency assessment method is the comprehension ability assessment method, and the test data is a test question;

[0029] The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes:

[0030] Get at least one test question;

[0031] Inputting the test question into a preset large language model, so that the preset large language model rewrites the test question in a preset rewriting manner to generate a plurality of rewritten test questions, wherein the preset rewriting manner is used to rewrite at least one of the stem, options, and type of the test question;

[0032] Inputting each rewritten test question and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to each rewritten test question;

[0033] Based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question, the comprehension ability evaluation result corresponding to the model to be evaluated is determined.

[0034] Optionally, the preset rewriting method includes at least one of the following:

[0035] If the test question is a single-choice question, rewrite the test question into a multiple-choice question or a true-or-false question;

[0036] If the test question is a multiple-choice question, rewrite the test question into a single-choice question or a true-or-false question;

[0037] Replacing the words included in the test questions with synonyms;

[0038] Without changing the substantial content of the test question stem, rewriting the test question stem from an affirmative sentence to a negative sentence;

[0039] Change the order of the options included in the test question.

[0040] Optionally, the preset evaluation method is the knowledge mastery degree evaluation method;

[0041] The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes:

[0042] Obtain at least one test data;

[0043] Inputting the test data and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test data;

[0044] Inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generates in-depth test data corresponding to the test data;

[0045] Inputting the in-depth test data and its corresponding prompt words into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the in-depth test data;

[0046] By using a preset judging model, based on the most recently outputted predicted answer and the corresponding in-depth test data, determining the judging result of the predicted answer from multiple preset judging dimensions;

[0047] Return to the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model until the number of cycles reaches the preset number, and determine the knowledge mastery evaluation result corresponding to the model to be evaluated based on the judgment result of the predicted answer.

[0048] Optionally, the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data includes:

[0049] Input the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model extracts key information from the test data, and generates a plurality of candidate in-depth test data corresponding to the test data based on the key information;

[0050] A candidate in-depth test data is randomly selected from the plurality of candidate in-depth test data as the in-depth test data corresponding to the test data.

[0051] Optionally, the test data is a test question and the preset in-depth test data generation model is a preset large language model, or the test data is a test graph and its corresponding test question and the preset in-depth test data generation model is a preset multimodal large model;

[0052] The step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data comprises:

[0053] In the case where the test data is a test question, inputting the predicted answer most recently output by the model to be evaluated and the corresponding test question into a preset large language model, so that the preset large language model generates an in-depth test question corresponding to the test question;

[0054] In the case where the test data is a test image and its corresponding test question, the test image and test question corresponding to the predicted answer most recently output by the model to be evaluated are input into the preset multimodal large model, so that the preset multimodal large model generates an in-depth test question corresponding to the test image.

[0055] Optionally, the preset evaluation method is the consistency evaluation method, the comprehension ability evaluation method and the knowledge mastery degree evaluation method;

[0056] The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes:

[0057] Based on the evaluation test data set, the model to be evaluated is evaluated by the consistency evaluation method, the comprehension ability evaluation method, and the knowledge mastery degree evaluation method to obtain a first score, a second score, and a third score;

[0058] An evaluation result of the model to be evaluated is determined based on the first score, the second score, and the third score.

[0059] Optionally, the step of obtaining an evaluation test data set includes:

[0060] Inputting a preset question matching the application field of the model to be evaluated into an answer generation device, so that the answer generation device returns an answer corresponding to the preset question, wherein the answer generation device is an information retrieval framework or a preset large language model;

[0061] Obtaining test data constructed by the user based on the preset questions and their corresponding answers and their corresponding standard answers;

[0062] The correspondence between the test data and its corresponding standard answer is stored to obtain an evaluation test data set.

[0063] Optionally, after the step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, the method further includes:

[0064] Based on the evaluation result, generating optimization direction prompt information corresponding to the model to be evaluated;

[0065] Wherein, when the evaluation result indicates that the correctness of the predicted answer does not meet the correctness requirement, the optimization direction prompt information is used to prompt: to add professional knowledge data in the technical field applied by the model to be evaluated to the training data of the model to be evaluated;

[0066] In the case where the evaluation result indicates that the correlation between the predicted answer and the corresponding test data does not meet the correlation requirement, the optimization direction prompt information is used to prompt: in the application process of the model to be evaluated, information related to the currently processed problem is obtained from an external knowledge base;

[0067] In the case where the evaluation result indicates that the detail level of the predicted answer does not meet the detail level requirement, the optimization direction prompt information is used to prompt: to add knowledge data in the technical field to which the predicted answer and the corresponding question belong in the training data of the model to be evaluated;

[0068] In the case where the evaluation result indicates that the logic of the predicted answer does not meet the logic requirement, the optimization direction prompt information is used to prompt: adding open source training data for enhancing the logic of the model to be evaluated to the training data of the model to be evaluated;

[0069] In a case where the evaluation result indicates that the language coherence of the predicted answer does not meet the language coherence requirement, the optimization direction prompt information is used to prompt: increase the richness of the vocabulary and sentence patterns used to generate the training data.

[0070] In a second aspect, an embodiment of the present invention provides a model evaluation device, the device comprising:

[0071] A data set acquisition module is used to acquire an evaluation test data set, wherein the evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied;

[0072] A model evaluation module, used to evaluate the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method;

[0073] The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, where the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, where the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

[0074] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0075] Memory, used to store computer programs;

[0076] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0077] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0078] Beneficial effects of the embodiments of the present invention:

[0079] In the scheme provided by the embodiment of the present invention, the electronic device can obtain an evaluation test data set, wherein the evaluation test data set includes multiple test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; the consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data, the comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, and the multiple rewritten test data are obtained by rewriting the same test data, and the knowledge mastery degree evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data corresponding to the test data, and the in-depth test data belongs to the same technical field as the test data, and the professional level is higher than the test data. With respect to consistency, assuming that the consistency of the model to be evaluated is high, then the consistency between the predicted answers output by the model to be evaluated based on multiple prompt words for the same test data is high. Therefore, the consistency of the model to be evaluated can be evaluated based on the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. For comprehension ability, since the professional level of the rewritten test data is similar to that of the test data, the difference is that the expression method is different. Therefore, assuming that the comprehension ability of the model to be evaluated is strong, then when the model to be evaluated masters one test data, the model to be evaluated can master multiple rewritten test data obtained by rewriting the test data. Therefore, the electronic device can evaluate the comprehension ability of the model to be evaluated based on the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. For the degree of knowledge mastery, since the in-depth test data and the test data belong to the same technical field, and the degree of professionalism is higher than the test data, it is assumed that the model to be evaluated has a higher degree of knowledge mastery for a certain technical field, then the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data is higher. Therefore, the knowledge mastery of the model to be evaluated can be evaluated by the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data. In this way, the consistency, comprehension ability and knowledge mastery of the large language model can be evaluated. Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0081] Figure 1 A flow chart of a model evaluation method provided by an embodiment of the present invention;

[0082] Figure 2 for Figure 1 A first specific flow chart of step S102 in the illustrated embodiment;

[0083] Figure 3 Based on Figure 1 A schematic diagram of a process for determining a consistency evaluation result in the illustrated embodiment;

[0084] Figure 4 for Figure 2 A first specific flow chart of step S201 in the illustrated embodiment;

[0085] Figure 5 Based on Figure 1 A flow chart of determining consistency evaluation results of the illustrated embodiment;

[0086] Figure 6 for Figure 1 A second specific flow chart of step S102 in the illustrated embodiment;

[0087] Figure 7 Based on Figure 1 A schematic diagram of a process for determining a comprehension ability evaluation result in the illustrated embodiment;

[0088] Figure 8 for Figure 1 A third specific flow chart of step S102 in the illustrated embodiment;

[0089] Fig. 9 Based on Figure 1 A schematic diagram of a process for determining a knowledge mastery evaluation result in the illustrated embodiment;

[0090] Fig.10 for Figure 8 A specific flow chart of step S803 in the illustrated embodiment;

[0091] Fig.11 Based on Figure 1 Another schematic diagram of a process for determining a knowledge mastery evaluation result in the illustrated embodiment;

[0092] Fig.12 for Figure 1 A fourth specific flow chart of step S102 in the illustrated embodiment;

[0093] Fig.13 Based on Figure 1 A schematic diagram of a process for determining an evaluation result in the illustrated embodiment;

[0094] Fig.14 for Figure 1 A specific flow chart of step S101 in the illustrated embodiment;

[0095] Fig.15 Based on Figure 1 A schematic diagram of a process of constructing an evaluation test data set in the illustrated embodiment;

[0096] Fig.16 A schematic diagram of the structure of a model evaluation device provided by an embodiment of the present invention;

[0097] Fig.17 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0099] In order to evaluate the consistency, comprehension ability and knowledge mastery of a large language model, an embodiment of the present invention provides a model evaluation method, device, electronic device and storage medium, computer-readable storage medium and computer program product. First, a model evaluation method provided by an embodiment of the present invention is introduced.

[0100] The model evaluation method provided in the embodiment of the present invention can be applied to any electronic device that needs to evaluate a large language model, for example, a model evaluation device, a model training device, etc., which is not specifically limited here. For the sake of clarity, it is subsequently referred to as an electronic device in this article.

[0101] like Figure 1 A model evaluation method is shown, the method comprising:

[0102] S101, obtaining an evaluation test data set;

[0103] The evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied.

[0104] S102, based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result.

[0105] Among them, the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method and a knowledge mastery level evaluation method. The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, and the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, and the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

[0106] It can be seen that in an embodiment of the present invention, an electronic device can obtain an evaluation test data set, wherein the evaluation test data set includes multiple test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; the consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data, the comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, and the multiple rewritten test data are obtained by rewriting the same test data, and the knowledge mastery degree evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data corresponding to the test data, and the in-depth test data belongs to the same technical field as the test data, and the professional level is higher than the test data. With respect to consistency, assuming that the consistency of the model to be evaluated is high, then the consistency between the predicted answers output by the model to be evaluated based on multiple prompt words for the same test data is high. Therefore, the consistency of the model to be evaluated can be evaluated based on the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. For comprehension ability, since the professional level of the rewritten test data is similar to that of the test data, the difference is that the expression method is different. Therefore, assuming that the comprehension ability of the model to be evaluated is strong, then when the model to be evaluated masters one test data, the model to be evaluated can master multiple rewritten test data obtained by rewriting the test data. Therefore, the electronic device can evaluate the comprehension ability of the model to be evaluated based on the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. For the degree of knowledge mastery, since the in-depth test data and the test data belong to the same technical field, and the degree of professionalism is higher than the test data, it is assumed that the model to be evaluated has a higher degree of knowledge mastery for a certain technical field, then the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data is higher. Therefore, the knowledge mastery of the model to be evaluated can be evaluated by the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data. In this way, the consistency, comprehension ability and knowledge mastery of the large language model can be evaluated.

[0107] In step S101, the electronic device may obtain an evaluation test data set. The evaluation test data set may include multiple test data, and the test data may include test questions and test diagrams. The types of test questions may include subjective questions and / or objective questions, and the objective questions may include at least one of single-choice questions, multiple-choice questions, and judgment questions.

[0108] In order to evaluate the model to be evaluated in a targeted manner, the technical field to which the content included in the evaluation test data set belongs can be matched with the technical field to which the model to be evaluated is applied. For example, assuming that the technical field to which the model to be evaluated is applied is the semiconductor display technology field, then the technical field to which the content included in the evaluation test set belongs is also the semiconductor display technology field.

[0109] Next, the electronic device can evaluate the model to be evaluated based on the evaluation test data set through a preset evaluation method to obtain an evaluation result, that is, execute step S102. The model to be evaluated can be a large language model, and the preset evaluation method can include at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method. The above three preset evaluation methods are introduced below:

[0110] For the above consistency evaluation method, the consistency evaluation method can be used to evaluate the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. Among them, the prompt words can be generated by a preset large language model or a Prompt template, and the preset large language model can be GPT-4o (GPT-4omni) or Claude 3.5Sonnet.

[0111] The prompt words can be used to indicate the corresponding answer method of the test question, and the answer method can be determined based on the type of test question. For example, assuming the test question is a multiple-choice question, the prompt word corresponding to the test question can be "The following is a multiple-choice question. Each test question contains four options, only one of which is correct. Please select the correct option." For another example, assuming the test question is a true or false question, the prompt word corresponding to the test question can be "The following is a true or false question. Each test question may be correct or incorrect. Please determine whether the statement of each test question is correct or incorrect."

[0112] In the actual interaction between the user and the large language model, the prompt words input by the user play a crucial role in whether the large language model can accurately and high-quality output the expected response. Therefore, in order to evaluate the consistency of the model to be evaluated, the predicted answers output by the model to be evaluated based on different prompt words for the same test data can be obtained. If the predicted answers are close to each other, it means that the consistency of the model to be evaluated is high. Correspondingly, if the predicted answers differ greatly, it means that the consistency of the model to be evaluated is poor.

[0113] With respect to the above-mentioned comprehension ability evaluation method, the comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. The rewritten test data can be generated by the above-mentioned preset large language model, and the multiple rewritten test data can be obtained by rewriting the same test data. The rewritten test data has a similar professional level to the test data, but has a different expression method.

[0114] For example, the test data may be rewritten by changing the order of options included in the test question, changing the type of the test question, replacing the words included in the test question with synonyms, and so on.

[0115] With respect to the above-mentioned knowledge mastery evaluation method, the knowledge mastery evaluation method is used to evaluate the accuracy of the predicted answer output by the model to be evaluated for the in-depth test data corresponding to the test data, where the in-depth test data belongs to the same technical field as the test data and has a higher professional level than the test data. The in-depth test data can be generated by the above-mentioned preset large language model.

[0116] For example, assuming the test data is "What is the working principle of PDP (Plasma Display Panel)?", then the in-depth test data corresponding to the test data may be "What is the difference in power consumption between ELD (Electroluminescent Display) and PDP?". The above in-depth test data and test data both belong to the field of semiconductor display technology.

[0117] It can be seen that in the embodiment of the present invention, for consistency, assuming that the consistency of the model to be evaluated is high, then the consistency between the predicted answers output by the model to be evaluated based on multiple prompt words for the same test data is high. Therefore, the consistency of the model to be evaluated can be evaluated based on the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. For comprehension ability, since the professional level of the rewritten test data and the test data is similar, the difference is that the expression method is different. Therefore, assuming that the comprehension ability of the model to be evaluated is strong, then when the model to be evaluated masters one test data, the model to be evaluated can master multiple rewritten test data obtained by rewriting the test data. Therefore, the electronic device can evaluate the comprehension ability of the model to be evaluated based on the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. For the degree of knowledge mastery, since the in-depth test data is the same as the technical field to which the test data belongs, and the degree of professionalism is higher than the test data, it is assumed that the knowledge mastery of the model to be evaluated for a certain technical field is high, then the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data is high. Therefore, the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data can be used to evaluate the knowledge mastery of the model to be evaluated. In this way, the consistency, comprehension ability, and knowledge mastery of the large language model can be evaluated.

[0118] As an implementation method of an embodiment of the present invention, the above-mentioned preset evaluation method is a consistency evaluation method, and the test data is a test question. In this case, Figure 2 As shown, the above step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain the evaluation result may include:

[0119] S201, obtaining at least one test question, and obtaining a plurality of different prompt words corresponding to the test question;

[0120] The electronic device may obtain at least one test question and a plurality of different prompt words corresponding to the test question. For example, the electronic device may obtain test question 1, prompt words 1 to 3 corresponding to test question 1, test question 2, and prompt words 4 to 7 corresponding to test question 2.

[0121] S202, for the test question, respectively combining a plurality of different prompt words corresponding to the test question with the test question according to a preset combination method to obtain a plurality of combination results;

[0122] In order to obtain the predicted answer output by the model to be evaluated based on different prompt words for the same test question, the electronic device can combine the test question and its corresponding multiple prompt words. Specifically, for the test question, the electronic device can combine the multiple different prompt words corresponding to the test question with the test question according to a preset combination method to obtain multiple combination results.

[0123] S203, inputting the plurality of combination results into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test question included in each combination result based on an indication of a prompt word included in the combination result;

[0124] Next, the electronic device may input multiple combination results into the model to be evaluated, and the model to be evaluated may output, for each combination result, a predicted answer to the test question included in the combination result based on the indication of the prompt word included in the combination result.

[0125] S204: Determine a consistency evaluation result corresponding to the model to be evaluated based on the differences between the multiple predicted answers corresponding to the test question.

[0126] If the differences between the multiple predicted answers corresponding to the test question are small, it means that when the test question remains unchanged, even if the prompt word changes, the model to be evaluated can maintain a high consistency each time the answer is given. Correspondingly, if the differences between the multiple predicted answers corresponding to the test question are large, it means that the consistency of the model to be evaluated is low each time the answer is given. Therefore, the electronic device can determine the consistency evaluation result corresponding to the model to be evaluated based on the differences between the multiple predicted answers corresponding to the test question.

[0127] For example, a flow chart for determining the results of a consistency assessment could be as follows: Figure 3 As shown, the following steps may be specifically included:

[0128] S301, adding or rewriting the template;

[0129] By adding or rewriting the template, the preset large language model or template can generate prompt word 1-prompt word n. The following only takes prompt word 1 and prompt word 2 as examples for explanation:

[0130] Prompt word 1 is "The following are single-choice questions. Each question contains four options, only one of which is correct. Please select the correct answer." Prompt word 2 is "The following are single-choice questions about the field of semiconductor display technology. Only one option meets the requirements of the question. Please select the option you think is correct. Please only output the correct answer without analysis." Among them, the above template can be a Prompt template.

[0131] S302, obtaining test questions from the evaluation test data set;

[0132] The electronic device may obtain a test question from the evaluation test data set. The test question may be as follows. The standard answer to the test question is B:

[0133] Among light-emitting display devices, which one consumes the least power?

[0134] A. Plasma display device; B. Electroluminescent display device;

[0135] C. cathode ray tube display device; D. field emission display device.

[0136] Among them, cathode ray tube display devices (Cathode Ray Tube, CRT) and field emission display devices (Field Emission Display, FED) are both light-emitting display devices.

[0137] S303, combining the test question with the prompt word;

[0138] The electronic device can combine the above test questions with prompt words 1 to prompt words n respectively, and then obtain combination results 1 to combination results n, wherein combination result 1 is as follows:

[0139] The following are multiple-choice questions. Each question contains four options, only one of which is correct. Please choose the correct answer.

[0140] Among light-emitting display devices, which one consumes the least power?

[0141] A. Plasma display device; B. Electroluminescent display device;

[0142] C. cathode ray tube display device; D. field emission display device.

[0143] The combined result 2 is as follows:

[0144] The following are multiple-choice questions about the field of semiconductor display technology. Only one option meets the requirements of the question. Please select the option you think is correct. Please only output the correct answer without analysis.

[0145] Among light-emitting display devices, which one consumes the least power?

[0146] A. Plasma display device; B. Electroluminescent display device;

[0147] C. cathode ray tube display device; D. field emission display device.

[0148] S304, inputting the combined result into the model to be evaluated;

[0149] Next, the electronic device can input combination results 1-combination results n into the model to be evaluated. Assuming that the model to be evaluated predicts that the answer to the test question in combination result 1 is B, and the answer to the test question in combination result 2 is D, then it can be determined that the score of the predicted answer corresponding to combination result 1 is 1, and the score of the predicted answer corresponding to combination result 2 is 0.

[0150] S305: Determine the consistency evaluation result.

[0151] Assuming that the consistency evaluation result is 1 only when the scores of the predicted answers corresponding to all combination results are 1, otherwise the consistency evaluation result is 0, then it can be determined that the consistency evaluation result corresponding to the model to be evaluated is 0.

[0152] It can be seen that in the embodiment of the present invention, the electronic device can obtain at least one test question and obtain multiple different prompt words corresponding to the test question; for the test question, according to the preset combination method, the multiple different prompt words corresponding to the test question are respectively combined with the test question to obtain multiple combination results; the multiple combination results are input into the model to be evaluated, so that the model to be evaluated outputs the predicted answer of the test question included in the combination result based on the indication of the prompt word included in each combination result; based on the difference between the multiple predicted answers corresponding to the test question, the consistency evaluation result corresponding to the model to be evaluated is determined. Since the same test question is combined with different multiple prompt words respectively, multiple predicted answers output by the model to be evaluated for the same test question based on different prompt words can be obtained. If the consistency between the multiple predicted answers is high, it means that the consistency of the model to be evaluated is high. Correspondingly, if the consistency between the multiple predicted answers is low, it means that the consistency of the model to be evaluated is poor. Therefore, it is possible to evaluate the consistency of the model to be evaluated.

[0153] As an implementation method of the embodiment of the present invention, Figure 4 As shown, the step of obtaining a plurality of different prompt words corresponding to the test question may include:

[0154] S401, inputting a prompt word generation instruction into a preset large language model, so that the preset large language model generates a plurality of candidate prompt words based on the prompt word generation instruction; if the candidate prompt words meet the prompt word condition, executing step S402; if the candidate prompt words do not meet the prompt word condition, executing step S403, and returning to step S401, until the generated candidate prompt words meet the prompt word condition;

[0155] In order to obtain the prompt word, the electronic device can input the prompt word generation instruction into the preset large language model, and the preset large language model can generate multiple candidate prompt words based on the prompt word generation instruction. Among them, the prompt word generation instruction can be a piece of text input by the user in a natural language expression, and the prompt word generation instruction can be used to indicate the prompt information of the answer method corresponding to the generated test question, and the above prompt word can also be generated by the Prompt template.

[0156] In one embodiment, the number of prompt words can be preset to obtain a preset number of prompt words. In this case, after receiving the prompt word generation instruction, the preset large language model or Prompt template can generate a preset number of prompt words based on the prompt word generation instruction. For example, the preset number of prompt words can be 4, 5, 6, and so on.

[0157] In the case where the test question is a single-choice question, the prompt word generation instruction can be used to instruct the generation of prompt information for selecting a correct option from multiple options. In the case where the test question is a multiple-choice question, the prompt word generation instruction can be used to instruct the generation of prompt information for selecting multiple correct options from multiple options. In the case where the test question is a judgment question, the prompt word generation instruction can be used to instruct the generation of prompt information for judging whether the statement of the test question is correct or incorrect. In the case where the test question is a subjective question, the prompt word generation instruction can be used to instruct the generation of prompt information for answering the test question stem.

[0158] S402, using the candidate prompt word as the prompt word corresponding to the test question;

[0159] When the candidate prompt word meets the prompt word condition, the electronic device can use the candidate prompt word as the prompt word corresponding to the test question. The prompt word condition can indicate that the candidate prompt word can indicate the answer method corresponding to the test question of the model to be evaluated when it conforms to the expression of natural language. That is to say, after receiving the candidate prompt word that meets the prompt word condition, the model to be evaluated can clearly know the answer method of the test question corresponding to the prompt word.

[0160] S403, obtaining a modified prompt word generation instruction.

[0161] If the candidate prompt words do not meet the prompt word conditions, the user can modify the prompt word generation instruction, and the electronic device can obtain the modified prompt word generation instruction. Next, the electronic device can return to the above step S401, that is, input the modified prompt word generation instruction into the preset large language model, and the preset large language model can generate multiple candidate prompt words based on the modified prompt word generation instruction.

[0162] Then, it can be determined whether the last generated alternative prompt word meets the prompt word condition. If the alternative prompt word meets the prompt word condition, the last generated alternative prompt word can be used as the prompt word. If the alternative prompt word still does not meet the prompt word condition, the user can continue to modify the prompt word generation instruction until the generated alternative prompt word meets the prompt word condition.

[0163] For example, if the goal is to generate prompt words for single-choice questions in the field of general technology or semiconductor display technology, the initial prompt word generation instruction may be as follows:

[0164] Design a prompt word, such as: "Please generate a multiple-choice prompt about semiconductor display technology. Each question has four options, only one of which is correct."

[0165] The electronic device can call the API (Application Programming Interface) of GPT-4o and pass the above prompt word generation instruction to GPT-4o. GPT-4o can then generate the prompt word. Next, the prompt word can be evaluated, specifically, to determine whether the prompt word is clearly expressed and meets the prompt word requirements.

[0166] If the prompt word does not meet the prompt word conditions, the user can modify the prompt word generation instruction, and the modified prompt word generation instruction can be as follows:

[0167] Please create a multiple-choice prompt about semiconductor display technology, requiring each question to have four options, and only one option is correct. The prompt should clearly require the correct answer to be selected.

[0168] After acquiring the modified prompt word generation instruction, the electronic device may input the modified prompt word generation instruction into a preset large language model, and the preset large language model may output prompt word 1 and prompt word 2 based on the modified prompt word generation instruction.

[0169] Among them, prompt word 1 can be "The following are multiple-choice questions. Each question contains four options, and only one option is correct. Please select the correct answer." Prompt word 2 can be "The following are multiple-choice questions in the field of semiconductor display. Only one option meets the requirements of the question. Please select the option you think is correct. Please only output the correct answer without analysis." Assuming that prompt word 1 and prompt word 2 both meet the prompt word conditions, then prompt word 1 and prompt word 2 can be used as prompt words.

[0170] It can be seen that in the embodiment of the present invention, the electronic device can input the prompt word generation instruction into the preset large language model, so that the preset large language model generates multiple candidate prompt words based on the prompt word generation instruction, wherein the prompt word generation instruction is used to indicate the prompt information of the answer method corresponding to the test question; when the candidate prompt word meets the prompt word condition, the candidate prompt word is used as the prompt word corresponding to the test question, wherein the prompt word condition represents that the candidate prompt word can indicate the answer method corresponding to the test question of the model to be evaluated when it meets the natural language expression method; when the candidate prompt word does not meet the prompt word condition, the modified prompt word generation instruction is obtained, and the step of inputting the prompt word generation instruction into the preset large language model is returned until the generated candidate prompt word meets the prompt word condition. In a loop, as long as the candidate prompt word does not meet the prompt word condition, the user can modify the prompt word generation instruction all the time, and let the preset large language model generate prompt words again based on the modified prompt word generation instruction until the candidate prompt word meets the prompt word condition. Therefore, prompt words that meet the prompt word condition can be accurately generated.

[0171] As an implementation manner of an embodiment of the present invention, the step of determining the consistency evaluation result corresponding to the model to be evaluated based on the difference between the multiple predicted answers corresponding to the test question may include at least one of the following three implementation manners:

[0172] In the case where the test question is an objective question, based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer, determining the consistency evaluation result corresponding to the model to be evaluated;

[0173] When the test question is a subjective question and there is a standard answer, based on the similarity between the multiple predicted answers corresponding to the test question and the standard answer, determine the consistency evaluation result corresponding to the model to be evaluated;

[0174] When the test question is a subjective question and there is no standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined from multiple preset judging dimensions through a preset judging model.

[0175] In the first embodiment, the test question is an objective question, and there is a clear standard answer for the objective question. If the predicted answer is the same as the standard answer, it is determined that the answer of the model to be evaluated is correct; if the predicted answer is different from the standard answer, it is determined that the answer of the model to be evaluated is wrong.

[0176] Therefore, the electronic device can determine the consistency evaluation result corresponding to the model to be evaluated based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer. The consistency evaluation result can be a number, and the size of the number can be positively correlated with the consistency of the model to be evaluated. Specifically, it can be implemented based on one of the following two specific implementations:

[0177] As a first specific implementation, the electronic device may determine the consistency evaluation result corresponding to the model to be evaluated based on the size relationship between the number of wrong answers given by the model to be evaluated and the preset number of times. Specifically, when the number of wrong answers given by the model to be evaluated reaches the preset number of times, the consistency evaluation result may be determined to be the first preset result; when the number of wrong answers given by the model to be evaluated does not reach the preset number of times, the consistency evaluation result may be determined to be the second preset result. The first preset result and the second preset result may be pre-set scores, and the first preset result may be greater than the second preset result.

[0178] For example, assuming the preset number is 2, the first preset result is 1, and the second preset result is 0. If, among the 5 predicted answers corresponding to the test question, 3 predicted answers are the same as the standard answer, and 2 predicted answers are different from the standard answer. Since the number of wrong answers given by the model to be evaluated reaches the preset number, it can be determined that the consistency evaluation result corresponding to the model to be evaluated is 0.

[0179] As a second specific implementation, the scores of the model to be evaluated for multiple predicted answers corresponding to the test question can be determined, and the average of the scores is used as the consistency evaluation result corresponding to the model to be evaluated. For example, when the predicted answer is correct, the score is 1, and when the predicted answer is wrong, the score is 0. Assuming that the scores of the combination results 1 to 5 corresponding to a test question by the model to be evaluated are 1, 0, 1, 1, and 1, respectively, then the average of the above scores can be calculated to obtain 0.8, and 0.8 is used as the consistency evaluation result of the model to be evaluated.

[0180] In the second embodiment, the test question is a subjective question and has a standard answer. When answering a subjective question, the predicted answer corresponding to the subjective question is not required to be completely consistent with the standard answer, but the similarity between the predicted answer and the standard answer is high. Therefore, the electronic device can determine the consistency evaluation result corresponding to the model to be evaluated based on the similarity between the multiple predicted answers corresponding to the test question and the standard answer.

[0181] Specifically, the ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence) values ​​between multiple predicted answers corresponding to the test question and the standard answer can be calculated, and the average of the ROUGE-L values ​​corresponding to each predicted answer can be used as the consistency evaluation result corresponding to the model to be evaluated.

[0182] In the third implementation, the test questions are subjective questions and there is no standard answer. Since there is no standard answer, it is necessary to introduce a referee model to judge the predicted answer, so as to determine the consistency evaluation result corresponding to the model to be evaluated.

[0183] Specifically, the consistency evaluation result corresponding to the model to be evaluated can be determined from multiple preset referee dimensions through a preset referee model. Among them, the preset referee model can be a large language model, for example, it can be GPT-4o, Claude 3.5Sonnet. The preset referee dimension can be used to reflect the degree of match between the predicted answer and the test question. For example, the content of the test question is about the power consumption of ELD. Assuming that the predicted answer includes keywords such as ELD and power consumption, it can be determined that the predicted answer and the test question have a high degree of match. Assuming that the predicted answer does not include ELD or power consumption, it can be determined that the predicted answer and the test question have a low degree of match.

[0184] It can be seen that in the embodiments of the present invention, the electronic device can, when the test question is an objective question, determine the consistency evaluation result corresponding to the model to be evaluated based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer; when the test question is a subjective question and there is a standard answer, determine the consistency evaluation result corresponding to the model to be evaluated based on the similarity between the multiple predicted answers corresponding to the test question and the standard answer; when the test question is a subjective question and there is no standard answer, determine the consistency evaluation result corresponding to the model to be evaluated from multiple preset judgment dimensions through a preset referee model, wherein the preset referee dimension is used to reflect the degree of match between the predicted answer and the test question. Through the above three implementation methods, corresponding methods can be adopted according to different situations of the test questions to quickly and accurately determine the consistency evaluation result of the model to be evaluated.

[0185] As an implementation manner of an embodiment of the present invention, the step of determining the consistency evaluation result corresponding to the model to be evaluated based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer may include:

[0186] If the multiple predicted answers corresponding to the test question are all consistent with the standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined to be the first preset result; if among the multiple predicted answers corresponding to the test question, there is a predicted answer that is different from the standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined to be the second preset result.

[0187] The electronic device can determine whether the multiple predicted answers corresponding to the test question are consistent with the standard answer. If the multiple predicted answers corresponding to the test question are consistent with the standard answer, it means that the model to be evaluated can answer the same test question correctly based on different prompt words, then it can be determined that the consistency of the model to be evaluated is high, and then the consistency evaluation result corresponding to the model to be evaluated is determined to be the above-mentioned first preset result.

[0188] If there is a predicted answer different from the standard answer among the multiple predicted answers corresponding to the test question, it means that the model to be evaluated will give an incorrect answer to the same test question based on different prompt words. Then, it can be determined that the consistency of the model to be evaluated is low, and then the consistency evaluation result corresponding to the model to be evaluated is determined to be the second preset result.

[0189] For example, assuming that the first preset result is 1 and the second preset result is 0. If among the 5 predicted answers corresponding to the test question, 4 predicted answers are the same as the standard answer and 1 predicted answer is different from the standard answer. Since among the multiple predicted answers corresponding to the test question, there is a predicted answer that is different from the standard answer, then it can be determined that the consistency evaluation result corresponding to the model to be evaluated is 0.

[0190] It can be seen that in the embodiment of the present invention, if the multiple predicted answers corresponding to the test question are consistent with the standard answer, the electronic device can determine that the consistency evaluation result corresponding to the model to be evaluated is the first preset result; if there is a predicted answer different from the standard answer among the multiple predicted answers corresponding to the test question, the consistency evaluation result corresponding to the model to be evaluated is determined to be the second preset result. Since the consistency evaluation result is determined to be the first preset result only when all the predicted answers are consistent with the standard answer, the model to be evaluated can be evaluated with a higher standard.

[0191] As an implementation method of the embodiment of the present invention, Figure 5 As shown, the above step of determining the consistency evaluation result corresponding to the model to be evaluated from multiple preset judgment dimensions through the preset judgment model may include:

[0192] S501, inputting the test question and the multiple predicted answers corresponding to the test question into a preset judging model, so that the preset judging model generates a score corresponding to each predicted answer in each preset judging dimension;

[0193] The electronic device can input the test question and the multiple predicted answers corresponding to the test question into the preset judging model, and the preset judging model can generate the score corresponding to each predicted answer in each preset judging dimension. The preset judging dimension can include at least one of the correctness of the predicted answer, the correlation between the predicted answer and the corresponding test data, the language coherence of the predicted answer, the detail of the predicted answer, and the logic of the predicted answer, which is not specifically limited here.

[0194] Specifically, the electronic device can combine the test question, the multiple predicted answers corresponding to the test question, and the preset prompt template, and input the combined result into the preset referee model. Then, the preset referee model can score the predicted answer according to the preset referee dimension, the degree of conformity between the predicted answer and the evaluation index, and the scoring standard. Among them, the preset prompt template can be used to indicate the preset referee model's preset referee dimension, evaluation index, and scoring standard for the predicted answer.

[0195] For example, the preset judging dimensions, evaluation indicators, scoring criteria, scoring methods, and preset prompt templates may be shown in Table 1:

[0196] Table 1

[0197] Taking the case in Table 1 where the preset referee dimension is the correctness of the predicted answer as an example, the scoring process of the preset referee model is explained. Assuming the test question is "What is the working principle of PDP?" The predicted answer is "The working principle of PDP is based on the plasma luminescence phenomenon generated by gas discharge." Then the preset prompt template shown below can be input into the preset referee model:

[0198] “Please rate the accuracy of the predicted answers based on the following criteria:

[0199] 1. Completeness: Evaluate whether the predicted answer covers all the key points of the test question?

[0200] 2. Correctness: Is the information provided by the model to be evaluated correct and reliable?

[0201] Please rate each dimension and briefly explain your reasons for your assessment.

[0202] Test question: How does PDP work?

[0203] Predicted answer: The working principle of PDP is based on the plasma luminescence phenomenon generated by gas discharge.

[0204] Rating (1-5):

[0205] reason:"

[0206] Next, the preset referee model can evaluate the correctness of the predicted answer of the model to be evaluated from the perspective of completeness and correctness. Assuming that the preset referee model scores s1 as 4 for completeness and s2 as 4 for correctness, it can be determined that the score S1 for the correctness of the predicted answer is 4. The preset referee model can also output the reason for the score, "The predicted answer basically covers the key points of the test question, and the information is relatively correct and reliable."

[0207] S502, for each preset judging dimension, calculating the average of the scores of each predicted answer corresponding to the preset judging dimension;

[0208] In one embodiment, in order to determine the capability of the model to be evaluated in each preset judging dimension, the electronic device may calculate, for each preset judging dimension, the average value of the score corresponding to each predicted answer in the preset judging dimension.

[0209] For example, suppose that the scores corresponding to the preset referee dimension 1 for the predicted answer 1-predicted answer 3 are 1, 3, and 2, the scores corresponding to the preset referee dimension 2 are 4, 4, and 1, and the scores corresponding to the preset referee dimension 3 are 5, 4, and 3. Then, taking the preset referee dimension 1 as an example, the scoring process is explained, and the average score corresponding to the preset referee dimension 1 for the predicted answer 1-predicted answer 3 is calculated to be 2, so it can be determined that the average value corresponding to the preset referee dimension 1 is 2. Similarly, it can be determined that the average values ​​corresponding to the preset referee dimension 2 and the preset referee dimension 3 are 3 and 4 respectively.

[0210] In another embodiment, the electronic device may calculate, for each preset judging dimension, an average value of the scores of each predicted answer in the scores corresponding to the preset judging dimension, excluding the maximum value and the minimum value.

[0211] For example, assuming that the scores corresponding to the predicted answers 1-predicted answers 5 in the preset judging dimension 1 are 3, 2, 3, 5, 3, and the scores corresponding to the preset judging dimension 2 are 1, 4, 4, 5, 4, then for the preset judging dimension 1 and the preset judging dimension 2, the average values ​​of the scores excluding the maximum and minimum values ​​are 3 and 4 respectively.

[0212] S503, based on the average value corresponding to each preset referee dimension and the preset weight, weighted summation is performed on the average value to obtain a consistency evaluation result corresponding to the model to be evaluated.

[0213] After determining the average value corresponding to each preset referee dimension, the average values ​​can be weighted and summed based on the preset weight corresponding to each preset referee dimension to obtain the consistency evaluation result corresponding to the model to be evaluated. The size of the preset weight can be set according to the importance of the preset referee dimension in the consistency evaluation result corresponding to the model to be evaluated. The higher the consistency evaluation result, the better the robustness of the model to be evaluated.

[0214] The above process can be expressed in the form of a formula: S=w1×S1+…+w i ×S i +…+w n ×S n , where S is the consistency assessment result, w i is the preset weight corresponding to the i-th preset referee dimension, S i is the average value corresponding to the i-th preset referee dimension, i ranges from 1 to n, w1+…+w i +…+w n =1.

[0215] For example, assuming that the average value and preset weight corresponding to the preset judging dimension 1 are 3 and 0.4 respectively, the average value and preset weight corresponding to the preset judging dimension 2 are 4 and 0.4 respectively, and the average value and preset weight corresponding to the preset judging dimension 3 are 2 and 0.2 respectively, then it can be determined that the consistency assessment result corresponding to the model to be evaluated is 3.2.

[0216] It can be seen that in the embodiment of the present invention, the electronic device can input the test question and the multiple predicted answers corresponding to the test question into the preset referee model, so that the preset referee model generates a score corresponding to each predicted answer in each preset referee dimension, wherein the preset referee dimension includes at least one of the correctness of the predicted answer, the correlation between the predicted answer and its corresponding test data, the language coherence of the predicted answer, the detail of the predicted answer, and the logic of the predicted answer; for each preset referee dimension, the average value of the score corresponding to each predicted answer in the preset referee dimension is calculated; based on the average value corresponding to each preset referee dimension and the preset weight, the average value is weighted and summed to obtain the consistency evaluation result corresponding to the model to be evaluated. When the test question is a subjective question and there is no standard answer, by introducing a preset referee model, the predicted answer output by the model to be evaluated is evaluated from the preset referee dimension, so as to achieve the consistency evaluation of the model to be evaluated.

[0217] As an implementation method of an embodiment of the present invention, the above consistency evaluation method may be a comprehension ability evaluation method, and the test data may be a test question. In this case, Figure 6As shown, the above step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain the evaluation result may include:

[0218] S601, obtaining at least one test question;

[0219] The electronic device may obtain at least one test question from the evaluation test data set.

[0220] S602, inputting the test question into a preset large language model, so that the preset large language model rewrites the test question in a preset rewriting manner to generate a plurality of rewritten test questions;

[0221] Next, in order to generate a rewritten test question with a similar professional level to the test question and a different expression method, the electronic device can input the test question into a preset large language model. Then, the preset large language model can rewrite the test question using a preset rewriting method, thereby generating multiple rewritten test questions. Among them, the preset rewriting method can be used to rewrite at least one of the stem, options, and type of the test question.

[0222] Specifically, the preset rewriting method may include at least one of the following five implementation modes:

[0223] In a first implementation manner, the electronic device may rewrite the test question into a multiple-choice question or a true-or-false question when the test question is a single-choice question.

[0224] For example, suppose the test question is a multiple-choice question, as shown below:

[0225] Among light-emitting display devices, which one consumes the least power?

[0226] A.PDP; B.ELD;

[0227] C.CRT; D.FED.

[0228] Then you can change the single-choice question into a true-or-false question to get a rewritten test question, which is as follows:

[0229] Among light-emitting display devices, ELD consumes the least power.

[0230] In a second implementation manner, when the test question is a multiple-choice question, the test question is rewritten as a single-choice question or a true-or-false question.

[0231] For example, suppose the test question is a multiple-choice question, as shown below:

[0232] Which of the following are common defects of MDL (Mask Defect Limited)?

[0233] A. Line defect; B. Point defect;

[0234] C. Horizontal black Block (occlusion); D. Accident Lot (batch).

[0235] Then you can change the multiple-choice questions into single-choice questions to get the rewritten test questions, which are as follows:

[0236] Which of the following is not included in the common defects of MDL (Mask Defect Limited)?

[0237] A. Line defect; B. Point defect;

[0238] C. Horizontal black block; D. Accident lot.

[0239] In the third embodiment, synonyms are replaced for the words included in the test questions, for example, "power consumption" is replaced with "power consumption".

[0240] In the fourth implementation, the stem of the test question is rewritten from an affirmative sentence to a negative sentence without changing the substantive content of the stem of the test question. For example, the stem is changed from "Which of the following are common defects of MDL?" to "Which of the following are not common defects of MDL?"

[0241] In a fifth embodiment, the order of the options included in the test question is changed.

[0242] For example, suppose the test question is as follows:

[0243] Among light-emitting display devices, which one consumes the least power?

[0244] A.PDP; B.ELD;

[0245] C.CRT; D.FED.

[0246] Then we can rewrite the order of the options in the above test question, swap A and B, and swap C and D, and get a rewritten test question, as shown below:

[0247] Among light-emitting display devices, which one consumes the least power?

[0248] A.ELD; B.PDP;

[0249] C.FED; D.CRT.

[0250] S603, inputting each rewritten test question and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to each rewritten test question;

[0251] In order to determine whether the model to be evaluated can master the rewritten test question corresponding to the test question after mastering the test question, the electronic device can input each rewritten test question and its corresponding prompt word into the model to be evaluated. Then, the model to be evaluated can output the predicted answer to each rewritten test question.

[0252] S604: Determine the comprehension ability evaluation result corresponding to the model to be evaluated based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question.

[0253] The difference between the predicted answer and the standard answer corresponding to the rewritten test question can represent the difference in the comprehension ability of the model to be evaluated when facing different rewritten test questions. Therefore, the electronic device can determine the comprehension ability evaluation result corresponding to the model to be evaluated based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question.

[0254] Specifically, the electronic device can determine the score corresponding to each rewritten test question based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question, and then take the average of the scores corresponding to all rewritten test questions as the comprehension ability evaluation result corresponding to the model to be evaluated.

[0255] For example, if the predicted answer is the same as the standard answer, the score corresponding to the predicted answer is 1; if the predicted answer is different from the standard answer, the score corresponding to the predicted answer is 0. Assuming that predicted answer 1-predicted answer 3 are the same as the standard answer, and predicted answer 4-predicted answer 5 are different from the standard answer, it can be determined that the score corresponding to predicted answer 1-predicted answer 3 is 1, and the score corresponding to predicted answer 4-predicted answer 5 is 0, and then the average value of the above scores, 0.6, can be used as the comprehension ability evaluation result corresponding to the model to be evaluated.

[0256] Exemplarily, a flow chart for determining the comprehension ability assessment result may be as follows: Figure 7 As shown, the following steps may be specifically included:

[0257] S701, obtaining test questions from an evaluation test data set;

[0258] The electronic device can obtain test questions from the evaluation test data set. The test questions are as follows:

[0259] Among light-emitting display devices, which one consumes the least power?

[0260] A. Plasma display device; B. Electroluminescent display device;

[0261] C. cathode ray tube display device; D. field emission display device.

[0262] S702, rewriting the test questions;

[0263] Next, at least one of the stem, options, and type of the test question can be rewritten by presetting a large language model. First, synonyms can be replaced for the words included in the test question to obtain a rewritten test question 1:

[0264] Among light-emitting display devices, which type has the lowest power consumption?

[0265] A. Plasma display device; B. Electroluminescent display device;

[0266] C. cathode ray tube display device; D. field emission display device.

[0267] The predicted answer output by the model to be evaluated for the rewritten test question 1 is: B. Electroluminescent display devices generally have lower power consumption. The score of the predicted answer corresponding to the rewritten test question 1 is 1. Next, the words included in the test question can be replaced with synonyms again, and the order of the options can be changed to obtain the rewritten test question 2:

[0268] Among light-emitting display devices, which type has the lowest power consumption?

[0269] A. Field emission display device; B. Electroluminescent display device;

[0270] C. cathode ray tube display device; D. plasma display device.

[0271] The predicted answer output by the model to be evaluated for the rewritten test question 2 is: B. Among the light-emitting display devices, the power consumption of the electroluminescent display device is relatively small. The score of the predicted answer corresponding to the rewritten test question 2 is 1. Then, the stem of the test question can be rewritten to obtain the rewritten test question 3:

[0272] Among light-emitting display devices, which one consumes the most power?

[0273] A. Plasma display device; B. Electroluminescent display device;

[0274] C. cathode ray tube display device; D. field emission display device.

[0275] The predicted answer output by the model to be evaluated for the rewritten test question 3 is: C. Among the four light-emitting display devices, cathode ray tube display devices generally consume the most power. The score of the predicted answer corresponding to the rewritten test question 3 is 1.

[0276] S703, determining the comprehension ability assessment result.

[0277] Furthermore, it can be determined that the comprehension ability evaluation result corresponding to the model to be evaluated is 1.

[0278] It can be seen that in an embodiment of the present invention, the electronic device can obtain at least one test question; input the test question into a preset large language model, so that the preset large language model rewrites the test question through a preset rewriting method to generate multiple rewritten test questions, wherein the preset rewriting method is used to rewrite at least one of the stem, options, and type of the test question; input each rewritten test question and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs the predicted answer of each rewritten test question; based on the difference between the predicted answer corresponding to each rewritten test question and the standard answer, determine the comprehension ability evaluation result corresponding to the model to be evaluated. Since the professional level of the rewritten test question is similar to that of the test question, and the only difference is the expression method, the comprehension ability of the model to be evaluated can be determined by the predicted answer output by the model to be evaluated for the rewritten test question.

[0279] As an implementation method of an embodiment of the present invention, the above-mentioned preset evaluation method can be a knowledge mastery degree evaluation method. In this case, Figure 8 As shown, the above step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain the evaluation result may include:

[0280] S801, obtaining at least one test data;

[0281] The electronic device may acquire at least one test data, wherein the test data may include a test question and / or a test diagram.

[0282] S802, inputting the test data and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test data;

[0283] The electronic device can input any acquired test data and its corresponding prompt words into the model to be evaluated, and use the test data as the beginning of the conversation. The model to be evaluated can then output the predicted answer to the test data.

[0284] S803, inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data;

[0285] Next, the electronic device can input the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model. If the model to be evaluated has only output a predicted answer once, the predicted answer most recently output is the predicted answer. If the model to be evaluated has output predicted answers multiple times, the predicted answer most recently output is the predicted answer output last time. Next, the preset in-depth test data generation model can generate in-depth test data corresponding to the test data. Among them, the preset in-depth test data generation model has professional knowledge in the technical field to which the test data belongs.

[0286] S804, inputting the in-depth test data and its corresponding prompt words into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the in-depth test data;

[0287] Since the in-depth test data is more professional than the test data, in order to evaluate the knowledge level of the model to be evaluated in the technical field to which the in-depth test data belongs, the electronic device can input the in-depth test data and its corresponding prompt words into the model to be evaluated, and the model to be evaluated will output the predicted answer to the in-depth test data.

[0288] S805, determining the judgment result of the predicted answer from multiple preset judgment dimensions based on the most recently outputted predicted answer and the corresponding in-depth test data through a preset judgment model; if the number of cycles does not reach the preset number, returning to step S803; if the number of cycles reaches the preset number, executing step S806;

[0289] In the above-mentioned dialogue process between the electronic device and the model to be evaluated, a preset referee model can be introduced to referee the predicted answer output by the model to be evaluated. Specifically, the electronic device can use the preset referee model to determine the referee result of the predicted answer from multiple preset referee dimensions based on the most recently output predicted answer and the corresponding in-depth test data.

[0290] In order to more accurately evaluate the knowledge mastery level of the model to be evaluated, a continuous dialogue content can be gradually constructed in a cyclical manner, so as to continuously explore and verify the knowledge mastery level of the model to be evaluated in the technical field to which the test data belongs.

[0291] Specifically, after obtaining the referee result, the electronic device can determine whether the current number of cycles reaches the preset number. If the number of cycles does not reach the preset number, then the above step S803 can be returned to continue generating in-depth test data. If the number of cycles reaches the preset number, then step S806 can be executed. The preset number can be 5, 6, 7, etc., which is not specifically limited here.

[0292] When the test data is a test question, the way in which the preset referee model determines the referee result of the predicted answer is the same as the way in which the preset referee model determines the score of the predicted answer in the aforementioned embodiment, so it will not be repeated here.

[0293] In the case where the test data is a test image and its corresponding test questions, assuming that the test questions are answered by discussion or analysis based on the information included in the test image, the preset judgment model can use the method of determining the score of the predicted answer by the preset judgment model in the aforementioned embodiment to determine the judgment result of the predicted answer.

[0294] Assuming that the test question instructs the model to be evaluated to answer a numerical answer, since the numerical answer can only be correct or incorrect, the preset referee model can directly generate a referee result indicating whether the predicted answer is correct or incorrect. For example, the referee result can be 1 or 0, 1 indicates that the predicted answer is correct, and 0 indicates that the predicted answer is incorrect.

[0295] S806: Based on the judgment result of the predicted answer, determine the knowledge mastery level evaluation result corresponding to the model to be evaluated.

[0296] After the cycle is completed, the electronic device can determine the knowledge mastery evaluation result corresponding to the model to be evaluated based on the judgment result of the predicted answer. Specifically, the electronic device can use the average value of the judgment results as the knowledge mastery evaluation result corresponding to the model to be evaluated.

[0297] For example, a flow chart for determining the results of a knowledge mastery assessment could be as follows: Fig. 9 As shown, the following steps may be specifically included:

[0298] S901, obtaining test questions from the evaluation test data set;

[0299] First, the electronic device can obtain test questions from the evaluation test data set. The test questions are specifically:

[0300] Among light-emitting display devices, which one consumes the least power?

[0301] A. Plasma display device; B. Electroluminescent display device;

[0302] C. cathode ray tube display device; D. field emission display device.

[0303] S902, generating in-depth test questions;

[0304] Next, a preset in-depth test data generation model can be used to generate multiple in-depth test questions in combination with the stem and options of the above test questions. Specifically, the above test questions and the predicted answers output by the model to be evaluated for the above test questions are input into the preset in-depth test data generation model, and the preset in-depth test data generation model can generate an in-depth test question 1 as shown below:

[0305] How does a plasma display device work?

[0306] Input the in-depth test question 1 into the model to be evaluated, and the model to be evaluated can output the predicted answer as shown below:

[0307] The working principle of plasma display devices is based on the plasma luminescence phenomenon generated by gas discharge.

[0308] In the process of the model to be evaluated outputting the predicted answer to the in-depth test question 1, the predicted answer can be scored by multiple preset referee models. Next, the above in-depth test question 1 and its corresponding predicted answer can be input into the preset in-depth test data generation model, and the preset in-depth test data generation model can generate the in-depth test question 2 as shown below:

[0309] What is the difference in power consumption between electroluminescent display devices and plasma display devices?

[0310] Input the in-depth test question 2 into the model to be evaluated, and the model to be evaluated can output the predicted answer as shown below:

[0311] Generally, electroluminescent display devices are more energy-efficient than plasma display devices.

[0312] When the model to be evaluated outputs the predicted answer to the in-depth test question 2, the predicted answer can also be scored by multiple preset referee models.

[0313] S903, determining the knowledge mastery level assessment result.

[0314] After the above process is repeated for a preset number of times, the knowledge mastery evaluation result 0.7 corresponding to the model to be evaluated can be generated based on the scoring of each round by the preset referee model.

[0315] It can be seen that in the embodiment of the present invention, the electronic device can obtain at least one test data; input the test data and its corresponding prompt words into the model to be evaluated so that the model to be evaluated outputs the predicted answer of the test data; input the test data corresponding to the predicted answer output by the model to be evaluated the most recently, into the preset in-depth test data generation model, so that the preset in-depth test data generation model generates the in-depth test data corresponding to the test data; input the in-depth test data and its corresponding prompt words into the model to be evaluated so that the model to be evaluated outputs the predicted answer of the in-depth test data; through the preset referee model, based on the predicted answer output the most recently and the corresponding in-depth test data, from multiple preset referee dimensions, determine the referee result of the predicted answer; return to the step of inputting the test data corresponding to the predicted answer output by the model to be evaluated the most recently, into the preset in-depth test data generation model, until the number of cycles reaches the preset number of times, based on the referee result of the predicted answer, determine the knowledge mastery evaluation result corresponding to the model to be evaluated. Through the above implementation, multiple in-depth test data can be generated in a cyclic manner, and the referee result of the predicted answer output by the model to be evaluated for the in-depth test data can be determined by the preset referee model. Since the professional level of the in-depth test data is higher than that of the test data, and the technical field to which it belongs is the same as that of the test data, it can be used to evaluate the knowledge mastery of the model to be evaluated.

[0316] As an implementation method of the embodiment of the present invention, Fig.10 As shown, the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data may include:

[0317] S1001, inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model extracts key information from the test data, and generates a plurality of candidate in-depth test data corresponding to the test data based on the key information;

[0318] In the case where the test data is a test question, the electronic device can input the test question corresponding to the predicted answer output by the model to be evaluated most recently into the preset in-depth test data generation model, and the preset in-depth test data generation model can extract key information from the test question to determine the technical field to which the test question belongs. Next, the preset in-depth test data generation model can generate multiple candidate in-depth test data corresponding to the test question based on the key information.

[0319] In the case where the test data is a test image and its corresponding test question, the electronic device can input the test image corresponding to the predicted answer output by the model to be evaluated the most recently, and the test question corresponding to the test image, into a preset in-depth test data generation model, and the preset in-depth test data generation model can extract key information from the test image. Next, the preset in-depth test data generation model can generate multiple candidate in-depth test data corresponding to the test image based on the key information.

[0320] S1002: Randomly select a candidate in-depth test data from the plurality of candidate in-depth test data as the in-depth test data corresponding to the test data.

[0321] After acquiring multiple candidate in-depth test data corresponding to the test data, the electronic device may randomly select one candidate in-depth test data from the multiple candidate in-depth test data, and use the candidate in-depth test data as the in-depth test data corresponding to the test data.

[0322] For example, in the case where the test data is a test question, assume that the test question is as follows:

[0323] Among light-emitting display devices, which one consumes the least power?

[0324] A.PDP; B.ELD;

[0325] C.CRT; D.FED.

[0326] Then, the electronic device can obtain query 1 based on the Prompt template and the test question:

[0327] query1="""

[0328] Identify key words in the following text.

[0329] Among light-emitting display devices, which one consumes the least power?

[0330] A.PDP; B.ELD;

[0331] C.CRT; D.FED.

[0332] """

[0333] Next, the electronic device can input query1 into Qwen2.5-7B-Instruct (Qwen2.5-7B instruction model), and Qwen2.5-7B-Instruct will output the following:

[0334] The key words in this text include: light-emitting display device, power consumption, PDP, ELD, CRT, FED.

[0335] Then, the electronic device can obtain query2 based on the Prompt template and the above keywords:

[0336] query2=""

[0337] Generate questions using one or more of the following keywords.

[0338] Emitting display devices, power consumption, PDP, ELD, CRT, FED.

[0339] """

[0340] Qwen2.5-7B-Instruct can generate the following 10 alternative in-depth test questions based on query2:

[0341] 1. What are the main types of light-emitting display devices?

[0342] 2. What are the differences in power consumption among different types of light-emitting display devices (such as PDP, ELD, CRT, FED)?

[0343] 3. How does PDP work?

[0344] 4. What are the advantages and disadvantages of ELD compared with PDP?

[0345] 5. How does CRT work?

[0346] 6. What are the advantages and disadvantages of FED compared with traditional CRT display devices?

[0347] 7. Among the light-emitting display devices, which type is most widely used in modern display technology?

[0348] 8. What are the differences in display effects between different types of light-emitting display devices (such as PDP, ELD, CRT, FED)?

[0349] 9. Among the light-emitting display devices, which types are more suitable for use in mobile devices?

[0350] 10. What is the development trend of light-emitting display devices?

[0351] Finally, the electronic device can randomly sample the above-mentioned alternative in-depth test questions, that is, randomly select one alternative in-depth test question from the above-mentioned 10 alternative in-depth test questions as the in-depth test question corresponding to the above-mentioned test question.

[0352] For another example, when the test data is a test image and its corresponding test question, the electronic device may obtain the test image and its corresponding test question, and obtain query 1 based on the Prompt template and the test image (image):

[0353] query1="""

[0354] Generate questions based on the key information of the following image.

[0355]

[0356] """

[0357] The electronic device can input query1 into Qwen2-VL-7B-Instruct (Qwen2-Vision Language-7B-Instruct, Qwen2 Vision Language 7B Instruct Model), and Qwen2-VL-7B-Instruct can output the following alternative in-depth test questions:

[0358] 1. What is the position of semiconductor equipment in the semiconductor industry chain?

[0359] 2. From a global perspective, which region currently has the highest sales volume of semiconductor equipment?

[0360] 3. By what percentage did semiconductor equipment sales in Region A decrease in 2015 compared to 2014?

[0361] 4. By what percentage did semiconductor equipment sales in Region B decrease in 2015 compared to 2014?

[0362] 5. How many billion US dollars was the sales volume of semiconductor equipment in Region C in 2015?

[0363] 6. How many billion US dollars was the sales volume of semiconductor equipment in Region D in 2015?

[0364] 7. How many billion US dollars was the sales volume of semiconductor equipment in Region E in 2015?

[0365] 8. How many billion US dollars was the sales volume of semiconductor equipment in Region F in 2015?

[0366] 9. How many billion US dollars was the sales volume of semiconductor equipment in Region G in 2015?

[0367] 10. How many billion U.S. dollars was the sales volume of semiconductor equipment in other regions (except the above regions) in 2015?

[0368] Finally, the electronic device can randomly sample the above-mentioned alternative in-depth test questions, that is, randomly select one alternative in-depth test question from the above-mentioned 10 alternative in-depth test questions as the in-depth test question corresponding to the above-mentioned test question.

[0369] It can be seen that in an embodiment of the present invention, the electronic device can input the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model, so that the preset in-depth test data generation model extracts key information in the test data, and based on the key information, generates multiple candidate in-depth test data corresponding to the test data; randomly selects one candidate in-depth test data from the multiple candidate in-depth test data as the in-depth test data corresponding to the test data. Since the key information in the test data can reflect the technical field to which the test data belongs, the preset in-depth test data generation model can first extract the key information from the test data, and then generate the alternative in-depth test data based on the key information. In this way, it can be ensured that the alternative in-depth test data is the same as the technical field to which the test data belongs, and then one candidate in-depth test data is randomly selected from the multiple candidate in-depth test data as the in-depth test data, so that the in-depth test data can be efficiently generated.

[0370] As an implementation mode of an embodiment of the present invention, the above-mentioned test data may be a test question and the preset in-depth test data generation model may be a preset large language model, or the above-mentioned test data may be a test graph and its corresponding test question and the preset in-depth test data generation model may be a preset multimodal large model. In this case, the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data may include at least one of the following:

[0371] In the case where the test data is a test question, inputting the predicted answer most recently output by the model to be evaluated and the corresponding test question into a preset large language model, so that the preset large language model generates an in-depth test question corresponding to the test question;

[0372] In the case where the test data is a test image and its corresponding test question, the test image and test question corresponding to the predicted answer most recently output by the model to be evaluated are input into the preset multimodal large model, so that the preset multimodal large model generates an in-depth test question corresponding to the test image.

[0373] In a first embodiment, the test data is a test question, and the preset in-depth test data generation model is a preset large language model. The electronic device can input the predicted answer output by the model to be evaluated most recently and the corresponding test question into the preset large language model, and the preset large language model can generate an in-depth test question corresponding to the test question.

[0374] In a second embodiment, the test data is a test image and its corresponding test questions, and the preset in-depth test data generation model is a preset multimodal large model, and the model to be evaluated is a multimodal large model. Since the multimodal large model can process information in the image, in order to determine the degree of mastery of the information contained in the image by the model to be evaluated, the test image and the test question corresponding to the predicted answer output by the model to be evaluated most recently can be input into the preset multimodal large model, so that the preset multimodal large model generates an in-depth test question corresponding to the test image.

[0375] In this case, the model to be evaluated can be recorded as E_LLM (Evaluation_Large Language Model), the preset multimodal large model can be recorded as G_LLM (Generate_LargeLanguage Model), and the preset referee model can be recorded as Judge LLM (JudgeLarge Language Model).

[0376] The above-mentioned preset multimodal large model can be GPT-4o, InternVL2 Pro (Intern Visual-Language2 Professional Version), and Qwen2-VL-7B-Instruct, without specific limitation here.

[0377] The above test graphs and their corresponding test questions may be data language benchmark data, for example, MMBench (Multimodal Model Benchmark), OCRBench (Optical Character Recognition Benchmark), ChartBench (Chart Understanding Benchmark), and CMMMU (Cross-Modal Multi-Task Unified Benchmark).

[0378] For example, assuming that the test graph is a line graph, the line graph is used to reflect the number of tourists in Region A-Region C from 2019 to 2021. The electronic device can input the line graph and the corresponding test question "How many tourists were there in Region A in 2019?" into the model to be evaluated, and the model to be evaluated can output a predicted answer of 270 million based on the line graph. Assuming that the number of tourists in Region A in 2019 is 270 million, multiple preset referee models can be scored separately, and the average score obtained in this answer is 1.

[0379] Next, the electronic device can input the test image and the corresponding test question "How many tourists were there in region A in 2019?" into the preset multimodal large model, and the preset multimodal large model can generate a new test question "How many tourists were there in region C in 2021?" Assuming that the predicted answer output by the model to be evaluated is 650,000,000, and the number of tourists in region C in 2021 reflected in the test image is 250,000,000, then multiple preset judging models can be scored separately, and the average score obtained for this answer is 0.3.

[0380] Finally, the electronic device can perform a weighted summation on the average values ​​of the two rounds of answers of the model to be evaluated. The weights corresponding to the average values ​​of the two rounds of answers are equal, so the knowledge mastery evaluation result corresponding to the model to be evaluated can be obtained as 0.65.

[0381] For another example, a flowchart of the method for determining the results of the knowledge mastery evaluation can be as follows: Fig.11 As shown, the following steps may be specifically included:

[0382] S1101, obtaining benchmark test data;

[0383] The electronic equipment obtains the benchmark test data. The basic test data includes test chart 1101 and the corresponding test question 1 "How much is the sales volume of semiconductor equipment in region B in 2014 (billion)?" Among them, the title of test chart 1101 is: Semiconductor upstream analysis - semiconductor equipment, and the text content is: Semiconductor equipment is the foundation of the development of the semiconductor industry and the "crown" at the top of the semiconductor industry value chain. From a global perspective, XX continent is currently the region with the highest sales volume of semiconductor equipment, and the overall sales volume is slowly increasing. Region A ranks first in total sales volume, and the sales volume of semiconductor equipment in region D fell sharply by 37% in 2015.

[0384] Test chart 1101 also includes a bar chart, which reflects the sales volume of semiconductor equipment in each region. The unit of sales volume is (billion yuan), and the scale includes 0, 2, 4, 6, 8, 10 and 12. The white rectangle represents the sales volume in 2014, and the black rectangle represents the sales volume in 2015. It can be seen from the bar chart that for the sales volume in 2014 and 2015, the sales volume in region A is 9.42 and 9.64 respectively, in region B is 6.64 and 7.47 respectively, in region C is 4.18 and 5.49 respectively, and in region D is 8.16 and 5.12 respectively.

[0385] S1102, inputting the benchmark data into the model to be evaluated;

[0386] The electronic device can input basic test data into the model to be evaluated.

[0387] S1103, output the predicted answer;

[0388] The model to be evaluated can output the predicted answer 6.64 corresponding to the basic test data.

[0389] S1104, generating test questions;

[0390] Next, the electronic device can input the basic test data into the preset multimodal large model, and the preset multimodal large model generates test question 2: "What is the sales volume of semiconductor equipment in Region C in 2015 (billions)?"

[0391] S1105, input the test questions into the model to be evaluated;

[0392] The model to be evaluated can output the predicted answer 5.12 corresponding to test question 2.

[0393] S1106, evaluating whether the predicted answer is accurate through multiple preset referee models; if the number of cycles does not reach the preset number, returning to step S1104; if the number of cycles reaches the preset number, executing step S1107;

[0394] The electronic device can evaluate whether the predicted answer is accurate through multiple preset referee models. Specifically, the electronic device can input the test image 1101, test question 2 and predicted answer 5.12 into preset referee model 1-preset referee model n. The output of preset referee model 1: incorrect, the output of preset referee model 2: correct... The output of preset referee model n: incorrect, combined with the output of multiple referee models: incorrect. In a loop, multiple test questions can be generated, and the predicted answers output by the model to be evaluated for the test questions can be obtained, and the judgment results of the predicted answers output by the preset referee model can be obtained.

[0395] S1107, determine the comprehension ability assessment result.

[0396] When the number of cycles reaches the preset number, the judgment results output by multiple preset judgment models in each round can be combined to output the knowledge mastery assessment result 0.7.

[0397] It can be seen that in an embodiment of the present invention, when the test data is a test question, the electronic device can input the predicted answer and the corresponding test question output by the model to be evaluated most recently into the preset large language model, so that the preset large language model generates an in-depth test question corresponding to the test question; when the test data is a test image and its corresponding test question, the electronic device can input the test image and the test question corresponding to the predicted answer output by the model to be evaluated most recently into the preset multimodal large model, so that the preset multimodal large model generates an in-depth test question corresponding to the test image. When the test data is a test question, since the in-depth test question is generated based on the predicted answer and the corresponding test question output by the model to be evaluated most recently, the preset large language model can dynamically generate the in-depth test question based on the answer of the model to be evaluated. When the test data is a test image and its corresponding test question, since the in-depth test question is generated based on the test image and the test question, the in-depth test question can be used to evaluate the degree of mastery of the model to be evaluated for the image.

[0398] As an implementation of an embodiment of the present invention, the above-mentioned preset evaluation method can be a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method. In this case, Fig.12 As shown, the above step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain the evaluation result may include:

[0399] S1201, based on the evaluation test data set, respectively evaluating the model to be evaluated by the consistency evaluation method, the comprehension ability evaluation method, and the knowledge mastery degree evaluation method to obtain a first score, a second score, and a third score;

[0400] When the preset evaluation methods are consistency evaluation method, comprehension ability evaluation method and knowledge mastery level evaluation method, the electronic device can evaluate the model to be evaluated based on the evaluation test data set by consistency evaluation method, comprehension ability evaluation method and knowledge mastery level evaluation method, and obtain a first score, a second score and a third score.

[0401] Since the consistency assessment method, the comprehension ability assessment method and the knowledge mastery degree assessment method have been described in detail in the above embodiments, they will not be described in detail here.

[0402] S1202: Determine an evaluation result of the model to be evaluated based on the first score, the second score, and the third score.

[0403] In order to make the evaluation result reflect the consistency, comprehension ability and knowledge mastery of the model to be evaluated, the electronic device can determine the evaluation result of the model to be evaluated based on the first score, the second score and the third score. Specifically:

[0404] In a first implementation, the sum of the first score, the second score and the third score can be used as the evaluation result of the model to be evaluated. In a second implementation, the average of the first score, the second score and the third score can be used as the evaluation result of the model to be evaluated.

[0405] For example, a schematic diagram for determining the evaluation results could be as follows Fig.13 As shown, the following steps may be specifically included:

[0406] S1301, obtaining test questions from an evaluation test data set;

[0407] The electronic device can obtain test questions from the evaluation test data set. The test questions are as follows:

[0408] Among light-emitting display devices, which one consumes the least power?

[0409] A. Plasma display device; B. Electroluminescent display device;

[0410] C. cathode ray tube display device; D. field emission display device.

[0411] S1302, adding or rewriting the template;

[0412] By adding or rewriting the template, you can get combination result 1 and combination result 2, as shown below:

[0413] Combination result 1:

[0414] The following are multiple-choice questions. Each question contains four options, only one of which is correct. Please choose the correct answer.

[0415] Among light-emitting display devices, which one consumes the least power?

[0416] A. Plasma display device; B. Electroluminescent display device;

[0417] C. cathode ray tube display device; D. field emission display device.

[0418] Combination result 2:

[0419] The following are multiple-choice questions about the field of semiconductor display technology. Only one option meets the requirements of the question. Please select the option you think is correct. Please only output the correct answer without analysis.

[0420] Among light-emitting display devices, which one consumes the least power?

[0421] A. Plasma display device; B. Electroluminescent display device;

[0422] C. cathode ray tube display device; D. field emission display device.

[0423] Through the combination result 1 and the combination result 2, it can be determined that the consistency evaluation result corresponding to the model to be evaluated is 0.

[0424] S1303, rewriting the test questions;

[0425] By rewriting the test questions, we can get rewritten test questions 1 to rewritten test questions 3, which are as follows:

[0426] Rewrite Test 1:

[0427] Among light-emitting display devices, which type has the lowest power consumption?

[0428] A. Plasma display device; B. Electroluminescent display device;

[0429] C. cathode ray tube display device; D. field emission display device.

[0430] Rewrite Test 2:

[0431] Among light-emitting display devices, which type has the lowest power consumption?

[0432] A. Field emission display device; B. Electroluminescent display device;

[0433] C. cathode ray tube display device; D. plasma display device.

[0434] Rewrite Test 3:

[0435] Among light-emitting display devices, which one consumes the most power?

[0436] A. Plasma display device; B. Electroluminescent display device;

[0437] C. cathode ray tube display device; D. field emission display device.

[0438] By rewriting test question 1-rewriting test question 3, it can be determined that the comprehension ability evaluation result corresponding to the model to be evaluated is 1.

[0439] S1304, generating in-depth test questions;

[0440] By generating in-depth test questions, in-depth test question 1 and in-depth test question 2 can be obtained, as shown below:

[0441] In-depth test question 1: What is the working principle of plasma display devices?

[0442] In-depth test question 2: What is the difference in power consumption between electroluminescent display devices and plasma display devices?

[0443] By inputting in-depth test questions 1 and 2 into the model to be evaluated, the predicted answer output by the model to be evaluated for in-depth test question 1 can be obtained: "The working principle of plasma display devices is based on the plasma luminescence phenomenon generated by gas discharge." And the predicted answer output for in-depth test question 2 can be obtained: "Generally speaking, electroluminescent display devices are more energy-efficient than plasma display devices." In the above process, the predicted answers can be scored by multiple preset referee models. Finally, it can be determined that the knowledge mastery evaluation result corresponding to the model to be evaluated is 0.7.

[0444] By adding the above consistency assessment results, comprehension ability assessment results, and knowledge mastery level assessment results, we can get the overall assessment result of the model to be evaluated, which is 1.7.

[0445] It can be seen that in the embodiment of the present invention, the electronic device can evaluate the model to be evaluated based on the evaluation test data set through the consistency evaluation method, the understanding ability evaluation method and the knowledge mastery degree evaluation method, respectively, to obtain the first score, the second score and the third score; based on the first score, the second score and the third score, determine the evaluation result of the model to be evaluated. Since the evaluation result is determined based on the first score, the second score and the third score, and the first score, the second score and the third score are determined by the consistency evaluation method, the understanding ability evaluation method and the knowledge mastery degree evaluation method, the evaluation result can simultaneously reflect the consistency, understanding ability and knowledge mastery degree of the model to be evaluated, and realize the integration of the scores corresponding to the consistency, understanding ability and knowledge mastery degree to obtain a more comprehensive evaluation result.

[0446] As an implementation method of the embodiment of the present invention, Fig.14 As shown, the above step of obtaining the evaluation test data set may include:

[0447] S1401, inputting a preset question matching the application field of the model to be evaluated into an answer generating device, so that the answer generating device returns an answer corresponding to the preset question;

[0448] In order to construct an evaluation test data set, the electronic device can input a preset question matching the application field of the model to be evaluated into the answer generation device, and the answer generation device can return the answer corresponding to the preset question. The answer generation device can be an information retrieval framework or a preset large language model, and the information retrieval framework can be a rag (Retrieval Augmented Generator) yield analysis platform.

[0449] S1402, obtaining test data constructed by the user based on the preset questions and their corresponding answers and their corresponding standard answers;

[0450] After obtaining the answers to the preset questions, the user can construct test data and their corresponding standard answers based on the preset questions and their corresponding answers, and the electronic device can obtain the test data constructed by the user and its corresponding standard answers.

[0451] S1403, storing the corresponding relationship between the test data and its corresponding standard answer to obtain an evaluation test data set.

[0452] Next, the electronic device can store the correspondence between the test data and its corresponding standard answer to obtain an evaluation test data set. For example, the electronic device can input the preset question "What are the Mura (uneven brightness of the display) type defects?" into the answer generation device, and the answer generation device can output the answer "1. White Gap; 2. Black Mura; 3. Horizontal stripes." Then, the user can construct the following test data based on the preset question and answer:

[0453] Which of the following is not a mura defect?

[0454] A, White Gap; B, Black Mura;

[0455] C. Horizontal lines; D. Lead Short (lead short circuit).

[0456] The annotated answer to the above test data is D. The electronic device can store the corresponding relationship between the above test data and the standard answer to obtain an evaluation test data set.

[0457] For another example, GPT-4o can be used to verify the model’s understanding of industry terms. The electronic device can input the preset question “What does H-Noise (Horizontal Noise) mean?” into GPT-4o, and GPT-4o can output the answer as shown below:

[0458] "H-Noise" usually refers to a specific type of noise that may be related to signal processing, audio engineering, or communication systems. In these fields, noise refers to interference introduced during signal transmission or processing that may affect the quality and clarity of the signal.

[0459] Since GPT-4o's understanding of H-Noise is different from that of the semiconductor display field, this industry term can be added to the inspection test set, and the following test data can be constructed based on this industry term:

[0460] In the field of semiconductor displays, what does "H-Noise" mean?

[0461] A. Horizontal interference

[0462] B. Random noise

[0463] C. High frequency noise

[0464] D. Parallel interference

[0465] The annotated answer to the above test data is A. The electronic device can store the corresponding relationship between the above test data and the standard answer to obtain an evaluation test data set.

[0466] For example, a flowchart for constructing an evaluation test dataset can be as follows: Fig.15 As shown, the following steps may be specifically included:

[0467] S1501, obtaining preset questions from a knowledge base;

[0468] Electronic devices can obtain preset questions from the knowledge base

[0469] S1502, generating answers corresponding to preset questions;

[0470] Input the preset question into the answer generation device, and the answer generation device can generate the answer corresponding to the preset question. The answer generation device can be GPT-4o or rag yield analysis platform.

[0471] S1503, user review feedback;

[0472] The test data and the corresponding standard answers constructed by the user based on the preset questions and their corresponding answers are obtained.

[0473] S1504, modification and update;

[0474] The electronic device can establish a correspondence between the test data and its corresponding standard answer.

[0475] S1505, generate an evaluation test data set.

[0476] The electronic device can store the above correspondence to obtain an evaluation test data set.

[0477] For example, the evaluation test data set may include multiple-choice questions and single-choice questions, wherein the multiple-choice questions and their corresponding standard answers may be as follows:

[0478] Which of the following are common defects in mask defect rate?

[0479] A. Line defect; B. Point defect;

[0480] C. Horizontal black occlusion; D. Accident batch.

[0481] Standard answer: ABC.

[0482] Multiple-choice questions and their corresponding standard answers can be shown as follows:

[0483] In the field of semiconductor displays, what does active disconnect mean?

[0484] A. Actively open the connection; B. Active layer missing;

[0485] C. Passively open; D. The pixel enters the on state.

[0486] Standard answer: B.

[0487] It can be seen that in the embodiment of the present invention, the electronic device can input the preset question matching the application field of the model to be evaluated into the answer generation device, so that the answer generation device returns the answer corresponding to the preset question, wherein the answer generation device is an information retrieval framework or a preset large language model; obtain the test data and the corresponding standard answer constructed by the user based on the preset question and its corresponding answer; store the corresponding relationship between the test data and its corresponding standard answer to obtain the evaluation test data set. In this way, the evaluation test data set can be quickly constructed in combination with the answer generated by the answer generation device for the preset question.

[0488] As an implementation manner of an embodiment of the present invention, after the step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, the method may further include:

[0489] Based on the evaluation result, optimization direction prompt information corresponding to the model to be evaluated is generated.

[0490] After obtaining the evaluation result of the model to be evaluated, in order to optimize the model to be evaluated in a targeted manner, the electronic device can generate optimization direction prompt information corresponding to the model to be evaluated. In the case where the evaluation result indicates that the correctness of the predicted answer does not meet the correctness requirement, the optimization direction prompt information can be used to prompt: to add professional knowledge data in the technical field applied by the model to be evaluated to the training data of the model to be evaluated.

[0491] In the case where the evaluation result indicates that the correlation between the predicted answer and its corresponding test data does not meet the correlation requirement, the optimization direction prompt information can be used to prompt: in the application process of the model to be evaluated, information related to the problem being processed is obtained from the external knowledge base. Specifically, the model to be evaluated can obtain information related to the problem being processed from the external knowledge base through the rag technology, thereby improving the correlation between the predicted answer and the problem.

[0492] When the level of detail of the predicted answer represented by the evaluation result does not meet the level of detail requirement, the optimization direction prompt information can be used to prompt: to add knowledge data of the technical field to which the predicted answer and the corresponding question belong in the training data of the model to be evaluated.

[0493] When the evaluation result represents that the logic of the predicted answer does not meet the logic requirements, the optimization direction prompt information can be used to prompt: add open source training data used to enhance the logic of the model to be evaluated to the training data of the model to be evaluated. The open source training data can be CoT (Chain of Thought) data.

[0494] In the case where the evaluation result indicating that the language coherence of the predicted answer does not meet the language coherence requirements, the optimization direction hint information can be used to hint: increase the richness of the vocabulary and sentence patterns used to generate the training data.

[0495] It can be seen that in the embodiment of the present invention, the electronic device can generate optimization direction prompt information corresponding to the model to be evaluated based on the evaluation result. In this way, after obtaining the evaluation result of the model to be evaluated, the model to be evaluated can be optimized in a targeted manner.

[0496] Corresponding to the above-mentioned model evaluation method, an embodiment of the present invention further provides a model evaluation device. The following is an introduction to a model evaluation device provided by an embodiment of the present invention.

[0497] like Fig.16 As shown, a model evaluation device, the device comprising:

[0498] The data set acquisition module 1601 is used to acquire an evaluation test data set, wherein the evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied;

[0499] A model evaluation module 1602 is used to evaluate the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method;

[0500] The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, where the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, where the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

[0501] It can be seen that in an embodiment of the present invention, an electronic device can obtain an evaluation test data set, wherein the evaluation test data set includes multiple test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; the consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data, the comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, and the multiple rewritten test data are obtained by rewriting the same test data, and the knowledge mastery degree evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data corresponding to the test data, and the in-depth test data belongs to the same technical field as the test data, and the professional level is higher than the test data. With respect to consistency, assuming that the consistency of the model to be evaluated is high, then the consistency between the predicted answers output by the model to be evaluated based on multiple prompt words for the same test data is high. Therefore, the consistency of the model to be evaluated can be evaluated based on the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. For comprehension ability, since the professional level of the rewritten test data is similar to that of the test data, the difference is that the expression method is different. Therefore, assuming that the comprehension ability of the model to be evaluated is strong, then when the model to be evaluated masters one test data, the model to be evaluated can master multiple rewritten test data obtained by rewriting the test data. Therefore, the electronic device can evaluate the comprehension ability of the model to be evaluated based on the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. For the degree of knowledge mastery, since the in-depth test data and the test data belong to the same technical field, and the degree of professionalism is higher than the test data, it is assumed that the model to be evaluated has a higher degree of knowledge mastery for a certain technical field, then the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data is higher. Therefore, the knowledge mastery of the model to be evaluated can be evaluated by the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data. In this way, the consistency, comprehension ability and knowledge mastery of the large language model can be evaluated.

[0502] As an implementation of an embodiment of the present invention, the preset evaluation method is a consistency evaluation method, and the test data is a test question. In this case, the model evaluation module 1602 includes:

[0503] A first test question acquisition submodule, used to acquire at least one test question and acquire a plurality of different prompt words corresponding to the test question;

[0504] A combination submodule is used for combining the test questions with a plurality of different prompt words corresponding to the test questions according to a preset combination mode to obtain a plurality of combination results;

[0505] A first predicted answer acquisition submodule, configured to input the plurality of combined results into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test question included in each combined result based on an indication of a prompt word included in the combined result;

[0506] The consistency evaluation result determination submodule is used to determine the consistency evaluation result corresponding to the model to be evaluated based on the differences between the multiple predicted answers corresponding to the test question.

[0507] As an implementation of an embodiment of the present invention, the test question acquisition submodule may include:

[0508] A prompt word generating unit, used for inputting a prompt word generating instruction into a preset large language model, so that the preset large language model generates a plurality of candidate prompt words based on the prompt word generating instruction, wherein the prompt word generating instruction is used for indicating the prompt information of the answering method corresponding to the test question to be generated;

[0509] a prompt word determination unit, configured to use the candidate prompt word as the prompt word corresponding to the test question if the candidate prompt word meets a prompt word condition, wherein the prompt word condition indicates that the candidate prompt word can indicate the answer method corresponding to the test question of the model to be evaluated if it meets the expression method of natural language;

[0510] The instruction modification unit is used to obtain a modified prompt word generation instruction when the candidate prompt word does not meet the prompt word condition, and return to the step of inputting the prompt word generation instruction into the preset large language model until the generated candidate prompt word meets the prompt word condition.

[0511] As an implementation manner of an embodiment of the present invention, the above-mentioned consistency evaluation result determination submodule may include:

[0512] A first result determination unit, configured to determine, when the test question is an objective question, a consistency evaluation result corresponding to the model to be evaluated based on the consistency between a plurality of predicted answers corresponding to the test question and a standard answer;

[0513] A second result determination unit is used to determine the consistency evaluation result corresponding to the model to be evaluated based on the similarity between multiple predicted answers corresponding to the test question and the standard answer when the test question is a subjective question and there is a standard answer;

[0514] The third result determination unit is used to determine the consistency evaluation result corresponding to the model to be evaluated from multiple preset judgment dimensions through a preset judgment model when the test question is a subjective question and there is no standard answer, wherein the preset judgment dimension is used to reflect the degree of match between the predicted answer and the test question.

[0515] As an implementation manner of an embodiment of the present invention, the first result determination unit includes:

[0516] The result determination subunit is used to determine that the consistency evaluation result corresponding to the model to be evaluated is a first preset result if the multiple predicted answers corresponding to the test question are all consistent with the standard answer; if there is a predicted answer different from the standard answer among the multiple predicted answers corresponding to the test question, determine that the consistency evaluation result corresponding to the model to be evaluated is a second preset result.

[0517] As an implementation manner of an embodiment of the present invention, the third result determination unit includes:

[0518] a score determination subunit, configured to input the test question and a plurality of predicted answers corresponding to the test question into a preset judging model, so that the preset judging model generates a score corresponding to each predicted answer in each preset judging dimension, wherein the preset judging dimension includes at least one of the correctness of the predicted answer, the correlation between the predicted answer and the test data corresponding to it, the language coherence of the predicted answer, the detail level of the predicted answer, and the logic of the predicted answer;

[0519] The average value calculation subunit is used to calculate the average value of the score corresponding to each predicted answer in each preset judging dimension; based on the average value corresponding to each preset judging dimension and the preset weight, the average value is weighted summed to obtain the consistency evaluation result corresponding to the model to be evaluated.

[0520] As an implementation of an embodiment of the present invention, the above consistency evaluation method is a comprehension ability evaluation method, and the test data is a test question. In this case, the above model evaluation module 1602 includes:

[0521] A second test question obtaining submodule, used to obtain at least one test question;

[0522] A test question rewriting submodule, used for inputting the test question into a preset large language model, so that the preset large language model rewrites the test question in a preset rewriting manner to generate a plurality of rewritten test questions, wherein the preset rewriting manner is used for rewriting at least one of the stem, options and type of the test question;

[0523] A second predicted answer acquisition submodule is used to input each rewritten test question and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to each rewritten test question;

[0524] The comprehension ability evaluation result determination submodule is used to determine the comprehension ability evaluation result corresponding to the model to be evaluated based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question.

[0525] As an implementation of an embodiment of the present invention, the above-mentioned preset rewriting method includes at least one of the following:

[0526] If the test question is a single-choice question, rewrite the test question into a multiple-choice question or a true-or-false question;

[0527] If the test question is a multiple-choice question, rewrite the test question into a single-choice question or a true-or-false question;

[0528] Replacing the words included in the test questions with synonyms;

[0529] Without changing the substantial content of the test question stem, rewriting the test question stem from an affirmative sentence to a negative sentence;

[0530] Change the order of the options included in the test question.

[0531] As an implementation of an embodiment of the present invention, the above-mentioned preset evaluation method is a knowledge mastery degree evaluation method. In this case, the above-mentioned model evaluation module 1602 includes:

[0532] A test data acquisition submodule, used to acquire at least one test data;

[0533] A third predicted answer acquisition submodule is used to input the test data and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs the predicted answer of the test data;

[0534] The in-depth test data acquisition submodule is used to input the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data;

[0535] A fourth predicted answer acquisition submodule is used to input the in-depth test data and its corresponding prompt words into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the in-depth test data;

[0536] A judgment result determination submodule, for determining the judgment result of the predicted answer from a plurality of preset judgment dimensions through a preset judgment model based on the most recently outputted predicted answer and the corresponding in-depth test data;

[0537] The return submodule is used to return to the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model until the number of cycles reaches the preset number, and based on the judgment result of the predicted answer, determine the knowledge mastery evaluation result corresponding to the model to be evaluated.

[0538] As an implementation of an embodiment of the present invention, the in-depth test data acquisition submodule includes:

[0539] A key information determination unit, configured to input the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model extracts key information from the test data, and generates a plurality of candidate in-depth test data corresponding to the test data based on the key information;

[0540] The in-depth test data determining unit randomly selects one candidate in-depth test data from the plurality of candidate in-depth test data as the in-depth test data corresponding to the test data.

[0541] As an implementation mode of an embodiment of the present invention, the test data is a test question and the preset in-depth test data generation model is a preset large language model, or the test data is a test graph and its corresponding test question and the preset in-depth test data generation model is a preset multimodal large model. In this case, the in-depth test data acquisition submodule includes:

[0542] A first in-depth test question generating unit is configured to input the predicted answer outputted most recently by the model to be evaluated and the corresponding test question into a preset large language model when the test data is a test question, so that the preset large language model generates an in-depth test question corresponding to the test question;

[0543] The second in-depth test question generating unit is used to input the test picture and the test question corresponding to the predicted answer most recently output by the model to be evaluated into the preset multimodal large model when the test data is a test picture and its corresponding test question, so that the preset multimodal large model generates an in-depth test question corresponding to the test picture.

[0544] As an implementation of an embodiment of the present invention, the preset evaluation method is a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method. In this case, the model evaluation module 1602 includes:

[0545] A score determination submodule is used to evaluate the model to be evaluated based on the evaluation test data set by using the consistency evaluation method, the comprehension ability evaluation method, and the knowledge mastery evaluation method to obtain a first score, a second score, and a third score;

[0546] An evaluation result determination submodule is used to determine the evaluation result of the model to be evaluated based on the first score, the second score and the third score.

[0547] As an implementation of an embodiment of the present invention, the data set acquisition module 1601 includes:

[0548] An answer generation submodule, used for inputting a preset question matching the application field of the model to be evaluated into an answer generation device, so that the answer generation device returns an answer corresponding to the preset question, wherein the answer generation device is an information retrieval framework or a preset large language model;

[0549] The test data construction submodule is used to obtain the test data and the corresponding standard answers constructed by the user based on the preset questions and their corresponding answers;

[0550] The corresponding relationship storage submodule is used to store the corresponding relationship between the test data and its corresponding standard answer to obtain an evaluation test data set.

[0551] As an implementation of the embodiment of the present invention, the above device may further include:

[0552] A prompt information generation submodule, used to generate optimization direction prompt information corresponding to the model to be evaluated based on the evaluation result;

[0553] Wherein, when the evaluation result indicates that the correctness of the predicted answer does not meet the correctness requirement, the optimization direction prompt information is used to prompt: to add professional knowledge data in the technical field applied by the model to be evaluated to the training data of the model to be evaluated;

[0554] In the case where the evaluation result indicates that the correlation between the predicted answer and the corresponding test data does not meet the correlation requirement, the optimization direction prompt information is used to prompt: in the application process of the model to be evaluated, information related to the currently processed problem is obtained from an external knowledge base;

[0555] In the case where the evaluation result indicates that the detail level of the predicted answer does not meet the detail level requirement, the optimization direction prompt information is used to prompt: to add knowledge data in the technical field to which the predicted answer and the corresponding question belong in the training data of the model to be evaluated;

[0556] In the case where the evaluation result indicates that the logic of the predicted answer does not meet the logic requirement, the optimization direction prompt information is used to prompt: adding open source training data for enhancing the logic of the model to be evaluated to the training data of the model to be evaluated;

[0557] In a case where the evaluation result indicates that the language coherence of the predicted answer does not meet the language coherence requirement, the optimization direction prompt information is used to prompt: increase the richness of the vocabulary and sentence patterns used to generate the training data.

[0558] The embodiment of the present invention further provides an electronic device, such as Fig.17 As shown, it includes a processor 1701 , a communication interface 1702 , a memory 1703 and a communication bus 1704 , wherein the processor 1701 , the communication interface 1702 , and the memory 1703 communicate with each other via the communication bus 1704 .

[0559] Memory 1703, used for storing computer programs;

[0560] The processor 1701 is used to implement the model evaluation method described in any of the above embodiments when executing the program stored in the memory 1703.

[0561] It can be seen that in an embodiment of the present invention, an electronic device can obtain an evaluation test data set, wherein the evaluation test data set includes multiple test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; the consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data, the comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, and the multiple rewritten test data are obtained by rewriting the same test data, and the knowledge mastery degree evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data corresponding to the test data, and the in-depth test data belongs to the same technical field as the test data, and the professional level is higher than the test data. With respect to consistency, assuming that the consistency of the model to be evaluated is high, then the consistency between the predicted answers output by the model to be evaluated based on multiple prompt words for the same test data is high. Therefore, the consistency of the model to be evaluated can be evaluated based on the consistency between the predicted answers output by the model to be evaluated based on different prompt words for the same test data. For comprehension ability, since the professional level of the rewritten test data is similar to that of the test data, the difference is that the expression method is different. Therefore, assuming that the comprehension ability of the model to be evaluated is strong, then when the model to be evaluated masters one test data, the model to be evaluated can master multiple rewritten test data obtained by rewriting the test data. Therefore, the electronic device can evaluate the comprehension ability of the model to be evaluated based on the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data. For the degree of knowledge mastery, since the in-depth test data and the test data belong to the same technical field, and the degree of professionalism is higher than the test data, it is assumed that the model to be evaluated has a higher degree of knowledge mastery for a certain technical field, then the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data is higher. Therefore, the knowledge mastery of the model to be evaluated can be evaluated by the accuracy of the predicted answers output by the model to be evaluated for the in-depth test data. In this way, the consistency, comprehension ability and knowledge mastery of the large language model can be evaluated.

[0562] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0563] The communication interface is used for communication between the above electronic device and other devices.

[0564] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0565] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0566] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model evaluation methods are implemented.

[0567] In another embodiment provided by the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model evaluation method in the above embodiments.

[0568] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0569] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0570] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium, computer-readable storage medium and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0571] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire an evaluation test data set, wherein the evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; Based on the evaluation test data set, the model to be evaluated is evaluated by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, where the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, where the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

2. The method according to claim 1, characterized in that: The preset evaluation method is the consistency evaluation method, and the test data is a test question; The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes: Obtain at least one test question, and obtain a plurality of different prompt words corresponding to the test question; For the test question, according to a preset combination method, a plurality of different prompt words corresponding to the test question are respectively combined with the test question to obtain a plurality of combination results; Inputting the plurality of combined results into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test question included in each combined result based on the indication of the prompt word included in the combined result; Based on the differences between the multiple predicted answers corresponding to the test question, a consistency evaluation result corresponding to the model to be evaluated is determined.

3. The method according to claim 2, characterized in that The step of obtaining a plurality of different prompt words corresponding to the test question includes: Inputting a prompt word generation instruction into a preset large language model, so that the preset large language model generates a plurality of candidate prompt words based on the prompt word generation instruction, wherein the prompt word generation instruction is used to instruct the generation of prompt information of an answer method corresponding to the test question; When the candidate prompt word meets the prompt word condition, the candidate prompt word is used as the prompt word corresponding to the test question, wherein the prompt word condition indicates that the candidate prompt word can indicate the answer method corresponding to the test question of the model to be evaluated when it meets the expression method of natural language; In the case that the candidate prompt word does not meet the prompt word condition, a modified prompt word generation instruction is obtained, and the process returns to the step of inputting the prompt word generation instruction into the preset large language model until the generated candidate prompt word meets the prompt word condition.

4. The method according to claim 2, characterized in that: The step of determining the consistency evaluation result corresponding to the model to be evaluated based on the difference between the multiple predicted answers corresponding to the test question includes at least one of the following: In the case where the test question is an objective question, based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer, determining the consistency evaluation result corresponding to the model to be evaluated; When the test question is a subjective question and there is a standard answer, based on the similarity between the multiple predicted answers corresponding to the test question and the standard answer, determining the consistency evaluation result corresponding to the model to be evaluated; When the test question is a subjective question and there is no standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined from multiple preset judgment dimensions through a preset judgment model, wherein the preset judgment dimension is used to reflect the degree of match between the predicted answer and the test question.

5. The method according to claim 4, characterized in that The step of determining the consistency evaluation result corresponding to the model to be evaluated based on the consistency between the multiple predicted answers corresponding to the test question and the standard answer includes: If the multiple predicted answers corresponding to the test question are all consistent with the standard answer, the consistency evaluation result corresponding to the model to be evaluated is determined to be the first preset result; if there is a predicted answer different from the standard answer among the multiple predicted answers corresponding to the test question, the consistency evaluation result corresponding to the model to be evaluated is determined to be the second preset result; The step of determining the consistency evaluation result corresponding to the model to be evaluated from multiple preset judgment dimensions by using a preset judgment model includes: Inputting the test question and the multiple predicted answers corresponding to the test question into a preset judging model, so that the preset judging model generates a score corresponding to each predicted answer in each preset judging dimension, wherein the preset judging dimension includes at least one of the correctness of the predicted answer, the correlation between the predicted answer and the corresponding test data, the language coherence of the predicted answer, the detail level of the predicted answer, and the logic of the predicted answer; For each preset judging dimension, calculate the average value of the score corresponding to each predicted answer in the preset judging dimension; based on the average value corresponding to each preset judging dimension and the preset weight, perform weighted summation on the average values ​​to obtain the consistency evaluation result corresponding to the model to be evaluated.

6. The method according to claim 1, characterized in that The consistency assessment method is the comprehension ability assessment method, and the test data is a test question; The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes: Get at least one test question; Inputting the test question into a preset large language model, so that the preset large language model rewrites the test question in a preset rewriting manner to generate a plurality of rewritten test questions, wherein the preset rewriting manner is used to rewrite at least one of the stem, options, and type of the test question; Inputting each rewritten test question and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to each rewritten test question; Based on the difference between the predicted answer and the standard answer corresponding to each rewritten test question, the comprehension ability evaluation result corresponding to the model to be evaluated is determined.

7. The method according to claim 6, characterized in that The preset rewriting method includes at least one of the following: If the test question is a single-choice question, rewrite the test question into a multiple-choice question or a true-or-false question; If the test question is a multiple-choice question, rewrite the test question into a single-choice question or a true-or-false question; Replacing the words included in the test questions with synonyms; Without changing the substantial content of the test question stem, rewriting the test question stem from an affirmative sentence to a negative sentence; Change the order of the options included in the test question.

8. The method according to claim 1, characterized in that: The preset evaluation method is the knowledge mastery degree evaluation method; The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes: Obtain at least one test data; Inputting the test data and its corresponding prompt word into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the test data; Inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generates in-depth test data corresponding to the test data; Inputting the in-depth test data and its corresponding prompt words into the model to be evaluated, so that the model to be evaluated outputs a predicted answer to the in-depth test data; By using a preset judging model, based on the most recently outputted predicted answer and the corresponding in-depth test data, determining the judging result of the predicted answer from multiple preset judging dimensions; Return to the step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into the preset in-depth test data generation model until the number of cycles reaches the preset number, and determine the knowledge mastery evaluation result corresponding to the model to be evaluated based on the judgment result of the predicted answer.

9. The method according to claim 8, characterized in that The step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data comprises: Input the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model, so that the preset in-depth test data generation model extracts key information from the test data, and generates a plurality of candidate in-depth test data corresponding to the test data based on the key information; A candidate in-depth test data is randomly selected from the plurality of candidate in-depth test data as the in-depth test data corresponding to the test data.

10. The method according to claim 8, characterized in that The test data is a test question and the preset in-depth test data generation model is a preset large language model, or the test data is a test graph and its corresponding test question and the preset in-depth test data generation model is a preset multimodal large model; The step of inputting the test data corresponding to the predicted answer most recently output by the model to be evaluated into a preset in-depth test data generation model so that the preset in-depth test data generation model generates in-depth test data corresponding to the test data comprises: In the case where the test data is a test question, inputting the predicted answer most recently output by the model to be evaluated and the corresponding test question into a preset large language model, so that the preset large language model generates an in-depth test question corresponding to the test question; In the case where the test data is a test image and its corresponding test question, the test image and test question corresponding to the predicted answer most recently output by the model to be evaluated are input into the preset multimodal large model, so that the preset multimodal large model generates an in-depth test question corresponding to the test image.

11. The method according to claim 1, characterized in that: The preset evaluation methods are the consistency evaluation method, the comprehension ability evaluation method and the knowledge mastery degree evaluation method; The step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result includes: Based on the evaluation test data set, the model to be evaluated is evaluated by the consistency evaluation method, the comprehension ability evaluation method, and the knowledge mastery degree evaluation method to obtain a first score, a second score, and a third score; An evaluation result of the model to be evaluated is determined based on the first score, the second score, and the third score.

12. The method according to any one of claims 1 to 11, characterized in that: The step of obtaining the evaluation test data set includes: Inputting a preset question matching the application field of the model to be evaluated into an answer generation device, so that the answer generation device returns an answer corresponding to the preset question, wherein the answer generation device is an information retrieval framework or a preset large language model; Obtaining test data constructed by the user based on the preset questions and their corresponding answers and their corresponding standard answers; The correspondence between the test data and its corresponding standard answer is stored to obtain an evaluation test data set.

13. The method according to any one of claims 1 to 11, characterized in that: After the step of evaluating the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, the method further includes: Based on the evaluation result, generating optimization direction prompt information corresponding to the model to be evaluated; Wherein, when the evaluation result indicates that the correctness of the predicted answer does not meet the correctness requirement, the optimization direction prompt information is used to prompt: to add professional knowledge data in the technical field applied by the model to be evaluated to the training data of the model to be evaluated; In the case where the evaluation result indicates that the correlation between the predicted answer and the corresponding test data does not meet the correlation requirement, the optimization direction prompt information is used to prompt: in the application process of the model to be evaluated, information related to the currently processed problem is obtained from an external knowledge base; In the case where the evaluation result indicates that the detail level of the predicted answer does not meet the detail level requirement, the optimization direction prompt information is used to prompt: to add knowledge data in the technical field to which the predicted answer and the corresponding question belong in the training data of the model to be evaluated; In the case where the evaluation result indicates that the logic of the predicted answer does not meet the logic requirement, the optimization direction prompt information is used to prompt: adding open source training data for enhancing the logic of the model to be evaluated to the training data of the model to be evaluated; In a case where the evaluation result indicates that the language coherence of the predicted answer does not meet the language coherence requirement, the optimization direction prompt information is used to prompt: increase the richness of the vocabulary and sentence patterns used to generate the training data.

14. A model evaluation device, characterized in that: The device comprises: A data set acquisition module is used to acquire an evaluation test data set, wherein the evaluation test data set includes a plurality of test data, and the technical field to which the content included in the evaluation test data set belongs matches the technical field to which the model to be evaluated is applied; A model evaluation module, used to evaluate the model to be evaluated based on the evaluation test data set by a preset evaluation method to obtain an evaluation result, wherein the preset evaluation method includes at least one of a consistency evaluation method, a comprehension ability evaluation method, and a knowledge mastery degree evaluation method; The consistency evaluation method is used to evaluate the consistency between the predicted answers output by the model to be evaluated for the same test data based on different prompt words. The comprehension ability evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for multiple rewritten test data, where the multiple rewritten test data are obtained by rewriting the same test data. The knowledge mastery level evaluation method is used to evaluate the accuracy of the predicted answers output by the model to be evaluated for in-depth test data corresponding to the test data, where the in-depth test data is in the same technical field as the test data and has a higher level of professionalism than the test data.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-13 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 13 are implemented.

Images