Model evaluation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model evaluation method and device, a storage medium and a program product. In the model evaluation method, after a reply result output by a target model for a target question is obtained, a question type corresponding to the target question can be obtained, and a target evaluation model matched with the question type in a plurality of evaluation models is called to evaluate the reply result. In the embodiment, in the multiple evaluation models, different evaluation models correspond to different question types, and different evaluation models have different parameter quantities. Based on this, a scheme for cooperatively evaluating the reply results output by the target model for different types of questions based on the evaluation models with different parameter quantities is realized, so that the evaluation model with low parameter magnitude can be conveniently adopted to evaluate the reply results of some questions with low complexity, the resource consumption is effectively reduced, and the evaluation efficiency is improved. And an evaluation model with relatively high parameter quantity can be adopted to evaluate reply results of some questions with relatively high complexity, so that the evaluation accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model evaluation method, device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI technology is increasingly being applied to various complex decision-making tasks. For example, in education, test assessment, public affairs, law and other industries, AI technology can be used to learn a large number of questions and answers, and train AI models that can answer specified questions, such as question-and-answer models, expert models, customer service models, etc.

[0003] In order to improve the performance of AI models in question answering tasks, an evaluation model is usually used to evaluate the AI ​​model. A commonly used evaluation method is to obtain the answer results output by the AI ​​model on a large number of evaluation questions, assemble the evaluation questions and answer results into prompt words, and call the evaluation model in the form of dialogue questions and answers. The evaluation model can perform the evaluation task according to the prompt words and output the evaluation results. Usually, the evaluation model is a high-performance model with many parameters. When calling the evaluation model on a large number of evaluation questions, it often consumes more resources. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Various aspects of the present application provide a model evaluation method, device, storage medium, and program product to reduce the resource cost required for testing AI models.

[0005] An embodiment of the present application provides a model evaluation method, including: responding to a model evaluation operation, obtaining a reply result output by a target model for a target question; obtaining a question type corresponding to the target question; calling a target evaluation model adapted to the question type among multiple evaluation models, evaluating the reply result, and obtaining an evaluation result of the target model on the target question; among the multiple evaluation models, different evaluation models correspond to different question types, and different evaluation models have different parameter quantities.

[0006] Optionally, a target evaluation model adapted to the question type among multiple evaluation models is called to evaluate the answer result, including: if the question type is an objective question, calling the first evaluation model to evaluate the answer result corresponding to the target question, the first evaluation model is a lightweight model with a parameter amount less than a set first threshold.

[0007] Optionally, calling the first evaluation model to evaluate the answer result corresponding to the target question to obtain the evaluation result of the target model on the target question, including: calling the first evaluation model, and evaluating the answer result corresponding to the target question according to the rule engine corresponding to the objective question, to obtain the evaluation result of the target model on the target question; the rule engine is used to match the answer result with the reference answer of the target question, and calculate the score of the answer result based on the matching result and predefined scoring rules.

[0008] Optionally, a target evaluation model adapted to the type of the target question is called to evaluate the answer result, including: if the question type is a subjective question, a second evaluation model is called to evaluate the answer result corresponding to the target question, and the second evaluation model is a large language model with a parameter amount greater than a set second threshold.

[0009] Optionally, a second evaluation model is called to evaluate the answer result corresponding to the target question, including: generating prompt words adapted to the question type according to the question type, the target question and the corresponding answer result; the prompt words corresponding to different question types have different structures and / or contents; and calling the second evaluation model to evaluate the answer result corresponding to the target question according to the prompt words.

[0010] Optionally, prompt words adapted to the question type are generated according to the question type, the target question and the corresponding answer result, including: if the question type is a subjective question of a knowledge quiz type, prompt words adapted to the subjective question of the knowledge quiz type are generated according to the target question, the answer result, the reference answer to the question and the scoring rules; if the question type is a subjective question of a content creation type, prompt words adapted to the subjective question of the content creation type are generated according to the creation question, the creation result output by the target model for the creation question, the creation requirements and the scoring rules.

[0011] Optionally, obtaining the question type corresponding to the target question includes: obtaining the question type corresponding to the target question according to the question type setting information of the target question; or, calling a question type classifier to identify the question type corresponding to the target question, the question type classifier being trained on a question type training set using a machine learning algorithm.

[0012] Optionally, before calling a target evaluation model among multiple evaluation models that is adapted to the question type and evaluating the answer result, it also includes: obtaining a matching degree between the answer result corresponding to the target question and the target question; if the matching degree is greater than a set matching degree threshold, calling a target evaluation model among the multiple evaluation models that is adapted to the question type to evaluate the answer result.

[0013] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided in the embodiment of the present application.

[0014] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which can implement the steps in the method provided in the embodiment of the present application when the computer program is executed by a processor.

[0015] The embodiment of the present application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in the embodiment of the present application.

[0016] In an embodiment of the present application, in the process of evaluating the target model, after obtaining the answer result output by the target model for the target topic, the topic type corresponding to the target topic can be obtained, and the target evaluation model adapted to the topic type in multiple evaluation models is called to evaluate the answer result. In this embodiment, among the multiple evaluation models, different evaluation models correspond to different topic types, and the parameter quantities of different evaluation models are different. Based on this, a scheme is implemented for collaboratively evaluating the answer results output by the target model for different types of topics based on evaluation models with different parameter quantities, which facilitates the use of evaluation models with lower parameter magnitudes to evaluate the answer results of some topics with lower complexity, effectively reducing resource consumption, and facilitates the use of evaluation models with higher parameter magnitudes to evaluate the answer results of some topics with higher complexity, thereby improving the evaluation accuracy, thereby achieving a balance between accuracy, cost, and time efficiency to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a model evaluation method provided for an exemplary embodiment of the present application; Figure 2 A schematic diagram of the structure of an evaluation system provided for an exemplary embodiment of the present application; Figure 3 is a flowchart of a model evaluation method provided by another exemplary embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0019] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0020] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0021] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the product or system including the element.

[0022] The purpose of evaluating an AI model is to assess the performance of the AI ​​model on a specific task to determine whether its task execution capability meets expectations. One method for evaluating an AI model is to obtain the response results output by the AI ​​model on a large number of evaluation questions, assemble the evaluation questions and response results into prompt words, and call the evaluation model in the form of a dialogue question and answer. The evaluation model can perform evaluation tasks according to the prompt words and output evaluation results. When the evaluation model performs tasks according to the prompt words, it is necessary to split the prompt words into small processing units (tokens). The longer the prompt words, the more tokens are obtained by splitting, and the evaluation model requires more time and resources to process these tokens. In this evaluation method, converting evaluation questions of different types and their response results into question and answer prompt words will generate a large number of tokens, thereby affecting the evaluation efficiency of the evaluation model and increasing the resource consumption required for the evaluation.

[0023] Secondly, in this evaluation method, in order to evaluate the AI ​​model's ability to solve questions with higher complexity, the number of parameters of the evaluation model is usually increased. The larger the number of parameters of the model, the more resources are consumed to run the model. When an evaluation model with a large number of parameters is used to evaluate the AI ​​model's ability to solve questions of different complexity, even if the AI ​​model's ability to solve questions with lower complexity is evaluated, it will cost more resources. In addition, in some scenarios, when evaluating some AI models with poor capabilities, such models may not be able to accurately understand or follow the given instructions, which may lead to the generation of interference information unrelated to the question during the problem-solving process. If an evaluation model with a large number of parameters is used to evaluate the AI ​​model in this case, more unnecessary resource loss will be generated.

[0024] In view of the above technical problems, a solution is provided in some embodiments of the present application. The technical solutions provided by the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0025] Figure 1 is a flow chart of a model evaluation method provided by an exemplary embodiment of the present application, which may include: Figure 1 Steps shown: Step 101, respond to the model evaluation operation and obtain the answer result output by the target model for the target question.

[0026] Step 102: Obtain the topic type corresponding to the target topic.

[0027] Step 103: call a target evaluation model adapted to the question type from among multiple evaluation models, evaluate the response result, and obtain an evaluation result of the target model on the target question; among the multiple evaluation models, different evaluation models correspond to different question types, and different evaluation models have different parameter quantities.

[0028] The execution subject of this embodiment is an electronic device, for example, it can be a terminal device such as a mobile phone, a computer, a tablet computer, or a server device such as a physical server or a cloud server, and this embodiment does not limit it. An evaluation tool can be run on the electronic device, and the evaluation tool can be a plug-in or an independent application. In this embodiment, the evaluation tool can call other evaluation models to perform evaluation operations on the target model. The following will be exemplified in combination with the above steps.

[0029] In step 101, the target model can be any AI model that can output a response result according to the target topic. The target model can also be called a participating model or a model to be tested. In different application scenarios, the implementation form of the target model is different. For example, in some application scenarios, the target model can be a knowledge question and answer model, a translation model, a customer service model, an intelligent problem-solving model, a teaching and tutoring model, a question and answer model related to public affairs, a question and answer model related to corporate services, and so on.

[0030] In this embodiment, the model evaluation operation may be initiated by the user, or may be initiated by other applications or system timer events, which is not limited in this embodiment. In some embodiments, the evaluation tool may respond to the model evaluation operation, and obtain the response result output by the target model for a given evaluation question through the interface provided by the target model. In some evaluation scenarios, the evaluation tool may obtain one or more evaluation sets, and different evaluation sets may correspond to different application fields. For example, in some embodiments, evaluation questions applied in different fields such as academic, engineering, and business may be obtained to generate evaluation sets corresponding to multiple different fields. The evaluation questions in any evaluation set may cover multiple different question types, and each type of evaluation question may cover different difficulty levels to improve the coverage of the evaluation set. In some embodiments, for some evaluation questions with standard answers, the standard answers of such evaluation questions may be marked by manual or automatic marking, so that the standard answers can be used as reference answers in subsequent evaluation links.

[0031] The evaluation tool can input the evaluation questions in the evaluation set into the target model to obtain the answer results output by the target model for the evaluation questions. The evaluation tool can score the answer results of the evaluation questions to test the performance of the target model in the problem-solving tasks corresponding to the evaluation questions.

[0032] This embodiment will take any evaluation topic as an example to exemplify the model evaluation process. For the convenience of description and distinction, any evaluation topic is described as a target topic. The target topic can be any evaluation topic in the evaluation set, or it can be any topic obtained by the target model during operation. In some embodiments, when the target model is implemented as an intelligent problem-solving model or a teaching and tutoring model, the target topic can be a test question of various different types. When the target topic has different types, the answer result output by the target model is also different. For example, the answer result of the multiple-choice question is the target option, the answer result of the writing question is a text paragraph, and the answer result of the painting question is a picture.

[0033] After obtaining the response result output by the target model for the target question, in step 102, the evaluation tool may obtain the question type corresponding to the target question. In some optional embodiments, the question types that the evaluation tool can identify may include: traditional question types such as objective questions and subjective questions. Among them, objective questions are questions that require the problem-solving model to make judgments or choices based on facts or data, such as multiple-choice questions, judgment questions, etc. Multiple-choice questions may include single-choice questions and multiple-choice questions. Subjective questions are questions that require the problem-solving model to answer based on its own views or experiences, such as knowledge questions, writing questions, decision-making guidance questions, etc. Optionally, in addition to traditional question types, the question types that the assessment tool can identify may also include some non-traditional question types, such as situational questions that require making decisions or solving problems based on given information (such as questions related to the return process in an intelligent customer service scenario), matching questions that require matching different objects, sorting questions that require sorting out processes or timelines, chart analysis questions that require analyzing charts or graphs, case study questions that require in-depth analysis of a case in light of the case context, programming questions that require using code to solve specific problems, debate questions that require debate based on designated controversial topics, etc. This embodiment includes but is not limited to these.

[0034] In some optional embodiments, the assessment tool may obtain the question type corresponding to the target question according to the question type setting information of the target question, wherein the question type setting information may be set by a user (eg, a tester) or may be marked in the assessment center, which is not limited in this embodiment.

[0035] In other optional embodiments, the evaluation tool may use a question type classifier to identify the question type corresponding to the target question. Optionally, the question type classifier may be obtained by training on a question type training set using a machine learning algorithm, and the machine learning algorithm may include but is not limited to: a naive Bayes classifier, a support vector machine, etc. In addition to machine learning methods, the question type classifier may also be trained using a rule-based method, such as a classification algorithm based on keyword matching, etc. When training a question type classifier, a large number of questions of various different question types may be collected as sample data, and the sample data may be divided into a question type training set and a question type test set. Among them, the question type training set is used to train the question type classifier, and the question type test set is used to evaluate the accuracy and recall rate of the question type classifier, which will not be repeated.

[0036] After obtaining the question type based on the above implementation method, in step 103, the evaluation tool can call the target evaluation tool adapted to the question type, evaluate the response result corresponding to the target question, and obtain the evaluation result of the target model on the target question. In this embodiment, the evaluation tool can provide different evaluation models for different question types, and the parameter amounts of different evaluation models are different. The parameters of the model refer to the number of adjustable variables used to learn data patterns within the model. Generally, the more parameters a model has, the greater the capacity of the model to capture complex patterns, and thus perform better in complex tasks. Accordingly, the more parameters a model has, the greater the consumption of resources, such as computing resources and storage resources, required by the model during operation.

[0037] In some optional embodiments, different evaluation models have different parameter quantities, which may include: different evaluation models have different parameter magnitudes. Among them, parameter magnitude refers to the order of magnitude of the parameters in the model, which is usually used to describe the scale and complexity of the model. Models with high parameter magnitudes are more complex and can capture more details and patterns in the data. Models with lower parameter magnitudes are relatively simple and have good performance on simple tasks. When running models with different parameter magnitudes, models with high parameter magnitudes require more computing resources, and models with low parameter magnitudes require fewer computing resources and have lower running costs.

[0038] In this embodiment, after identifying the type of question corresponding to the target question, the evaluation tool can select an evaluation model that is adapted to the question type from multiple evaluation models to evaluate the target question and its answer results. Among them, the adaptation of the evaluation model to the question type means that the evaluation model can have a good performance on the evaluation task corresponding to the question type. Among the multiple evaluation models available for selection by the evaluation tool, any test model is a high-performance model with a rich knowledge base, excellent reasoning ability and scalability. The multiple evaluation models may include but are not limited to: evaluation models corresponding to any two or more types of questions among objective questions, subjective questions, scenario questions, matching questions, sorting questions, chart analysis questions, case study questions, programming questions and debate questions. For example, the multiple evaluation models that the evaluation tool can select may include: a first evaluation model corresponding to an objective question, a second evaluation model corresponding to a subjective question, a third evaluation model corresponding to a scenario question, and a fourth evaluation model corresponding to a matching question, etc., which are no longer listed one by one.

[0039] In this embodiment, considering the different complexity of the evaluation tasks corresponding to different question types, different evaluation models among the multiple evaluation models can be set as evaluation models with different parameter amounts to balance the resource cost required for model operation and the evaluation accuracy. Some evaluation models with lower parameter magnitudes have lower operation costs and can have better execution effects on tasks with lower complexity. Based on this, evaluation models with lower parameter magnitudes can be used to perform evaluation tasks of question types with lower complexity, such as evaluation tasks of objective questions, evaluation tasks of matching questions, evaluation tasks of sorting questions, etc., thereby effectively reducing resource consumption. Some evaluation models with higher parameter magnitudes have higher operation costs and can have better execution effects on tasks with higher complexity. Evaluation models with higher parameter magnitudes can be used to perform evaluation tasks of question types with higher complexity, such as evaluation tasks of subjective questions, evaluation tasks of chart analysis questions, evaluation tasks of case study questions, evaluation tasks of programming questions, and evaluation tasks of debate questions, etc., to ensure evaluation accuracy. For example, in some embodiments, the difficulty of solving objective questions, matching questions, situational questions, and subjective questions increases in sequence. The parameter amount of the first evaluation model corresponding to the objective questions can be set to be smaller than the parameter amount of the fourth evaluation model corresponding to the matching questions, the parameter amount of the fourth evaluation model can be set to be smaller than the parameter amount of the third evaluation model corresponding to the situational questions, and the parameter amount of the third evaluation model can be set to be smaller than the parameter amount of the second evaluation model corresponding to the subjective questions.

[0040] The following will provide further illustrative explanations using subjective questions and objective questions as examples.

[0041] In some optional embodiments, when the evaluation tool calls a target evaluation model adapted to the question type corresponding to a target question to evaluate the answer result of the target question, if the question type is an objective question, the evaluation tool may call a first evaluation model to evaluate the answer result corresponding to the target question to obtain the evaluation result of the target model on the target question, and the first evaluation model is a lightweight model with a parameter amount less than a set first threshold.

[0042] Optionally, the first threshold may be a threshold of hundreds of billions or tens of billions. For example, in some embodiments, the first evaluation model may be a 7B / 14B level model, where a 7B / 14B level model refers to a model with approximately 7 billion parameters, and a 14B level model refers to a large language model with approximately 14 billion parameters. Compared to larger-scale models, 7B or 14B models are more lightweight, require less computing resources for training and reasoning, and have lower operating costs.

[0043] Based on this implementation, by identifying the question type and calling the first evaluation model according to the question type, the calling instructions provided to the first evaluation model can be made clearer, so that the first evaluation model can accurately understand the calling instructions and provide more accurate evaluation results.

[0044] Objective questions may include but are not limited to single-choice questions, multiple-choice questions, judgment questions, etc. This type of question is mainly scored using rules. When the evaluation tool calls the first evaluation model to evaluate the objective question, it can call the first evaluation model, and evaluate the answer result corresponding to the target question according to the rule engine corresponding to the objective question, and obtain the evaluation result of the target model on the target question. Optionally, the first evaluation model can preprocess the answer result of the target question to make the description format of the answer result more standardized. Afterwards, the first evaluation model can call the rule engine to score the answer result corresponding to the preprocessed target question according to the rule. In this embodiment, the rule engine is used to match the answer result of the target question with the reference answer of the target question, and calculate the score of the answer result based on the matching result and the predefined scoring rules. Optionally, the rule engine can use at least one judgment method among regular expressions, edit distance and semantic relevance to match the reference answer and the answer result to judge the accuracy of the answer result. Optionally, for multiple-choice questions in objective questions, the rule engine can use regular expressions to match the answer results of multiple-choice questions with reference answers to evaluate whether the answer results of multiple-choice questions are accurate; for judgment questions in objective questions, the rule engine can use regular expressions to match the answer results of judgment questions with reference answers to evaluate whether the answer results of judgment questions are accurate. For fill-in-the-blank questions in objective questions, the rule engine can use methods such as edit distance and semantic relevance to match the answer results of fill-in-the-blank questions with reference answers to evaluate whether the answer results of fill-in-the-blank questions are accurate.

[0045] For example, taking a single-choice question as an example, assuming that the reference answer is option A, the rule engine can construct the following regular expression based on the reference answer: reference_pattern_single = r"^[A]$", where the symbol "^" indicates the start position of the matching string, and the symbol "$" indicates the end position of the matching string, to ensure that the entire answer contains only "A". Assuming that the user's answer result user_answer_single = "A", the rule engine can use the match() function or the fullmatch() function to compare the string in the regular expression reference_pattern_single = r"^[A]$" with the string in the answer result user_answer_single = "A" to determine whether the string in the answer result completely matches the string in the regular expression. Similarly, for multiple-choice questions and true-or-false questions, the rule engine can construct regular expressions based on the reference answers to the multiple-choice questions and true-or-false questions, and execute the above-mentioned matching method based on the regular expression and the answer result, which will not be described separately.

[0046] For another example, taking fill-in-the-blank questions as an example, after obtaining the reference answers and reply results of the fill-in-the-blank questions, the rule engine can pre-process the reference answers and reply results to obtain the character string corresponding to the reference answers and the character string corresponding to the reply results. Among them, the pre-processing includes but is not limited to: case conversion, removal of redundant spaces, word segmentation, removal of stop words, etc. Afterwards, the rule engine can use the edit distance algorithm to quantify the character-level differences between the character string corresponding to the reply result and the character string corresponding to the reference answer, and obtain the edit distance between the reply result and the reference answer. Among them, the lower the value of the edit distance, the more similar the two character strings are. If the edit distance between the reply result and the reference answer is less than a set minimum threshold, the rule engine can determine that the reply result is accurate.

[0047] Among them, the scoring rules for different types of objective questions are different. For example, for single-choice questions, if the answer result is accurate, 1 point is added. For multiple-choice questions, if the answer result is all correct, 3 points are added, and if it is partially correct, 1 point is added. In some optional embodiments, the standard answer to the target question is provided in the evaluation set as a reference answer. In this case, the rule engine can directly perform the scoring operation based on the difference between the reference answer and the answer result in combination with the scoring rules. In other optional embodiments, when the reference answer of the target question is not provided in the evaluation set, the first evaluation model can parse the target question and generate a reference answer that conforms to common sense corresponding to the target question. Optionally, the first evaluation model can parse the entities in the target question and the relationship between the entities based on the target question and its context information. The first evaluation model can query the knowledge information related to the parsed entities and the relationship between the entities in the rich knowledge base, and based on excellent reasoning ability, generate a reference answer based on the parsing results and the queried knowledge information. Furthermore, the rule engine can perform the scoring operation based on the difference between the reference answer generated by the first evaluation model and the answer result in combination with the scoring rules.

[0048] Based on this implementation, by combining the first evaluation model and the rule engine, the evaluation score can be accurately obtained even when the number of parameters of the first evaluation model is small, effectively taking into account both evaluation efficiency and evaluation cost.

[0049] In some optional embodiments, when the evaluation tool calls a target evaluation model adapted to the question type corresponding to a target question to evaluate the answer result of the target question, if the question type is a subjective question, the evaluation tool may call a second evaluation model to evaluate the answer result corresponding to the target question to obtain the evaluation result of the target model on the target question, where the second evaluation model is a large language model with a parameter amount greater than a set second threshold.

[0050] Among them, the large language model is a natural language processing (NLP) model that has been trained on a large scale. Large language models are usually built based on deep learning technology and trained on large-scale training data sets, so that they can show strong performance in processing natural language tasks. Given a piece of text (ie, context), the large language model will try to predict the words that are most likely to appear next. The number of parameters of the large language model is greater than the set threshold, which is usually in the order of millions or billions. During the training process, these parameters are continuously adjusted and optimized based on the difference between the predicted results of the large language model and the actual results to improve the prediction accuracy of the large language model. In some embodiments, the large language model usually uses an advanced neural network architecture, such as a Transformer architecture, to build a model structure. This model structure enables the large language model to capture complex patterns in the text and handle long-distance dependencies. After sufficient training, the large language model has a strong generation capability and can produce coherent and contextual text content based on given prompts.

[0051] In this embodiment, a large language model pre-trained on a large number of general data sets can be used as the base model of the second evaluation model. To adapt to the specific application scenario of this application, that is, the evaluation scenario of the target model for automatic problem solving, the pre-trained large language model can be fine-tuned on a large number of data sets formed by various question types and their answer results, so that the fine-tuned large language model is more suitable for performing the task of evaluating the problem-solving ability of the target model. Furthermore, the evaluation tool can obtain more accurate evaluation results based on the advantages of the fine-tuned large language model in the evaluation task.

[0052] Subjective questions may include, but are not limited to: knowledge questions and content creation. Among them, knowledge questions are an objective question with reference answers, and content creation is an objective question without reference answers. For example, content creation can be questions such as writing articles, reports, and drawings. The evaluation focuses of the above two objective questions are different. When evaluating questions related to knowledge questions, the evaluation focus is to verify the difference between the response results and the reference answers to evaluate the accuracy of the response results of the target model, so as to facilitate the debugging of the target model according to the evaluation results to reduce the degree of fantasy of the target model. When evaluating questions related to content creation, the evaluation focus is: based on the requirements of the question, writing ideas and industry common sense, evaluate the writing content with a more open mind.

[0053] In some optional embodiments, when a reference answer to the target question is provided in the evaluation set, the second evaluation model can directly perform a scoring operation based on the difference between the reference answer and the response result, combined with the pre-learned scoring rules. In some optional embodiments, when the rule engine does not provide a reference answer to the target question, the second evaluation model can parse the target question and generate a reference answer corresponding to the target question. Optionally, the second evaluation model can parse the entities in the target question and the relationships between the entities based on the target question and its contextual information, and query the knowledge base for knowledge information related to the parsed entities and the relationships between the entities. Based on the parsing results and knowledge information, the second evaluation model can generate a reference answer using the pre-learned content generation capability. Furthermore, the rule engine can perform a scoring operation based on the difference between the reference answer generated by the second evaluation model and the response result, combined with the scoring rules.

[0054] Optionally, the second evaluation model may adopt a rule-based scoring method or a deep learning-based scoring method when performing the scoring operation. For example, the second evaluation model may use a regular expression to match the reference answer and the reply result, and output a score based on the matching result. Alternatively, the second evaluation model may use a pre-trained sub-model and a fine-tuned sub-model to score according to the difference between the reference answer and the reply result. Taking the scoring method based on regular expressions as an example, assuming that the reference answer to the knowledge quiz is paragraph T0, the second evaluation model may perform word segmentation on paragraph T0 to obtain a text unit. When constructing a regular expression, the second evaluation model may use the text unit obtained by word segmentation as the value of the regular expression, and may add appropriate wildcards or quantifiers to some text units to allow reasonable variations. For example, the "\s*" symbol may be used to match any number of blank characters, or the "[\w\s]*" symbol may be used to match a combination of words and spaces. If the knowledge quiz allows synonym replacement, corresponding synonyms may be added to some text units with synonyms. After obtaining the answer result of the knowledge quiz, the second evaluation model can use the constructed regular expression to match the answer result to obtain a matching result. The matching result may include the number of text units overlapping in the answer result and the value of the regular expression. Among them, the more the number of overlapping text units, the higher the score of the answer result. Optionally, the second evaluation model can further calculate the semantic similarity score between the reference answer and the answer result, and judge the consistency of the answer result and the reference answer based on the matching result of the regular expression and the semantic similarity score. In this case, the first evaluation model can set the weight coefficients of the matching result and the semantic similarity score respectively, and calculate the comprehensive score of the answer result based on the number of overlapping text units in the matching result, the semantic similarity score and the weight coefficient. In some optional embodiments, in order to accurately evaluate the answer results corresponding to different subjective questions more accurately, the test tool can distinguish different subjective questions when the second evaluation model performs the evaluation task by constructing prompt words, and perform the evaluation tasks corresponding to different subjective questions based on the evaluation methods with different focuses.

[0055] Optionally, the evaluation tool may generate prompt words with different contents and / or structures according to the question type, so as to effectively guide the evaluation model to generate evaluation results that better meet the task requirements. Continuing with the target question as an example, optionally, when the evaluation tool calls the second evaluation model to evaluate the answer result corresponding to the target question, it may generate prompt words adapted to the question type according to the question type, the target question and the corresponding answer result, and call the second evaluation model to evaluate the answer result corresponding to the target question according to the prompt words. Among them, the prompt words are adapted to the question type, which means that the structure of the prompt words is adapted to the question type, and / or the content contained in the prompt words is adapted to the task instructions corresponding to the question type, so that the evaluation model can perform the evaluation task corresponding to the question type in the expected manner.

[0056] Optionally, when the question type corresponding to the target question is a subjective question of the knowledge question type, the structure of the prompt word can be composed of the target question, the answer result of the target model to the question, the reference answer to the target question, and the scoring rule. For example, the prompt word can be: {Please score the answer result based on the difference between the answer result of the following question and the reference answer; Question = Please list three renewable energy sources, Answer result = solar energy, wind energy, geothermal energy, Reference answer = any three of solar energy, wind energy, hydropower, biomass energy, ocean energy, geothermal energy, and hydrogen energy, Scoring rule = 1 point for any one correct answer}.

[0057] Optionally, when the topic type is a subjective topic of content creation, the structure of the prompt word can be composed of the creation topic, the creation result output by the target model for the creation topic, the creation requirements and the scoring rules. For example, the prompt word can be: {Please score the creation result according to the creation result and creation requirements of the following creation topic; creation topic = write an argumentative essay on renewable energy, creation result = xxx, creation requirements = meet the topic requirements, reliable citation facts, strict logic, complete structure, fluent language, meet the style of argumentative essays, and no less than 1,000 words; scoring rules = meet the topic requirements 0-20 points, reliable citation facts 0-20, strict logic 0-20, complete structure 0-10, fluent language 0-10, meet the style of argumentative essays 0-10, no less than 1,000 words 0-10}.

[0058] After the prompt words corresponding to the knowledge question and answer and content creation are input into the second evaluation model, the second evaluation model can execute different scoring logics according to the instructions and rules contained in the prompt words. In other words, different scoring logics with different focuses can be executed on different subcategories under the same question type, making the evaluation results more reliable.

[0059] In some optional embodiments, the second evaluation model may provide one or more expert roles, and different expert roles have different emphases when performing tasks. The evaluation tool may also add information about expert roles that are adapted to the question type in the prompt word, so that the second evaluation model calls the sub-models corresponding to different expert roles according to the prompt word to perform the evaluation task. Optionally, the sub-models corresponding to different expert roles are good at evaluating different subjective questions. For example, the knowledge question and answer expert in the second evaluation model is good at evaluating knowledge question and answer questions, and the writing expert is good at evaluating content creation questions. Optionally, multiple expert roles may also correspond to multiple different field types. For example, the industrial knowledge question and answer expert in the second evaluation model is good at evaluating knowledge question and answer questions in the industrial field, the computer knowledge question and answer expert is good at evaluating knowledge question and answer questions in the field of computer technology, the industrial writing expert is good at evaluating content creation questions in the industrial field, and the computer writing expert is good at evaluating content creation questions in the field of computer technology.

[0060] Based on this implementation, by constructing prompt words adapted to different question types, the second evaluation model can be prompted to have different emphases when performing evaluation tasks of different subjective questions, thereby facilitating the second evaluation model to accurately understand the evaluation instructions and give accurate and reliable evaluation results.

[0061] The above-mentioned implementation methods can be executed in combination. For the evaluation tool, when it is determined that the corresponding question type of the target question is an objective question, the lightweight first evaluation model is called to evaluate the answer result corresponding to the target question. When it is determined that the question type is a subjective question, the second evaluation model is called to evaluate the answer result corresponding to the target question, realizing a "one large and one small" collaborative evaluation scheme. In this collaborative evaluation scheme, the first evaluation model has a good performance in the evaluation tasks of objective questions with lower complexity, and the first evaluation model has a low parameter magnitude, which can effectively reduce resource consumption. The second evaluation model has a good performance in the evaluation tasks of subjective questions with higher complexity, which effectively improves the evaluation accuracy of subjective questions, thereby achieving a balance between accuracy, cost and time efficiency to a certain extent.

[0062] It is worth noting that, in some optional embodiments, the evaluation tool may further obtain the matching degree between the answer result of the target question and the target question before executing the above step 103. Among them, the matching degree between the question and its answer result can be calculated by the text similarity between the question and its answer result. If the matching degree is less than or equal to the set matching degree threshold, it can be considered that the target model fails to accurately understand the question or the target model has a model hallucination in the process of solving the question. In this case, the evaluation tool can directly output a lower evaluation score to quickly obtain the evaluation result. If the matching degree is greater than the set matching degree threshold, step 103 is executed to call the target evaluation model that is adapted to the question type in the evaluation model to evaluate the answer result. Based on this implementation method, the evaluation tool can preliminarily filter the evaluation questions and evaluation result processes before calling the target evaluation model, which can reduce the frequency of unnecessary calls to the evaluation model, which is conducive to further reducing the resource cost consumed by running the model.

[0063] The following will be combined with the attached Figure 2 And attached Figure 3 , taking the evaluation model for solving a problem as an example, the model evaluation method provided in the embodiment of the present application is further exemplified. Among them, the answer result output by the evaluation model for solving a problem for the target question can be described as the problem solving result.

[0064] Figure 2 is a schematic diagram of an evaluation system provided by an exemplary embodiment of the present application. Figure 2 As shown in , the evaluation system includes: an evaluation model, a question type classifier, and multiple evaluation models. Figure 2 As shown, during the evaluation process, the evaluation tool can take evaluation questions from the evaluation set. In some embodiments, a portion of the evaluation questions are marked with the type of question. When another portion of the evaluation questions is not marked with the type of question, the evaluation tool can enter this portion of the evaluation questions into the evaluation set. Figure 2 The question type classifier shown in the figure can obtain the corresponding question type. After that, the evaluation tool can input the evaluation question into the participating model to obtain the problem-solving result output by the participating model.

[0065] After obtaining the evaluation model's solution to the evaluation question and the question type corresponding to the evaluation question, if it is determined that the evaluation question is an objective question, the evaluation tool can call Figure 2 The first evaluation model shown evaluates the solution results corresponding to the evaluation questions. Figure 2 As shown, the first evaluation model can pre-process the evaluation questions to convert the evaluation questions into a specified standardized format. Through the pre-processing operation, the first evaluation model can convert the evaluation questions into single-choice questions, multiple-choice questions, or true-or-false questions with standardized formats. Afterwards, the first evaluation model can call the rule engine to score the solution results of the evaluation questions according to the rules. If the evaluation question is a subjective question, the evaluation tool can call Figure 2 The second evaluation model shown scores the solution results corresponding to the evaluation questions. Among them, for essay questions with reference answers, the evaluation tool can construct prompt words corresponding to the essay questions, reference answers and scoring rules, and call the second evaluation model according to the prompt words. The second evaluation model can use a sub-model adapted to the essay questions to score the solution results. For creative questions without reference answers, the evaluation tool can construct prompt words corresponding to the creative questions and scoring rules, and call the second evaluation model according to the prompt words, so that the second evaluation model uses a sub-model adapted to the creative questions to score the solution results.

[0066] In some other embodiments, the evaluation tool may also call the question type classifier to classify the evaluation questions after obtaining the solution results output by the participating model for the evaluation questions. Figure 3 As shown in the figure, after the evaluation tool obtains the evaluation questions and the solution results of the evaluation questions output by the participating models, the evaluation tool can classify the evaluation questions into question types. The question type classification operation can be implemented using a question type classifier. The evaluation tool can input the evaluation questions and their solution results into different evaluation models according to the question types corresponding to the evaluation questions. Figure 3 As shown, for any evaluation model, if the reference answer corresponding to the evaluation question is not input, the evaluation model can parse the evaluation question. Specifically, the evaluation model can parse the entities in the evaluation question and the relationship between the entities based on the evaluation question and its contextual information, and generate a reference answer based on the parsing result. The evaluation model can evaluate and score based on the generated reference answer and the input problem-solving result. Among them, when any evaluation model evaluates and scores the reference answer and the problem-solving result, a rule-based scoring method can be adopted, such as using regular expressions to match the reference answer and the problem-solving result, and outputting the score based on the matching result. Alternatively, the evaluation model can use a deep learning-based method for scoring, for example, a pre-trained sub-model and a fine-tuned sub-model can be used to score based on the difference between the reference answer and the problem-solving result.

[0067] In this implementation, by differentiating the assessment questions into different types and calling different assessment models to perform the assessment according to the question types, the assessment can be completed at a faster speed and with lower token consumption while ensuring the assessment accuracy.

[0068] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0069] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0070] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0071] Figure 4 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown. Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402 .

[0072] The memory 401 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0073] Processor 402 is coupled to memory 401 and is used to execute a computer program in memory 401, so as to: respond to a model evaluation operation, obtain a response result output by a target model for a target question; obtain a question type corresponding to the target question; call a target evaluation model adapted to the question type among multiple evaluation models, evaluate the response result, and obtain an evaluation result of the target model on the target question; among the multiple evaluation models, different evaluation models correspond to different question types, and different evaluation models have different parameter quantities.

[0074] Optionally, the processor 402 calls a target evaluation model adapted to the question type among multiple evaluation models to evaluate the answer result, including: if the question type is an objective question, calling a first evaluation model to evaluate the answer result corresponding to the target question, and the first evaluation model is a lightweight model with a parameter amount less than a set first threshold.

[0075] Optionally, when the processor 402 calls the first evaluation model to evaluate the answer result corresponding to the target question and obtains the evaluation result of the target model on the target question, it is specifically used to: call the first evaluation model, and evaluate the answer result corresponding to the target question according to the rule engine corresponding to the objective question, and obtain the evaluation result of the target model on the target question; the rule engine is used to match the answer result with the reference answer of the target question, and calculate the score of the answer result based on the matching result and predefined scoring rules.

[0076] Optionally, when the processor 402 calls the target evaluation model adapted to the type of the target question to evaluate the answer result, it is specifically used to: if the question type is a subjective question, call the second evaluation model to evaluate the answer result corresponding to the target question, and the second evaluation model is a large language model with a parameter amount greater than a set second threshold.

[0077] Optionally, when the processor 402 calls the second evaluation model to evaluate the answer result corresponding to the target question, it is specifically used to: generate a prompt word adapted to the question type according to the question type, the target question and the corresponding answer result; the structure and / or content of the prompt words corresponding to different question types are different; call the second evaluation model to evaluate the answer result corresponding to the target question according to the prompt word.

[0078] Optionally, prompt words adapted to the question type are generated according to the question type, the target question and the corresponding answer result, including: if the question type is a subjective question of a knowledge quiz type, prompt words adapted to the subjective question of the knowledge quiz type are generated according to the target question, the answer result, the reference answer to the question and the scoring rules; if the question type is a subjective question of a content creation type, prompt words adapted to the subjective question of the content creation type are generated according to the creation question, the creation result output by the target model for the creation question, the creation requirements and the scoring rules.

[0079] Optionally, when obtaining the question type corresponding to the target question, the processor 402 is specifically used to: obtain the question type corresponding to the target question according to the question type setting information of the target question; or, call a question type classifier to identify the question type corresponding to the target question, the question type classifier being trained on a question type training set using a machine learning algorithm.

[0080] Optionally, before calling a target evaluation model adapted to the question type among multiple evaluation models to evaluate the answer result, the processor 402 is also used to: obtain the matching degree between the answer result corresponding to the target question and the target question; if the matching degree is greater than a set matching degree threshold, call the target evaluation model adapted to the question type among the multiple evaluation models to evaluate the answer result.

[0081] Further, if Figure 4 As shown, the electronic device also includes: a communication component 403, a power component 404, a display component 405, an audio component 406 and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown. Figure 4 Components in the dashed box are optional components, not mandatory components, and may depend on the product form of the electronic device. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or may be a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it may include Figure 4 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 4 Components within the dashed box.

[0082] Among them, the memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0083] The communication component 403 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM)), 3G (such as Wideband Code Division Multiple Access (WCDMA), 4G (such as Long Term Evolution (LTE)), 4G+ (such as upgraded Long Term Evolution (LTE-Advanced, LTE-A)) or 5G (5th Generation Mobile Communication Technology), or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0084] The power supply component 404 is used to provide power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power to the device where the power supply component is located.

[0085] The display assembly includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0086] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0087] In this embodiment, after obtaining the answer result output by the target model for the target question, the question type corresponding to the target question can be obtained, and the target evaluation model adapted to the question type in multiple evaluation models can be called to evaluate the answer result. In this embodiment, among the multiple evaluation models, different evaluation models correspond to different question types, and the parameter quantities of different evaluation models are different. Based on this, a scheme is implemented for collaboratively evaluating the answer results output by the target model for different types of questions based on evaluation models with different parameter quantities, which makes it easier to use evaluation models with lower parameter magnitudes to evaluate the answer results of some questions with lower complexity, effectively reducing resource consumption, and making it easier to use evaluation models with higher parameter magnitudes to evaluate the answer results of some questions with higher complexity, thereby improving evaluation accuracy, thereby achieving a balance between accuracy, cost, and time efficiency to a certain extent.

[0088] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0089] The embodiment of the present application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in the embodiment of the present application.

[0090] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program codes.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0095] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Parallel Random Access Machine (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, product or device including the element.

[0098] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model evaluation method, characterized in that: include: Respond to the model evaluation operation and obtain the answer result output by the target model for the target question; Obtaining the topic type corresponding to the target topic; Call a target evaluation model adapted to the question type from among multiple evaluation models, evaluate the answer result, and obtain the evaluation result of the target model on the target question; among the multiple evaluation models, different evaluation models correspond to different question types, and different evaluation models have different parameter quantities.

2. The method according to claim 1, characterized in that Calling a target evaluation model adapted to the question type from among multiple evaluation models to evaluate the answer result includes: If the question type is an objective question, a first evaluation model is called to evaluate the answer result corresponding to the target question, and the first evaluation model is a lightweight model with a parameter amount less than a set first threshold.

3. The method according to claim 2, characterized in that Calling the first evaluation model to evaluate the answer result corresponding to the target question to obtain the evaluation result of the target model on the target question, including: Call the first evaluation model, and evaluate the answer result corresponding to the target question according to the rule engine corresponding to the objective question to obtain the evaluation result of the target model on the target question; the rule engine is used to match the answer result with the reference answer of the target question, and calculate the score of the answer result based on the matching result and predefined scoring rules.

4. The method according to claim 1, characterized in that: Calling a target evaluation model adapted to the type of the target question to evaluate the answer result, including: If the question type is a subjective question, a second evaluation model is called to evaluate the answer result corresponding to the target question, and the second evaluation model is a large language model with a parameter quantity greater than a set second threshold.

5. The method according to claim 4, characterized in that Calling the second evaluation model to evaluate the answer result corresponding to the target question includes: According to the question type, the target question and the corresponding answer result, a prompt word adapted to the question type is generated; the structure and / or content of the prompt word corresponding to different question types are different; The second evaluation model is called to evaluate the answer result corresponding to the target question according to the prompt word.

6. The method according to claim 5, characterized in that According to the question type, the target question and the corresponding answer result, a prompt word adapted to the question type is generated, including: If the question type is a subjective question of the knowledge quiz type, then generating prompt words adapted to the subjective question of the knowledge quiz type according to the target question, the answer result, the reference answer to the question and the scoring rule; If the question type is a subjective question of content creation, a prompt word for the subjective question of content creation is generated according to the creation question, the creation result output by the target model for the creation question, the creation requirements and the scoring rules.

7. The method according to any one of claims 1 to 6, characterized in that: Get the topic type corresponding to the target topic, including: According to the question type setting information of the target question, obtaining the question type corresponding to the target question; or, A question type classifier is called to identify the question type corresponding to the target question, wherein the question type classifier is trained on a question type training set using a machine learning algorithm.

8. The method according to any one of claims 1 to 6, characterized in that: Calling a target evaluation model adapted to the question type from among multiple evaluation models, and before evaluating the answer result, further comprising: Obtaining the matching degree between the answer result corresponding to the target question and the target question; If the matching degree is greater than a set matching degree threshold, a target evaluation model adapted to the question type among the multiple evaluation models is called to evaluate the answer result.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the method described in any one of claims 1 to 8 can be implemented.

11. A computer program product, characterized in that include: A computer program / instruction, which, when executed by a processor, can implement the steps of the method according to any one of claims 1 to 8.

Images